You could use something like grinder (http://grinder.sourceforge.net/) to create a loadtest scenario which you can use to establish a baseline before a change and run after a change to assess impact.

True loadtesting requires quite a bit of thought to go into your test-setup, but for those kind of differential tests it is usually enough to make sure that your test-client and network connectivity to the application are constant factors and good enough to generate a decent load.

You won't catch the subtleties with those tests, but you will normally catch the 'oops, my perfomance dropped by 50%' type of errors. The subtleties you can then work out during normal operations.