Load Testing The Right Way - Some Tips From The Trenches
There have been two of my clients recently, who outsourced load-testing to third party companies, different ones in actual fact. I would recommend against that for several reasons.
First and foremost, the load-testing will typically start from the wrong location; where the user sits, which is where we should end up, not where we should begin. The reason, is if there are issues that could relate to network issues, time-to-first-byte (TTFB) or socket related issues, being where the user sits makes those almost impossible to pin down. As another tip in this regard, there are several companies now offering Cloud-based, distributed testing, which definitely has its place but is not in of itself enough. All load testing should ideally start in the same sub-net as the application tier, directed at the application tier, which in turn will exercise the data tier. This will test the basic infrastructure with little or no network latency. If we get TTFB errors or socket errors at this point, that confirms that there are issues with the application tier which can be dealt with quickly before we move further out via the network. Ideally we should also test from inside and outside any firewalls whilst still in the hosting or network location. If there is clustering (load-balancing/failover) we should test each server individually then in the cluster. Also, if there is a content delivery system (CDN) we should also test with and without that. All of this testing should happen before moving to testing from where the user exists.
Secondly, load-testing an application where there is no knowledge of how the application was written or even in what language it was created is far from ideal. We had a client recently where a single request spawned 15 other requests and this was not easily apparent from browsing the web site but showed up in the details of the load test script after careful investigation. If you are going to use a third-party company to carry out load-testing for you, make sure you get to see the scripts they create and pay particular attention to the URL's in the test.
Lastly, creating load-tests is not longer a multi-thousand dollar process. J-Meter is an adequate open-source utility and commercial products such as Paessler Web Server Stress Tool are in the hundreds and not thousands of dollars to buy. If you encounter a vendor wanting thousands of dollars for a load-testing tool, in my opinion and experience, it is over-priced. In addition, if a tool takes hours or days to produce a usable script, something is wrong.