Determining a realistic measure of requests per second for a web server

Question

I'm setting up a nginx stack and optimizing the configuration before going live. Running ab to stress test the machine, I was disappointed to see things topping out at 150 requests per second with a significant number of requests taking > 1 second to return. Oddly, the machine itself wasn't even breathing hard.

I finally thought to ping the box and saw ping times around 100-125 ms. (The machine, to my surprise, is across the country). So, it seems like network latency is dominating my testing. Running the same tests from a machine on the same network as the server (ping times < 1ms) and I see > 5000 requests per second, which is more in-line with what I expected from the machine.

But this got me thinking: How do I determine and report a "realistic" measure of requests per second for a web server? You always see claims about performance, but shouldn't network latency be taken into consideration? Sure I can serve 5000 request per second to a machine next to the server, but not to a machine across the country. If I have a lot of slow connections, they will eventually impact my server's performance, right? Or am I thinking about this all wrong?

Forgive me if this is network engineering 101 stuff. I'm a developer by trade.

Update: Edited for clarity.

Solution 1

If you care about your server's performance when accessed from somewhere in the world, ask a friend somewhere in the world (should have good bandwith) to install sproxy + siege on his linux box. Just download, configure, make. These tools are small, they compile within seconds.

First of, start sproxy on the linux box. By default, it will run on port 9001 on localhost (127.0.0.1). If you want to access it from outside, just pass it the outbound ip address as a parameter.
Now connect to sproxy by setting your browser to use this ip and port as proxy for HTTP. Everything you do from now on is recorded by sproxy and can be replayed later. Now surf around on your site, do things your customers would do and try to do "expensive" things which use your server.
When done, end sproxy by hitting CTRL^C. It recorded your actions to $HOME/urls.txt. Move the file to where siege resides. To start stress testing, run siege -f urls.txt -d NUM -c NUM. d stands for delay between requests, when doing performance tests, use 1 (second). c stands for the number of simulated concurrent users. Choose at will, but start low. Siege will show you the number of transactions per second, errorrate, how long average requests took etc. It is a powerful and easy to use tool.
If you need more info on parameters (there are many), check the siege manual an the sproxy manual

To get more realistic results, let many people test your server from various countries at once and let them send you the statistics.

Solution 2

The realistic requests/sec measure should be taken from the access logs. IMO, the request latency has nothing to do with server load, as the server processes all the requests at the same speed irrespectively of their origin.

Solution 3

Consider using services such as Soasta Cloudtest. With it you can get pretty detailed reports on your tests and you can run performance tests from various public cloud/virtualization providers. You can configure how hard and for how long do you want to hammer your servers. They also have a free "lite" version so you can see what it can do before committing any money.

Comments

[Ladadadada]

ab has a concurrency option. What did you set it to? Also, if you're testing from a domestic ADSL connection, the test is likely to be dominated by your bandwidth and will not be testing anything on the server at all.

[Don]

I'm familiar with ab's concurrency option and tried a wide-range of values to discover the limits of the box. As I wrote above, I understand that my initial tests were dominated by the network and didn't reflect the capability of the server. But my question still stands: Where do most people run their tests from to get realistic metrics? Tests run from a box on the same network as the server (effectively dropping any network latency out of the equation) return great numbers, but they don't seem like "fair" numbers since real users will be coming from outside the network.

[cyberx86]

Tests run from the same network may in fact be more 'fair' as they essentially ignore the network. Your users are likely all on different networks - so the aggregated bandwidth of all those networks should easily exceed your server's available bandwidth. Considering all the users together, the bottleneck is therefore the capabilities of the server - whereas considering an individual user, the bottleneck may be that single user's bandwidth. (The ideal test, perhaps, would be run from multiple remote locations to best simulate actual circumstances, although, most cases should not need that).

[Don]

"Considering all the users together, the bottleneck is therefore the capabilities of the server" -- that makes sense and seems like the right way to think about it. I suppose the server could be sitting behind crappy network equipment, limiting its response rate to the outside world, but that's not really the server's problem and needs to be addressed separately. Something like Pingdom could be used to run the ideal test, I suppose.