How significant is the effect of network latency in Amazon Web Services between EC2 and RDS instances, especially for Magento?

Question

Please keep in mind I have very little experience with any of this stuff.

I've been trying to set up a multi-server Magento site in Elastic Beanstalk, using RDS as the database and EC2s for the separate admin and frontend servers. I was testing the speed of a few different setups on tools.pingdom.com, mostly looking at what they call "wait" (DNS, connect, send, WAIT, receive), assuming that is a rough measure of how long it takes for Magento to generate the html for a page. I was puzzled by how on different setups, the wait time could vary so dramatically for creating roughly the same page using similar server instances. I was getting values between 980ms and 1.8s.

I thought I started to notice a pattern. It seemed that the setups in which the EC2s were in the same availability zones as the RDS instances would be faster and more consistently faster. So I changed the elastic beanstalk configurations so that the EC2s would be in the same zone as the database. My unscientific findings were that I would consistently get wait times of around 1s after this change. It seems to me that the fairly significant differences in speed were due to network latency between the application server and the database.

There are three parts to my question. First, is this what one would expect from keeping instances in the same zones, or am I reading too much into a small set of test results? Second, is this a significant real world difference in speed? Because it seems to be to me, and it also seems that it would only be made worse by using things like NFS to share the media folders. Third, are there any advantages to allowing application servers to be launched in different zones, and are those advantages worth the increase in wait time?

Also, if I'm doing something wrong, please tell me.

Solution

The latency between zones should be in the single millisecond range. While this may a problem from some scenarios, in most cases, you can compensate for it by making good use of caching.

The primary benefit of using multiple zones is availability. The system is designed so that there is very little shared between each zone. If something causes a single zone to fail, it shouldn't cascade to other zones.

While this mostly works, there have been some cases (especially with EBS volumes) that have shown the limits of that isolation.

Comments

[The Phil Lee]

Cool. I tried pinging between EC2 instances in the same and in different zones. Between ec2s both in us-east-1d, I'm getting about .5 ms. Between us-east-1d and 1c, I'm getting about 2 ms. That seems like a really small difference to me, but in your opinion, would that 1.5 ms difference add up (with enough database queries) to the 200 ms -800 ms differences I'm seeing on my test sites? Is that how it works?

[Michael Shaw]

pings are good at detecting latency, but how critical latency is for you will depend on how you use the network connection. Send lots of short "quick" SQL requests that are executed sequentially in code and latency hurts really bad. If you can organize your SQL requests into larger but fewer requests, and run these requests in parallel in your code then the latency will have much less impact.