How to troubleshoot slow performance on AWS EC2/RDS?

Question

We recently moved our web servers from some 10 year old boxes to AWS EC2.

Usage of the site is currently higher now (it's our busy season) and the site has become much slower, which is unexpected because our instance sizes are much higher than what we had previously.

We run a pretty small site that only gets a few hundred at a time. We're running a c3.large instance on our webserver, and db.m1.large for our RDS MySQL database. We don't have any read replicas and or multiple webservers (load balancing). According to Google Analytics, we only had 18,106 page views for the whole day.

Our users (external and internal) keep seeing the site time out in their browser. It's pretty much across the board instead of any particular page. MySQL PROCESS LIST is also nearly empty without any table locks or whatnot.

If you look at our stats in CloudWatch, everything should be fine. We have very CPU utilization, and what I think is pretty low Network I/O. Likewise on the RDS side, nothing here is screaming "bottleneck".

EC2 Usage (c3.large)

RDS Usage (db.m1.large)

Any ideas how I should go about troubleshooting this issue?

Solution

Finally tracked down the cause of our problems. Apache was misconfigured to have a Keep Alive timeout for 30 seconds. That combined with an overly aggressive AJAX script was causing the site to hang and wait for a connection.

Turning KeepAliveTimeout down to 7, as well as taming the AJAX script, brought everything back to normal.

Comments

[MDMoore313]

RDS? As in Windows Server? Have you tried using PerfMon to see what the OS sees? Also look at the SQL stuff too, it may be stuck running a query.

[ceejayoz]

18,106 pageviews a day means one every 5 seconds on average, yet you're averaging 50-100 writes and 200-400 reads per second. I'd troubleshoot that first.

[DOOManiac]

Sorry, forgot to add: EC2 instance is Amazon Linux w/ Apache/PHP/Memcached (although we just started using Memcached and only have it on 2 or 3 places so far)

[chubby.skelton]

@DOOManiac it seems that your EC2 instance is using EBS, but it seems that the statistics of that is missing. Could you post that as well?

[DOOManiac]

Yes, it is an EBS backed instance.

[David Levesque]

Are the RDS and EC2 instances in the same availability zone?