How to solve AWS ELB/EC2 HTTP 503 with timeout settings?

Question

I'm getting intermittent but regular 503 errors ("Service Unavailable: Back-end server is at capacity") from a site consisting of 2 t2.medium instances behind an ELB. None are under particularly heavy load and all monitoring seems normal.

The AWS docs here: http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/ts-elb-error-message.html say that a potential cause is mismatched timeout settings between the ELB and EC2s: "set the keep-alive timeout to greater than or equal to the idle timeout settings of your load balancer"

Apache conf on the EC2s has:

• KeepAlive On
• MaxKeepAliveRequests 100
• KeepAliveTimeout 5

Idle timeout on the load balancer is 60 seconds.

This would seem, then, to be a cause, but I'm unsure about the fix. Increasing the Apache KeepAliveTimeout isn't - I understand - normally recommended, and I'm equally unsure about the effect that reducing the idle timeout on the ELB will have on site performance.

What's the recommended approach? How can I get an idea of what the ideal settings are for my setup and the sort of traffic level (currently about 30-50 requests/min) it deals with?

Solution

I would lower the idle timeout in the ELB. Clients will need to open new connections more often, but its just slightly slower than reusing an keepalive connection.

Raising the keepalive to 60 in Apache would also fix the 503, but you need to be careful not to run out of connections or memory, especially with prefork mpm because you will get a lot more slots used in keepalive. Use worker mpm (or event mpm if youre not scared of the "This MPM is experimental" warning), make sure you have high enough MaxClients to handle all requests but low enough to not run out of memo.