Instances slow when connected to AWS load balancer

Question

I have setup 5 t1.micro EC2's behind an AWS load balancer.

All the instances slow down once they are connected to the LB. Eight out of 10 requests takes more than 30s even when I do the requests directly to the instances (bypassing LB). The CPU load of the instances doesn't seem to be the problem. When I remove one of the instances from the LB, it responds just fine. As soon as it is added back to the LB the requests slow down again.

Can anyone give me some clues as to why the instances respond so much slower once the LB is hooked up?

Thanks in advance

Solution

Sounds like requests queuing in your ELBs due to (maybe) long Keep-Alive headers. ELBs will try to minimize the number of connections to the backend by honoring the Keep-Alive header.

Short (or disabled) keep-alive will force ELB to open a single connection per request to your backend, should your backend nodes have free resources to handle that incoming connection. Long keep-alives will reduce backend connections and will enforce the ELBs to serialize requests as much as possible, which may result in requests waiting for others being processed.

Check out your ELB queues in Cloudwatch during load tests and, if they grow while your backends are idle, try lowering the number of requests per connection and reducing the Keep-Alive timeouts. The exact values will vary depending of your application requirements.

PS: t1.micro is evil PS(2): Stickyness is evil

Comments

[quintencls]

As temporary solution I have configured HAProxy on an ec2. Worked great from the first try :)

[Drew Khoury]

Micros have very limited performance. They're not intended for any heavily lifting, production or performance tests. Why not use a small instance?

[quintencls]

I wanted to use test the load balancer in the free tier. And no matter how slow t1.micros are, if i put 5 behind a lb it should increase throughput and this is not happening :s

[Drew Khoury]

Once again, I don't think it's a great idea to do performance tests on micros. Do the same tests with a higher powered machine, and see what the results are.