EC2 Failing Load Balancer Health Check

Question

One of my projects is an internal web app which we are putting behind a load balancer. As of now there is only one ec2 (will be more in production) and the load balancer is not able to reach the ec2-Instance.

The load balancer is setup to accept connections over port 448 and forward them to the ec2-Instance on port 80. The ec2-Instance's are listed in the load balancer in the console but they keep failing the health check.

To debug further I created a test ec2-Instance (54.68.255.208) through the console with a security group that allows connections from anywhere in the world on any port. I installed apache on it and had it serve the default page. I then created a new load balancer (http://lb-test-lb-512274268.us-west-2.elb.amazonaws.com/) that allows all traffic on tcp ports. I added the ec2-Instance to the load balancer, but for no apparent reason the health check fails even though I can connect to the ec2-Instance over port 80 with no problems.

Nobody else in our office has this issue but nobody else has any explanation for what is causing it. Does it sound like I'm missing anything here?

Solution

The best way to debug an ELB issue is to do the following:

1. Ensure the security groups on the instance(s) are setup to allow the incoming port (80)

2. Make sure your health check is setup correctly

   • tail -f (or similar) the logs of the instance(s) you're trying to add to the balancer.
   • See that the ELB is correctly pinging the instance(s)

   An example ping from the health checker in the instance logs:

   10.1.2.3 - - [16/Dec/2014:15:03:57 +0000] "GET /health HTTP/1.1" 200 13 "-" "ELB-HealthChecker/1.0"

3. Ensure the "Listener" configuration is correct: 80 to 80 (for example)

Comments

[c4urself]

What/How is the health check configured?

[Colin Murphy]

80 (HTTP) forwarding to 80 (HTTP) right now the load balancer for the internal project is down until I can fix whatever problem is breaking any load balancer I make.

[c4urself]

Health check usually consists of a Ping Protocol, Ping Port, and Ping Path -- canonical example "HTTP:80/health"

[Ajo Mathew]

I suggest you remove IP\Load Balance information from your question.

[Colin Murphy]

Checking the logs revealed the error, it turns out the default page returns a 403 error without giving any indication of a problem. I added an index.html and the load balancer error on the debug server went away. Thanks