Why does Elastic Load Balancing report 'Out of Service'?

Question

I am trying to set up Elastic Load Balancing (ELB) in AWS to split the requests between multiple instances. I have created several images of my webserver based on the same AMI, and I am able to ssh into each individually and access the site via each distinct public DNS.

I have added each of my instances to the load balancer, but they all come back with the Status: Out of Service because they failed the health check. I'm mostly confused because I can access each instance from its public DNS, but I get a timeout whenever I visit the load balancer DNS name.

I've been trying to read through all the docs and googling it, but I'm stuck. Any pointers or links in the right direction would be greatly appreciated.

Solution 1

I contacted AWS support about this same issue. Apparently their system doesn't know how to handle cases were all of the instances behind the ELB are stopped for an extended amount of time. AWS support can manually refresh the statuses, if you need them up immediately.

The suggested fix it to de-register the ec2 instances from the ELB instead of just stopping them and re-register them when you start again.

Solution 2

Health check is (by default) made by accessing index.html on each instance incorporated in load balancer. If you don't have index.html in document root of instance - default health check will fail. You can set custom protocol, port and path for health check when creating elastic load balancer.

Solution 3

Finally I got this working. The issue was with the Amazon Security Groups, because I've restricted the access to port 80 to few machines on my development area and the load balancer could not access the apache server on the instance. Once the load balancer gained access to my instance, it gets In Service.

I checked it with tail -f /var/log/apache2/access.log in my instance, to verify if the load balancer was trying to access my server, and to see the answer the server is giving to the load balancer.

Hope this helps.

Solution 4

If your web server is running fine, then it means the health check goes on a url that doesn't return 200.

A trick that works for me : go on the instance, type curl localhost:80/pathofyourhealthcheckurl

After you can adapt your health check url to always have a 200 response.

Solution 5

In my case, the rules on security groups assigned to the instance and the load balancer were not allowing traffic to pass between the two. This caused the health check to fail.

View more solutions

Comments

[Parmaia]

I have the same problem, removing the instances (that were stopped when I created the ELB) and adding when they are running, doesn't seem to solve the issue. I still get Out of Service

[retromuz]

This is the same solution I found that worked for me.

[Upendra]

I got this "GET / HTTP/1.1" 302 625 "-" "ELB-HealthChecker/1.0"... can you help me in that

[Parmaia]

That means that the LoadBalancer got access to the instance. It should be In Service on any moment.

[Jonathon Rossi]

It might help someone else. Make sure to also check your elb security group, mine didn't allow outbound traffic from the elb to ec2 instances.

[Mike Purcell]

It sucks that we have to have manual interaction in order to get the ELB back into a default working state if one of the instances goes down. It seems to me it should be able to notify the ELB that it's back online and the ELB marks it back in service. I shudder to think what would happen in the off chance that if you only have 2 instances (varying AZ) and they both went down overnight...

[hexicle]

On the EC2 console I set Load Balancers -> Health Check -> Edit Health Check -> Ping Path: /index.html. I read the error logs in sudo tail -f /var/log/httpd/error_log and needed to touch /var/www/html/index.html because the error said Cannot serve directory /var/www/html/: No matching DirectoryIndex (index.html,index.php) found, and server-generated directory index forbidden by Options directive. 30 seconds later my load balancers were healthy.

[konse]

That really helped me find my issue: I had a .htaccess password protection enabled on my ec2 instance. Of course the loadbalancer couldn't get passed that. So it got 401 back. :-)

[geotheory]

Also if you haven't yet installed apache/nginx.. Cheers for tip.

[Chad]

this was my issue, my entire web application required a login, so health check was getting a 302 redirect to the login page!

[C Murphy]

I had this issue, and it was due to both my inbound and outbound rule only allowing HTTP on port 80. I needed to add another rule for HTTPS on port 443.