AWS: None of the Instances are sending data

Question

I'm trying to set up an Elastic Beanstalk application with Amazon Web Services however I'm receiving a load of errors with the message None of the instances are sending data. I've tried deleting the Elastic Beanstalk Application and the EC2 instance several times with the sample application and trying again but I get the same error.

I also tried uploading a flask application with AWS Elastic Beanstalk command line tools but then I received the error below:

Environment health has transitioned from Pending to Severe. 100.0 % of the requests to the ELB are failing with HTTP 5xx. Insufficient request rate (0.5 requests/min) to determine application health (7 minutes ago). ELB health is failing or not available for all instances. None of the instances are sending data

Why do I get this error and how do I fix it? Thanks.

Solution 1

You are using Enhanced Health Monitoring. With enhanced health monitoring an agent installed on your EC2 instance monitors vital system and application level health metrics and sends them directly to Elastic Beanstalk.

When you see an error message like "None of the instances are sending data", it means either the agent on the instance has crashed or it is unable to post data to Elastic Beanstalk due to networking error or some other error.

For debugging this, I would recommend downloading "Full logs" from the AWS console. You can follow the instructions for getting logs in the section "Downloading Bundle Logs from Elastic Beanstalk Console" here. If you are unable to download logs using the console for any reason you can also ssh to the instance and look at the logs in /var/log.

You will find logs for the health agent in /var/log/healthd/daemon.log. Additional logs useful for this situation are /var/log/cfn-init.log, /var/log/eb-cfn-init.log and /var/log/eb-activity.log. Can you look at the logs and give more details of the errors you see? This should hopefully give you more details regarding the error "None of the instances are sending data".

Regarding other health "causes" you are seeing:

• Environment health has transitioned from Pending to Severe - This is because initially your environment health status is Pending. If the instances do not go healthy within grace period health status transitions to Severe. In your case since none of the instances is healthy / sending data, the health transitioned to Severe.

• 100.0 % of the requests to the ELB are failing with HTTP 5xx. Insufficient request rate (0.5 requests/min) to determine application health (7 minutes ago). Elastic Beanstalk monitors other resources in addition to your EC2 instances when using enhanced health monitoring. For example, it monitors cloudwatch metrics for your ELB. This error means that all requests sent to your environment CNAME/load balancer are failing with HTTP 5xx errors. At the same time the request rate is very low only 0.5 requests per minute, so this indicates that even though all requests are failing, the request rate is pretty low. "7 minutes ago" means that information about ELB metrics is slightly old. Because Elastic Beanstalk monitors cloudwatch metrics every few minutes, so the data can be slightly stale. This is as opposed to health data we get directly from the EC2 instances which is "near real time". In your case since the instances are not sending data the only available source for health is ELB metrics which is delayed by about 7 minutes.

• ELB health is failing or not available for all instances Elastic Beanstalk is looking at the health of your ELB, i.e. it is checking how many instances are in service behind ELB. In your case either all instances behind ELB are out of service or the health is not available for some other reason. You should double check that your service role is correctly configured. You can read how to configure service role correctly here or in the documentation. It is possible that your application failed to start.

In your case I would suggest focusing on the first error "None of the instances are sending data". For this you need to look at the logs as outlined above. Let me know what you see in the logs. The agent is started fairly early in the bootstrap process on the instance. So if you see an error like "None of the instances are sending data", it is very likely that bootstrap failed or the agent failed to start for some reason. The logs should tell you more.

Also make sure you are using an instance profile with your environment. Instance profile allows the health agent running on your EC2 instance to authenticate with Elastic Beanstalk. If instance profile is not associated with your environment then the agent will not be able to send data to Elastic Beanstalk. Read more about Instance Profiles with Elastic Beanstalk here.

Update One common reason for the health cause "None of the instances are sending data" can be that your instance is in a VPC and your VPC does not allow NTP access. Typical indicator of this problem is the following message in /var/log/messages: ntpdate: Synchronizing with time server: [FAILED]. When this happens the clock on your EC2 instance can get out of sync and the data is considered invalid. You should also see a health cause on the instances on the health page on the AWS web console that tells you that instance clock is out-of-sync. The fix is to make sure that your VPC allows access to NTP.

---

Solution 2

There can be many reasons why the health agent is not able to send any data, so this may not be the answer to your problem, but it was to mine and hopefully can help somebody else:

I got the same error and looking into /var/log/healthd/daemon.log the following was repeatedly reported:

sending message(s) failed: (Aws::Healthd::Errors::GroupNotFoundException) Group 97c30ca2-5eb5-40af-8f9a-eb3074622172 does not exist

This was caused by me making and using an AMI image from an EC2 instance inside an Elastic Beanstalk environment. That is, I created a temporary environment with one instance the same configuration as my production environment, went into the EC2 console and created an image of the instance, terminated the temporary environment, and then created yet another environment using the new custom AMI.

Of course (in hindsight) this meant some settings of the temporary environment were still being used. In this case specifically /etc/healthd/config.yaml, resulting in the health agent trying to send messages to a no longer existing health group.

To fix this and make sure there was no other stale configuration around, I instead started a new EC2 instance by hand from the default AMI used in the production environment (find it under the 'Instances' configuration page of your environment), provision that, then create a new image from that and use that image in my new EB environment.

Solution 3

Check if your instance type's RAM is enough for app + os + amazon tooling. We suffered from this for a long time, when we discovered that t2.micro is barely enough for our use cases. The problem went away right after using t2.small (2GB).

Solution 4

I solved this by adding another security group (the default one for my Elastic Beanstalk).

Solution 5

It appears my problem was that I didn't associate a public ip address to my instance... after I set it it worked just fine.

View more solutions

Comments

[Pav Sidhu]

Thanks for your reply, I try to access to logs for my Elastic Beanstalk application like you said and it restarts the EB app and then nothing happens. I'm unable to get the logs at all.

[Rohit Banga]

Another possibility is that you have incorrect VPC configuration so the bootstrap failed. Again if you can ssh to the instance and get the logs you will have details.

[Pav Sidhu]

I'm trying to ssh into the EC2 instance, but it times out. debug1: connect to address 52.17.213.23 port 22: Operation timed out

[Rohit Banga]

Can you double check your VPC configuration? This strongly suggests that somehow networking is not working correctly on your EC2 instance. Can you also look at the EC2 console logs from the EC2 console. To do this you can follow these steps: 1. Open the Amazon EC2 console. 2. Select the instance. 3. Choose Actions, then Instance Settings, and then Get System Log. docs.aws.amazon.com/AWSEC2/latest/UserGuide/…

[Pav Sidhu]

I've not gone into the VPC before, what am I checking for once I found the VPC configuration? I've also got the system log, would you like to see a copy of it?

[Rohit Banga]

Yes copy of system log will be useful

[Rohit Banga]

Yep confirmed. Initial bootstrap is failing for the instance because the logs show "curl s3-eu-west-1.amazonaws.com/…" is failing. Can you double check that the VPC in which your EC2 instance lives has access to internet through an internet gateway? Did you use any special configuration for VPC on your environment?

[Pav Sidhu]

I haven't touched the VPC at all. I checked if there was an internet gateway associated with the VPC for my EC2 instance and there was?

[Rohit Banga]

Can you post your question on AWS Forums? forums.aws.amazon.com/forum.jspa?forumID=86 There you can PM me your instance id and account ID and then we can look at the network configuration.

[Pav Sidhu]

Here is the AWS forums question: forums.aws.amazon.com/thread.jspa?threadID=216107

[yogeshagr]

Hi @sgvd, i am also facing the same issue. Is there a way we can change the group_id in /etc/healthd/config.yaml. Can we find the group_id of new environment and update it there?

[Matt Fletcher]

Added another security group to what, the EC2 instance? It already had an Elastic Beanstalk security group added but it seems to produce two. Does it need to be in both?

[Greg Holst]

Thanks for that! My memory problem was also caused by jobs initiated by SQS with too short inactivity timeout, so that several jobs were running in parallel. Solved it by restarting the EC2 instance, getting rid of the jobs (for the meanwhile) and afterwards changing the instance type to small and then to medium.

[Steven Yong]

This really saves us. In our scenario, we spawn up the EC2 using an image we created from another environment.

[Arefe]

This is very helpful answer