Amazon ELB for EC2 instances in private subnet in VPC

Question

I'm using Amazon EC2, and I want to put an internet-facing ELB (load balancer) to 2 instances on a private subnet. I am using VPC with public and private subnets.

• If I just add the private subnet to the ELB, it will not get any connections.
• If I attach both subnets to the ELB then it can access the instances, but it often will get time-outs. (Refer Screenshot 1)
• If I attach to only public subnet then my instance attached to ELB gets OutOfService because I do not have any instance in the Public Subnet, instance count shows 0. (Refer Screenshot 2)

Screenshot 1: Both subnets attached

Screenshot 2: Only public subnet attached

My question is actually an extension to this question. After following all 6 steps mentioned in the accepted answer, I am still getting struck, my instance attached to ELB gets OutOfService. I have even tried with allowing ports in the Security Groups for EC2 instances and ELB, but it did not help.

Please help, I am breaking my head with this.

Solution 1

The other SO question you referenced is spot on. Double/Triple check the following

• You need to attach only public subnets to your ELB, making sure that the availability zones those subnets are aligned with the availability zones of the private subnets that your instances are in.
• Make sure that the security group of your instances allows access from the security group of your load balancer
• The load balancer security group should have an egress rule allowing the health check to reach the instance
• Make sure that your health check is working locally on the instance. For example, if your health check in the ELB is HTTP:8080/health_check, on the instance you can curl x.x.x.x:8080/health_check (where x.x.x.x is the private IP of the instance) and get a 200 response code.
• The public subnet routing table should route 0.0.0.0/0 to the internet gateway attached to your VPC.
• The private subnet routing table should route 0.0.0.0/0 to a NAT instance or gateway in a public subnet

Solution 2

The other SO question helped me as well. For me I kept forgetting that I needed to install software to server (i.e Apache) and if you don't create a NAT GW or some other method to allow software installs it will fail. If you want to try an automated fashion w/o much hassle, you can try a script:

https://github.com/jouellnyc/AWS/tree/master/create_aws_vpc3

Note there's plenty of good docs and discussions already but:

HTTP codes I got from the ELB:

• 504 - Probably security groups - allow access to the port 80 of the instances
• 503 - Probably the wrong target group setup
• 502 - Probably Apache/Server not running b/c it’s not installed because there's no nat gw or method to install software.

Comments

[Manish Singh]

Hi Ben, thanks for your answer. I have triple checked all the points you mentioned and all of them seem to be in place. I have attached some screenshots to the question, could you please check them. When I attach only public subnets to the ELB, then the instance count is 0 and the status is OutOfService. If I attach both subnets then the health check is working fine, but the ELB often gives timeouts on connecting. (Here in the screenshots, I am adding only one instance to the ELB).

[Ben Whaley]

I notice in screenshot 1 that the healthy instance is in AZ ap-southeast-1b. Make sure that the public subnet for you ELB is in the same zone. E.g. public subnet for ELB is in ap-southeast-1b, and private subnet for instance is in ap-southeast-1b. If the zones do not match it will not work.

[Michael - sqlbot]

@BenWhaley is correct. To expand this just a bit: Your ELB should only be on public subnets, and only on subnets in the same availability zone(s) as your instance(s).

[Manish Singh]

Thanks Ben and Michael, a million times! My public and private subnets were in different availability zones. That was the problem.

[d11wtq]

So doesn't the "CrossZone" setting on the ELB allow the ELB and the instances to be in different AZs? It doesn't appear to actually work for me. Not a major issue anyway, as I'll just ensure AZ-alignment.

[Ben Whaley]

Cross Zone ensures that a load balancer interface within one zone will send traffic to instances in other AZs. Docs

[jmcollin92]

Very helpful. Thank's for this

[seanmcl]

The reason for this requirement is that AWS puts an ELB "node" in each subnet associated with the ELB. The network interfaces of these nodes have public IPs and need to be able to talk to the Internet, and thus need to be in a subnet with an IGW route.

[havak5]

If traffic is coming from the ELB to the EC2 instances, does the reply from the EC2 instances go back to the ELB? If it does, this means that the ELB would then forward to the response back to the client. If this is true, then instead of needing a NAT instance or gateway, couldn't we just allow inbound HTTP/HTTPS access on our private EC2 instances security group from the security group that the ELB sits in and the rest would be taken care of?

[Ben Whaley]

Yes, if you don't need other outbound access from your private instances then you don't need a NAT.

[Brett Green]

I've found this official source pretty dead-on, but this question helped me prove to my DevOps team that their setup was causing us problems in our QA environment, so thanks! aws.amazon.com/premiumsupport/knowledge-center/…

[A. Kutluozen]

What do you mean by zones should "align"?

[Ben Whaley]

I mean that if you have instances in zones A and B, make sure your load balancer is present in subnets in zones A and B. If it is instead in only zone C, things will not work.

[dancl]

One extra thing to check: the load balancer security group should have an egress rule allowing the health check to reach the instance

[Raphael Titus]

@BenWhaley Will the load balancer will not work if Load balancer is attached with private subnets?

[smile]

This link might be helpful, "Public load balancer to private subnet ec2" aws.amazon.com/premiumsupport/knowledge-center/…