can't connect aws VPC private instance to internet through NAT

Question

trying to install a LAMP stack on an aws instance in a VPC private subnet
using Scenario 2 described here: http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Scenario2.html

I've set everything up with the relevant subnets, security groups, internet gateway and routing table.

With my set-up I can successfully:
- ping NAT
- ssh to NAT
- ping external website from NAT (eg ping ietf.org)
- ping private instance from NAT (eg ping 10.0.X.X)
- ssh to private instance
- ping NAT from private instance

However I cannot ping from the private instance to the internet (ie ping ietf.org doesn't work)
and when I try sudo yum update from the private instance I get the following error:
Cannot find a valid baseurl for repo: amzn-main/latest

I presume this is either a routing or security group error... ?

Notes:
- Source/Destination Check is disabled for NAT instance
- the NAT instance has a public IP assigned but not an elastic IP
- ACL is not implemented (left as default)
- route table for public subnet:

10.0.X.X/XX local  
0.0.0.0/0 internet gateway

- route table for private subnet:

10.0.X.X/XX local  
0.0.0.0/0 nat instance

- security groups are as follows:

Private Instance Inbound

ALL Traffic =  vpc_private_sg    
SSH(22) = vps_nat_sg  
MySQL(3306) = vpc_public_sg  
ALL ICMP = 0.0.0.0/0  

Private Instance Outbound

SSH(22) = vps_nat_sg  
HTTP(80) = 0.0.0.0/0  
HTTPS(443) = 0.0.0.0/0  
ALL ICMP = 0.0.0.0/0  

NAT Instance Inbound

ALL Traffic =  vpc_private_sg    
SSH(22) = vps_nat_sg  
SSH(22) = 10.0.X.X/XX  
HTTP(80) = 10.0.X.X/XX  
HTTPS(443) = 10.0.X.X/XX  
ALL ICMP = 0.0.0.0/0  

NAT Instance Outbound

SSH(22) = 10.0.X.X/XX  
HTTP(80) = 0.0.0.0/0  
HTTPS(443) = 0.0.0.0/0  
ALL ICMP = 0.0.0.0/0  

Solution

The problem:

standard AMI instance not working correctly as a NAT instance

If you try to use a standard AMI instance as a NAT instance it'll have the same symptoms as described in the question above:

• SSH connection to the NAT = OK
• SSH connection to private subnet instance = OK
• ping to/from private subnet instance = OK
• private subnet instance outbound calls = FAIL (even with valid route tables/security groups)

---

The hard solution:

To get a standard AMI instance working as a NAT instance you need to customise it:

• modify the iptables as shown here
• ensure IPv4 forwarding is enabled and ICMP redirects are disabled as noted here

---

The easy solution:

use a Community amzn-ami-vpc-nat instance already configured for use as a NAT instance

For most people (who are using the NAT instance for admin connection purposes) a customised NAT instance is simply unnecessary.

AWS provides standard instances configured for NAT use (ie modified as described above) as community AMI instances.

• click Launch Instance in the EC2 control panel
• in Step 1: Choose an Amazon Machine Image (AMI) click the Community AMIs selector on the left
• type amzn-ami-vpc-nat 2017 or more simply nat 2017 (note below) in the search input box
• launch your NAT and ensure your route tables + security groups are correct - see here

Note:
The reason you include the year in the Community AMI instance search is that AWS keeps all the old NAT AMI versions (21 at the time of writing, going back to 2013) - which mostly include a year in the version number
...it's best / easiest to simply choose the latest version.

Comments

[EEAA]

Do you have the requisite iptables rules and ip forwarding turned on on the NAT box?

[goredwards]

not sure... is this something that needs to be added separately - ie something that isn't included by default - i've been following these instructions: docs.aws.amazon.com/AmazonVPC/latest/UserGuide/… and docs.aws.amazon.com/AmazonVPC/latest/UserGuide/… ? no mention of iptables in either of the relevant AWS documentation.

[Michael - sqlbot]

The Amazon-provided NAT instance AMI already has the correct iptables config and should need no changes. Remove all the outbound security group settings and set them back to the default, allow everything, and see what you get.

[goredwards]

good stuff - thanks @Michael-sqlbot - I'd completely ignored the part in the NAT user guide where it says: On the Choose an Amazon Machine Image (AMI) page, select the Community AMIs category, and search for amzn-ami-vpc-nat..

[PHP Avenger]

I have same exact problem, with same configuration and still unable to resolve this issue, the only difference might be I am using "ami-9a562df2" for Nating. Could you please shed some light how you fixed it