EC2 DNS not resolving hostname to internal IP

Question

I already posted this on the AWS forums without success (https://forums.aws.amazon.com/message.jspa?messageID=502441). I'm wondering if I'll be more lucky here.

We have an internal DNS setup that uses the official AWS EC2 DNS 172.16.0.23 as a forwarder. Intermittently the forwarder stops resolving AWS hostnames with their internal IPs and returns the public IPs instead. A script in the crontab checks the DNS every minute and restarts the service to clear the cache when needed. The host against which we check for correct name resolution is an RDS instance. DNS software is Bind version 9.7.0. The problem is not unique to this instance or account since we had the same problem on another instance from another account. Also, this particular instance has been shut down and restarted so it doesn't seem to bee any hardware issue since shutdown/restart should migrate instances to different hardware.

Same (unsolved) problem here: https://forums.aws.amazon.com/thread.jspa?threadID=76515

Does anyone else have this problem? Can someone point me in the right direction for a solution?

Thanks, Jeremy

Solution

DNS in a VPC works as follows:

When an instance starts, it registers with the DNS forwarder in the subnet.

Thereafter, when any other instance in the VPC queries any hostname for that instance, it will return the private ip.

When the instance is stopped, the record with the forwarder will expire.

Thereafter, when any other instance in the VPC queries any hostname for that instance, it will return the private ip. This is because the local DNS forwarder doesn't have a record and forwards the request to the primary DNS servers for Amazon.

Comments

[kaji]

Can you test it out if the failure is at your DNS instance or the Amazon resolver itself ? The latter is very unlikely. Try command dig @172.16.0.23 ec2-url-blah-blah.amazoneaws.com within the amazon instance to check if the fault is in their side.

[jeremyjr]

I'll try that. But the point is my DNS should never know anything about public IPs. The only way for it to resolve is through the AWS DNS which is supposed to return private IPs when queried via a private IP. Thanks. Jeremy

[jeremyjr]

OK so querying 172.16.0.23 directly does in fact return the private IP, even when the caching server returns the public IP. I'm wondering why since queries are only allowed from internal addresses, so public IPs should not be involved at all.

[kaji]

well it is designed that way so that the internal servers get to those servers via local route and external machine get to that machine too albeit via public route :)