Why are DNS lookups not respecting /etc/nsswitch.conf & /etc/host.conf?

Question

I have an issue where hostnames are being looked up from DNS even though they are present in /etc/hosts.

I have the following configured:

/etc/host.conf:

order hosts,bind

/etc/nsswitch.conf:

hosts:      files dns

/etc/resolv.conf:

nameserver <nameserver one>
nameserver <nameserver two>

The application running on the host makes some internal and external API requests.

From tcpdump, I'm seeing DNS queries to some of the internal service hostnames that are listed in /etc/hosts. The tcpdump command I'm using is:

tcpdump -tqAlU -s0 port 53 and host <nameserver one>

In the dump I'm seeing requests like the following:

IP 10.0.80.11.domain > app004-private.51308: UDP, length 102
E...I2..>...
.P.
.........I.1E...:...Q.. localhost............   [email protected]..
IP app004-private.33828 > 10.0.80.11.domain: UDP, length 39
E..Ca.@[email protected].
.2.
.P..$.5./..3e.......... localhost.site.com.....
IP 10.0.80.11.domain > app004-private.33828: UDP, length 96
E..|....>.T;
.P.
.2..5.$.hU.3e.......... localhost.site.com................-.ns10.dnshost.com...dns.8w.............u.....

Notice that localhost is being sent to DNS as well as localhost.site.com. The /etc/hosts entry for localhost is:

127.0.0.1 localhost.localdomain localhost

And

IP 10.0.80.11.domain > app004-private.51664: UDP, length 93
E..yx...>.m.
.P.
.2..5...e.<N2...........api.site.com................-.ns10.dnshost...dns.5w.............u.....
IP app004-private.51664 > 10.0.80.11.domain: UDP, length 48
E..L`.@[email protected].
.2.
.P....5.8..n............api.site.com.site.com.....
IP 10.0.80.11.domain > app004-private.48878: UDP, length 76
E..h&e..>..w
.P.
.2..5...TQ..............11.80.0.10.in-addr.arpa.............Q............p.... .        :...Q.
IP 10.0.80.11.domain > app004-private.51664: UDP, length 105
E...VX..>..g
.P.
.2..5...qJ.n............api.site.com.site.com................-.ns10.dnshost.'.dns.Aw.............u.....

Where api.site.com is in /etc/hosts. Running getent to query api.site.com returns:

$ getent hosts api.site.com
10.36.176.114   api001-private api001-private.site.com api001 api.site.com api

I'm stumped. Everything appears to be configured correctly (as far as I'm aware) to use /etc/hosts first then DNS. Any insight as to why /etc/nsswitch.conf and /etc/host.conf are not being respected?

The main application running on the system is http (apache 2.2.15 and PHP 5.3.8 with curl 7.30.0). The OS is Centos 5.6 running with kernel 2.6.18-238.9.1.el5 and glibc 2.5-58.el5_6.3.

Thanks in advance!

Solution 1

We were able to resolve this by disabling ipv6. We disabled ipv6 by adding the following to /etc/modprobe.conf and rebooting.

alias net-pf-10 off
alias ipv6 off
options ipv6 disable=1

After reboot, we no longer see DNS lookups for hosts listed in /etc/hosts.

It's not clear to me exactly why this resolves the issue.

Solution 2

There are many applications out there that do not use the OS-api to query names. Instead they do an explicit DNS-query.

If that happens - they will not go via the resolver library.

On Linux you can do the same on the command line:

• host YOURHOST will try to resolve DNS - no matter what.
• gethostip YOURHOST will use the defined resolver settings in the configured order.

Comments

[Damon Snyder]

One wrinkle that we have observed is that we have one system (out of about 10) that is behaving as expected with our application. This suggests that it's not the application, but something about the OS. As best we can tell, the two systems are configured exactly the same and have the same apache, php, curl, kernel, libc, and centos version.

[Nils]

@drsnyder there are config-items in apache/httpd and propably also in php for name-resolving. Did you compare all direct and included configurations?

[Admin]

That's interesting.. I'd consider filing that as a bug upstream.

[Damon Snyder]

all of the php and apache configs are uniform across all of the servers. They are distributed via puppet.

[Nils]

And that one system that behaved correctly already had IPv6 disabled?

[Damon Snyder]

Yes, the one system that behaved correctly had IPv6 disabled.

[Nils]

One more reason to disable IPv6 - which I do on every installation.

[Andrew B]

Interesting. Is an IPV6 record defined for the hosts that were going to DNS instead of the hosts file? Just trying to narrow down how this quirk behaves.

[Damon Snyder]

There were no IPV6 records defined in the hosts file.

[IsaaC]

Then, defining ::1 localhost or similar (that is, a ipv6 address for each name) should avoid the DNS resolutions needed to find IPv6 addresses.