Why don't my server query the 2nd entry in my resolv.conf?

Question

I have a CentOS 6.4 box.

As with all the other boxes in this rack, it has two NICs: one internal (192.168.1.x) and one external (visible to the world).

We run our own cacheing resolver (using Unbound) on one of the servers in our rack (192.168.1.11), and it has some local DNS entries configured (mario.local, luigi.local, etc.) so that we can simply ping/ssh/ftp to "hostname" FROM any local box TO any other local box without going over one of the external switches (which incurs charges from our ISP).

If my /etc/resolv.conf looks like this:

search local
nameserver 192.168.1.11
nameserver 8.8.8.8
nameserver 74.82.42.42

and I ping "mario" I get:

# ping mario
PING mario.local (192.168.1.3) 56(84) bytes of data.
64 bytes from 192.168.1.3: icmp_seq=1 ttl=64 time=0.738 ms

However, the local DNS server is much slower than the Google public DNS server, so I'd prefer to have that one first in the list. So if I change /etc/resolve.conf to this:

search local
nameserver 8.8.8.8
nameserver 192.168.1.11
nameserver 74.82.42.42

I would expect a ping to "mario" to attempt resolution of mario.local on 8.8.8.8, fail, then query the 2nd DNS server in the list (192.168.1.11) and resolve. But instead, I get:

# ping mario
ping: unknown host mario

Any idea what I'm doing wrong - or am I misunderstanding how resolve.conf is supposed to work? I'm wondering if it could it be related to routing.

My expectation is that if the first DNS server can't resolve an IP, the second resolv.conf entry gets a shot, but that's not working. Help!

Solution

The resolver will query the second name server only if the attempt to reach the first name server times out. In your case, it is not a time out issue, it is a resolution failure, so there is no need to query the remaining name servers.

You can test this by adding an IP which doesn't have a name server running in the first line, and the real name server below it - like this

 search local
 nameserver 1.2.3.4
 nameserver 192.168.1.11
 nameserver 8.8.8.8

The first one will definitely time out, then the remaining name server will be queried in that order.

Comments

[Zoredache]

If your local DNS server is too slow, then you have broken something on it. You probably need to configure it properly with forwarders and so on.

[Andrew B]

His point still stands; if your local is slower than remote, something is definitely not configured right.

[SteveJ]

Thanks, Daniel. This answers the question (and I will mark it as such). Do you know of a way to get the functionality I'm looking for? If nameserver #1 doesn't time out, but doesn't resolve, to query a subsequent nameserver?

[SteveJ]

Our local resolver does work, does perform recursive lookups, and does cache the results. However, when I run namebench to compare the speed of a large number of lookups, the public DNS server outperforms it every time, as it already has a number of queries cached. The catch is, as you say, "lower latency responses after resolved domain names are cached." It's the "after they're cached" part that is causing the issue. Most of the domain lookups from our servers are first time lookups. That's why I'm (still) looking for a way to autoquery a 2nd DNS server after failing to resolve on the 1st.

[Daniel t.]

That is how dns resolution works, so if you are using commands like ping, they rely on internal resolvers, which work on the principle i described above. But if you are considering writing your own scripts, it would be possible to parse the nameservers in /etc/resolve.conf, get the list, and do, say "dig +short @nameserver yourdomain". Then capture the out put of dig, if it doesn't resolve (status: NXDOMAIN ), then query the next nameserver.

[Andrew B]

Daniel's statement is correct under a default configuration, but it's also possible to change this behavior in /etc/resolv.conf. (i.e. options rotate) This introduces inconsistent behavior with your other servers though, so use very cautiously.

[ewokx]

so basically, in order to get the OP's idea working, you just setup a local caching server and put that in it? (especially true if the host is on two nets).. am I correct?