Reverse DNS lookups slowing down network operations on LAN

Question

Environment

My LAN setup is quite basic:

• A router connected to the ISP's modem and the internet
• My development pc directly connected to the router

The router provides DHCP but does not run its own DNS server. In fact, there is no DNS server hosted anywhere on my LAN (typical home network setup). The router is configured to send the ISP's DNS servers as part of the DHCP lease information.

I set up a VirtualBox machine on my development PC and installed Debian Squeeze (6.0.4) on it. The VirtualBox network mode is Bridged Adapter to simulate a standalone server on my LAN. Being a VirtualBox server instead of a physical server is not really important, but I mention it for completeness.

The Problem

Every time a network operation executes a DNS reverse lookup of a LAN ip prior to executing, the server has long delays. Some examples of slow network operations:

• SSH connection to the server from my dev PC
• Connection to admin port of Glassfish server
• netstat -l (netstat -nl is very fast)
• Starting MTA: exim4 on boot takes a long time to complete

Some of these have workarounds like adding my dev pc's Ip to /etc/hosts or adding a command-specific option to avoid doing DNS reverse lookups. Obviously, using /etc/hosts only goes so far because it is at odds with DHCP.

However, I can't help but think that I'm missing something. Do I really need to setup a DNS server somewhere on my LAN? That seems like a huge and useless effort for my needs and I can't believe there isn't another option in a DHCP environment like mine.

I searched the net a lot for this and maybe I don't have the right search terms, but I can't find the solution...

update 1 following BillThor's answer

Using host (dig gives the same results):

# ip of stackoverflow.com
$ time host -v 64.34.119.12
Trying "12.119.34.64.in-addr.arpa"
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 15537
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;12.119.34.64.in-addr.arpa.     IN      PTR

;; ANSWER SECTION:
12.119.34.64.in-addr.arpa. 143  IN      PTR     stackoverflow.com.

Received 74 bytes from 192.168.1.1#53 in 15 ms

real    0m0.020s
user    0m0.008s
sys     0m0.000s

# ip of dev pc
$ time host -v 192.168.1.50
Trying "50.1.168.192.in-addr.arpa"
;; connection timed out; no servers could be reached

real    0m10.004s
user    0m0.004s
sys     0m0.000s

My /etc/resolv.conf (was automatically created during installation)

nameserver 192.168.1.1

Both host and dig return very fast for a public ip but take 10s to timeout for a LAN ip. I guess 10s is my current timeout value.

update 2

With dev-pc in /etc/hosts file:

$ time getent hosts 192.168.1.50
192.168.1.50    dev-pc

real    0m0.001s
user    0m0.000s
sys     0m0.000s

Without dev-pc in /etc/hosts file:

$ time getent hosts 192.168.1.50

real    0m10.012s
user    0m0.004s
sys     0m0.000s

It looks more and more like I'll have to find piecewise program options or parameters for each one trying to do reverse DNS lookups! None of the machines (virtual or not) can act as a DNS server on my LAN since they are not always up. Unfortunately, the router's firmware doesn't include a DNS server.

Solution 1

Is 192.168.1.1 your router's IP address?

nameserver 192.168.1.1 suggests your router is advertising itself as a DNS server, rather than "sending the ISP's DNS servers".

What brand and model of router do you have? Does the web interface show log messages?

I'm wondering if your router is forwarding the request to your ISP's nameservers, but your ISP's nameservers are dropping the request, because they don't want you to know what their machine with IP 192.168.1.50 is called.

Suggestions:

• Double check your router's settings. It should answer requests for your own private network. Maybe you can add a static host entry in your router's web interface?
• Try installing Avahi on all the systems on your network.
• Tell your router to use Google Public DNS (8.8.8.8 and 8.8.4.4) or OpenDNS

Solution 2

If you don't want to go through the exercise of setting up a local DNS (and potentially DHCP) server for your home network, you could tell the services where possible to not perform reverse DNS lookups.

For example, to disable reverse dns lookup for SSHD, add the following to the sshd_config file (or similar depending on distro):

UseDNS no

You could remove this line from the Exim configuration, though I'm not clear if this will have impact on the startup speed, it will prevent reverse dns lookup on connecting clients:

host_lookup = *

It looks like you can export this environment variable for the user that starts the Glassfish process:

export AS_NO_REVERSE_DNS=true

Solution 3

Failing DNS lookups should fail fast. However, I generally run dnsmasq which will read the hosts file and serve those entries for forward and reverse lookups. It also acts as a DNS caching server to reduce load on your upstream DNS servers.

You may want to use the host or dig command to see which lookups are slow. If you are redirected to a DNS server which isn't running DNS will take quite a while to fail. You can tune the timeouts and retries in /etc/resolv.conf to speed up the failures.

EDIT: To test the resolver response try using time getent hosts 192.168.1.50. This should return lookups from your /etc/hosts file as well as DNS. The host and dig commands only check DNS. If this returns but takes a few seconds, you may want to change the order of the hosts entry in /etc/nsswitch.conf move files to the front of the list.

Setting up dnsmasq and using it as your primary DNS nameserver in /etc/resolve.conf should resolve the issue if you have any programs which rely only on DNS for name lookups. Exim should use the resolver for is base lookups. You will need a /etc/hosts file, unless you can disable the DHCP server on your router, you can use dnsmasq for DHCP. dnsmasq will automatically register names if they are provided in DHCP requests it handles.

EDIT2: If none of your computers are always up, you may want to look at installing samba and using wins in your nsswitch configuration. It should be fast, so it may work if you put it ahead of dns in the hosts entry.

You could also look at installing the avahi utilities which will enable automatic discovery on link-local networks. I have it working on the IPv6 side, but it is not publishing IPv4 addresses. The default domain for hosts is '.local'. This may not be suitable in your case, but can be overridden. It also seems slower doing lookups than wins so putting mdns ahead of dns in the nsswitch configuration may not be suitable.

Solution 4

Had same problem after changing from dynamic to static IP. My Network configured now: STATIC - NO IPv6 and not intending of using a mailserver.

I resolved the problem by doing:

1: run dpkg-reconfigure exim4-config

2: When you get to the DNS lookups question -> Set the DNS lookups to minimum (Dial up)

It gave me an error that this kind of lookups wasn't going to work and the suggestion to edit /etc/hosts - but the only thing I did with /etc/hosts was commenting out the ip6-allnodes and ip6-allrouters lines.

Reboot should be fast now (for me it is at least).

View more solutions

Comments

[evanda]

I think he said all (and only) LAN looks up are slow, and he's trying to avoid a hosts file altogether.

[bernie]

Since this is only a test server (and a virtual one too!), I can't rely on it as a DHCP server for the whole LAN. There are other occasional wireless clients too that need an IP even when my dev pc, and therefore the test server, is not running.

[bernie]

Updated my post again. Thanks for your help so far.

[bernie]

My router is a "NETGEAR Router WNR3500L" @ 192.168.1.1. The logs only show DHCP leases and logins to the admin interface. You are right, the router sends 192.168.1.1 as the dns server when handling out DHCP leases (confirmed with Wireshark). On the router's admin interface, I'm using "Get Automatically from ISP" for the DNS adresses. The other option is to specify directly DNS servers by IP. I guess this means DNS requests are actually forwarded by the router.

[bernie]

I wish I could set up a DNS server with the actual equipment I have, but it's not really feasible unless I load an alternate firmware on the router. Thanks for the config options; I hadn't found out about the AS_NO_REVERSE_DNS option for Glassfish so I'll have a look.

[Mikel]

Yeah, I'm guessing it's forwarding the DNS requests. What if you change it to use Google Public DNS (8.8.8.8 and 8.8.4.4) instead?

[bernie]

Wow this actually makes a BIG difference! Apparently these DNS servers (Google's and OpenDNS') fail fast when given a private IP address. I'll run a few more tests, but I think this solves my problem!

[bernie]

Marking this as accepted answer. The 3 answers were helpful, but changing the dns servers solved the problem "at large".