Why would 127.0.0.1 in resolv.conf cause problems in DNS resolution?

Question

[update and update II: see my awkward and proper solutions at the end of the question, based on one of the answers]

I have an extremely slow DNS lookup when my resolv.conf file includes the 127.0.0.1 IP address, actually, many names do not get resolved at all as it (most certainly) times out.

I found one question with what looks like a valid answer here:

Extremely slow DNS lookup

except that I can see the the dnsmasq tool is running for the IP range 192.168.122.2 to 192.168.122.254. This looks like the IPs used by VirtualBox and qemu, so I would imagine that if I turn off dnsmasq, accessing the Internet from a virtual system will fail!

Would there be another reason for the slowness and/or timeout? (note that the problem is more prominent with specific systems such as the user pictures on all stackoverflow websites, including askubuntu, when the problem occurs.)

At this point I remove the IP from the resolv.conf file and that bypasses the issue while I'm browsing, but I don't think that's the best solution (and of course that IP is reinstalled there on each reboot!) I'd like a more permanent solution to this problem that still allows me to run the virtual systems as expected.

P.S. I do not run the Network Manager.

---

Contents of /etc/network/interfaces

Note that I had the problem before adding the 2nd IP on eth1 (i.e. eth1:0).

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth1
iface eth1 inet static
    address 162.226.130.121
    netmask 255.255.255.248
    network 162.226.130.120
    broadcast 162.226.130.127
    gateway 162.226.130.126

auto eth1:0
iface eth1:0 inet static
    name Local network
    address 192.168.1.1
    netmask 255.255.255.0
    network 192.168.1.0
    broadcast 192.168.1.255
    gateway 192.168.1.254

# bridge for virtual box
auto br0
iface br0 inet static
    address 192.168.2.1
    netmask 255.255.255.0
    network 192.168.2.0
    broadcast 192.168.2.255
    bridge_ports    eth3
    bridge_stp      off
    bridge_maxwait  0
    bridge_fd       0

Contents of the /etc/resolv.conf (which is still a soft link as expected) after a boot:

# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 127.0.0.1
search m2osw.com
nameserver 192.168.122.1
nameserver 206.13.31.12
nameserver 206.13.28.12

---

Update:

I do not write an answer myself because I very much used another answer here to resolve my issue, although it is really not what I wanted to do, it is the easiest solution at this point. The bug referenced by the resolvconf author below, found here:

https://bugs.launchpad.net/ubuntu/+source/bind9/+bug/933723

clearly states that if you want to use bind9, you WILL get the namespace 127.0.0.1 in your resolv.conf file. No choice. (RESOLVCONF=no does not seem to do anything, although I may have a surprise after a reboot which I will do very soon!)

As a side note: If I point the /etc/resolvconf/resolv.conf.d/tail softlink to /dev/null (as mentioned below) then I get a resolv.conf that looks like this:

# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 127.0.0.1

In other words, I still get the 127.0.0.1 which is the culprit at this point. But also I completely lose the other two nameservers, even though they are clearly specified in my eth1 interface. This being said, looking in the /run/resolvconf/interface/ directory, I see an eth1.net and lo.named files. The lo.named is copied to the resolv.conf, but not the eth1.net. I have no clue how those files get created and then copied to resolv.conf but that is the dynamism used by resolvconf...

Anyway, there are two solutions mentioned in the bug:

(1) Delete the /etc/resolv.conf soft link and replace it with a plain file with exactly what you want. This may work for you, but I thought that keeping the default dynamic file would be best.

(2) Change your bind9 settings so BIND can respond to those local DNS requests. The interesting thing with that is that names will now be cached on your computer. So it is not all bad. This being said, I was not too interested in opening my nameserver to all domain names... But this works and I do not have to destroy the dynamic resolv.conf.

Just in case, here is an example setup for the bind to make that work:

options {
    [...]
    forwarders {
          // Google DNSes
          8.8.8.8;
          8.8.4.4;
    };
    [...]
};

---

Update II:

Okay! The reboot got rid of the lo.named file in the /run/resolvconf/interface directory. That must be because bind knows to create it if RESOLVCONF=yes, but it does not delete it if RESOLVCONF=no. However, a reboot takes care of that because the /run directory is a RAM disk.

Without that file in the way, the /etc/resolv.conf is setup with exactly what I'd expect which is the list of nameservers as defined in my eth1 definition found in /etc/network/interface as shown earlier.

So... it's a big mess because there are many factors and in some cases a reboot is required!

Solution 1

1: Add dns-nameservers and dns-search options to /etc/network/interfaces.

auto eth1
iface eth1 inet static
    address 162.226.130.121
    netmask 255.255.255.0
    gateway 162.226.130.126
    dns-nameservers 8.8.8.8 162.226.130.126
    dns-search m2osw.com

2: Remove all dns- options from files in /etc/resolvconf/resolv.conf.d/. That resolv.conf includes nameserver options after nameserver 127.0.0.1 indicates that such cruft is present. If /etc/resolvconf/resolv.conf.d/tail is a symbolic link, make it a symbolic link to /dev/null.

3: Downup eth1.

sudo ifdown eth1
sudo ifup eth1

4: Look in /etc/resolv.conf. Is nameserver 127.0.0.1 still there and are replies to DNS queries still delayed? If so then figure out where the nameserver 127.0.0.1 line is coming from. Something is registering the listen address 127.0.0.1 without starting a local nameserver at 127.0.0.1. (i) One possibility is the bind9 package. If you aren't running a local BIND nameserver then purge the bind9 package (sudo apt-get purge bind9). If you are running a BIND nameserver that doesn't provide general Internet name service then edit /etc/default/bind9 and set RESOLVCONF=no, then restart the nameserver. See: https://bugs.launchpad.net/ubuntu/+source/bind9/+bug/933723 (ii) Another possibility is that you have remnants of dnsmasq or similar package on the system. Purge that package. Also purge network-manager since you aren't using it.

5: Reboot and see if things have improved, then report back here.

Solution 2

Using a dns server with a loopback address (e.g. 127.0.0.1) causes problems:

All other dns server with a lower priority are ignored by resolvconf.

The priority of dns servers is defined by the network interfaces with which the dns-server is defined.

See /etc/resolvconf/interface-order and man 5 interface-order.

Luckily there is an environment variable to change this behaviour:

TRUNCATE_NAMESERVER_LIST_AFTER_LOOPBACK_ADDRESS

See man 8 resolvconf

If put

TRUNCATE_NAMESERVER_LIST_AFTER_LOOPBACK_ADDRESS=no

in /etc/default/resolvconf and restart the resolvconf service all other dns server will show up in /etc/resolv.conf.

Comments

[chili555]

Please edit your question to add: cat /etc/network/interfaces

[Alexis Wilke]

I also added my resolv.conf file. Note that this content starting from the search comes from the /etc/resolvconf/resolv.conf.d/original file.

[chili555]

Notwithstanding what appears in various files, now that you've declared DNS nameservers, is DNS lookup back to normal as expected or no?

[steeldriver]

The /etc/resolvconf/resolv.conf.d/original file is only supposed to be a backup of your static /etc/resolv.conf configuration at the time that the resolvconf service is installed - it's not supposed to get used after that. Likewise if you have anything in /etc/resolvconf/resolv.conf.d/tail you should probably remove that - see jtdhood's detailed explanation here --> ubuntuforums.org/…

[Alexis Wilke]

Yes, the gateway is .126. That's how networks are setup by large companies offering static IPs such as AT&T. One question: any idea how to trigger the regeneration of the resolv.conf file without having to reboot or ifup/ifdown an interface?

[chili555]

The only way I am aware of to get the system to re-read and use the changes is: sudo ifdown eth0 && sudo ifup eth0. resolv.conf should then use the declared DNS nameservers.

[Alexis Wilke]

Hmm... I also found this page askubuntu.com/questions/224966/… but at this point the resolv.conf file doesn't get updated properly. I can see the dns-nameservers defined in the interfaces file, but that's just in /var/run/resolvconf/interface/eth1.inet and not in the resolv.conf file. I tried (just in case) to not have them in both files, but that did not help at all... (i.e. the eth1.inet file is ignored while resolving.)

[chili555]

After you down/upped eth0, did resolv.conf change at al?

[Alexis Wilke]

Yes. The defaults appeared in there, but the list eth1 dns-nameservers did not make it...

[Alexis Wilke]

In regard to point 4, I am running a local bind9 which is a second DNS for another server. Bind itself seems to work fine, but I'm not entirely sure how to test with 127.0.0.1. As for where the 127.0.0.1 comes from, one was coming from that backup file that tail was linking to (As I mentioned somewhere else) and the other from ... somewhere, I guess since you mentioned it, probably bind, but not the resolv.conf.d directory.

[jdthood]

OK, assuming that the local BIND named isn't set up to provide general Internet name service, stop BIND from registering 127.0.0.1 as a nameserver address: set RESOLVCONF=no in /etc/default/bind9 and reboot. If the local BIND named is set up to provide general Internet name service then leave RESOLVCONF=yes and configure named to listen on 127.0.0.1.

[Alexis Wilke]

Well... as mentioned in the bug you posted a link about, the problem is still around (since I'm running 13.10 and had the problem.) I have to run bind9. Although I tried to remove all the other software you mentioned here, no difference. Until I decided to setup bind as a forwarder, it kept failing. That is, now I just get nameserver 127.0.0.1 in my resolv.conf file and bind9 answers the queries. Not what I had in mind, but well... the other way around doesn't work anyway.

[Alexis Wilke]

Okay, I suggest you clearly specify that at this time bind9 is not smart enough to delete the lo.named file and a reboot is therefore required after you change RESOLVCONF to RESOLVCONF=no. Without the reboot the 127.0.0.1 stays there.

[Alexis Wilke]

As a side note, I also uninstalled all those other software (network-manager and dnsmasq-base) but that made no difference, since the culprit was anyway bind.

[Alexis Wilke]

Another note, removing the dnsmasq-base breaks the MASQUERADE FORWORDING capability. It also turns off the default ip_forward flag! Luckily I had a page on how to turn on masquerading so I could restore that part quickly: linux.m2osw.com/install_ubuntu

[Alexis Wilke]

Ah! That variable sounds a bit like a hack, but it is good to know about the priority scheme. My system works now, so I won't touch anything, but next time, I may look into that closer.