16.10 fail to resolve DNS

Question

After upgrading my 16.04-installation to 16.10, I have trouble with DNS.

First I got problems a couple of times when connected to WiFi, while it worked on ethernet. Now it seems to work on WiFi also. Not sure why, and if it is in any way related to the problem I face now:

When connecting to a VPN host with Cisco Anyconnect VPN, it adds a line in '/etc/resolv.conf'. I understand that Ubuntu is now using systemd-resolve, and the man page says that there are three different modes for handling /etc/resolv.conf. My /etc/resolv.conf is not a symlink, and does not list 127.0.0.53 as a DNS server, so as far as I understand systemd-resolved should "read it for DNS configuration data". However, it does not seem to care about it.

dig

The strange thing (for me) is that dig host.customer.tld, returns a nice answer with an ANSWER SECTION showing the ip of the requested host, and it refers to the dns server added to /etc/resolv.conf by vpn client as the SERVER. When vpn connection is disabled I get no answer. I.e. dig reads /etc/resolv.conf.

ping

The browser, on the other side, does not get to /etc/resolv.conf, and is not able to resolve the host name. Neither is ping/curl, by the way.

nmcli

I found a related post, and tried running

nmcli device show <interfacename> | grep IP4.DNS

but it lists no dns for the cscotun0 device. (It does not in 16.04 neither, though.) Also, nmcli lists my dhcp server (my router) as IP4.DNS host for my eth/wlan connections. Using dig @192.168.0.1 xxx for any public domain works fine.

configuration

There are some other DNS servers listed in my /run/systemd/resolve/resolv.conf:

nameserver 8.8.8.8
nameserver 8.8.4.4
nameserver 2001:4860:4860::8888
# Too many DNS servers configured, the following entries may be ignored.
nameserver 2001:4860:4860::8844

These are not served by my DHCP server. the file /etc/systemd/resolved.conf contains only commented lines, except the section header:

[Resolve]
#DNS=
#FallbackDNS=8.8.8.8 8.8.4.4 2001:4860:4860::8888 2001:4860:4860::8844

The man page for resolved.conf says that

DNS= A space-separated list of IPv4 and IPv6 addresses to use as system DNS servers. ... For compatibility reasons, if this setting is not specified, the DNS servers listed in /etc/resolv.conf are used instead, if that file exists and any servers are configured in it. This setting defaults to the empty list.

FallbackDNS= A space-separated list of IPv4 and IPv6 addresses to use as the fallback DNS servers. Any per-link DNS servers obtained from systemd-networkd.service(8) take precedence over this setting, as do any servers set via DNS= above or /etc/resolv.conf. This setting is hence only used if no other DNS server information is known. If this option is not given, a compiled-in list of DNS servers is used instead.

Seems like the fallback ends up in /run/systemd/resolve/resolv.conf in my case.

EDIT: I was not certain what was the problem, and to be honest I still don't know exactly how this works, but at least it turned out that the solution in my case was to disable the systemd-resolved service. I thought that service was required, that it was the component that provided DNS service to all the local applications, but apparently there are something else in there doing that job.

Solution 1

I experienced similar problems, for example with adding an extra USB wifi dongle. First I disabled dnsmasq in networkmanager as described above and I stopped dnsmasq (service dnsmasq stop)

I noticed that when resolving broke during my VPN connecting, the routing table looks slightly different (output of route command). The name of the Gateway is DD-WRT in the case it does not work and simply 'gateway' when it does work. The output of this did not change:

nmcli device show wlp1s0 | grep IP4.DNS

It kept showing my router IP. A workaround to get it to work for a while is to restart systemd-resolvd:

sudo service systemd-resolved restart

Since dnsmasq is out of the equation, it is either systemd-resolvd that is the cause of the issue, or anything changing the routing table.

So this is the only difference I see:

ubuntu@ubuntu-Lenovo-Yoga-2-11:~$ route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         gateway         0.0.0.0         UG    601    0        0 

which works. And this when it does NOT work:

ubuntu@ubuntu-Lenovo-Yoga-2-11:~$ route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         DD-WRT          0.0.0.0         UG    601    0        0 wlp1s0

And the same name difference on the VPN line :

vpn-dns.name gateway         255.255.255.255 UGH   0      0        0 wlp1s0

Who knows what may influence the routing table? It would be great if we can identify this so a bug report can be filed. I am getting seriously sick and tired of pursuing all these bugs, but I would like to get them fixed so future users and us will be happy :).

[update] It seems stopping systemd-resolved may fix this and not negatively impact other stuff. You can try that and let it know if it does break stuff. I saw when running systemd-resolvd in debug when it broke:

Removing scope on link wlp1s0, protocol llmnr, family AF_INET
Removing scope on link wlp1s0, protocol llmnr, family AF_INET6
Removing scope on link *, protocol dns, family *

To disable:

sudo systemctl disable systemd-resolved.service

I updated the Ubuntu report with suggestions. [/update] Add: Note: the bug report : https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1624317 has a patch for 17.04 for some issues. Please check the bug report and if possible test the patch. Thank you!

[update]

Please check the above mentioned bug report, the issue seems to be resolved for 17.10 and with a simple command DNS leakage can be disabled too.

[/update]

Solution 2

DNS behavior during OpenVPN connection improved immediately when I followed a suggestion on ubuntuforums:

1. Open /etc/NetworkManager/NetworkManager.conf in an editor with root rights.
2. Delete (or comment out with a hash #) the line that reads dns=dnsmasq
3. Restart NetworkManager via sudo service NetworkManager restart

Solution 3

Ran into the same issue. Somehow I must have installed DNSmasq with some application. Simply removing dnsmasq solved the issue for me.

sudo apt-get remove dnsmasq 

Since then, no more disconnects or some sites not being able to load anymore (I've had an issue loading gmail, i.e. all of the sudden it couldn't connect to gmail, though other sites worked).

Solution 4

Edit /etc/nsswitch.conf and change

hosts:          files mdns4_minimal [NOTFOUND=return] dns

to

hosts:          files dns mdns4_minimal [NOTFOUND=return]

Edit:

I got the same problems for quite some time. I was able to resolve domain names from vpn but I was not able to ping or curl those or use them in other applications. The described change above solved it for me.

View more solutions

Comments

[Mark Stosberg]

Do you have a problem with DNS if you aren't using the VPN?

[aweibell]

I had DNS problems on WiFi right after installation, but somehow it started working... VPN also actually worked once, but I don't know why. The main problem now, though, seems to be that additions in /etc/resolv.conf is not taken into account.

[aweibell]

I haven't tried that workaround involving manual build. Looks like the version referred to there is quite old compared to the one that is in the repo now.

[philcolbourn]

I have similar DNS resolving issues that were not present with 16.04. My suggestion is to start by removing (backup) /etc/resolv.conf; uninstall resolvconf package; reboot; and use dig, systemd-resolve with and without VPN to see what is working or not.

[jdthood]

Compare this bug report: bugs.launchpad.net/ubuntu/+source/systemd/+bug/1624317

[aweibell]

@philcolbourn Uninstall resolvconf? aptitude suggests to replace it with openresolv instead? Can I go without any of them?

[philcolbourn]

systemd-resolve establishes a live resolv.conf in /run/systemd/resolve/resolv.conf that systemd-resolved man page recommends that /etc/resolv.conf be linked to.

[aweibell]

@philcolbourn I know, but even if I link to that, the vpn client renames the link and adds a new /etc/resolv.conf that the OS does not seem to use.

[philcolbourn]

@aweibell, Then what is changing /etc/resolv.conf? Mine is 'wrong' since I'm experimenting, but establishing VPN to PIA does not change /etc/resolv.conf symlink

[aweibell]

I assume it must be the VPN client that is "kindly" making a backup of the old version before modifying /etc/resolv.conf.

[jmartinez]

sudo systemctl disable systemd-resolved.service seems to work fine for me.

[aweibell]

That did not change anything for me. The Anyconnect vpn client still replaces the symlink with a text file containing the vpn dns references.

[aweibell]

When trying to remove dnsmasq-base package, aptitude tells me that it is required by network-manager and ubuntu-fan, and removing it will result in a lot of additional packages being removed..

[Nitai]

What distro and version? I'm on Ubuntu 16.10 and had no issues removing it. Else I wouldn't have posted it :)

[aweibell]

Thanks. Tried this now,but it didn't work. Actually, DNS is working fine, except when I start Cisco VPN client, which replaces the symlink /etc/resolve.conf with a plain text file.

[aweibell]

I'm also on Ubuntu 16.10! Strange. apt remove dnsmasq-base ... The following packages will be REMOVED: account-plugin-ubuntuone checkbox-converged checkbox-gui dnsmasq-base indicator-network network-manager network-manager-gnome network-manager-openconnect network-manager-openconnect-gnome network-manager-openvpn network-manager-openvpn-gnome network-manager-pptp network-manager-pptp-gnome network-manager-vpnc pay-service plainbox-provider-checkbox plainbox-provider-resource-generic ubuntu-desktop ubuntu-fan ubuntu-push-client ....

[postfuturist]

This fix worked for me, I was having DNS issues with OpenVPN. After this change, my /etc/resolve.conf has changed. It's very odd since I don't even have dnsmasq installed.

[aweibell]

Thanks for your thorough report! I see different changes in routing table than you – my VPN seem to add a lot of routes, dynamically as they are being used, I suspect. However, disabling systemd-resolved worked very well for my problem as well!

[Vincent Gerris]

I think eventually the name in the routing table didn't matter, it was systemd-resolved messing up the DNS resolving somehow. I also had to disable the dnsmasq service to prevent it from starting but it all works now. Let's hope someone will fix the proper set of dependencies between packages that are needed to make it all work properly. It's such an annoying bug to deal with.

[Dave Kincaid]

Same here with 16.10. It wants to remove all those other packages too.

[Nitai]

I just had a disconnect again the other day. Somehow one app must have reinstalled dnsmasq again. In any case, this time I simply disabled it with systemd. So far it's not running anymore and I get no disconnects either.

[Vincent Gerris]

@aweibell : I think you cannot remove that package, but if you have a package just called 'dnsmasq' it should be removable. Given the dependancies, the best is to disable it for now. Hopefully the ubuntu devs will look at this and determine the set of packages that need to be included and enabled.

[fd8s0]

worth noting I struggled for a full day with this issue, restarting systemd-resolved service didn't do anything for me, but disabling it completely and no problems any more!

[BairDev]

This might work for issues with the NM and openvpn, but at least it slows down the connections. Like it is guessed here.

[Japheth Ongeri - inkalimeva]

Just to reiterate, I have had network issues for several days since upgrading to 17.04 from 16.10 I have tried most answers here, most work for a while and then the issue crops up again what finally worked was disabling systemd-resolved using sudo systemctl disable systemd-resolved.service and setting the dns to 8.8.8.8 in /etc/resolv.conf

[Sergio Abreu]

This simple line solved for me: sudo service systemd-resolved restart, thanks!

[Vincent Gerris]

Please check the bug report I posted for the options for 17.10!

[asgs]

Amazing! This is one solution that worked flawless for me. in my case, I was trying to connect to my work VPN and none of those VPN hosts were reachable and that's because DNS resolution failed; hitting them with their IP Addresses worked. so I found this article on OpenVPN and your SE answer! Now my VPN works perfectly fine :-)