DNS Resolve is not working on 18.04 server

Question

I have done some fairly extensive searching and can't seem to find the needle in the haystack that fixes this issue.

I have a server running Ubuntu 18.04

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.1 LTS
Release:    18.04
Codename:   bionic

I am running LXC/LXD on the server currently with only a single container which is actually a 16.04 image. DNS works fine from within the container. I believe this eliminates any potential network issue as the problem.

In the 18.04 install the following happens when using nslookup

nslookup google.com
;; connection timed out; no servers could be reached

However when including a dns server directly as such I get a lookup to work. Again seemingly ruling out firewall/network issues

nslookup google.com 1.1.1.1
Server:     1.1.1.1
Address:    1.1.1.1#53

Non-authoritative answer:
Name:   google.com
Address: 172.217.5.238
Name:   google.com
Address: 2607:f8b0:4006:802::200e

As part of following tips/tricks/guides some of the following things I tried are below, as well as various outputs that might be useful in helping nail this down.

I modified the following file to look as such. I only added the name servers. I did this following one of the fixes out there.

$ cat /etc/netplan/50-cloud-init.yaml
network:
version: 2
ethernets:
    ens3:
        dhcp4: true
        match:
            macaddress: <redacted for post>
        set-name: ens3
        nameservers:
            addresses: [8.8.4.4, 8.8.8.8, 1.1.1.1, 1.1.0.0]

This did seem to add the dns servers to the device

sudo systemd-resolve --status
Global
      DNS Domain: openstacklocal
      DNSSEC NTA: 10.in-addr.arpa
                  16.172.in-addr.arpa
                  168.192.in-addr.arpa
                  17.172.in-addr.arpa
                  18.172.in-addr.arpa
                  19.172.in-addr.arpa
                  20.172.in-addr.arpa
                  21.172.in-addr.arpa
                  22.172.in-addr.arpa
                  23.172.in-addr.arpa
                  24.172.in-addr.arpa
                  25.172.in-addr.arpa
                  26.172.in-addr.arpa
                  27.172.in-addr.arpa
                  28.172.in-addr.arpa
                  29.172.in-addr.arpa
                  30.172.in-addr.arpa
                  31.172.in-addr.arpa
                  corp
                  d.f.ip6.arpa
                  home
                  internal
                  intranet
                  lan
                  local
                  private
                  test

Link 5 (vethTR4JCU)
      Current Scopes: none
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

Link 3 (lxdbr0)
      Current Scopes: none
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

Link 2 (ens3)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 8.8.4.4
                      8.8.8.8
                      1.1.1.1
                      1.1.0.0
                      <redacted for post>
         DNS Domain: openstacklocal

Even with the dns servers listed there, no lookups are possible using either dig, or nslookup.

I did install resolvconf as part of a guide, though I think that was unnecessary and just proved to make a bigger mess.

$ ls -al /etc/resolv.conf 
lrwxrwxrwx 1 root root 29 Jan 29 12:55 /etc/resolv.conf -> ../run/resolvconf/resolv.conf

cat /run/resolvconf/resolv.conf 
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
# 127.0.0.53 is the systemd-resolved stub resolver.
# run "systemd-resolve --status" to see details about the actual nameservers.

nameserver 127.0.0.53
search openstacklocal

This is as far as I seem to be able to get. If I add valid nameservers (8.8.8.8, 8.8.4.4, 1.1.1.1, etc) to /run/resolveconf/resolv.conf file:

# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
# 127.0.0.53 is the systemd-resolved stub resolver.
# run "systemd-resolve --status" to see details about the actual nameservers.

nameserver 127.0.0.53
nameserver 8.8.8.8   # manually added in for testing
search openstacklocal

I can get lookups to work as shown below. If course as is stated in the file, these changes are overwritten on reboot.

nslookup google.com
Server:     8.8.8.8
Address:    8.8.8.8#53

Non-authoritative answer:
Name:   google.com
Address: 172.217.15.78
Name:   google.com
Address: 2607:f8b0:4004:810::200e

EDIT: output of apply command

sudo netplan --debug apply
** (generate:15710): DEBUG: 14:11:34.829: Processing input file /etc/netplan/50-cloud-init.yaml..
** (generate:15710): DEBUG: 14:11:34.830: starting new processing pass
** (generate:15710): DEBUG: 14:11:34.878: ens3: setting default backend to 1
** (generate:15710): DEBUG: 14:11:34.879: Generating output files..
** (generate:15710): DEBUG: 14:11:34.879: NetworkManager: definition ens3 is not for us (backend 1)
DEBUG:netplan generated networkd configuration exists, restarting networkd
DEBUG:no netplan generated NM configuration exists
DEBUG:ens3 not found in {}
DEBUG:Merged config:
network:
  bonds: {}
  bridges: {}
  ethernets:
    ens3:
      dhcp4: true
      match:
        macaddress: <redacted for post>
      nameservers:
        addresses:
        - 8.8.4.4
        - 8.8.8.8
        - 1.1.1.1
        - 1.1.0.0
      set-name: ens3
  vlans: {}
  wifis: {}

DEBUG:Skipping non-physical interface: lo
DEBUG:device ens3 operstate is up, not changing
DEBUG:Skipping non-physical interface: lxdbr0
DEBUG:Skipping non-physical interface: vethTR4JCU
DEBUG:{}
DEBUG:netplan triggering .link rules for lo
DEBUG:netplan triggering .link rules for ens3
DEBUG:netplan triggering .link rules for lxdbr0
DEBUG:netplan triggering .link rules for vethTR4JCU

EDIT: Requsted

sudo iptables -L -n -v

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     tcp  --  lxdbr0 *       0.0.0.0/0            0.0.0.0/0            tcp dpt:53 /* generated for LXD network lxdbr0 */
    0     0 ACCEPT     udp  --  lxdbr0 *       0.0.0.0/0            0.0.0.0/0            udp dpt:53 /* generated for LXD network lxdbr0 */
    0     0 ACCEPT     udp  --  lxdbr0 *       0.0.0.0/0            0.0.0.0/0            udp dpt:67 /* generated for LXD network lxdbr0 */
    0     0 ACCEPT     tcp  --  ens3   *       0.0.0.0/0            0.0.0.0/0            tcp dpt:8443 /* allow connection to lxd */
 2336  152K ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
    1    60 ACCEPT     tcp  --  lxdbr0 *       10.100.106.40        0.0.0.0/0            tcp dpt:22
 1279 73342 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
 8207 2604K ACCEPT     all  --  *      lxdbr0  0.0.0.0/0            0.0.0.0/0            /* generated for LXD network lxdbr0 */
 9496 3318K ACCEPT     all  --  lxdbr0 *       0.0.0.0/0            0.0.0.0/0            /* generated for LXD network lxdbr0 */

Chain OUTPUT (policy ACCEPT 70 packets, 8606 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     tcp  --  *      lxdbr0  0.0.0.0/0            0.0.0.0/0            tcp spt:53 /* generated for LXD network lxdbr0 */
    0     0 ACCEPT     udp  --  *      lxdbr0  0.0.0.0/0            0.0.0.0/0            udp spt:53 /* generated for LXD network lxdbr0 */
    0     0 ACCEPT     udp  --  *      lxdbr0  0.0.0.0/0            0.0.0.0/0            udp spt:67 /* generated for LXD network lxdbr0 */

Anyone know the link/solution to this problem. I'm at a loss.

Solution 1

TL;DR: allow port 53 tcp & udp to lo interface.

Even though the default policy on INPUT is ACCEPT, there is a final rule that drops anything not yet accepted. The only rules accepting traffic on port 53 are on the lxdbr0 interface. You could blanket allow everything on lo interface or just allow ports as needed.

To push a rule to allow everything on lo interface ahead of the other rules:

iptables -I INPUT 1 -i lo -j ACCEPT

Solution 2

Frankly the only correct answer to this modern b***** was to :

apt remove ifupdown
apt install cloud-init
# comment out settings in /etc/network/interfaces
# complete settings in /etc/netplan/config.yaml

# Apply settings or reboot
netplan apply

The removal of ifupdown is needed to make the DNS resolver function properly.

Solution 3

While upgrading from 16.04 to 18.04 I had this same issue. Tried the above plus many other solutions that revolved around the resolv.conf.

My true culprit: POSTFIX. Somehow/someway postfix was interfering with my DNS configuration. After removing postfix with the --auto-remove, magically the DNS started to resolve again.

I can now ping and apt-get again. What a wonderful day :)

Solution 4

In my case upgrade from 16.04 to 18.04 mysteriously disabled systemd-resolved service. Fix was a matter of enabling it again.

View more solutions

Comments

[ubfan1]

See my answer askubuntu.com/questions/1113360/… Try adding the libnss-resolve package. The other workaround to the bugs listed (do add yourself to the bug "does this affect me?" list) is to use the nameserver(s) directly, cutting systemd-resolv out of the loop by changing /etc/resolv.conf

[virullius]

are you able to contact local DNS cache? output of iptables -L -n -v?

[Dave]

Updated question with requested output

[Dave]

Those were done when I made the changes.

[Dave]

Made no difference. I didn't think it would as the configuration file is actually being picked up and used as shown in the added domain server output. There seems to be a disconnect between the resolv 127.0.0.53 and the actual nameservers. I'm just not sure how that's supposed to work to even know how to troubleshoot further.

[Dave]

I can't believe I missed that rule. It's even in my notes! Thank you!

[jgitter]

Unfortunately, I went down the same path Dave did, but I am not having a problem with my iptables... This is fantastically hard to debug now

[Tallandtree]

I stumbled with the same and solved it by strict syntax in the /etc/netplan/99-netcfg-vmware.yaml: I added spaces around the nameservers and before the square brackets like this: addresses: [ 10.103.4.50,10.103.4.51 ] instead of addresses: [10.103.4.50,10.103.4.51].

[dardo]

cloud-init was missing...but not missing somehow for me

[LondonRob]

Yep. Worked for me after about a year of suffering intermittently DNS during which nothing else has ever worked. See this AskUbuntu question which was never resolved. I didn't comment out any settings, just installed cloud-init and removed ifupdown then did netplan apply.

[mats1995]

Sorry, I do not understand the answer - or how this could have solved Daves problem: in Dave's INPUT chain, there is an ACCEPT anything rule - two lines above the mentioned DROP anything rule. This rule should also have accepted packets for "lo" port 53. Can anyone please explain? Thanx.

[virullius]

It's been a few years and I don't remember, but I think there was something funny about the wildcard not matching the loopback interface.