Ubuntu 16.04 Cannot resolve hostnames

16.04 networking network-manager dns resolv.conf

11,617

Finally I found the problem place: it was iptables issue. I didn't remember where I change this rule, but all UDP packets were set to DROP:

iptables -A INPUT -p udp -i $EXT_INTERFACE -j DROP

Setting to ACCEPT resolve problem:

iptables -A INPUT -p udp -i $EXT_INTERFACE -j ACCEPT

11,617

zhoriq

Updated on September 18, 2022

Comments

zhoriq over 1 year

I have weird problem with resolving hostnames on Ubuntu 16.04. I'm searching for solution from friday (4 days!), so I think it's totally not duplicate to similar questions.

I have a server in local network with two interfaces: external (to the internet) and internal (to local network). Both are static configured. I'll show config below.

It was working perfectly since installed in March. Several times I've successfully made 'apt update' with no problem. On friday I tried to update it again, and here the main story begins...

apt update

Welcome to Ubuntu 16.04.2 LTS (GNU/Linux 4.4.0-93-generic x86_64)

~$ sudo apt update
Err:1 http://us.archive.ubuntu.com/ubuntu xenial InRelease
  Temporary failure resolving 'us.archive.ubuntu.com'
Err:2 http://security.ubuntu.com/ubuntu xenial-security InRelease
  Temporary failure resolving 'security.ubuntu.com'
Err:3 http://us.archive.ubuntu.com/ubuntu xenial-updates InRelease
  Temporary failure resolving 'us.archive.ubuntu.com'
Err:4 http://us.archive.ubuntu.com/ubuntu xenial-backports InRelease
  Temporary failure resolving 'us.archive.ubuntu.com'
Reading package lists... Done
Building dependency tree
Reading state information... Done
195 packages can be upgraded. Run 'apt list --upgradable' to see them.
W: Failed to fetch http://us.archive.ubuntu.com/ubuntu/dists/xenial/InRelease  Temporary failure resolving 'us.archive.ubuntu.com'
W: Failed to fetch http://us.archive.ubuntu.com/ubuntu/dists/xenial-updates/InRelease  Temporary failure resolving 'us.archive.ubuntu.com'
W: Failed to fetch http://us.archive.ubuntu.com/ubuntu/dists/xenial-backports/InRelease  Temporary failure resolving 'us.archive.ubuntu.com'
W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/xenial-security/InRelease  Temporary failure resolving 'security.ubuntu.com'
W: Some index files failed to download. They have been ignored, or old ones used instead.

ping/telnet

~$ ping google.com
ping: unknown host google.com

~$ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=58 time=3.87 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=58 time=3.93 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=58 time=3.88 ms

--- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 3.870/3.898/3.939/0.077 ms

~$ telnet 8.8.8.8 53
Trying 8.8.8.8...
Connected to 8.8.8.8.
Escape character is '^]'.
Connection closed by foreign host.

On previous updates there was no such problems with resolving hostnames. Server is primarily used in local network by ip address, so i don't know when this problem arise. So here are some commands, that i executed (external ip address replaced with #):

ifconfig

~$ ifconfig
enp29s0   Link encap:Ethernet  HWaddr 00:10:18:25:cd:40
          inet addr:#.#.#.#  Bcast:#.#.#.#  Mask:255.255.255.248
          inet6 addr: fe80::210:18ff:fe25:cd40/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:145862 errors:0 dropped:0 overruns:0 frame:0
          TX packets:119991 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:14777641 (14.7 MB)  TX bytes:22823397 (22.8 MB)


enp3s0    Link encap:Ethernet  HWaddr 00:1a:64:c9:93:f8
          inet addr:10.0.35.115  Bcast:10.255.255.255  Mask:255.0.0.0
          inet6 addr: fe80::21a:64ff:fec9:93f8/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:779951 errors:0 dropped:0 overruns:0 frame:0
          TX packets:608340 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:514425482 (514.4 MB)  TX bytes:189891768 (189.8 MB)


lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:2145438 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2145438 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1
          RX bytes:1185976997 (1.1 GB)  TX bytes:1185976997 (1.1 GB)

cat /etc/network/interfaces

~$ cat /etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).


source /etc/network/interfaces.d/*


# The loopback network interface
auto lo
iface lo inet loopback


# The primary network interface - Internal
auto enp3s0
iface enp3s0 inet static
        address 10.0.35.115
        netmask 255.0.0.0
        network 10.0.0.0
        broadcast 10.255.255.255
#       gateway 10.1.10.102
#       # dns-* options are implemented by the resolvconf package, if installed
#       dns-nameservers 10.1.10.102
        metric 20


# The secondary network interface - External
auto enp29s0
iface enp29s0 inet static
        address #.#.#.#
        netmask 255.255.255.248
#       network #.#.#.#
#       broadcast #.#.#.#
        gateway #.#.#.#
        dns-nameservers 8.8.8.8 8.8.4.4
        metric 10


#auto enp6s0
iface enp6s0 inet manual

/etc/resolv.conf

~$ ls -la /etc/resolv.conf
lrwxrwxrwx 1 root root 27 Oct 14 01:46 /etc/resolv.conf -> /run/resolvconf/resolv.conf


~$ cat /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 8.8.8.8
nameserver 8.8.4.4

But nmcli didn't show any DNS configured: nmcli

~$ nmcli dev show | grep 'DNS'

~$ nmcli dev show | grep 'IP4'
IP4.ADDRESS[1]:                         #.#.#.#/29
IP4.GATEWAY:                            #.#.#.#

What i did:

several times restarted server.
several times restarted systemd-resolved, NetworkManager.
comment and uncomment "dns=dnsmasq" in /etc/NetworkManager/NetworkManager.conf (with restart service and server).
found advice about switch off DNSSEC, but as i found it's already switched off.
made /etc/resolv.conf static file (not symbolic link), get back to symbolic link - all with restarts.

Nothing of this helps...

Today 16 Oct

Today I've made 'apt update' and 'apt upgrade' with some trick: changed in /etc/apt/source.list domains names to their ip addresses. So system updated and become 16.04.3. It doesn't help.

/etc/nsswitch.conf

Remove some settings for "hosts" and leave minimal values.

~$ cat /etc/nsswitch.conf
# /etc/nsswitch.conf
#
# Example configuration of GNU Name Service Switch functionality.
# If you have the `glibc-doc-reference' and `info' packages installed, try:
# `info libc "Name Service Switch"' for information about this file.

passwd:         compat
group:          compat
shadow:         compat
gshadow:        files

#hosts:          files mdns4_minimal [NOTFOUND=return] dns
hosts:          files dns
networks:       files

protocols:      db files
services:       db files
ethers:         db files
rpc:            db files

netgroup:       nis

nslookup on custom DNS server

This thing finally drives me crazy. I run nslookup on custom DNS server...

~$ nslookup yandex.ru 208.67.222.222
;; connection timed out; no servers could be reached

tcpdump on 53 port

...check tcpdump - it was showing too much records with "bad udp cksum"...

~# sudo tcpdump -vvv -B 4096 -i enp29s0 host 208.67.222.222 and port 53
tcpdump: listening on enp29s0, link-type EN10MB (Ethernet), capture size 262144 bytes
16:01:57.043326 IP (tos 0x0, ttl 64, id 30657, offset 0, flags [none], proto UDP (17), length 55)
#.#.#.#.55778 > 208.67.222.222.domain: [bad udp cksum 0x445e -> 0x6d74!] 47660+ A? yandex.ru. (27)

ethtool

...execute ethtool to avoid such records...

~$ sudo ethtool --offload enp29s0 rx off tx off
Actual changes:
rx-checksumming: off
tx-checksumming: off
    tx-checksum-ipv4: off
tcp-segmentation-offload: off
    tx-tcp-segmentation: off [requested on]
    tx-tcp-ecn-segmentation: off [requested on]

...and check tcpdump again:

~# cat tcpdump.log
15:26:08.451181 IP (tos 0x0, ttl 64, id 65257, offset 0, flags [none], proto UDP (17), length 55)
    #.#.#.#.54228 > 208.67.222.222.domain: [udp sum ok] 23065+ A? yandex.ru. (27)
15:26:08.498299 IP (tos 0x0, ttl 57, id 4778, offset 0, flags [DF], proto UDP (17), length 119)
    208.67.222.222.domain > #.#.#.#.54228: [udp sum ok] 23065 q: A? yandex.ru. 4/0/0 yandex.ru. [4m4s] A 77.88.55.80, yandex.ru. [4m4s] A 5.255.255.80, yandex.ru. [4m4s] A 77.88.55.50, yandex.ru. [4m4s] A 5.255.255.60 (91)
15:26:13.451133 IP (tos 0x0, ttl 64, id 537, offset 0, flags [none], proto UDP (17), length 55)
    #.#.#.#.54228 > 208.67.222.222.domain: [udp sum ok] 23065+ A? yandex.ru. (27)
15:26:13.498225 IP (tos 0x0, ttl 57, id 5523, offset 0, flags [DF], proto UDP (17), length 119)
    208.67.222.222.domain > #.#.#.#.54228: [udp sum ok] 23065 q: A? yandex.ru. 4/0/0 yandex.ru. [3m59s] A 5.255.255.80, yandex.ru. [3m59s] A 77.88.55.50, yandex.ru. [3m59s] A 5.255.255.60, yandex.ru. [3m59s] A 77.88.55.80 (91)
15:26:18.451231 IP (tos 0x0, ttl 64, id 1389, offset 0, flags [none], proto UDP (17), length 55)
    #.#.#.#.54228 > 208.67.222.222.domain: [udp sum ok] 23065+ A? yandex.ru. (27)
15:26:18.498305 IP (tos 0x0, ttl 57, id 6088, offset 0, flags [DF], proto UDP (17), length 119)
    208.67.222.222.domain > #.#.#.#.54228: [udp sum ok] 23065 q: A? yandex.ru. 4/0/0 yandex.ru. [3m54s] A 77.88.55.50, yandex.ru. [3m54s] A 5.255.255.60, yandex.ru. [3m54s] A 77.88.55.80, yandex.ru. [3m54s] A 5.255.255.80 (91)

So it seems that Ubuntu works well with DNS server, sending request, getting answers. But somehow cannot parse this answer...

For now I have no any more ideas how to get DNS resolving back to work.

Am I did all right, or something was not, and what else I can do to make it work?

Upd 17 Oct:

ip route output:

~$ ip route
default via x.x.x.41 dev enp29s0  metric 10 onlink 
10.0.0.0/8 dev enp3s0  proto kernel  scope link  src 10.0.35.115 
x.x.x.40/29 dev enp29s0  proto kernel  scope link  src x.x.x.42 
169.254.0.0/16 dev enp3s0  scope link  metric 1000 

~$ ip route get 8.8.8.8
8.8.8.8 via x.x.x.41 dev enp29s0  src x.x.x.42 
    cache

traceroute to Google DNS:

~$ traceroute 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
 1  x.x.x.41 (x.x.x.41)  1.413 ms  1.404 ms  1.420 ms
 2  89.221.63.15 (89.221.63.15)  3.451 ms  3.473 ms  3.463 ms
 3  195.208.208.232 (195.208.208.232)  3.360 ms  2.851 ms  2.843 ms
 4  108.170.250.129 (108.170.250.129)  3.582 ms 108.170.250.97 (108.170.250.97)  4.029 ms 108.170.250.33 (108.170.250.33)  3.737 ms
 5  108.170.226.91 (108.170.226.91)  3.679 ms 66.249.94.195 (66.249.94.195)  4.103 ms 108.170.227.65 (108.170.227.65)  3.828 ms
 6  8.8.8.8 (8.8.8.8)  4.506 ms  3.786 ms  3.979 ms

x.x.x.42 - server address.

x.x.x.40 and x.x.x.41 - "x" is the same as in server address.

Jaime over 6 years

What is the route used by your machine to access 8.8.8.8, the Google DNS server ? What is the response of ip route?
zhoriq over 6 years

@Jaime updated question with traceroute and ip route output.
Jaime over 6 years

The routes look ok. -- what happened when you use nslookup using the Google DNS in the command line ? It does not work ? -- have you tried to set the Google DNS in both interfaces at /etc/network/interfaces ?
zhoriq over 6 years

@Jaime Found the problem. See my own answer. Thanks!