Installing docker-ce in Ubuntu 18.04 breaks internet connectivity of host

Question

I've installed docker on a fresh ubuntu desktop 18.04 machine and I lost internet connection in the host immediately after the installation finished. More specifically, I can ping 8.8.8.8 but I can't ping www.google.com.

The steps to reproduce the error are the following:

1. Install ubuntu 18.04
2. Install docker-ce using a local DNS server.

EDIT:

The most important difference between a clean install and my situation is that my machine is using a local DNS server, which has a fixed address at 172.17.100.3 and 172.17.100.70 in a corporate network.

From what I've read so far, either the problem comes from NetworkManager's attempt to manage the docker0 interface or something in the docker installation process breaks systemd resolve capabilities. Note that systemd-resolve uses 127.0.0.53 by default

So far I've attempted the following:

• Hardcoding nameserver 8.8.8.8 in /etc/resolv.conf works
• Uninstalling docker fixes this problem. Installing/removing docker does not change resolv.conf.
• Removing dnsmasq from /etc/NetworkingManager.conf: I used this solution as I show here https://github.com/moby/moby/issues/36153, which shows the same problem on ubuntu 16.04. My fresh installation of 18.04 does not have this line, so this fix is not valid anymore.
• Docker docs suggest disabling or removing NetworkManager completely as shown in https://docs.docker.com/install/linux/linux-postinstall/#dns-resolver-found-in-resolvconf-and-containers-cant-use-it but I'm not sure if this refers to my problem. I also don't know if this will break other stuff.
• This suggests disabling management for docker0 https://gist.github.com/lanrat/458066dbdeb460b9cef40dc2af639a24 but it does not fix the problem in my case
• This approach suggest something different https://github.com/moby/moby/issues/32836 but I believe is not related to this problem

So far the only thing that works is hardcoding the google dns server address in resolv.conf, which is not a clean approach in my opinion. I also have to automate this process on startup, which is something I don't like.

How does one properly fix the NetworkingManager problem in a development machine that has docker?

Current environment

My default resolv.conf has

nameserver 127.0.0.53

and systemd-resolve --status returns (I've checked with a colleague and we have the same output)

Global
          DNSSEC NTA: 10.in-addr.arpa
                      16.172.in-addr.arpa
                      168.192.in-addr.arpa
                      17.172.in-addr.arpa
                      18.172.in-addr.arpa
                      19.172.in-addr.arpa
                      20.172.in-addr.arpa
                      21.172.in-addr.arpa
                      22.172.in-addr.arpa
                      23.172.in-addr.arpa
                      24.172.in-addr.arpa
                      25.172.in-addr.arpa
                      26.172.in-addr.arpa
                      27.172.in-addr.arpa
                      28.172.in-addr.arpa
                      29.172.in-addr.arpa
                      30.172.in-addr.arpa
                      31.172.in-addr.arpa
                      corp
                      d.f.ip6.arpa
                      home
                      internal
                      intranet
                      lan
                      local
                      private
                      test

Link 4 (docker0)
      Current Scopes: none
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

Link 3 (eno1)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 172.17.100.3
                      172.17.100.70

Link 2 (eno2)
      Current Scopes: none
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

ifconfig docker0

bluesmonk@laptop:/etc/NetworkManager$ ifconfig docker0
docker0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 172.17.0.1  netmask 255.255.0.0  broadcast 172.17.255.255
        ether 02:42:86:e0:f0:94  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

and the status of the docker service

$ sudo service docker status
● docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2018-07-03 10:17:35 -04; 4min 52s ago
     Docs: https://docs.docker.com
 Main PID: 14992 (dockerd)
    Tasks: 31
   CGroup: /system.slice/docker.service
           ├─14992 /usr/bin/dockerd -H fd://
           └─15015 docker-containerd --config /var/run/docker/containerd/containerd.toml

jul 03 10:17:32 laptop dockerd[14992]: time="2018-07-03T10:17:32.981563020-04:00" level=warning msg="Your kernel does not support swap memory limit"
jul 03 10:17:32 laptop dockerd[14992]: time="2018-07-03T10:17:32.981595408-04:00" level=warning msg="Your kernel does not support cgroup rt period"
jul 03 10:17:32 laptop dockerd[14992]: time="2018-07-03T10:17:32.981603807-04:00" level=warning msg="Your kernel does not support cgroup rt runtime"
jul 03 10:17:32 laptop dockerd[14992]: time="2018-07-03T10:17:32.982040899-04:00" level=info msg="Loading containers: start."
jul 03 10:17:34 laptop dockerd[14992]: time="2018-07-03T10:17:34.403909997-04:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used
jul 03 10:17:35 laptop dockerd[14992]: time="2018-07-03T10:17:35.083366302-04:00" level=info msg="Loading containers: done."
jul 03 10:17:35 laptop dockerd[14992]: time="2018-07-03T10:17:35.458766295-04:00" level=info msg="Docker daemon" commit=9ee9f40 graphdriver(s)=overlay2 version=18.03.1-ce
jul 03 10:17:35 laptop dockerd[14992]: time="2018-07-03T10:17:35.458857295-04:00" level=info msg="Daemon has completed initialization"
jul 03 10:17:35 laptop dockerd[14992]: time="2018-07-03T10:17:35.470042819-04:00" level=info msg="API listen on /var/run/docker.sock"
jul 03 10:17:35 laptop systemd[1]: Started Docker Application Container Engine.
lines 1-20/20 (END)

Related to my machine

Client:
 Version:      18.03.1-ce
 API version:  1.37
 Go version:   go1.9.5
 Git commit:   9ee9f40
 Built:        Wed Jun 20 21:43:51 2018
 OS/Arch:      linux/amd64
 Experimental: false
 Orchestrator: swarm

Server:
 Engine:
  Version:      18.03.1-ce
  API version:  1.37 (minimum version 1.12)
  Go version:   go1.9.5
  Git commit:   9ee9f40
  Built:        Wed Jun 20 21:42:00 2018
  OS/Arch:      linux/amd64
  Experimental: false

docker info:

docker info
Containers: 1
 Running: 0
 Paused: 0
 Stopped: 1
Images: 1
Server Version: 18.03.1-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 773c489c9c1b21a6d78b5c538cd395416ec50f88
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.15.0-23-generic
Operating System: Ubuntu 18.04 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 31.38GiB
Name: bluesmonk-ubuntu
ID: FRUH:57KI:POWV:EAHH:CY6Y:J3UH:OWBH:AIYF:BONF:DH4Q:5Y2P:RZ6T
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

about the distro

$ cat /etc/*release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04 LTS"
NAME="Ubuntu"
VERSION="18.04 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

Probably related

• https://development.robinwinslow.uk/2016/06/23/fix-docker-networking-dns/
• https://github.com/moby/moby/issues/23910
• https://stackoverflow.com/questions/35693117/how-can-i-give-docker-containers-access-to-a-dnsmasq-local-dns-resolver-on-the-h

Thanks!

ps: I've opened a question in the docker forums with the same info https://forums.docker.com/t/cant-access-internet-after-installing-docker-in-a-fresh-ubuntu-18-04-machine/53416

Solution

So here's your problem:

Docker has assigned the range 172.17.0.1/16 to your docker0 interface. This includes all addresses from 172.17.0.1 through 172.17.255.255. You will note that this range includes your DNS servers (172.17.100.3 and 172.17.100.70). What you have is a routing problem:

Whenever you host needs to reach your DNS servers, it sees that it has an interface already on the same network (docker0), so it tries to route packets using that interface...which of course go nowhere, which is why your DNS stops working.

Docker doesn't have a simple mechanism for excluding an address range from it's automatic selection process, so you'll probably need to do two things to resolve the problem:

First, explicitly set the address of docker0 in your /etc/docker/daemon.json. Use any network that doesn't conflict your internal networks. E.g.:

{
  "bip": "172.31.0.1/16"
}

You'll need to restart Docker.

Next, to prevent Docker from selecting the same network range for a user defined network (one that you create explicitly using docker network create or implicitly using, e.g., docker-compose or docker stack ...), create a new network and then never use it:

docker network create --subnet 172.17.0.0/16 --config-only do_not_use

This should both resolve your problem and prevent it from cropping back up in the future.

Update

Docker actually documents an arguably better way of accomplishing this in How do I influence which network address ranges Docker chooses during a 'docker network create'?.

This requires setting persistent static routes on your system, which varies between Linux distributions.

Comments

[larsks]

Does your host have an interface on the same network as the DNS server? What is the address that docker assigns to the docker0 interface?

[bluesmonk]

Adresses of DNS Servers are 172.17.100.3 and 172.17.100.70 and if I'm answering your question right, ip a show docker0 (or ifconfig docker0) shows inet 172.17.0.1/16

[bluesmonk]

and ip a show eno2 which is my host interface shows inet 172.16.254.111/25

[jozxyqk]

More answers here, but this is a really frustrating problem to work around especially for a vpn: serverfault.com/questions/916941/…

[bluesmonk]

I'm adding this for the sake of completeness. After changing "bip", some stray networks left were still conflicting with my DNS (and later I found out also with the VPN server) So I did the following: 1. stopped containers using those networks 2. docker network prune until no conflicting network remained 3. next time I run compose up it would fail because some networks were obviously missing, so I did compose up --force-recreate

[Kevin Vasko]

I am running into this same exact case on 18.04. We have 16.04 systems that this does NOT happen on. Why?

[bluesmonk]

I'm not sure but my guess is that network management works differently between versions of Ubuntu

[barro32]

I'm having this problem on 19.10, I don't have a /etc/docker/daemon.json file! Should I create it? And not really sure how to add a route that the article on docker.com is talking about.

[larsks]

You can just create the daemon.json file.

[jozxyqk]

Also set default-address-pools