How to configure docker default bridge work with unusual network configuration?

Question

Basically, at work I have a dhcp address assigned as:

eth0      Link encap:Ethernet  HWaddr 5c:26:0a:5a:b8:48  
          inet addr:10.10.10.193  Bcast:10.10.10.255  Mask:255.255.255.0
          inet6 addr: <addr here>/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:3591236 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2057576 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:3449424352 (3.4 GB)  TX bytes:384131635 (384.1 MB)
          Interrupt:20 Memory:e2e00000-e2e20000 

and with this, my host, can connect to the internet just fine. But none of my docker machines can connect to the internet at work. Their configuration looks like this:

docker0   Link encap:Ethernet  HWaddr 56:84:7a:fe:97:99  
          inet addr:172.17.42.1  Bcast:0.0.0.0  Mask:255.255.0.0
          inet6 addr: fe80::5484:7aff:fefe:9799/64 Scope:Link
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:117799 errors:0 dropped:0 overruns:0 frame:0
          TX packets:170586 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:4858816 (4.8 MB)  TX bytes:122237788 (122.2 MB)

Everything works when I'm at home sitting beneath a traditional 192.168 router switch.

So, I'm thinking, if I somehow get docker0 interface to sit natted behind eth0, then everything would work, both at home and at work. But I'm not familiar with configuring linux interfaces. I found an article that talked about almost the exact same problem, but changing following those commands to add interface br0 to 10.10.10.200/24 made the following symptoms arise:

1. My host no longer can resolve a domain name. Removing the interface br0 made this immediately work again
2. The dockerized apps can now ping 4.2.2.1, but not 8.8.8.8 or 8.8.4.4 or resolve a domain name. Adding --dns 4.2.2.1 tp DOCKER_OPTS in /etc/default/docker.io does not solve the problem.
3. The dockerized apps no longer can ping 4.2.2.1 or 8.8.8.8 or 8.8.4.4 after the br0 interface is removed

I haven't changed iptables; it's using the default docker configuration changes for a basic ubuntu 14.04 host.

How do I best configure the interfaces in order that docker allow the dockerized applications to connect to the internet both at home and work?

Solution

A lot of details about networking and bridge can be found at:

• Ubuntu: https://help.ubuntu.com/community/NetworkConnectionBridge
• Docker: https://docs.docker.com/articles/networking/#building-your-own-bridge.

As I have a VM with Ubuntu 14.04, I'm not sure if that would reproduce the solution. However, I have the same exact situation in my office, where some VPN servers give the same exact default network IP that Docker uses by default on docker0 bridge. The behavior of being able to use Docker from the office and not being able to use docker when VPN'ed was really frustrating.

So, I have used the same strategy described at the link you used http://jpetazzo.github.io/2013/10/16/configure-docker-bridge-network/, but on RHEL 6.5 servers. However, I did try many different options to get it working:

1. Used a different IP range
2. Used a different mask
3. Try manual setup first, then automate the permanent solution.

I have the solution on RHEL 6.5 as follows:

[root@pppdc9prd6dq newww]# cat /etc/sysconfig/network-scripts/ifcfg-bridge0
TYPE=Bridge
DEVICE=bridge0
NETMASK=255.255.252.0
IPADDR=192.168.5.1
ONBOOT=yes
BOOTPROTO=none
NM_CONTROLLED=no
DELAY=0

Manually add bridge

Here are the steps for you to create a bridge manually:

1. Stop Docker

$ sudo service docker stop

2. Create the bridge

$ ip link add bridge0 type bridge
$ ip addr add 192.168.5.1/20 dev bridge0
$ ip link set bridge0 up

3. Update the Docker daemon to use the bridge

$ vim /etc/docker/daemon.json
$ { "bridge": "bridge0"}

4. Restart Docker

sudo service start docker

If everything is working fine, just permanently add the fix

Persistent

1. Same as manual

2. Update the following file

$ vim /etc/network/interfaces
auto bridge0
iface bridge0 inet static
    address 192.168.5.1
    netmask 255.255.252.0
    bridge_ports dummy0
    bridge_stp off
    bridge_fd 0

3. Same as manual

4. Same as manual

And make sure that bridge-utils are installed on the server, otherwise the bridge interface won't come up.

Maybe that would work? Anyway, try anything here and we can discuss and change this solution. I'm sure more people will have problems with this when they start using Docker internally behind a VPN.