Docker on CentOS with bridge to LAN network

Question

I have a server VLAN of 10.101.10.0/24 and my Docker host is 10.101.10.31. How do I configure a bridge network on my Docker host (VM) so that all the containers can connect directly to my LAN network without having to redirect ports around on the default 172.17.0.0/16? I tried searching but all the howtos I've found so far have resulted in losing SSH session which I had to go into the VM from a console to revert the steps I did.

Solution

There's multiple ways this can be done. The two I've had most success with are routing a subnet to a docker bridge and using a custom bridge on the host LAN.

Docker Bridge, Routed Network

This has the benefit of only needing native docker tools to configure docker. It has the down side of needing to add a route to your network, which is outside of dockers remit and usually manual (or relies on the "networking guy").

1. Enable IP forwarding

   /etc/sysctl.conf: net.ipv4.ip_forward = 1
   sysctl -p /etc/sysctl.conf
   

   Create a docker bridge with new subnet on your VM network, say 10.101.11.0/24

   docker network create routed0 --subnet 10.101.11.0/24
   

2. Tell the rest of the network that 10.101.11.0/24 should be routed via 10.101.10.X where X is IP of your docker host. This is the external router/gateway/"network guy" config. On a linux gateway you could add a route with:

   ip route add 10.101.11.0/24 via 10.101.10.31
   

3. Create containers on the bridge with 10.101.11.0/24 addresses.

   docker run --net routed0 busybox ping 10.101.10.31
   docker run --net routed0 busybox ping 8.8.8.8
   

Then your done. Containers have routable IP addresses. If you're ok with the network side, or run something like RIP/OSPF on the network or Calico that takes care of routing then this is the cleanest solution.

Custom Bridge, Existing Network (and interface)

This has the benefit of not requiring any external network setup. The downside is the setup on the docker host is more complex. The main interface requires this bridge at boot time so it's not a native docker network setup. Pipework or manual container setup is required.

Using a VM can make this a little more complicated as you are running extra interfaces with extra MAC addresses over the main VM's interface which will need additional "Promiscuous" config first to allow this to work.

The permanent network config for bridged interfaces varies by distro. The following commands outline how to set the interface up and will disappear after reboot. You are going to need console access or a seperate route into your VM as you are changing the main network interface config.

1. Create a bridge on the host.

   ip link add name shared0 type bridge
   ip link set shared0 up
   

   In /etc/sysconfig/network-scripts/ifcfg-br0

   DEVICE=shared0
   TYPE=Bridge
   BOOTPROTO=static
   DNS1=8.8.8.8
   GATEWAY=10.101.10.1
   IPADDR=10.101.10.31
   NETMASK=255.255.255.0
   ONBOOT=yes
   

2. Attach the primary interface to the bridge, usually eth0

   ip link set eth0 up
   ip link set eth0 master shared0
   

   In /etc/sysconfig/network-scripts/ifcfg-eth0

   DEVICE=eth0
   ONBOOT=yes
   TYPE=Ethernet
   IPV6INIT=no
   USERCTL=no
   BRIDGE=shared0
   

3. Reconfigure your bridge to have eth0's ip config.

   ip addr add dev shared0 10.101.10.31/24
   ip route add default via 10.101.10.1
   

4. Attach containers to bridge with 10.101.10.0/24 addresses.

   CONTAINERID=$(docker run -d --net=none busybox sleep 600)
   pipework shared1 $CONTAINERID 10.101.10.43/[email protected]
   

   Or use a DHCP client inside the container

   pipework shared1 $CONTAINERID dhclient
   

Docker macvlan network

Docker has since added a network driver called macvlan that can make a container appear to be directly connected to the physical network the host is on. The container is attached to a parent interface on the host.

docker network create -d macvlan \
  --subnet=10.101.10.0/24 \
  --gateway=10.101.10.1  \
  -o parent=eth0 pub_net

This will suffer from the same VM/softswitch problems where the network and interface will need be promiscuous with regard mac addresses.

Comments

[Eroji]

Wow, thank you for the detailed guide. I followed your first method but I must not be doing something right with the static route. I created the network on the host, added the network and created the static route in my pfsense firewall. The L3 switch I'm using has a default route to pfsense, so I figured that should be enough. From the busybox I can ping just fine but when I spin up a container using the routed0 network, it attaches to an IP but I cannot access or ping it.

[Eroji]

I believe I tracked it down to my switch not routing it correctly. I will do more troubleshooting.

[Eroji]

Sweet, I got it working. I did a subnet of 10.101.20.x/26, which splits it in 4 sections for 4 docker hosts. Had to create the static routes on the core switch.

[Matt]

Nice. Yeah it's a bit easier in GCE or AWS where you can create a route that applies everywhere with an API call.

[ealeon]

could macvlan network driver fix this issue as well?

[Matt]

@ealeon Yes, the addition of macvlan to Docker covers the same use case. It has the same VM/softswitch issue of needing to allow traffic to and from a mac address that is not on the main interface.

[33Fraise33]

I've tried the first option. When I add the route to my router I'm able to ping the default gateway of the docker network but I'm unable to ping ip addresses taken by my docker containers. My host is able to ping those though.

[Matt]

@33Fraise33 sounds like net.ipv4.ip_forward might not be set on the host? or if you are using VM's it could be MAC filtering for the containers.