Docker + Bridges + DHCP

Question

I have a lot of docker containers that I need addressable on the the same LAN as their hosts. Up until now, I have been achieving this by using setting up a bridge and manually assigning them IPs, and managing the IPs myself. An example startup would be like so:

docker run \
--net="none" \ \
--lxc-conf="lxc.network.type = veth" \
--lxc-conf="lxc.network.ipv4 = 192.168.1.3/24" \
--lxc-conf="lxc.network.ipv4.gateway = 192.168.1.254" \
--lxc-conf="lxc.network.link = br0" \
--lxc-conf="lxc.network.name = eth0" \
--lxc-conf="lxc.network.flags = up" \
-d [Docker Image ID]

With the host having the bridge defined in /etc/network/interfaces (ubuntu) like so:

auto eth0
iface eth0 inet manual

auto br0
iface br0 inet static
        address 192.168.1.2
        netmask 255.255.255.0
        gateway 192.168.1.254
        bridge_ports eth0
        bridge_stp off
        bridge_fd 0
        bridge_maxwait 0

Since I discovered serf, I have been trying to move over to using automatic discovery within the containers, so that DHCP can keep track of IPs and hand them out to containers. I since changed the startup command to:

docker run \
--net="none" \
--lxc-conf="lxc.network.type = veth" \
--lxc-conf="lxc.network.link = br0" \
--lxc-conf="lxc.network.flags = up" \
-d [Docker Image ID]
/bin/bash

and the bridge to:

auto br0
iface br0 inet dhcp
        bridge_ports eth0
        bridge_stp off
        bridge_fd 0
        bridge_maxwait 0

This resulted in the container starting up, but not having an IP. I then took advice from an online post who managed to get it done with Fedora, by calling dhclient. Unfortunately this is not working for me within ubuntu based containers.

Below are the following error messages I get under different conditions:

• Running dhclient when I have enabled --privileged in starting the container:

  dhclient: error while loading shared libraries: libc.so.6: cannot open shared object file: Permission denied
  

• Running sudo dhclient eth0 when not in --privileged

  RTNETLINK answers: Operation not permitted
  mv: cannot move '/etc/resolv.conf.dhclient-new.31' to '/etc/resolv.conf': Device or resource busy
  

• Running sudo dhclient or dhclient (no interface specified).

  Returns immediately and there is still no IP or network connectivity.

How can I get docker containers to grab dynamic IPs from the same subnet as their hosts, such that I can deploy containers across multiple hosts without tracking IPs?

Extra Info

• Running DOCKER_OPTS="-e lxc" in /etc/default/docker
• Host is Ubuntu 14.04
• Docker containers are built using from ubuntu:14.04 in the Dockerfile.

Solution 1

It appears that this is an open issue and is this is specific to Ubuntu containers and apparmor.

A workaround was posted in there from bprodoehl:

• Start the container as privileged with --privileged
• Add the following line to the dockerfile: RUN mv /sbin/dhclient /usr/sbin/dhclient
• Run dhclient eth0 and you will still see the error message: mv: cannot move '/etc/resolv.conf.dhclient-new.29' to '/etc/resolv.conf': Device or resource busy, but you will now have an IP and you can use the network.

Solution 2

if you are trying to get a dhcp address on an ubuntu docker container just do the following:

1. set dns option on you docker daemon command (--dns <my_dns_ip>)
2. open /etc/dhcp/dhclient.conf and edit the line which contains request subnet-mask, broadcast-address... and remove the words domain-name, domain-name-servers
3. then after applying service networking restart you'll get a new dhcp address without error messages

Solution 3

I've found a script

https://github.com/jkrauska/tech-notes/blob/master/docker-dhclient.md

that does exactly what you want (the workaround mentioned by Programster).

The mv command is needed, because when you run docker container in privileged mode, docker not define AppArmor profile for container. So machine-default AppArmor profile is used, and it prevents you from running dhclient at it's default path.

Comments

[Federico Bonelli]

it works like a charm for my case, but why is that mv needed? Can you explain please?

[Programster]

That was what was in the solution the guy posted (which I linked to) and it appears to work. Perhaps someone else can explain.

[terdon]

Hi and welcome to the site. We like answers to be a bit more comprehensive here. Could you edit and explain what pipework is, where one can find it, how it can be installed, perhaps give an example of its use? As it stands, your answer is a comment and not an answer.

[paprika]

@Federico Bonelli: Apparmor applies security restrictions to the applications it manages. It manages only applications that it has policy definitions for. These policies match applications based on their path. When you move dhclient from /sbin to /usr/sbin the Apparmor policy definition will not match anymore and Apparmor will not apply any security restrictions to this "unknown" application.

[Goblinhack]

I had to resort to "sudo apparmor_parser -R /etc/apparmor.d/sbin.dhclient" as the workaround didn't, this was an ubuntu 14.04 container fyi. However the error message above was seen and an IP address was then assigned.