Routing from docker containers using a different physical network interface and default gateway

Question

Background Information

I have a server with two network interfaces that is running Docker. Docker, like some virtualization tools, creates a Linux bridge interface called docker0. This interface is configured by default with an IP of 172.17.42.1 and all Docker containers communicate with this interface as their gateway and are assigned IP addresses in the same /16 range. As I understand it, all network traffic to/from containers goes through a NAT, so outbound it appears to come from 172.17.42.1, and inbound it gets sent to 172.17.42.1.

My Setup looks like so:

                                          +------------+        /
                                          |            |       |
                            +-------------+ Gateway 1  +-------
                            |             | 10.1.1.1   |     /
                     +------+-------+     +------------+    |
                     |     eth0     |                      /
                     |   10.1.1.2   |                      |
                     |              |                      |
                     | DOCKER HOST  |                      |
                     |              |                      | Internet
                     |   docker0    |                      |
                     |   (bridge)   |                      |
                     |  172.17.42.1 |                      |
                     |              |                      |
                     |     eth1     |                      |
                     |  192.168.1.2 |                      \
                     +------+-------+     +------------+    |
                            |             |            |     \
                            +-------------+ Gateway 2  +-------
                                          | 192.168.1.1|       |
                                          +------------+            

The Problem

I want to route all traffic from/to any Docker containers out of the second eth1 192.168.1.2 interface to a default gateway of 192.168.1.1, while having all traffic from/to the host machine go out the eth0 10.1.1.2 interface to a default gateway of 10.1.1.1. I've tried a variety of things so far to no avail but the one thing that I think is the closest to correct is to use iproute2 like so:

# Create a new routing table just for docker
echo "1 docker" >> /etc/iproute2/rt_tables

# Add a rule stating any traffic from the docker0 bridge interface should use 
# the newly added docker routing table
ip rule add from 172.17.42.1 table docker

# Add a route to the newly added docker routing table that dictates all traffic
# go out the 192.168.1.2 interface on eth1
ip route add default via 192.168.1.2 dev eth1 table docker

# Flush the route cache
ip route flush cache

# Restart the Docker daemon so it uses the correct network settings
# Note, I do this as I found Docker containers often won't be able
# to connect out if any changes to the network are made while it's     
# running
/etc/init.d/docker restart

When I bring up a container I cannot ping out from it at all after doing this. I'm uncertain if bridge interfaces are handled the same way physical interfaces are for this sort of routing, and just want a sanity check as well as any tips on how I might accomplish this seemingly simple task.

Solution 1

A friend and I ran into this exact problem where we wanted to have docker support multiple network interfaces servicing requests. We were specifically working with the AWS EC2 service where we were also attaching/configuring/bringing up the additional interfaces. In this project, there is more than what you need so I will try to only include what you need here.

First, what we did was create a separate route table for eth1:

ip route add default via 192.168.1.2 dev eth1 table 1001

Next we configured the mangle table to set some connection marks coming in from eth1:

iptables -t mangle -A PREROUTING -i eth1 -j MARK --set-xmark 0x1001/0xffffffff
iptables -t mangle -A PREROUTING -i eth1 -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask 0xffffffff

Finally we add this rule for all fwmarks to use the new table we created.

ip rule add from all fwmark 0x1001 lookup 1001

The below iptables command will restore the connection mark and then allow the routing rule to use the correct routing table.

iptables -w -t mangle -A PREROUTING -i docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j CONNMARK --restore-mark --nfmask 0xffffffff --ctmask 0xffffffff

I believe this is all that is needed from our more complex example where (like I said) our project was attaching/configuring/bringing up the eth1 interface at boot time.

Now this example would not stop connections from eth0 from servicing requests through to docker0 but I believe you could add a routing rule to prevent that.

Solution 2

You might have to look more on the iptables setup also. Docker masquerades all traffic originating from the container subnet, say 172.17.0.0/16, to 0.0.0.0. If you run iptables -L -n -t nat, you can see the POSTROUTING chain under nat table which does this -

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
MASQUERADE  all  --  172.17.0.0/16        0.0.0.0/0

Now, you can remove this rule and replace it with a rule which masquerades all traffic originating from the containers' subnet to your second interface's IP - 192.168.1.2, as that is what you desire. The delete rule will be, assuming that it is the first rule under POSTROUTING chain -

iptables -t nat -D POSTROUTING 1

Then you add this custom rule -

iptables -t nat -A POSTROUTING -s 172.17.0.0/16 -j SNAT --to-source 192.168.1.2

Comments

[hookenz]

Your comment about the NAT isn't right. It's set to masquerade on the 172.17.0.0/16 source address where the output interface isn't docker0. i.e. -A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE. Which means if the packet originating from a docker container exits via either eth0 or eth1 it'll take on that interfaces IP address, not the 172.17.x.x IP.

[Admin]

Thanks for your comment. So that would change my approach for routing out of a specific interface then, correct? Do you know how I could go about accomplishing this? As it stands Docker seems to just use the default gateway of the host system, and this isn't the desired behavior. I want all things Docker to route out of a specific interface.

[hookenz]

I'm not sure why what I suggested isn't working. But another option could be to use pipework. github.com/jpetazzo/pipework Also the advanced networking section of the docker help might be useful to read. docs.docker.com/articles/networking

[Admin]

@Matt I thank you for your suggestions on this. From what I've read pipework requires additional commands after a container is started to get it access to the correct gateway which seems like a security vulnerability. It was originally created to connect containers together across multiple hosts, which really isn't my goal here. As for the Docker advanced networking docs, I'll take a look again, but it hasn't been helpful in this situation so far (I had read it before posting here). I am going to post a link to this post on the docker Github Issues page and see how it goes from there.

[Admin]

Here is a link back to the Github Issue: github.com/docker/docker/issues/13762

[Stefan Lasiewski]

@stuntmachine Did you ever figure this out? We're having trouble here convincing Docker to use a static route on a host with 2+ interfaces.

[mhristache]

Did you find a solution?

[user696752]

Unfortunately this did not work. My containers were still routing out to the same default gateway as the host. Here's what I ran: ip rule add from 172.17.0.0/16 table docker ip route add default via 192.168.1.2 dev eth1 table docker ip route flush cache /etc/init.d/docker restart From in the container I ran a traceroute and the first hop was 10.1.1.1 when it should have been 192.168.1.1

[hookenz]

Actually it masquarades it to the output interface. That destination part is just saying that any traffic originating from source subnet to any destination will be masquaraded.

[Admin]

Unfortunately this did not work. My complete process was to run: ip rule add from 172.17.0.0/16 table docker ip route add default via 192.168.1.2 dev eth1 table docker ip route flush cache iptables -t nat -D POSTROUTING 1 iptables -t nat -A POSTROUTING -s 172.17.0.0/16 -j SNAT --to-source 192.168.1.2 /etc/init.d/docker restart. Then I tried to run a ping out and a traceroute from a container, and was unable to reach anything.

[williamsbdev]

@stuntmachine just curious if you had a chance to try this out? If you have already figured out something on your own, would you be able to share your solution?

[Jolly Roger]

Hi @williamsbdev - old post and a long shot, but do you think this solution would also work for my issue on SO? stackoverflow.com/questions/51312310/…

[williamsbdev]

Jolly Roger - I do believe this will solve your issue. I’ve recently had another team ask me about this blog post (essentially the same solution as here) and they said it worked great. williamsbdev.com/posts/docker-connection-marking

[Ram]

I had a similar requirement, host has secondary ip setup to reach out to a public IP. And I wanted docker container to follow the same. This worked in that case: "iptables -t nat -I POSTROUTING 1 -s 172.17.0.0/16 -d PUBIP/32 -j SNAT --to-source SECONDARYIP"

[sashk]

@williamsbdev you would be surprised, but in 2021 this solution is still valid and works. I would expect docker will have support for this scenario internally, but nope. thanks for sharing and explaining!