No network connectivity to/from Docker CE container on CentOS 8

Question

I just installed the latest release of docker-ce on CentOS, but I can't reach published ports from a neighboring server and can't reach the outside from the container itself.

Running a plain vanilla CentOS 8 with NetworkManager and FirewallD enabled. Default firewall zone is public.

Versions:

• docker-ce 19.03.3 (official Docker RPM)
• containerd.io 1.2.6 (official Docker RPM for CentOS 7 - not available for CentOS 8 yet)
• CentOS 8.0.1905 (minimal install)

Solution 1

After spending a couple of days looking at logs and configurations for the involved components, I was about to throw in the towel and revert back to Fedora 30, where this seems to work straight out of the box.

Focusing on firewalling, I realized that disabling firewalld seemed to do the trick, but I would prefer not to do that. While inspecting network rules with iptables, I realized that the switch to nftables means that iptables is now an abstraction layer that only shows a small part of the nftables rules. That means most - if not all - of the firewalld configuration will be applied outside the scope of iptables.

I was used to be able to find the whole truth in iptables, so this will take some getting used to.

Long story short - for this to work, I had to enable masquerading. It looked like dockerd already did this through iptables, but apparently this needs to be specifically enabled for the firewall zone for iptables masquerading to work:

# Masquerading allows for docker ingress and egress (this is the juicy bit)
firewall-cmd --zone=public --add-masquerade --permanent

# Specifically allow incoming traffic on port 80/443 (nothing new here)
firewall-cmd --zone=public --add-port=80/tcp
firewall-cmd --zone=public --add-port=443/tcp

# Reload firewall to apply permanent rules
firewall-cmd --reload

Reboot or restart dockerd, and both ingress and egress should work.

Solution 2

What's missing from the answers before is the fact that you first need to add your docker interface to the zone you configure, e.g. public (or add it to the "trusted" zone which was suggested here but I doubt that's wise, from a security perspective). Because by default it's not assigned to a zone. Also remember to reload the docker daemon when done.

# Check what interface docker is using, e.g. 'docker0'
ip link show

# Check available firewalld zones, e.g. 'public'
sudo firewall-cmd --get-active-zones

# Check what zone the docker interface it bound to, most likely 'no zone' yet
sudo firewall-cmd --get-zone-of-interface=docker0

# So add the 'docker0' interface to the 'public' zone. Changes will be visible only after firewalld reload
sudo nmcli connection modify docker0 connection.zone public

# Masquerading allows for docker ingress and egress (this is the juicy bit)
sudo firewall-cmd --zone=public --add-masquerade --permanent
# Optional open required incomming ports (wasn't required in my tests)
# sudo firewall-cmd --zone=public --add-port=443/tcp
# Reload firewalld
sudo firewall-cmd --reload
# Reload dockerd
sudo systemctl restart docker

# Test ping and DNS works:
docker run busybox ping -c 1 172.16.0.1
docker run busybox cat /etc/resolv.conf
docker run busybox ping -c 1 yourhost.local

Solution 3

To be able to set fine-grained rules for Docker, I did not need to set docker0 to any zone.

# 1. Stop Docker
systemctl stop docker

# 2. Recreate DOCKER-USER chain in firewalld. 
firewall-cmd --permanent \
             --direct \
             --remove-chain ipv4 filter DOCKER-USER

firewall-cmd --permanent \
             --direct \
             --remove-rules ipv4 filter DOCKER-USER

firewall-cmd --permanent \
             --direct \
             --add-chain ipv4 filter DOCKER-USER

# (Ignore any warnings)

# 3. Docker Container <-> Container communication

firewall-cmd --permanent \
             --direct \
             --add-rule ipv4 filter DOCKER-USER 1 \
             -m conntrack --ctstate RELATED,ESTABLISHED \
             -j ACCEPT \
             -m comment \
             --comment 'Allow docker containers to connect to the outside world'

firewall-cmd --permanent \
             --direct \
             --add-rule ipv4 filter DOCKER-USER 1 \
             -j RETURN \
             -s 172.17.0.0/16 \
             -m comment \
             --comment 'allow internal docker communication'

# Change the Docker Subnet to your actual one (e.g. 172.18.0.0/16)

# 4. Add rules for IPs allowed to access the Docker exposed ports.

firewall-cmd --permanent \
             --direct \
             --add-rule ipv4 filter DOCKER-USER 1 \
             -o docker0 \
             -p tcp \
             -m multiport \
             --dports 80,443 \
             -i eth0 \
             -o docker0 \
             -s 1.2.3.4/32 \
             -j ACCEPT \
             -m comment \
             --comment 'Allow IP 1.2.3.4 to docker ports 80 and 443'

# 5. log docker traffic (if you like)

firewall-cmd --direct \
             --add-rule ipv4 filter DOCKER-USER 0 \
             -j LOG \
             --log-prefix ' DOCKER: '

# 6. Block all other IPs. 
This rule has lowest precedence, so you can add allowed IP rules later.

firewall-cmd --permanent \
             --direct \
             --add-rule ipv4 filter DOCKER-USER 10 \
             -j REJECT \
             -m comment \
             --comment 'reject all other traffic to DOCKER-USER'

# 7. Reload firewalld, Start Docker again
firewall-cmd --reload
systemctl start docker

This ends in rules defined in /etc/firewalld/direct.xml:

<?xml version="1.0" encoding="utf-8"?>
<direct>
  <chain ipv="ipv4" table="filter" chain="DOCKER-USER"/>
  <rule ipv="ipv4" table="filter" chain="DOCKER-USER" priority="0">-m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT -m comment --comment 'Allow docker containers to connect to the outside world'</rule>
  <rule ipv="ipv4" table="filter" chain="DOCKER-USER" priority="0">-j RETURN -s 172.17.0.0/16 -m comment --comment 'allow internal docker communication'</rule>
  <rule ipv="ipv4" table="filter" chain="DOCKER-USER" priority="0">-p tcp -m multiport --dports 80,443 -s 1.2.3.4/32 -j ACCEPT -m comment --comment 'Allow IP 1.2.3.4 to docker ports 80 and 443'</rule>
  <rule ipv="ipv4" table="filter" chain="DOCKER-USER" priority="0">-j LOG --log-prefix ' DOCKER TCP: '</rule>
  <rule ipv="ipv4" table="filter" chain="DOCKER-USER" priority="10">-j REJECT -m comment --comment 'reject all other traffic to DOCKER-USER'</rule>
</direct>

Drawback still is that you need to install containerd.io from CentOS7 as stated by Saustrup

Solution 4

I have changed the FirewallBackend variable to iptables again and it works for me.

With this update, the nftables filtering subsystem is the default firewall backend for the firewalld daemon. To change the backend, use the FirewallBackend option in the /etc/firewalld.conf file.

Link: Centos8 Deprecated_functionality

I don't have too much information about this behavior change. Some of the iptables rules that Docker tries to use are not working according to the CentOS8 logs:

WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -D FORWARD -i docker0 -o docker0 -j DROP' failed: iptables: Bad rule (does a matching rule exist in that chain?).

View more solutions

Comments

[blissweb]

Mine doesn't even work with firewalld completely disabled. Anyone have any ideas ??

[Kevin]

I have seen professional blogs only recommending to disable firewalld. Not going to do that. After some digging, I found people recommending to add the docker0 interface to the trusted zone. That does not seem much better, but I will admit to not knowing if that could be made viable under some environments. Using masquerade seems like a good solution, but I unfamiliar with the term outside of NetworkManager DNS caching. Does it simply mean that the zone is allowed to configure forwarding?

[EKOlog]

It helped mi with Sql Server running in docker container. Thanks!

[user528025]

remember people, docker is only supports centos 7. :angryface:

[Bertl]

docker0 is in public zone by default (implicite)

[omni]

Where do you see that information / implicite hierarchy? In my tests it would not work without explicitly adding it to the zone in use.

[Bertl]

To be able to set fine-grained rules, I did not need to set docker0 to any zone.

[Bertl]

See my answer below.

[jamshid]

Are you sure --add-port is necessary? Your first --add-masquerade command and reload (and restart of docker) exports all published container ports automatically for me, at least on docker 19.03.12 on centos 8.2.2004.

[Kali Kimanzi]

worked like a charm for me, i had spend so much time with this issue

[Samyok Nepal]

Spent all day today working on different solutions. This worked. Thanks so much!!!

[Brian Sidebotham]

Thank you! This really helped me trying to figure out why gilab-runner's docker executor was not playing ball with docker-in-docker configurations!

[Incömplete]

You are a gift to the humanity, Thanks.

[Jose Cabrera Zuniga]

Man, this trick saved my neck today!!

[balping]

This was the solution after 4 hours of misery on openSUSE 15.3. The only difference was that I needed to run the first command for zone docker ie.: firewall-cmd --zone=docker --add-masquerade --permanent

[Admin]

This worked for me on RHEL 8.6.