Docker macvlan network, unable to access internet

Question

I have a dedicated server with multiple IP addresses, some IP's have mac address associated while others(in a subnetwork) doesn't have mac addresses. I have created docker macvlan network using:

docker network create -d macvlan -o macvlan_mode=bridge --subnet=188.40.76.0/26 --gateway=188.40.76.1 -o parent=eth0 macvlan_bridge

I have ip: 88.99.102.115 with mac: 00:50:56:00:60:42. Created a container using:

docker run --name cont1 --net=macvlan_bridge --ip=88.99.102.115 --mac-address 00:50:56:00:60:42 -itd nginx

This works, I can access nginx hosted at that ip address from outside.

Case with IP which doesn't have mac address and the gateway is out of subnet.

subnet: 88.99.114.16/28, gateway: 88.99.102.103

Unable to create network using:

docker network create -d macvlan -o macvlan_mode=bridge --subnet=88.99.114.16/28 --gateway=88.99.102.103 -o parent=eth0 mynetwork

Throws error:

no matching subnet for gateway 88.99.102.103

Tried with increasing subnet scope to include gateway:

docker network create -d macvlan -o macvlan_mode=bridge --subnet=88.99.0.0/16 --gateway=88.99.102.103 -o parent=eth0 mynetwork

Network got created, then started nginx container using 'mynetwork' and well I dont have mac address for 88.99.114.18 so used some random mac address 40:1c:0f:bd:a1:d2.

docker run --name cont1 --net=mynetwork --ip=88.99.114.18 --mac-address 40:1c:0f:bd:a1:d2 -itd nginx

Can't reach nginx(88.99.102.115).

1. How do I create a macvlan docker network if my gateway is out of my subnet?
2. How do I run a container using macvlan network when I have only IP address but no mac address?

I don't have much knowledge in networking, it will be really helpful if you explain in detail.

My /etc/network/interfaces file:

### Hetzner Online GmbH - installimage
# Loopback device:
auto lo
iface lo inet loopback
iface lo inet6 loopback
# device: eth0
auto  eth0
iface eth0 inet static
  address   88.99.102.103
  netmask   255.255.255.192
  gateway   88.99.102.65
  # default route to access subnet
  up route add -net 88.99.102.64 netmask 255.255.255.192 gw 88.99.102.65 eth0

iface eth0 inet6 static
  address 2a01:4f8:221:1266::2
  netmask 64
  gateway fe80::1

Solution

You might want to start by reading up on simple routing concepts or subnets and routing

How do I create a macvlan docker network if my gateway is out of my subnet?

A gateway address must be on the same subnet as an interface. To use this new subnet you will need to use up one of the IP addresses and assign it somewhere on the host as a gateway.

Subnet routing to a bridge network.

From the hosting screen shot, the 88.99.114.16/28 subnet has been setup to route via your host 88.99.102.103. You need to create an interface somewhere on your host to use as the gateway if you want Docker to use the rest of the IP addresses in the subnet.

Create a bridge network for Docker to use, the bridge will be assigned the gateway address 88.99.114.17

docker network create \
  --driver=bridge \
  --subnet 88.99.114.16/28 \
  --gateway=88.99.114.17 \
  name0

You may also need to enable IP forwarding for routing to work. Configure ip forwarding in /etc/sysctl.conf:

net.ipv4.ip_forward = 1

and Apply the new setting

sysctl -p /etc/sysctl.conf

Then run a container on the new bridge with your routed network should be able to access the gateway and the internet

docker run --net=name0 --rm busybox \
  sh -c "ip ad sh && ping -c 4 88.99.114.17 && wget api.ipify.org"

You may need to allow access into the subnet in iptables, depending on your default FORWARD policy

iptables -I DOCKER -d 88.99.114.16/28 -j ACCEPT

Services on the subnet will be accessible from the outside world

docker run --net=name0 busybox \
  nc -lp 80 -e echo -e "HTTP/1.0 200 OK\nContent-Length: 3\n\nHi\n"

Then outside

○→ ping -c 2  88.99.114.18
PING 88.99.114.18 (88.99.114.18): 56 data bytes
64 bytes from 88.99.114.18: icmp_seq=0 ttl=63 time=0.527 ms
64 bytes from 88.99.114.18: icmp_seq=1 ttl=63 time=0.417 ms

--- 88.99.114.18 ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.417/0.472/0.527/0.055 ms

○→ curl 88.99.114.18
Hi

No need for macvlan interface mapping.

How do I run a container using macvlan network when I have only IP address but no mac address?

macvlan is use to map a physical/host interface into a container. As you don't have a physical interface for these addresses it will be hard to map one into a container.

Comments

[user3713466]

the ip alias for eth0 with addr 88.99.114.18 is working, this subnet(88.99.114.16/28) has host ip as the gateway(88.99.102.103). AFAIK when i create a container using macvlan, I am unable to access the internet due to random MAC address, macvlan is supposed to work based on mac addresses, so how could it contact the container if we assign random mac.

[Matt]

and again, a "gateway" has a very specific meaning. 88.99.102.103 is not the gateway for 88.99.114.16/28. The subnet might be routed via the 88.99.114.16 but the "gateway" must be an address inside the subnet.

[user3713466]

Im sure gateway is 88.99.102.103 and is out of my subnet(which seems possible), using macvlan type of network creates a separate virtual interface which frees from port binding all that stuff.

[Matt]

Sorry but it's not possible that 88.99.102.103 is the gateway address for 88.99.114.16/28.

[Matt]

I've had to setup macvlan interfaces on AWS, KVM and VirtualBox for the same reason which is how I've run into the MAC filtering problems. You appear to be in the fortunate position to not need to setup the macvlan interfaces to avoid mapping ports.

[user3713466]

postimg.org/image/trpphtqbh that shows 88.99.102.103 as my gateway

[Matt]

They look to be conflating the term "gateway" for where they are routing that subnet to. The gateway you are being asked to assign to docker is not the same type of gateway.

[Matt]

Also that screen grab confirms it's the subnet routing section in the answer.

[user3713466]

Thanks @Matt, I will confirm with our server provider about gateway term and may be gather more info to resolve this issue.

[user3713466]

i have stripped down the question removing docker specific commands, please check serverfault.com/questions/834105/…

[Matt]

@user3713466 Ok, my answer would be the same as above though except the manual steps to create and attach a container to the bridge. Did you try via the docker bridge network above?

[user3713466]

If I use docker bridge, I have to port forward and docker does this by adding iptables rules, my use case is to treat the container as a fullblown linux OS, so I need to use all the ports, forwarding these many ports takes about 10-20mins and messes up iptables.

[Matt]

No port forwarding, docker-proxy or iptables is required. The real world IP's will be assigned to the containers on that bridge and will be directly accessible.

[user3713466]

I have tried above configuration, I can't access the container from outside, eg: runngin...docker run --name nginx --net=name0 --ip=88.99.114.20 -itd nginx I cant access 88.99.114.20 from outside. But it works within host(88.99.102.103)

[Matt]

Your iptables FORWARD policy might default to drop? try inserting the rule above

[user3713466]

Wow man, finally this works, really thanks for your concern, have to learn all these interesting stuff.

[Matt]

Cool, got there in the end!