Forward traffic between VLANs with iptables

linux networking iptables routing vlan

23,717

Solution 1

The problem is now at least partly solved (I'm fine with this).

I was testing the network with Windows clients in the beginning, and both PCs had got the Windows 10 upgrade lately. The problem itself was really stupid: Windows didn't response to pings since there is some new security enabled by default in Windows 10. When I added one Linux laptop to the network I got to a situation where all others could ping the Linux laptop (default policy was ALLOW) but the Linux PC couldn't ping anything else (Windows didn't response to ping). Then I tried remote desktop to the Windows PCs in VLANs 10 and 50, from the Linux PC in VLAN 100 and boom - it worked!

So there was no problem at all with the iptables/netfilter rules.

Thank you very much for all answers and comments!

Here is my final, working and simplified iptables config:

# Generated by iptables-save v1.6.0 on Thu May 26 16:00:55 2016
*filter
:INPUT ACCEPT [359:39449]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [563:89333]
-A FORWARD -i eth1+ -o eth1.100 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i eth1.100 -o eth1+ -j ACCEPT
-A FORWARD -i eth1+ -o eth0 -j ACCEPT
-A FORWARD -i eth0 -o eth1+ -m state --state RELATED,ESTABLISHED -j ACCEPT
COMMIT
# Completed on Thu May 26 16:00:55 2016
# Generated by iptables-save v1.6.0 on Thu May 26 16:00:55 2016
*nat
:PREROUTING ACCEPT [5650:1147271]
:INPUT ACCEPT [91:14019]
:OUTPUT ACCEPT [325:31088]
:POSTROUTING ACCEPT [44:7161]
-A POSTROUTING -o eth0 -j MASQUERADE
COMMIT
# Completed on Thu May 26 16:00:55 2016

Solution 2

First, your default policy in the FORWARD chain is ACCEPT, so you're not actually denying any traffic. This explains why the 20 -> 50 forwarding works. You can fix this with iptables -P FORWARD DROP.

As far as the other pings, notice that in your log entries, the TTL is either 127 or 63, indicating the firewall has already made its routing decision and decremented the TTL. Also, your firewall is not configured to block any traffic. This suggests the problem is in the routing configuration of the clients. May I suggest you use a sniffer on the ping recipient to see if the original ping packet is arriving? Make sure the default gateway for clients (not the firewall) on 10.10.10.0/24 is 10.10.10.1, etc. Good luck!

And following up on the routing table comment, your routing table is correct, indicating that your firewall is directly connected to your VLANed networks.

23,717

AirPett

I'm a hobbyist programmer writing C, C#, PHP, HTML, CSS, JavaScript.

Updated on September 18, 2022

Comments

AirPett over 1 year

We're having the following 4 VLANs on my network, connected to an Ubuntu Linux box with DHCP. This Linux should act as the L3 router as well.

VLAN 10  on interface eth1.10  with subnet 10.10.10.0/24
VLAN 20  on interface eth1.20  with subnet 10.10.20.0/24
VLAN 50  on interface eth1.50  with subnet 10.10.50.0/24
VLAN 100 on interface eth1.100 with subnet 10.10.100.0/24

Here is the /etc/network/interfaces:

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
        address 192.168.10.169
        netmask 255.255.255.0
        gateway 192.168.10.1

auto eth1
iface eth1 inet static
        address 10.10.0.1
        network 10.10.0.0
        netmask 255.255.255.0
        broadcast 10.10.0.255

auto eth1.100
iface eth1.100 inet static
        address 10.10.100.1
        network 10.10.100.0
        netmask 255.255.255.0
        broadcast 10.10.100.255

auto eth1.10
iface eth1.10 inet static
        address 10.10.10.1
        network 10.10.10.0
        netmask 255.255.255.0
        broadcast 10.10.10.255

auto eth1.20
iface eth1.20 inet static
        address 10.10.20.1
        network 10.10.20.0
        netmask 255.255.255.0
        broadcast 10.10.20.255

auto eth1.50
iface eth1.50 inet static
        address 10.10.50.1
        network 10.10.50.0
        netmask 255.255.255.0
        broadcast 10.10.50.255

Now, all clients from all VLANs should be able to connect to the public internet via the interface eth0. That part actually works with the iptables rule -A POSTROUTING -o eth0 -j MASQUERADE. The DHCP server is working as well.

BUT since the VLAN 100 is going to be the network for administration computers, the clients in VLAN 100 should be able to access all other computers on the VLANs 10, 20 and 50. AND the clients in the VLANs 10, 20 and 50 should only be able to access computers inside their own VLANs.

I've tried so far the following iptables rules alongside with the MASQUERADE:

-A FORWARD -i eth1.100 -o eth1.10 -j ACCEPT
-A FORWARD -i eth1.100 -o eth1.20 -j ACCEPT
-A FORWARD -i eth1.100 -o eth1.50 -j ACCEPT
-A FORWARD -i eth1.10 -o eth1.100 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i eth1.20 -o eth1.100 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i eth1.50 -o eth1.100 -m state --state RELATED,ESTABLISHED -j ACCEPT

I tried to ping a PC with IP-address 10.10.20.100 from a PC with IP-address 10.10.100.101, but it didn't succeed. Nor was I able to ping 10.10.50.101 from the 100-subnet.

AND there is a weird side-behavior: The VLAN 20 is by accident (I have no idea why) acting like the 100 should. From there I can ping PCs in the VLANs 10 and 100, which should not be possible in the end.

I have IPv4 forwarding enabled in the kernel and the external internet works as expected.

Here is the complete output of iptables-save:

# Generated by iptables-save v1.6.0 on Thu May 26 09:28:59 2016
*filter
:INPUT ACCEPT [7375:724156]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [3695:415474]
-A FORWARD -i eth1.100 -o eth1.10 -j ACCEPT
-A FORWARD -i eth1.100 -o eth1.20 -j ACCEPT
-A FORWARD -i eth1.100 -o eth1.50 -j ACCEPT
-A FORWARD -i eth1.10 -o eth1.100 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i eth1.20 -o eth1.100 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i eth1.50 -o eth1.100 -m state --state RELATED,ESTABLISHED -j ACCEPT
COMMIT
# Completed on Thu May 26 09:28:59 2016
# Generated by iptables-save v1.6.0 on Thu May 26 09:28:59 2016
*nat
:PREROUTING ACCEPT [32796:9980970]
:INPUT ACCEPT [142:30526]
:OUTPUT ACCEPT [1829:211124]
:POSTROUTING ACCEPT [128:29756]
-A POSTROUTING -o eth0 -j MASQUERADE
COMMIT
# Completed on Thu May 26 09:28:59 2016

And the output of route:

Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         192.168.10.1    0.0.0.0         UG    0      0        0 eth0
10.10.0.0       *               255.255.255.0   U     0      0        0 eth1
10.10.10.0      *               255.255.255.0   U     0      0        0 eth1.10
10.10.20.0      *               255.255.255.0   U     0      0        0 eth1.20
10.10.50.0      *               255.255.255.0   U     0      0        0 eth1.50
10.10.100.0     *               255.255.255.0   U     0      0        0 eth1.100
link-local      *               255.255.0.0     U     1000   0        0 eth0
192.168.10.0    *               255.255.255.0   U     0      0        0 eth0

Thanks in advance, hope someone will figure out which iptables rules should I use to achieve what I want (if that's the problem at all).

EDIT:

As @Sanael requested, I did some more logging (-A FORWARD -o eth1+ -p icmp -j LOG --log-prefix "IPTABLES FORWARD: " --log-level 7) and here are the results:

Ping 10.10.20.101 --> 10.10.50.100 succeeded with the following log:

May 26 12:14:57 homeserver kernel: IPTABLES FORWARD: IN=eth1.20 OUT=eth1.50 MAC=00:0a:5e:50:7c:c1:c8:0a:a9:e5:f0:bc:08:00:45:00:00:3c SRC=10.10.20.101 DST=10.10.50.100 LEN=60 TOS=0x00 PREC=0x00 TTL=127 ID=23708 PROTO=ICMP TYPE=8 CODE=0 ID=1 SEQ=96
May 26 12:14:57 homeserver kernel: IPTABLES FORWARD: IN=eth1.50 OUT=eth1.20 MAC=00:0a:5e:50:7c:c1:00:1e:ec:fa:d1:10:08:00:45:00:00:3c SRC=10.10.50.100 DST=10.10.20.101 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=15185 PROTO=ICMP TYPE=0 CODE=0 ID=1 SEQ=96
May 26 12:14:58 homeserver kernel: IPTABLES FORWARD: IN=eth1.20 OUT=eth1.50 MAC=00:0a:5e:50:7c:c1:c8:0a:a9:e5:f0:bc:08:00:45:00:00:3c SRC=10.10.20.101 DST=10.10.50.100 LEN=60 TOS=0x00 PREC=0x00 TTL=127 ID=23709 PROTO=ICMP TYPE=8 CODE=0 ID=1 SEQ=97
May 26 12:14:58 homeserver kernel: IPTABLES FORWARD: IN=eth1.50 OUT=eth1.20 MAC=00:0a:5e:50:7c:c1:00:1e:ec:fa:d1:10:08:00:45:00:00:3c SRC=10.10.50.100 DST=10.10.20.101 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=15317 PROTO=ICMP TYPE=0 CODE=0 ID=1 SEQ=97
May 26 12:14:59 homeserver kernel: IPTABLES FORWARD: IN=eth1.20 OUT=eth1.50 MAC=00:0a:5e:50:7c:c1:c8:0a:a9:e5:f0:bc:08:00:45:00:00:3c SRC=10.10.20.101 DST=10.10.50.100 LEN=60 TOS=0x00 PREC=0x00 TTL=127 ID=23710 PROTO=ICMP TYPE=8 CODE=0 ID=1 SEQ=98
May 26 12:14:59 homeserver kernel: IPTABLES FORWARD: IN=eth1.50 OUT=eth1.20 MAC=00:0a:5e:50:7c:c1:00:1e:ec:fa:d1:10:08:00:45:00:00:3c SRC=10.10.50.100 DST=10.10.20.101 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=15349 PROTO=ICMP TYPE=0 CODE=0 ID=1 SEQ=98
May 26 12:15:00 homeserver kernel: IPTABLES FORWARD: IN=eth1.20 OUT=eth1.50 MAC=00:0a:5e:50:7c:c1:c8:0a:a9:e5:f0:bc:08:00:45:00:00:3c SRC=10.10.20.101 DST=10.10.50.100 LEN=60 TOS=0x00 PREC=0x00 TTL=127 ID=23711 PROTO=ICMP TYPE=8 CODE=0 ID=1 SEQ=99
May 26 12:15:00 homeserver kernel: IPTABLES FORWARD: IN=eth1.50 OUT=eth1.20 MAC=00:0a:5e:50:7c:c1:00:1e:ec:fa:d1:10:08:00:45:00:00:3c SRC=10.10.50.100 DST=10.10.20.101 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=15597 PROTO=ICMP TYPE=0 CODE=0 ID=1 SEQ=99

Ping 10.10.20.101 --> 10.10.100.100 failed with the following log:

May 26 12:09:06 homeserver kernel: IPTABLES FORWARD: IN=eth1.20 OUT=eth1.100 MAC=00:0a:5e:50:7c:c1:c8:0a:a9:e5:f0:bc:08:00:45:00:00:3c SRC=10.10.20.101 DST=10.10.100.100 LEN=60 TOS=0x00 PREC=0x00 TTL=127 ID=18715 PROTO=ICMP TYPE=8 CODE=0 ID=1 SEQ=85
May 26 12:09:11 homeserver kernel: IPTABLES FORWARD: IN=eth1.20 OUT=eth1.100 MAC=00:0a:5e:50:7c:c1:c8:0a:a9:e5:f0:bc:08:00:45:00:00:3c SRC=10.10.20.101 DST=10.10.100.100 LEN=60 TOS=0x00 PREC=0x00 TTL=127 ID=18716 PROTO=ICMP TYPE=8 CODE=0 ID=1 SEQ=86
May 26 12:09:16 homeserver kernel: IPTABLES FORWARD: IN=eth1.20 OUT=eth1.100 MAC=00:0a:5e:50:7c:c1:c8:0a:a9:e5:f0:bc:08:00:45:00:00:3c SRC=10.10.20.101 DST=10.10.100.100 LEN=60 TOS=0x00 PREC=0x00 TTL=127 ID=18717 PROTO=ICMP TYPE=8 CODE=0 ID=1 SEQ=87

Ping 10.10.50.100 --> 10.10.20.101 failed with the following log:

May 26 12:11:28 homeserver kernel: IPTABLES FORWARD: IN=eth1.50 OUT=eth1.20 MAC=00:0a:5e:50:7c:c1:00:1e:ec:fa:d1:10:08:00:45:00:00:54 SRC=10.10.50.100 DST=10.10.20.101 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=54378 DF PROTO=ICMP TYPE=8 CODE=0 ID=1903 SEQ=1
May 26 12:11:29 homeserver kernel: IPTABLES FORWARD: IN=eth1.50 OUT=eth1.20 MAC=00:0a:5e:50:7c:c1:00:1e:ec:fa:d1:10:08:00:45:00:00:54 SRC=10.10.50.100 DST=10.10.20.101 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=54495 DF PROTO=ICMP TYPE=8 CODE=0 ID=1903 SEQ=2
May 26 12:11:30 homeserver kernel: IPTABLES FORWARD: IN=eth1.50 OUT=eth1.20 MAC=00:0a:5e:50:7c:c1:00:1e:ec:fa:d1:10:08:00:45:00:00:54 SRC=10.10.50.100 DST=10.10.20.101 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=54693 DF PROTO=ICMP TYPE=8 CODE=0 ID=1903 SEQ=3

Ping 10.10.50.100 --> 10.10.100.100 failed with the following log:

May 26 12:12:24 homeserver kernel: IPTABLES FORWARD: IN=eth1.50 OUT=eth1.100 MAC=00:0a:5e:50:7c:c1:00:1e:ec:fa:d1:10:08:00:45:00:00:54 SRC=10.10.50.100 DST=10.10.100.100 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=8069 DF PROTO=ICMP TYPE=8 CODE=0 ID=1905 SEQ=1
May 26 12:12:25 homeserver kernel: IPTABLES FORWARD: IN=eth1.50 OUT=eth1.100 MAC=00:0a:5e:50:7c:c1:00:1e:ec:fa:d1:10:08:00:45:00:00:54 SRC=10.10.50.100 DST=10.10.100.100 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=8269 DF PROTO=ICMP TYPE=8 CODE=0 ID=1905 SEQ=2
May 26 12:12:26 homeserver kernel: IPTABLES FORWARD: IN=eth1.50 OUT=eth1.100 MAC=00:0a:5e:50:7c:c1:00:1e:ec:fa:d1:10:08:00:45:00:00:54 SRC=10.10.50.100 DST=10.10.100.100 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=8375 DF PROTO=ICMP TYPE=8 CODE=0 ID=1905 SEQ=3

Ping 10.10.100.100 --> 10.10.20.101 failed BUT didn't create any log output.

Ping 10.10.100.100 --> 10.10.50.100 succeeded BUT didn't create any log output.

setenforce 1 almost 8 years

It looks ok. Could you please log the packets with -A FORWARD -o eth1+ -p icmp -j LOG --log-prefix "IPTABLES FORWARD: " --log-level 7 and add to your question the logs of the 100 to 20 ping, and the ones from 20 to anything else ? You can get them with journalctl -xef -n 50 -k.
AirPett almost 8 years

Thanks for the suggestion @Sanael I did now the logging and please see my updated answer for the log data.
Sum1sAdmin almost 8 years

is there a reason you can't do this on your switch, do all of these VLANS terminate on the Linux box? - I would use openvswitch
AirPett almost 8 years

The switch is a Cisco Catalyst 2950G, which is a Layer 2 device. It can't do cross-VLAN routing since it's not a L3 router. And what do you mean with the terminating? All VLANs go to the public internet through this Linux box, which is connected to the switch by an 802.1Q VLAN trunk.
Sum1sAdmin almost 8 years

I would terminate them on an L3 device such as firewall router or openvswitch - that would be my preference for ease of admin, can you add virtual interfaces to your firewall? - just put the VLAN gateways there no?
AirPett almost 8 years

Ok, I don't have a physical L3 device, but I could explore openvswitch. Currently my firewall is iptables(netfilter) and the VLANs are virtual interfaces for the eth1 (i.e. eth1.20, eth1.50). And do you mean the VLAN trunk with the VLAN gateway?
Sum1sAdmin almost 8 years

well, the same as your current setup really, any host on those vlans will have a DF of you linux box via trunk, and you have out of band management on 192.168.10.0/24 - you are essentially building a firewall router the manual way - but wont get VPN etc,etc. I would have a look at some of these mondaiji.com/blog/other/it/…
Sum1sAdmin almost 8 years

One thing, if there is going to be a lot of inter-vlan traffic you will be doing it all at the edge of your network and this could slow down internet access for everyone.
AirPett almost 8 years

Ok, I'll check OVS, but the only inter-vlan traffic is going to be ssh connections from the 100-vlan to the others to maintain servers in those networks. That's the reason why I'd really like not to teach myself a completely new system just now, and it should be possible to do this little thing with netfilter/iptables as well.
Sum1sAdmin almost 8 years

no just hosts should have their gateways as the interface on the Linux box, they go to the linux interface for anything outside of their VLAN. something like this hitecksqud.blogspot.ie/2012/12/…

AirPett almost 8 years

Thanks for the answer, I now changed the default policy to DROP and that part of the problem is now solved. I also figured out the big problem, please see my own answer in a monent.
AirPett almost 8 years

Thanks a lot for this excellent answer! You actually made me think of a possible problem: I can ping all clients in all VLANs from the Linux box running this all. From there all PCs (the Windows ones as well) respond, but when pinging from another subnet (VLAN 100), only Linux responds to ping even though I'm able to connect to the Win10 PCs with remote desktop from the same 100-VLAN.