Linux: bridges, VLANs and RSTP

Question

I tried to figure out how to setup RSTP on Linux with VLANs and bridges involved and now am completely confused.

I am trying to bridge three interfaces, two of which are supposed to act as trunk (hdlc0 and hdlc1) and one is supposed to act as access port (eth0). I also need to enable RSTP on each interface included in bridge, but with the configuration listed below RSTP packets are sent via hdlc0 and hdlc1 tagged (!) so other devices rejects them. As Linux has no concept of 'native vlan', I have no idea how to fix it.

Here's my configuration:

ifconfig eth0 up


ifconfig hdlc0 up
ifconfig hdlc1 up

vconfig add hdlc0 42
vconfig add hdlc1 42
ifconfig hdlc0.42 up
ifconfig hdlc1.42 up


brctl addbr br1
brctl addif br1 eth0
brctl addif br1 hdlc0.42
brctl addif br1 hdlc1.42

ifconfig br1 up
brctl stp br1 on

Another question: I also wonder how to configure RSTP in scenarios where I have multiple bridges: say eth0 is trunk with vlan 42-42 allowed, vlan 42 is supposed to pass via hdlc0 and vlan 43 is supposed to go via hdlc1, so I have two bridges. If i enable RSTP on both bridges it will (probably) run independently on each bridge so I'll face problems soon?

Solution 1

In Linux, VLANs and bridges are completely separate constructs, and Linux bridges are are not "VLAN aware".

When you create a VLAN interface, Linux tags/untags packets on that interface before passing them to/from the underlying physical ("trunk") interface. However, you can still use the underlying physical interface to send untagged ("native VLAN") packets.

When you create a bridge, Linux switches packets between the associated interfaces without any concern for the VLAN tags (or lack thereof) on the packets. If you attach a trunk interface to a bridge, the bridge will happily switch VLAN tagged packets with no regard for the tags. When you enable STP on a bridge, Linux generates untagged STP packets and drops them on the bridge.

When a bridge is attached to a physical interface that also has associated VLAN interfaces, those VLAN interfaces will stop seeing any traffic that is not destined for the MAC address of the physical interface. This behavior is due to the order in which bridging and VLAN tagging are processed, and can be altered using ebtables as described at http://blog.rackspace.com/vms-vlans-and-bridges-oh-my-part-2 . However, as far as Spanning Tree is concerned, attaching bridges to both a physical interface and the associated VLAN interfaces will only work properly if you are using PVST+ anyway (because STP port blocking is managed independently for each bridge), so it is not really relevant here.

But you can also create VLAN interfaces on top of a bridge that is passing VLAN tagged packets, and then add those VLAN interfaces to other bridges.

So, to accomplish what you want, try:

ip link set dev hdlc0 up
ip link set dev hdlc1 up

brctl addbr br_native
brctl addif br_native hdlc0
brctl addif br_native hdlc1
brctl stp br_native on
ip link set dev br_native up

ip link add link br_native name br_native.42 type vlan id 42
ip link set dev br_native.42 up
ip link set dev eth0 up

brctl addbr br_42
brctl addif br_42 br_native.42
brctl addif br_42 eth0
ip link set dev br_42 up

Note that the Linux kernel bridging code only natively supports traditional 802.1D STP. To add support for RSTP and PVST+, use https://github.com/mstpd/mstpd (Some relevant documentation for mstpd can also be found on: https://docs.cumulusnetworks.com/display/DOCS/Spanning+Tree+and+Rapid+Spanning+Tree ). mstpd is also capable of speaking MSTP, but due to the way Linux implements its FIBs, it is currently impossible to map MSTP topologies onto Linux bridges, so MSTP is not actually functional.

To answer your second question, I don't believe it is possible (on any switch, not just when using Linux) to use STP or RSTP to direct each of two different VLANs on a single trunk through two other trunks. This can only be accomplished using PVST+ or MSTP, although as mentioned above MSTP is not supported in Linux.

Solution 2

Since Linux kernel 3.0 or so, it's no longer true that Linux bridges are "not VLAN aware". https://linux-blog.anracom.com/2017/10/30/fun-with-veth-devices-in-unnamed-linux-network-namespaces-i/ and the follow-up articles provide a fairly good explanation, but in summary:

You can use the bridge(8) tool to manipulate the bridge-vlan interactions, in a manner not dissimilar from most managed switches.

In your situation, you might try something like:

# ip link set br0 type bridge vlan_filtering 1 vlan_default_pvid 1 stp_state 1 priority 32768 nf_call_iptables 1 nf_call_arptables 1
# bridge vlan add vid 1 pvid untagged dev eth0
bridge vlan add vid 19 dev eth0

ip link set up dev eth0
ip link set up dev hdlc0
ip link set up dev hdlc1

# if you only want to bridge VLAN42, you don't need these, but here's how you'd create them:
# ip link add link hdlc0 name hdlc0.42 type vlan protocol 802.1q id 1 reorder_hdr on gvrp on mvrp on loose_binding off
# ip link add link hdlc1 name hdlc1.42 type vlan protocol 802.1q id 1 reorder_hdr on gvrp on mvrp on loose_binding off
# ip link set up dev hdlc0.42
# ip link set up dev hdlc1.42

# now for the actual bridging:
brctl addbr br1
brctl addif br1 eth0
brctl addif br1 hdlc0
brctl addif br1 hdlc1

# port eth0 is an untagged port; assuming it should be in VLAN42:
bridge vlan add vid 42 dev eth0 pvid untagged
# hdlc0 is a trunk interface that forwards vlan42 in a tagged manner:
bridge vlan add vid 42 dev hdlc0
# hdlc1 carries vlan42 and vlan43, both with tagging:
bridge vlan add vid 42 dev hdlc1
bridge vlan add vid 43 dev hdlc1

brctl stp br1 on
ip link set up dev br1

You can also add a vlan interface to a bridge, which will then do what is logical, if you think about it: the bridge generates an ethernet frame and passes it down to the port, which sends it out. Since the port in question is a vlan interface, it adds a vlan tag before sending the frame out. I'm sure there are valid uses for this but I can't think of any at the moment. :)

Comments

[grindelwaldus]

But if i create second bridge with physical interfaces included, this bridge will "consume" all the packets so they'll never reach vlan interface. It can be fixed with ebtables, but in that case RSTP is active on second bridge (it's blocking hdlc0, for example), but not on the first, so vlan interfaces are not blocked (so packets are able to go via hdlc0.42 and I still have a loop). I've tried it before, I can show you a code.

[Paul Donohue]

Hmmm... You are correct, and I just learned something new. Prior to Linux 2.6.37, if the NIC hardware and driver supported NETIF_F_HW_VLAN_RX, then this may have worked as expected. With Linux 2.6.37 and later, the native VLAN bridge always consumes all of the packets as described here: blog.rackspace.com/vms-vlans-and-bridges-oh-my-part-2

[Paul Donohue]

You are also correct that that the STP on the second bridge doesn't block traffic on the first bridge. One solution is to use PVST+ with mstpd if your other switches support that. Another possible solution is to add the VLANs on top of the bridge. I will update my answer with an example.

[grindelwaldus]

Not really sure including one bridge into another will actually work - seems tricky, but I'll check it next week.