Linux bridge network influenced by route table

Question

I was doing an experiment about linux bridge and my network topology is like:

As you can see, there are two hosts located in a LAN, Host1(10.74.68.58) and Host2(10.74.68.47). On Host1, I created a bridge br0 and assigned an IP for it (192.168.3.101). Then I attached eth0 to the bridge:

[[email protected]:~] # bridge link
2: eth0 state UP : <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 master br0 state forwarding priority 32 cost 2

I set the default route as br0 and it is ok to ping 10.74.68.47:

[[email protected]:~] # ip r
default dev br0  scope link
172.17.0.0/16 dev docker0  proto kernel  scope link  src 172.17.42.1
192.168.3.0/24 dev br0  proto kernel  scope link  src 192.168.3.101

But things became unexplainable when I set default route to eth0 :

[[email protected]:~] # ip r
default dev eth0  scope link
172.17.0.0/16 dev docker0  proto kernel  scope link  src 172.17.42.1
192.168.3.0/24 dev br0  proto kernel  scope link  src 192.168.3.101

When eth0 is the default route interface, I try to ping host2 in two different ways:

1, ping 10.74.68.47

Failed. After I checked tcpdump file (captured on interface br0), I found that br0 only received ARP response. So there is no ARP info on interface eth0, thus it can not get the mac of host2. I think this is the right behaviour, is my understanding right?

2, Then I tried ping -I br0 10.74.68.47

I wanted to use -I option to steer clear of default route, but also failed. After I check the tcpdump file (captured on interface br0), I found there is already a pair of icmp echo request and echo reply packet. This confused me a lot. Now that br0 have received echo reply, why can't I ping to host2 successfully?

[[email protected]:~] # ping -I br0 10.74.68.47
2 packets transmitted, 0 received, 100% packet loss, time 1006ms

Can you guys give me some pointers?

Solution

Bridging doesn't work the way you think it works. :-)

A bridge is only concerned with OSI level 2 (Ethernet frames). On this level, IP addresses etc. don't exist. Conceptually, you can think of a bridge as a collection of ethernet interfaces. Each interface is called a port, and a packet that goes into one port comes out on all other ports. (Actually, in the Linux implementation, there's an optimization that keeps a table of seen MAC addresses, but conceptually, it doesn't matter).

So a bridge can connect ("bridge") several ethernet segments into one big segment.

Then what does it mean to "give a Linux bridge an IP address"? In the Linux implementation, the bridge is not a separate hardware device (like they originally were), it's also accessible from the host itself. That means it acts like a kind of "super-ethernet interface" with many ports, but packets that go into the kernel or out from this kernel to or from any of these ports reach the Linux OS under a single IP address.

So as soon as you make an ethernet interface a slave (port) of a bridge, it ceases to have its own address. The only thing that counts is the IP address of the bridge.

In other words, making a bridge with only a single port makes no sense (you could have used the interface by itself). Trying to route packets to a port of a bridge makes no sense (as far as the kernel is concerned, the bridge is a single device).

If you want to play around with a bridge, you need a structure like this:

  10.0.2.1/23    10.0.2.2/23    10.0.3.254/23     10.0.3.1/23    10.0.3.2/23 

  ............   ............   ...............   ............   ............
  .  Host A  .   .  Host B  .   .  Host X     .   .  Host C  .   .  Host D  .
  .          .   .          .   . <-- br0 --> .   .          .   .          .
  .   eth0   .   .   eth0   .   . eth0   eth1 .   .   eth0   .   .   eth0   .
  .....|......   .....|......   ...|......|....   .....|......   .....|......
       |              |            |      |            |              |      
  -----+--------------+------------+      +------------+--------------+------

  <-------- left Segment  --------->      <------- right Segment ----------->

Here the left segment with hosts A and B is bridged by host X to the right segment with hosts C and D, and each host is accessible by a single IP address (which is assigned to interfaces or the bridge as a whole).

Comments

[ruanhao]

thank you very much for your instruction. You're right, my topology here is not the proper practice for linux bridge. I set up this weird topology in order to find out how packet flows when there exists a bridge. But after I read up some material about linux kernel, I found my question has something to do with 'rp_filter' option. And though this is a very very strange topology, it may happen to be an experiment environment to test 'rp_filter' function :)

[dirkt]

AFAIK, rp_filter is "reverse path filtering" and works on level 3 (IP packets, routing), not on level 2 (ethernet frames, bridging), so I don't think there's a connection. If you want to test Linux kernel routing, you need to set it up as a router, not a bridge.

[gcarvelli]

Thank you! This is a fantastic answer that cleared up some misconceptions I had about network bridging.

[Eric Woodruff]

"In other words, making a bridge with only a single port makes no sense (you could have used the interface by itself)." Except that if you need to configure the networking to make it visible to an application while still hotplugging an interface, a bridge can give you this level of indirection.