Bridging two tap interfaces

Question

I am a total noob, this is my first ever experiment with kernel networking. I am attempting to create a bridge between two tap interfaces, and try to send traffic through. It is more of an experiment than for any particular purpose.

$ brctl showstp br0
br0
bridge id      8000.46846e0c0ff9
designated root    8000.46846e0c0ff9
root port         0            path cost          0
max age          20.00         bridge max age        20.00
hello time        2.00         bridge hello time      2.00
forward delay        15.00         bridge forward delay      15.00
ageing time         300.00
hello timer           1.98         tcn timer          0.00
topology change timer     0.00         gc timer         115.04
flags          


tap1 (1)
port id        8001            state            forwarding
designated root    8000.46846e0c0ff9   path cost        100
designated bridge  8000.46846e0c0ff9   message age timer      0.00
designated port    8001            forward delay timer   10.34
designated cost       0            hold timer         0.98
flags          

tap2 (2)
port id        8002            state            forwarding
designated root    8000.46846e0c0ff9   path cost        100
designated bridge  8000.46846e0c0ff9   message age timer      0.00
designated port    8002            forward delay timer    0.00
designated cost       0            hold timer         0.98
flags          

I have the bridge br0 created, with both tap1 and tap2 added. I have a program injecting ARP packets into tap1 using libpcap. Wireshark correctly shows the packets entering tap1. However, no packet shows up at tap2. I tried adding the follwing rule in ebtables:

sudo ebtables -I INPUT --log --log-level debug

No packets show up in the logs. I'll appreciate any inputs.

EDIT: Adding more information. Injecting fake packets is indeed the application. My intention here is to simulate, fully in software and without VMs, how packets get forwarded through the linux kernel stack. I am not creating any new network namespaces. Perhaps that's the problem?

I only have two processes. The "read" process has a file descriptor open to tap2, and constantly tries to read from it. The write process has a file descriptor open to tap1 and waits for user prompt to send out the ARP query. The ARP query has a random source IP address. The source MAC address is set as the MAC address of tap1. Here is the output of tcpdump:

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tap1, link-type EN10MB (Ethernet), capture size 262144 bytes
^[[A07:19:57.752990 ARP, Request who-has google-public-dns-a.google.com tell 0.0.248.17, length 28
    0x0000:  ffff ffff ffff ba9c 0589 16ad 0806 0001
    0x0010:  0800 0604 0001 ba9c 0589 16ad 0000 f811
    0x0020:  0000 0000 0000 0808 0808

I have configured tap1 and tap2 to not have IP addresses. Could that be the problem?

brctl addbr br0
ip tuntap add name tap1 mode tap
ip tuntap add name tap2 mode tap
brctl addif br0 tap1
brctl addif br0 tap2
ifconfig tap1 0.0.0.0 up
ifconfig tap2 0.0.0.0 up
ifconfig br0 10.0.1.1 netmask 255.255.255.0 broadcast 10.0.1.255
ip link set br0 up
ip link set tap1 up
ip link set tap2 up

Based on the answer, I checked attaching various applications to tap2. I notice this: when no application is using tap1 or tap2, both interfaces don't have the LOWER_UP flag set.

4: br0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default 
    link/ether 46:84:6e:0c:0f:f9 brd ff:ff:ff:ff:ff:ff
25: tap1: <NO-CARRIER,BROADCAST,MULTICAST,PROMISC,UP> mtu 1500 qdisc pfifo_fast master br0 state DOWN mode DEFAULT group default qlen 500
    link/ether ba:9c:05:89:16:ad brd ff:ff:ff:ff:ff:ff
26: tap2: <NO-CARRIER,BROADCAST,MULTICAST,PROMISC,UP> mtu 1500 qdisc pfifo_fast master br0 state DOWN mode DEFAULT group default qlen 500

When I start the applications, the LOWER_UP flag becomes set:

4: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether 46:84:6e:0c:0f:f9 brd ff:ff:ff:ff:ff:ff
25: tap1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UP mode DEFAULT group default qlen 500
    link/ether ba:9c:05:89:16:ad brd ff:ff:ff:ff:ff:ff
26: tap2: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UP mode DEFAULT group default qlen 500
    link/ether 46:84:6e:0c:0f:f9 brd ff:ff:ff:ff:ff:ff

I'm sorry this is turning out to be lengthy, I'm just hoping that there is enough information to make sense of the issue.

Solution 1

Just in case, because you said you are a total noob: A tun (layer 3) or tap (layer 2) interface is the network interface endpoint of an application, and the application can read and write packets from this network interface. What you create with ip tuntap add ... or the outdated tunctl are persistent names for such endpoints, and you'll normally still run the application, and it won't do anything unless you run the application.

As the application interacts with the network interface as a matter of design, there's no need to "inject" packets with a third party application, unless you mean by "inject' this normal interaction I described.

Also, if you want to play around with networking, I can recommend using network namespaces and veth pairs. Basically you can set up a lot of virtual computers on your computer that can mimic communication between real computers on a network.

So if you want to do that, and don't want to play with your own application creating and receiving packets, you don't need a tun/tap interface.

That said, I just tested your setup, with a slight variation because you didn't say what you use to "inject" packets: I used two socats to create to tap endpoint tap0a and tap1a, then I bridged them, and used another two socats in two different namespaces to create the correct packets for me. They need to be in a different namespace, as local packets will always be delivered via loopback lo.

And as expected, bridging tap devices works just fine.

So I suppose the problem is in the packet you are injecting: Wrong ethernet address, or no broadcast. Please edit your question with with tcpdump -xx ... output when you inject the ARP packet.

Or possibly you'd like to create network namespaces and bridge two endpoints of two veth-pairs instead? That's a lot simpler.

Edit

The ARP packet looks good. It seems like there's no application connected to tap2. If you do ip link, you shouldn't see a LOWER_UP flag for tap2. Guess: The bridge detects that the device is only partially up, and doesn't send packets to this port.

Try replacing it with a tap that has an application connected to it, something like

sudo socat TUN:10.0.2.2/24,tun-name=tapx,tun-type=tap,iff-up - | hexdump -C

(the 10.0.2.2/24 address doesn't do anything, but socat won't work if you don't specify an address), and in another terminal

sudo ip link set tapx master br0

(that replaces brctl addif), then inject your packet several times and see if you get a hexdump in the first window. Also check for LOWER_UP with ip link show dev tapx.

BTW, ifconfig and brctl are outdated. Use ip and bridge instead.

Not assigning IP addresses to the ports of the bridge doesn't matter, because bridge ports don't have IP addresses (if they got some assigned to them before they became enslaved by a bridge, they are ignored). See e.g. here.

Solution 2

I struggled a bunch with this and think I've arrived at a solution. Or, at least, a better understanding of what's going on.

It's important to keep in mind that the only way to deliver a packet to a tap interface on the receive side ("RX") is by writing that packet to the socket descriptor created by the process that opened the tap (via open("/dev/net/tun",...) and ioctl). Only one process can have this file descriptor at once. If a tap device is open, and another process tries to re-open another tap device with the same name, that system call will fail.

So when you have any other process, like wireshark, opening a raw socket and then binding that to tap0, it's only capable of writing traffic out of the system (from the perspective of the kernel). That is, wireshark should be bumping the TX counter of tap0, and only RX packets will be forwarded to the bridged interface.

You can check the counters for each like:

#!/bin/bash
for if in tap{0,1}; do
    stats=/sys/class/net/$if/statistics/
    rx=$(cat $stats/rx_packets)
    tx=$(cat $stats/tx_packets)
    echo "$if: rx=$rx, tx=$tx"
done

which outputs something like:

tap0: rx=0, tx=6
tap1: rx=0, tx=5

So in your specific case, I suspect that the problem is here:

I have a program injecting ARP packets into tap1 using libpcap. Wireshark correctly shows the packets entering tap1. However, no packet shows up at tap2.

How are you injecting ARP packets into tap1? I assume wireshark isn't calling fd = open("/dev/net/tun", ...) and ioctl(fd, TUNSETIFF, ...), so the packets sent will be TX, and therefore not bridged.

You'll need the program that opens tap0 to write packets to tap0's file descriptor to get those packets to be forwarded on br0 and show up on tap1.

Comments

[SPMP]

I've added the details. Could you please check?

[SPMP]

Turns out the reverse is true. tap2 has an application using it, but not tap1. Whenever an application connects to an interface, LOWER_UP gets set. Is that an indication of the problem?

[SPMP]

Thank you for the the help so far! I haven't marked the answer as accepted, just in case someone can figure it out.