Why are incoming packets on a TAP interface seen with tcpdump but not with iptables?

Question

A program injects packets on a Linux TAP interface (these packets are coming from a virtual machine). Specifically, these are DHCP requests (so they're UDP). I can see the packets with tcpdump but not with iptables, and they don't reach the local DHCP server either. Why not, and how do I fix that?

Update: I tried injecting IP packets directed to the address of the tap0 interface. I see the ARP requests coming in from the VM in tcpdump -i tap0, but the network layer does not reply. If I send ARP requests to the VM, it sees them and replies to the host (and the replies show up in tcpdump but are otherwise lost).

Another observation: ifconfig tap0 shows that the TX dropped packet count is incremented for each packet that's injected onto the host. Why TX?

# ifconfig tap0
…
          TX packets:0 errors:0 dropped:958 overruns:0 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

---

The long story: On a Linux host (running Ubuntu 10.04), I'm running a virtual machine which amongst other things emulates an Ethernet card. It does so by communicating with a helper program that's in charge of injecting and capturing Ethernet packets onto the hosts's network stack. The virtual machine is an ARM chip emulator, and the helper program is called nicserver; all I know about it is what is found in the ARM documentation.

I want to establish an Ethernet link between the VM and the host, and above that I want an IP link. The VM obtains its IP address over DHCP. I do not want any communication between the VM and the rest of the world, only with the host, so I created a virtual network interface tap0 with

tunctl -u gilles
ifconfig tap0 192.168.56.1 netmask 255.255.255.0 up
nicserver -p 7801 -a tap0 &

Now I boot the VM, and I can see it's sending DHCP requests with tcpdump -n -i tap0 -vv (the DHCP client doesn't time out, I'm just showing one sample request here):

tcpdump: listening on tap0, link-type EN10MB (Ethernet), capture size 96 bytes
18:29:23.941574 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 576)
    0.0.0.0.68 > 255.255.255.255.67: [no cksum] BOOTP/DHCP, Request from 02:52:56:47:50:03, length 548, xid 0x238a7979, secs 46, Flags [none] (0x0000)
          Client-Ethernet-Address 02:52:56:47:50:03 [|bootp]

I've set up Dnsmasq on the host to serve the requests, but it's not seeing any incoming request. The Dnsmasq server doesn't even see the incoming requests (I straced it). So I tried observing the packets with Iptables. (All the filter/INPUT rules are shown; there are no mangle or nat rules).

Chain INPUT (policy ACCEPT 2366K packets, 5334M bytes)
 pkts bytes target     prot opt in     out     source               destination 
  119 39176 LOG        udp  --  *      *       0.0.0.0/0            0.0.0.0/0           udp dpt:67 LOG flags 4 level 4 prefix `[DHCP request] '
  119 39176 DROP       udp  --  eth1   *       0.0.0.0/0            0.0.0.0/0           udp dpt:67
    2   490 LOG        udp  --  tap0   *       0.0.0.0/0            0.0.0.0/0           LOG flags 4 level 4 prefix `[in=tap0] '
   26  6370 ACCEPT     udp  --  tap0   *       0.0.0.0/0            0.0.0.0/0   
    0     0 ACCEPT     all  --  tap0   *       0.0.0.0/0            0.0.0.0/0   
 3864  457K ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0   

All these incoming DHCP requests are on eth1 (and I'm careful not to ignore these so as not to anger my colleagues and my network admin). Those UDP packets on tap0 come from the local Samba server. The DHCP request packets that I see with tcpdump do not appear to go through the packet filter!

Why do I see incoming broadcast packets on tap0 with tcpdump but not with iptables (nor with programs listening on the machine)? And what do I need to fix so that these packets are seen, as they would be if they were coming on an Ethernet interface?

Solution

Here is further guesswork. Hope this is helpful, but it may as well be embarassingly wrong.

tap0 has two ends, the kernel network stack end and the program interface. It seems to me if you supply 'nicserver' with tap0, it won't attach to it the way it is intended with tap devices, using the program interface. Instead, nicserver will simply write to it from the network stack end, and with no application reading from the program interface end, you will end up overflowing the device queue. This explains the dropped packets. Also, no packets will be delivered, which might explain the iptables result.

I guess if you let tcpdump capture on tap0 it actually attaches to the program interface end of tap0 and, voila, you see the packets. I searched the internet but found no source to confirm such behaviour. To falsify this theory, capture tap0 and see how it affects packet drops and the iptables log.

Finally, an advice that addresses your original problem: what about using the loopback device, instead? Like so:

nicserver -p 7801 -a lo

Comments

[Admin]

maybe try using physdev matching iptables and see if that changes how you're seeing the traffic. iptables -A INPUT -p udp --dport 67 -m physdev --physdev-in tap+ -j LOG --log-level 4 --log-prefix "[in=tap0] "

[Admin]

-m physdev is usually used with bridges and tap interfaces. i know you're not trying to bridge the tap interface to your eth, but I'm wondering if using the physdev to match the interface somehow changes the traffic to make it visible? I have no idea if it will work, or change anything, though.

[Admin]

@TimKennedy doesn't change anything. More oddness: if I tell nicserver to use eth0, even tcpdump doesn't see any packets. This could be an issue with nicserver, but I still want to know how packets can be seen by tcpdump on tap0 and not by iptables.

[Admin]

if you run tcpdump with -p, can tcpdump still see the packets? (-p restricts tcpdump from putting the interface in promiscuous mode.) Since iptables doesn't put the interface into promiscuous mode, you'll get a closer comparison to what iptables sees.

[Admin]

@TimKennedy nicserver puts the interface in promiscuous mode permanently (ifconfig confirms this). I notice something peculiar: each “lost” DHCP request increments the TX dropped counter by 1. Could it be that the packets are somehow injected in a way that makes the kernel consider them as sent and not received? I tried logging UDP output, but the requests don't appear in the filter/OUTPUT chain either.

[Admin]

@shank netstat -s is global, and I can't stop unrelated traffic on eth1. In /sys/class/net/tap0/statistics, only tx_dropped` is nonzero. Anything else I should look at? (Note: further tests will have to wait until Monday.)

[Admin]

Are you sure the tap0 packets enter the input chain? They could as well be subject to routing.

[Admin]

@artistoex I can see they don't enter the input chain, and I'm trying to find out why. I know I have no set any routing table entries beyond the one implied by bringing up an interface with a netmask, plus a default route through eth1.

[Admin]

On a router no packet passing it would enter the input chain, but each would show up in a tcpdump. Maybe TAP traffic is likewise regarded as routed traffic? Another thought: Non-IP packets might not be handled at all by iptables (one could use ebtables, instead).

[Admin]

@artistoex What should I be looking at or doing then? I'm fairly sure there was nothing exotic about that machine's networking setup until I created the tap device.

[Admin]

Insert your log target into the prerouting chain. I also recommend the nice iptables flowchart on wikipedia.

[Admin]

@artistoex Thanks. No IP packet from the VM appears in the iptables raw/PREROUTING, nat/PREROUTING, mangle/PREROUTING or mangle/INPUT. No Ethernet packet from the VM's MAC appears in the ebtables broute/BROUTING, nat/PREROUTING or filter/INPUT.

[Admin]

It's been some time since you've posted this question. Have you had any success so far?

[Admin]

@artistoex A year later I came back to this and realized that I see no traffic with iptables because the first thing the VM does is send an ARP request which the host doesn't see. With no ARP response, there is no IP traffic for iptables to see. Duh. But that doesn't explain the root problem, which is that the host isn't communicating with the VM. I still don't understand what's going on, and I suspect a bug in nicserver. This is a very old utility, and we're running an unsupported version of Linux (far too recent and the wrong distribution to boot). I lack the time to investigate further.