qemu - two clients (tap0 & tap1) via br0 can't ping eachother but server pings ok

Question

I've got:

• A host system running Ubuntu with KVM (functional)
• Two clients, one Ubuntu and one Windows 7, nothing fancy.

I've also setup (see below) a bridge via br0 for the clients:

• br0: 192.168.2.1 (with DHCP running)
• eth0 -> br0
• tap0 -> br0
• tap1 -> br0

br0 can give IP to the clients, they get:

• Windows 7: 192.168.2.29
• Ubuntu: 192.168.2.30

Here's my problem:

• Windows ICMP -> br0: Works
• Ubuntu ICMP -> br0: Works
• Ubuntu ICMP -> Windows: Fails

http://www.youtube.com/watch?v=5C1VvJezouI

---

I played around with some NAT (which I'm getting the hang of via iptables) and that works like a charm! Without any issues both the clients can ping www.google.se and fetch data etc, but not between each other?!

---

Here's the commands I'm running from the beginning to end:

tc qdisc del dev wlan0 root 2>/dev/null
tc qdisc del dev wlan0 ingress 2>/dev/null
iptables -P INPUT ACCEPT
iptables -F INPUT
iptables -P OUTPUT ACCEPT
iptables -F OUTPUT
iptables -P FORWARD DROP
iptables -F FORWARD
iptables -t nat -F PREROUTING
iptables -t nat -F OUTPUT
iptables -t nat -F POSTROUTING
iptables -t nat -F
iptables -t mangle -F
ip route flush table main
iptables -t mangle -A PREROUTING -p tcp -j CONNMARK --restore-mark

modprobe ifb numifbs=1
modprobe ip_conntrack
modprobe xt_mark
modprobe iptable_nat

echo "1" > /proc/sys/net/ipv4/ip_forward
echo "0" > /proc/sys/net/bridge/bridge-nf-call-arptables
echo "0" > /proc/sys/net/bridge/bridge-nf-call-iptables
echo "0" > /proc/sys/net/bridge/bridge-nf-call-ip6tables

ifconfig br0 192.168.2.1 netmask 255.255.255.0
route add -net 192.168.2.0 netmask 255.255.255.0 dev br0
ip route add default via 192.168.2.1 dev wlan0

kpartx -av win7.img
kpartx -av ubuntu.img

tunctl -u `whoami` -t tap0
ip link set tap0 up
brctl addif br0 tap0

tunctl -u `whoami` -t tap1
ip link set tap1 up
brctl addif br0 tap1

kvm -no-reboot -m 1300 -hda /dev/mapper/loop0p1 -boot order=c -net nic,macaddr="DE:AD:BE:EF:EC:B6" -net tap,ifname=tap0,script=/etc/qemu-dummy

kvm -no-reboot -m 1300 -hda /dev/mapper/loop1p1 -boot order=c -net nic,macaddr="DE:AD:BE:EF:EC:B7" -net tap,ifname=tap1,script=/etc/qemu-dummy

At this point, I'm trying to ping between the clients without success, so I close down the systems and run:

brctl delif br0 tap0
brctl delif br0 tap1
ifconfig tap0 down
ifconfig tap1 down
tunctl -d tap0
tunctl -d tap1
kpartx -dv win7.img
kpartx -dv ubuntu.img

Solution

brctl delif br0 tap0
brctl delif br0 tap1

kvm -no-reboot -m 1300 -hda /dev/mapper/loop0p1 -boot order=c -net nic,macaddr="DE:AD:BE:EF:EC:B6" -net bridge,br=br0
kvm -no-reboot -m 1300 -hda /dev/mapper/loop1p1 -boot order=c -net nic,macaddr="DE:AD:BE:EF:EC:B7" -net bridge,br=br0

In your current scenario tap is being used.
If all VM's should be able to reach other VM's as well as virtualisation host letting br0 act as a switch is a option.

Another way would be to use macvtap in bridge mode or even using openvswitch.

Comments

[gertvdijk]

Do you really need to pass the traffic through the netfilter module of the host? How about just letting the bridge doing the forwarding (being a "switch")?

[Torxed]

I plan on using the netfilter for shaping/shuffling traffic around and experiment around with some stuff, but even with bridge-nf-call* and without all the iptable/tc stuff it doesn't work.

[Stéphane Chazelas]

Doesn't the windows firewall block pings by default? Have you tried tcpdump/wireshark on the br0, tap0, tap1? And in the VMs themselves?

[Torxed]

Yes it does, but Ubuntu doesn't so the argument here that the firewall is causing the issue is not the fact, sorry to say. It's a routing issue from tap0 -> br0 -> tap1 from what i can tell, and i don't think wireshark on the hosts would give me anything but as a last resort i'll consider it. Perhaps on the br0 interface or even better the tap interfaces to see what's popping out of those.

[peterph]

Do tap0 and tap1 have an IP in the host system?

[Torxed]

Humm nope, they don't. Being a network tech this might be the most retarded question i'll ever ask.. but should they? I mean they're just a virtual link patched into the virtual client, and since the KVM host can ping the KVM client and vice verse i thought that the tap didn't need to have an IP on its own?

[peterph]

@Torxed well, I really don't understand the routing in this case well enough, but I would expect the host kernel might be eating the packets somewhere between the guests. How about running tcpdump or iptraf on all devices on host (tapX, br0 and eth0) and in the guest(s) at the same time? That way you'd get some idea where the packets are getting lost.

[sunnysideup]

@torxed no, neither tap1 nor tap2 need an ip address. Can you try to enable net.ipv4.conf.all.log_martians, i.e. sysctl -w net.ipv4.conf.all.log_martians=1 and see if you get any messages in your log?

[Torxed]

It works so i'll give you the solution. However @gertcdijk also made a point which made me research my config.. i had totally destroyed the bridging functionality with "ghost" tc rules and IP table rules which i hadn't flushed/cleared properly. But your solution works.