Guest and host cannot see each other using linux-kvm and macvtap

Question

I'm migrating a kvm virtual machine from an old host (both hardware and OS) to a new one.

For networking, virt-manager proposed me a new option: macvtap. This looked a good alternative to setting up a bridge on eth0.

So now the guest boots just fine, gets an IP from my local network DHCP server, can reach the internet. The guest also sees other machines on the local network, I can ssh them, etc.

The problem is that the host and the guest do not see each other. I cannot reach the guest from the host using the guest IP, neither can I reach the host from the guest using the host IP. No ping, ssh, http, nothing.

Here is the route -n command from the host:

$ /sbin/route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.0.1     0.0.0.0         UG    0      0        0 eth0
192.168.0.0     0.0.0.0         255.255.255.0   U     1      0        0 eth0

(same output from the guest).

I could probably set up a new tun/tap interface dedicated to communication between host and guest but it looks a little bit overkill. Is there a way to make host and guest communicate?

Solution 1

I asked this question on IRC and it appears that macvtap

injects guest traffic into the network stack too low for that

The solution is then to add a network interface for the guest and the host to communicate, or stay with the old bridged solution...

Solution 2

virt-manager says explictly that macvtap does not work for host to guest network communications when you set it up. I simply added a second nat based interface, set it up in the guest, and use that to communicate with my host.

Solution 3

The solution is to configure a macvlan interface on the hypervisor, with the same IP address than the real hardware interface (very important), and to configure routing on the host to use it. In Qemu/KVM, use a macvtap interface on the hardware interface as usual.

For my config (192.168.1.0/24 network, p10p1 hardware interface, and 192.168.1.1 gateway), it gives (on the hypervisor):

ip link add link p10p1 address 00:19:d1:29:d2:58 macvlan0 type macvlan mode bridge
ip address add 192.168.1.100/24 dev macvlan0
ip link set dev macvlan0 up

ip route flush dev p10p1
ip route add default via 192.168.1.1 dev macvlan0 proto static

Solution 4

You want to end up with something like this:

Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.0.1     0.0.0.0         UG    0      0        0 macvlan0
192.168.0.0     0.0.0.0         255.255.255.0   U     1      0        0 macvlan0

Note that the last column contains "macvlan0" instead of "eth0".

To achieve that, you can use these commands, assuming 192.168.0.42 as your IP address:

ip link add link eth0 macvlan0 type macvlan mode bridge
ip address add 192.168.0.42 dev macvlan0
ip link set dev macvlan0 up
ip route flush dev eth0
ip route add default via 192.168.0.1

Mostly similar to the solution of npen. If you want a more sophisticated script, see my webpage about this subject.

Solution 5

As has been mentioned in previous answers, a solution to this problem is to add a macvlan network adapter on the host. However, I felt that manually rewiring the routes to the macvlan adapter was kinda hacky, especially since I wanted IPv6 support and manually set routes may become a problem when the prefix changes. So here's my configuration which leaves the kernel in control of the routing table:

_{(The particular configuration here is Debian and Upstart specific, but the basic steps should work on any GNU/Linux.)}

Creating the macvlan adapter on boot

First, you need to select a MAC address for your adapter. You may be able to just use a random one, but I suggest you manually create a macvlan adapter and use its MAC. This way, the MAC obliges to any conventions that might be there.

Setting a fixed MAC is advisable, since otherwise there is no way for e.g. an DHCP server to recognize your machine after a reboot and assign it the same IP address as before.

So create an adapter and look up the MAC:

root@host:~# ip link add link eth0 macvlan0 type macvlan mode bridge
root@host:~# ip addr show dev macvlan0
#: macvlan0@eth0:  mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1
    link/ether 12:34:56:78:90:ab brd ff:ff:ff:ff:ff:ff
...

The highlighted hexadecimal number is your MAC address.

Now you create an init script - which must be run before the networking is initalized - to create the macvlan adapter each startup. The command to do this is:

ip link add link eth0 address <MACADDRESS> macvlan0 type macvlan mode bridge

An example Upstart init script for this purpose would be:

start on starting networking

script
    ip link add link eth0 address <MACADDRESS> macvlan0 type macvlan mode bridge
end script

Just put this in e.g. /etc/init/macvlan.conf.

Setting up the network configuration

In /etc/network/interfaces, set your physical network adapter to manual (but leave it auto) and move its previous configuration (typically DHCP or a static IP address) to your macvlan adapter. E.g.:

auto eth0
iface eth0 inet manual

auto macvlan0
iface macvlan0 inet dhcp

Disabling IPv6 for the physical adapter

Lastly, you do not want the physical adapter to obtain an IP address. For IPv4 setting the adapter to manual prevents it from obtaining an address. However, I have not found a configuration which stops the kernel from obtaining/assigning an IPv6 address for/to the adapter. When it does, it also adds routes for them, which may cause problems. So the best way seems to be to disable IPv6 for the physical adapter. You can do this by adding the line

net.ipv6.conf.eth0.disable_ipv6=1

to /etc/sysctl.conf, by creating a file in /etc/sysctl.d/ with this line, or by adding

sysctl -w net.ipv6.conf.eth0.disable_ipv6=1

to your init script.

When you now reboot your machine, communicating from host to guest should work with both IPv4 and IPv6.

---

Be aware, that if you make a mistake while setting this up, your host may become unreachable via network even after a reboot. Only do this if you have physical access to the machine or other safeguards are in place so you can fix potential problems.

View more solutions

Comments

[HDave]

Here is step by step instructions on how to create the host/guest interface without having to disable network manager: wiki.libvirt.org/page/…

[HDave]

I couldn't get those instructions to work...

[mdd]

Don't forget to load the macvlan module: modprobe macvlan

[Marc.2377]

Related: Script to create macvlan bridge on the host doesn't work unless it's run twice - Unix SE