How to bridge a range of public IPs to KVM virtual machines on Debian/Ubuntu

Question

I've been trying for weeks to figure out the right network configurtion for sharing a range of public IPs with KVM virtual machines running on my server, but so far with little luck and with the help of the friendly ServerFault community, I've managed to make it work. You can find my working setup below:

My ISP routes all the traffic to 192.168.8.118 (so that needs to be the primary IP of eth0), but I have 192.168.239.160/28 to my disposition.

Here's /etc/network/interfaces on the host machine:

# Loopback device:
auto lo
iface lo inet loopback

# device: eth0
auto  eth0
iface eth0 inet static
  address   192.168.8.118
  broadcast 192.168.8.127
  netmask   255.255.255.224
  gateway   192.168.8.97
  pointopoint 192.168.8.97
  # This device acts as gateway for the bridge, so provide a route.
  up ip route add 192.168.8.118/32 dev eth0 scope host

# device: br0
auto  br0
iface br0 inet static
  bridge_stp      off
  bridge_maxwait  0
  bridge_fd       0
  address   192.168.239.174
  broadcast 192.168.239.175
  netmask   255.255.255.240
  gateway   192.168.8.118
  # Create and destroy the bridge automatically.
  pre-up brctl addbr br0
  post-down brctl delbr br0
  # Our additional IPs are allocated on the bridge.
  up ip route add to 192.168.239.160/28 dev br0 scope host

I have configured a virtual machine like this:

sudo ubuntu-vm-builder kvm precise \
                  --domain pippin \
                  --dest pippin \
                  --hostname pippin.hobbiton.arnor \
                  --flavour virtual \
                  --mem 8196 \
                  --user mikl \
                  --pass hest \
                  --bridge=br0 \
                  --ip 192.168.239.162 \
                  --mask 255.255.255.240 \
                  --net 192.168.239.160 \
                  --bcast 192.168.239.175 \
                  --gw 192.168.239.174 \
                  --dns 8.8.8.8 \
                  --components main,universe \
                  --addpkg git \
                  --addpkg openssh-server \
                  --addpkg vim-nox \
                  --addpkg zsh \
                  --libvirt qemu:///system ;

If I inspect the virtual machine's XML definition, its network interface is defined like this:

<interface type='bridge'>
  <mac address='52:54:00:b1:e9:52'/>
  <source bridge='br0'/>
  <model type='virtio'/>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>

When I (re)start the virtual machine, /var/log/syslog receives these lines:

Jul 20 03:13:02 olin kernel: [ 4084.652906] device vnet0 entered promiscuous mode
Jul 20 03:13:02 olin kernel: [ 4084.686388] br0: port 2(vnet0) entering forwarding state
Jul 20 03:13:02 olin kernel: [ 4084.686394] br0: port 2(vnet0) entering forwarding state

My server is running Ubuntu 12.04 64-bit with kernel 3.2.0-26-generic (from Ubuntu). I'm running libvirt-bin 0.9.8-2ubuntu1 and qemu-kvm 1.0+noroms-0ubuntu13.

iptables on the host machine is currently set up to allow all traffic (to eliminate that as a problem source), and I have enabled forwarding of both ipv4 and ipv6 traffic.

When I log in to the guest via SSH from the host, I have no internet connection inside the guest OS. The guest’s /etc/network/interfaces looks like this:

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
    address 192.168.239.162
    netmask 255.255.255.240
    network 192.168.239.160
    broadcast 192.168.239.175
    gateway 192.168.239.174
    # dns-* options are implemented by the resolvconf package, if installed
    dns-nameservers 8.8.8.8
    dns-search pippin

Now it works

The configuration outline above actually works as I want it to. Refer to the edit history if you want to see my earlier attempts.

Solution 1

If you're bridging, you shouldn't need to configure anything related to the VMs' IP addresses on the host machine. Just configure them to connect to the bridge, and configure the IPs within each VM in the usual way. Bridging joins networks together at the Ethernet layer, where IP addresses don't matter; from your ISP's standpoint it'll look like you have several computers plugged into a switch that's conected directly to the ISP.

But if your ISP is routing traffic to the host's .118 address, you'll need to configure your VM host to act as a router and forward the VMs' traffic. To do that, remove the bridge_ports eth0 line from your interfaces file, ip route add to 192.168.239.160/28 dev br0, and echo 1 > /proc/sys/net/ipv4/ip_forward. In the VMs, you'll need to configure 192.168.8.118 as the default gateway, and add a route that says 192.168.8.118 is directly reachable via eth0. (That's the VM's eth0, which connects to the host's br0.)

In neither case should you be adding the VMs' addresses directly to the br0 interface. In the bridging case, you want the VMs and not the host to answer ARP requests for those addresses, and in the routing case, you want the host to understand that when it receives a packet for one of those addresses, it needs to be routed somewhere else, not delivered locally.

Solution 2

You don't want to allocate the IP address of your VMs to the br0 interface of your host - that would just make that address belong to the host, not the VM.

However, your VMs will need a gateway address to route all outbound packets to. I'd recommend allocating one IP in that /28 to your host, and configure your VMs to use that IP as their default route. The first or last available IP in a subnet is a reasonable choice for a gateway address....

ip addr add 192.168.239.161/28 dev br0

have you enabled ip forwarding? e.g. uncomment the following in /etc/sysctl.conf (one or both for ipv4 and/or ipv6):

#net.ipv4.ip_forward=1
#net.ipv6.conf.all.forwarding=1

Finally, have you tried logging in at the console of the VM (e.g. with virt-manager or a VNC viewer like vinagre or xvnc4viewer)? if so, what IP address does it have (if any)? Is the VM configured to have a static IP or dhcp? If the latter, have you configured your dhcp server to give the appropriate IP address to the VMs MAC address?

Comments

[Dean Chen]

up ip route add to 192.168.239.160/28 dev br0 scope host. Why you use 'to' in this command, I didn't find to option in man route.

[mikl]

I have enabled forwarding for both ipv4 and ipv6. The server is running headless in a datacenter, so I'm not sure how I'd go about attaching a VNC viewer to it's virtual machines. I have tried virsh console, but that does not give me a shell, just seems to stall.

[mikl]

The exact words from their support were “Your Subnet is routed via the Main IP of your server so you should use this IP as the GW for you subnet.”, so the routing part is only my interpretation. I tried removing eth0 from bridge_port, but that caused my network not to come back up correctly. I suppose I should re-add eth0 as a separate interface (instead of just being part of the bridge)?

[mikl]

Okay, from following your advice, I got a step further, but still can't connect from the outside to the guest VM, and it doesn't have internet access. I've updated my question to reflect matters as they currently stand.

[Wyzard]

Yes, you should have eth0 as a separate interface, configured the same way you used to (before you started trying to set up VMs).

[Wyzard]

The "network is unreachable" message in this case probably means that the guest failed to find a route for the packet it wanted to send. What's in the guest's routing table? You need a route entry that says that 192.168.8.118 is reachable directly via the guest's eth0, and I don't see anything in your configuration that would create that route.

[mikl]

Thanks for the help, I managed to make my outgoing traffic work correctly by changing the gateway to be the bridge address and then adding a route on the host machine. Now, I just need to get the incoming traffic routed correctly (see my update above for details).