How to assign multiple public IP-Adresses for 2 KVM-Guests

Question

I am new to this whole topic and I try for days now to figure out how to assign multiple public ip-addresses to KVM-guests through a KVM host. I found tons of examples howto get such a setup with 1 public IP running.

Here is my setup: The Server has only one NIC/MAC and runs 2 KVM-Guests with apache(and other stuff). Both guest-environments are ubuntu server 11.10 and must run in separate VMs. The 5 public ip-addresses are used to handle SSL-certificates and other stuff. The first VM should use 3 of the 5 addresses/certificates. The second VM gets the rest. The apache-stuff is configured correctly.

I have tried a number of differend ways via iptables to route the traffic from the hosts NIC to the guest-NICs. In spite of the fact that one way was the right one but only wrong implemented, I leave the details untold to leave you unprepossessed. The question is: Whats the ideal way it should be done?

The following conditions should be met:

• Apache must get the original ip-address of the visitor
• Apache must know, what public-ip address was ultilized to use the right ssl-vhost
• The traffic must not be routed through a (reverse-)proxy on the host, since there are 2 other non-http-services, on other VM-guests, that should be accessible from public. And: Only sshd should listen directly on the host - nothing else
• Each VM should be able to access the internet directly.
• The network in the data-center is switched MAC-based. As I figured out, the only way to comunicate with the internet is through eth0 and its MAC-address.

If I would discard all the virtualization-stuff, this would be perfectly easy, as apache get the request directly from a specific ip-address.

I am open for any working solution.

Solution 1

Use a bridge on your dom0 (e.g. KVM Host) WAN interface. This requires installing bridge-utils package. Since this is Debian-based distro, you may configure it in /etc/network/interfaces:

iface eth0 inet manual

auto br_wan
iface br_wan inet dhcp
    # Assuming DHCP to get address, otherwise migrate all WAN connection options here
    #address 192.168.122.0
    bridge_ports eth0 tap_guest1
    bridge_stp      off
    bridge_maxwait  0
    bridge_fd       0
    pre-up ip tuntap add dev tap_guest1 user guest1 mode tap
    # This command is required if your ISP allocates static IPs depending on MAC address
    # You shouldn't use this but might be handy some time
    #pre-up sysctl -q -w net/ipv4/conf/tap_guest1/proxy_arp=1
    post-down ip tuntap del tap_guest1 mode tap

Pre-up commands set up TAP interface to connect your KVM guest to a bridge. Note that this setup allows to run kvm from non-privileged user guest1. Note that setting net.ipv4.ip_forward = 1 with sysctl might be usefull as well.

I have used ip tuntap command from iproute2 package. It's not yet documented in the Debian package but soon will be available in upstream's manual page. Since this package is installed on every Debian-based server, you won't need to install uml-utilities or openvpn package to just create these interfaces.

This approach sure lacks some elegance to manage lots of tap interfaces, because you'll need to create similar pre-up and post-down lines as for tap_guest1 interface. This can be fixed by writing additional scripts in /etc/network/pre-up.d and /etc/network/post-down.d. It is also a problem if you want to reconfigure br_wan interface with ifdown/ifup scripts while KVM guests are still running — you'll need either to remove all interfaces except eth0 from bridge configuration and detach them from bridge manually (don't forget to attach them back after bridge reconfiguration then) or shutdown all KVM instances running on a bridge.

Another way, perhaps more clean, is to write custom ifup script for KVM itself and use it in script option for your NIC. You can get an example in /etc/qemu-ifup. See kvm manual page for details.

Then you can run your KVM box like this:

kvm -net nic,model=virtio,macaddr=12:34:56:78:9a:bc \
    -net tap,ifname=tap_guest1,script=no,downscript=no \
    -boot c -nographic -display none -daemonize \
    guest1-drive.qcow2

Setting several IP addresses on one interface for your KVM guest can be done manually with command

ip address add aaa.bbb.ccc.101/24 dev eth0

Or permanently in /etc/network/interfaces like this:

auto eth0 eth0:1
iface eth0 inet static
    address aaa.bbb.ccc.100
    network aaa.bbb.ccc.0
    netmask 255.255.255.0
    broadcast aaa.bbb.ccc.255
    gateway aaa.bbb.ccc.1

iface eth0:1 inet static
    address aaa.bbb.ccc.101
    network aaa.bbb.ccc.0
    netmask 255.255.255.0
    broadcast aaa.bbb.ccc.255
    gateway aaa.bbb.ccc.1

Note that if your datacenter/provider does not expect you to reveal additional boxes on the same net he might not configure them and they will be unavailable. In this case you might want to create internal bridge and use iptables to forward packets between your WAN interface and this bridge using DNAT and SNAT. Assuming your local virtual bridge network is 10.0.0.0/8, your guest1 is 10.0.0.2 you'll need this:

iptables -t nat -A PREROUTING --dst aaa.bbb.ccc.100 -p tcp --dport 80 -j DNAT --to-destination 10.0.0.2
iptables -t nat -A PREROUTING --dst aaa.bbb.ccc.101 -p tcp --dport 80 -j DNAT --to-destination 10.0.0.2
...
iptables -t nat -A POSTROUTING -p tcp --dst 10.0.0.2 -j SNAT --to-source aaa.bbb.ccc.100

Note that you'll need as much DNAT commands as external IPs per KVM guest you have, but only one SNAT rule to give access to the internet. Also you can allow only HTTP/HTTPS/SSH traffic by allowing only desired ports. If you omit the --dport statement then all ports will be forwarded. Your KVM guest should have static network settings with KVM host as default gateway unless you're willing to host DHCP server.

Solution 2

You use a bridge instead of forwarding packets with iptables. After you started the guest you can just configure the IP's to the NIC(s) (if you have more than 1 virtual NIC) on your guest.

Refer to this documentation.

Solution 3

@jollyroger: Thank you so much for your comprehensive guide. Sadly, -in my case- it had not worked. But your guide helped me to better understand the functionality of linux-networking/bridging together with KVM.

So I reverted everything and got some help from my hosting-company directly. The Scenario was as easy as using virt-manager and virsh for adding a custom network-configuration. Here is the solution, that has finally worked for me:

I have a /29-subnet (+ another different IP for ssh on the host). The network in the data-center is mac-based-switched, so all traffic must go through a local gateway (as far as I understood it). My provider is Hetzner, in Germany. I used http://www.subnet-calculator.com/subnet.php?net_class=A to calculate all neccessary settings for a network-configuration-xml:

<network>
        <name>{some systemwide unique name. e.g.: subnet1}</name>
        <uuid>{> call uuidgen from shell to generate a uuid}</uuid>
        <forward dev="{interface e.g.: eth0}" mode="route" />
        <bridge name="{name of a _new_ virtual interface e.g.: virbr2}" stp="off" forwardDelay="0" />
        <ip address="{subnet-calc: Host Address Range Start Address}" netmask="subnet-calc: Subnet Mask">
                <dhcp>
                        <range start="{subnet-calc: Host Address Range Start Address+1}" end="{subnet-calc: Host Address Range End Address}" />
                </dhcp>
        </ip>
</network>

I saved this xml-data to something like /tmp/subnet1.xml and applied this new virtual-network by using:

sudo virsh net-define /tmp/subnet1.xml

I use virt-manager 0.9.0 (over x11-forwarding via putty on my windows-machine) to create and edit all VMs. So my 2 public-VMs both get this new virtual-network. In each VM, I've created a new virtual network-interface per public-ip. I've noted which MAC-address I have used for which public ip. So if I want to move an public IP-address to another VM, it is nothing more than deleting the virtual NIC in VM A and create it in VM B using the same MAC-address.

Another thing, if you try to reproduce this scenario: Before you start the VM(s), make sure the virtual-network "subnet1" (s.a.) is active. In the virt-manager's mainwindow go to Edit > Connection Details and jump to Tab Virtual Networks. If the Network "subnet1" is greyd out, just mark it and click on the play-button.

Now start the VM and type ifconfig. In my case everything worked directly.

If I missed anything in this report, just ask. If anyone has a better solution, please feel free to add it here.

Comments

[Kyle Smith]

Posted your image for you. Welcome to Serverfault!

[Ron]

Does that mean, that I have to create as much bridges as I own IP-Addresses?

[Steve Townsend]

Nope, you can assign as many IPs as you want on a single interface.

[Lucas Kauffman]

You create one bridge, when you create a VM you can point it at the bridge. Then you can install NIC(s) on the guest machine. Bridging means you allow the virtual NICs to access the network that your physical network card is connected to.

[Miguel Diaz Ballesteros]

So you created a local bridge and forwarded packets there but with somewhat high-level tool like virt-manager. Surprisingly I describbed that in only last two paragraphs and very briefly. Also you may try to forget those MACs, they are only meaningfull for internal DHCP. The real IP address forwarding is done with iptables and maps external IP address to internal IP in your bridge. Glad it helped you anyway.

[mikl]

I've been trying to follow this guide, but when I add the bridge to /etc/network/interfaces and reload the network connection, the network does not come back up. In my case, I have one primary public IP for the host server and a separate /28 subnet I want to assign to the guests. I kept the original address for eth0, and then set up the first IP of the /28 on the bridge. Is that the wrong approach? Should it be the same adress as eth0?

[Miguel Diaz Ballesteros]

Please paste your interfaces file and clarify which setup did you use (the DNAT/SNAT or bridging)

[Ron]

@jollyroger Why do I need that TAP device? It it not possible to connect the guests directly to the bridge?

[Miguel Diaz Ballesteros]

@Ron that tap device is an interface for QEMU/KVM. It's like the other side of the cord connected to your VM. While -net nic describes the end connected right to the VM (and its ethernet card), the other end described with -net tap is that interface which you can plug in your host machine in any ways you want. In this case bridge is acting (as always) like a switch where you plug all your VMs.