Keepalived for more than 20 virtual addresses

Question

I have set up keepalived on two Debian machines for high availability, but I've run into the maximum number of virtual IP's I can assign to my vrrp_instance. How would I go about configuring and failing over 20+ virtual IP's?

This is the, very simple, setup:

LB01: 10.200.85.1
LB02: 10.200.85.2
Virtual IPs: 10.200.85.100 -  10.200.85.200

Each machine is also running Apache (later Nginx) binding on the virtual IPs for SSL client certificate termination and proxying to backend webservers. The reason I need so many VIP's is the inability to use VirtualHost on HTTPS.

This is my keepalived.conf:

vrrp_script chk_apache2 {
        script "killall -0 apache2"
        interval 2
        weight 2
}

vrrp_instance VI_1 {
        interface eth0
        state MASTER
        virtual_router_id 51
        priority 101
        virtual_ipaddress {
            10.200.85.100
            .
            . all the way to
            .
            10.200.85.200
}

An identical configuration is on the BACKUP machine, and it's working fine, but only up to the 20th IP.

I have found a HOWTO discussing this problem. Basically, they suggest having just one VIP and routing all traffic "via" this one IP, and "all will be well". Is this a good approach? I'm running pfSense firewalls in front of the machines.

Quote from the above link:

ip route add $VNET/N via $VIP

or

route add $VNET netmask w.x.y.z gw $VIP

Thanks in advance.

EDIT:

@David Schwartz said it would make sense to add a route, so I tried adding a static route to the pfSense firewall, but that didn't work as I expected it would.

pfSense route:

Interface:            LAN
Destination network:  10.200.85.200/32 (virtual IP)
Gateway:              10.200.85.100    (floating virtual IP)
Description:          Route to VIP .100

I also made sure I had packet forwarding enabled on my hosts:

$ cat /etc/sysctl.conf
net.ipv4.ip_forward=1
net.ipv4.ip_nonlocal_bind=1

Am I doing this wrong? I also removed all VIPs from the keepalived.conf so it only fails over 10.200.85.100.

Solution

The simplest solution while not changing your current architecture is to make use of virtual_ipaddress_excluded. For example

vrrp_instance VI_1 {
    interface eth0
    state MASTER
    virtual_router_id 51
    priority 101

    virtual_ipaddress {
        10.200.85.100
    }

    virtual_ipaddress_excluded {
        10.200.85.101
        . all the way to
        10.200.85.200
    }
}

virtual_ipaddress_excluded contains a list of IP addresses that keepalived will bring up and down on the server, however they are not included in the VRRP packet itself so they don't count towards the 20 IP address limit.

In my configurations I like to allocate an IP specifically for virtual_ipaddress. i.e. the one that is included in the VRRP packets and put everything else in virtual_ipaddress_excluded. This is a good idea because you don't want to have to change the main IP just because a customer left.

Comments

[David Schwartz]

I think that approach makes good sense.

[Josh]

... or would I just make additional vrrp_instances each with a block of 20 IP's? I could probably group the hosted HTTPS sites some way giving it some additional meaning.

[David Schwartz]

The destination network should be the set of 20+ IPs, not a single IP.

[Josh]

Excellent. I'm doing this instead of multiple vrrp instances. You also made me read up on keepalived documentation. Thanks!