How to configure Failover IPs for High Availability with Hetzner Online

5,978

As this is an old answer I'm not sure if you are still looking for an answer. But I stumbled upon it looking for the best way on how to do this.

The way Hetzner assigns a failover IP to a dedicated server is not to allow it to be configurred on the server but to route the traffic to the original server IP. Therefore it is possible to not change anything on your server and manually switch the IP in their admin interface. However; this is not a suitable solution for most as I would not like to get out of bed to manually fail over. This should be done automatically and then notify the admin that the failover has been done. Maybe even with a small report which issues the system has seen and why it did fail over.

Keepalived can do this for you, the only thing required is to configure keepalived to run the script while failing over. But if there is no IP to failover, how can we then failover?

Simple; create an internal network between the servers and assign your own non-routed internal IP to keepalived. As this internal network uses the same interface as the external network it does not really matter. A benefit of this approach is that you can keep all internal traffic 100% internal by using this internal VIP.

Once Keepalived fails over you order it to run the script from Hetzner to also switch the external ip using: notify

An example:

vrrp_script chk_haproxy {
        script "killall -0 haproxy"     # cheaper than pidof
        interval 2                      # check every 2 seconds
        weight 2                        # add 2 points of prio if OK
}

vrrp_instance VI_1 {
    state MASTER
    interface enp0s31f6.4000
    virtual_router_id 51
    priority 101
    virtual_ipaddress {
        192.168.100.3/24 # this is the shared IP I was using
      }
    track_script {
        chk_haproxy
    }
    notify /usr/sbin/hetzner_failoverIP.sh database set $THIS_SERVER_IP
}

Ofcourse the Hetzner script can be adjusted to be much smarter, selecting the server IP by itself for example.

The downside that should be noted is that the external IP will take between 40 and 60 seconds. For me, a minimum of 40 seconds and a maximum of 1 minute is too long.

Another option is to use the Hetzner cloud instances to enable HA without the use of the failover IP and the script above. In the cloud there is another solution: Cloud floating IP.

This option will set you back for about €8,50 euro's per month for:

two cloud instances (1 basic cpu, 2GB memory and 20TB traffic each)
two cloud floating IP's

Then use keepalived to manage the cloud floating IP (virtual_ipaddress section) and HAProxy to send all traffic to the dedicated servers. HAProxy will then do the healthchecks and you don't need to worry about:

Switching IP's using the Hetzner API
40 to 60 seconds of additional downtime

It's is worth to mention that Hetzner cloud servers do not support an internal network. But it is not required if you use them in this way and it will not cost you extra as internal traffic is free of charge. For security, secure the load balancers (Keepalived+HAProxy cloud instances) with SELinux/AppArmor and Firewalld. Use encrypted traffic between the two clusters (cloud <-> Dedicated) to prefent packet sniffing. I would also encrypt the traffic between your dedicated servers even if you are using a private VLAN, the traffic is still being send out via the same NIC. Something to keep in mind.

Sources used:

5,978

merlin

Updated on September 18, 2022

Comments

merlin almost 2 years
I have a cluster of 3 Ubuntu nodes running in VMs in the lab and want to take it now to production. Hetzner Online hetzner.de offers some good value dedicated servers, so I rented 3 machines, connected with a gigabit switch.

My intention is to create a HA-Setup with two keepalived in front of 2 HAProxy Servers. Keepalived is configured with a VIP inside my setup. Unfortunatelly this does not work with Hetzner. However they provide a system called failover IP where one could switch with the help of a script to the other server: http://wiki.hetzner.de/index.php/Failover_Skript

My config for keepalived looks like this:
```
vrrp_script chk_haproxy {
        script "killall -0 haproxy"     # cheaper than pidof
        interval 2                      # check every 2 seconds
        weight 2                        # add 2 points of prio if OK
}

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 51
    priority 101
    virtual_ipaddress {
        192.168.56.101/24 # this is the shared IP I was using
      }
    track_script {
        chk_haproxy
    }
}
```
So how would their Failover Script fit into this?

It looks like I am not the only one with the problem, just the solution is not so obvious. https://www.howtoforge.com/community/threads/hetzner-to-stop-support-for-high-availability-setups.19988/
- womble almost 9 years
  
  It would replace your keepalived setup.
- womble almost 9 years
  
  I don't understand your objection. Your question it not about shared internal IPs, it's about failing over a Hetzner shared IP.
- merlin almost 9 years
  
  That's true. I have confounded this question with a similar one: serverfault.com/questions/721249/…
- cherouvim about 8 years
  
  Did you find a solution to this?
- merlin about 8 years
  
  apparently there is no keepalived/backup setup possible with Hetzner with automated IP-failover. You could however switch IPs manually through their backend. There is also a script that might do this automatically,