keepalived doesn't listen on virtual IP

Question

I'm learning to use keepalived; for now within a single machine - just to make sure the forwarding works.

I've started a mock webserver listening on 192.168.56.200:8080. It just replies with the hostname.

$ curl http://192.168.56.200:8080/
controller-1

Now I've configured keepalived (/etc/keepalived/keepalived.conf):

virtual_server 192.168.111.1 8082 {
  delay_loop 30
  lb_algo rr
  lb_kind DR
  persistence_timeout 50
  protocol TCP
  real_server 192.168.56.200 8080 {
    weight 1
    TCP_CHECK {
    }
  }
}

vrrp_instance VI_1 {
  state MASTER
  interface eth0
  virtual_router_id 1
  priority 100
  virtual_ipaddress {
    192.168.111.1/24
  }
}

It doesn't seem to work, however:

$ curl http://192.168.111.1:8082/
curl: (7) couldn't connect to host
$ telnet 192.168.111.1 8082
Trying 192.168.111.1...
telnet: Unable to connect to remote host: Connection refused

And indeed, according to netstat -apn | grep 8082, noone is listening on the port 8082, though keepalived probably should.

What's wrong with my configuration?

Some more useful info:

$ ip a
...
2: eth0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:4e:4b:c6 brd ff:ff:ff:ff:ff:ff
    inet 192.168.111.1/24 scope global eth0
    inet6 fe80::a00:27ff:fe4e:4bc6/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 1000
    link/ether 08:00:27:9d:c4:b0 brd ff:ff:ff:ff:ff:ff
    inet 192.168.56.200/24 scope global eth1
    inet6 fe80::a00:27ff:fe9d:c4b0/64 scope link 
       valid_lft forever preferred_lft forever

And syslog:

Aug 10 22:58:36 controller-1 Keepalived: Starting Keepalived v1.2.2 (12/23,2011)
Aug 10 22:58:36 controller-1 Keepalived: Starting Healthcheck child process, pid=5257
Aug 10 22:58:36 controller-1 Keepalived: Starting VRRP child process, pid=5258
Aug 10 22:58:36 controller-1 Keepalived_healthcheckers: Initializing ipvs 2.6
Aug 10 22:58:36 controller-1 Keepalived_healthcheckers: Registering Kernel netlink reflector
Aug 10 22:58:36 controller-1 Keepalived_healthcheckers: Registering Kernel netlink command channel
Aug 10 22:58:36 controller-1 Keepalived_healthcheckers: Opening file '/etc/keepalived/keepalived.conf'.
Aug 10 22:58:36 controller-1 Keepalived_healthcheckers: Configuration is using : 8775 Bytes
Aug 10 22:58:36 controller-1 Keepalived_vrrp: Registering Kernel netlink reflector
Aug 10 22:58:36 controller-1 Keepalived_vrrp: Registering Kernel netlink command channel
Aug 10 22:58:36 controller-1 Keepalived_vrrp: Registering gratutious ARP shared channel
Aug 10 22:58:36 controller-1 Keepalived_vrrp: Initializing ipvs 2.6
Aug 10 22:58:36 controller-1 Keepalived_vrrp: Opening file '/etc/keepalived/keepalived.conf'.
Aug 10 22:58:36 controller-1 Keepalived_vrrp: Configuration is using : 59356 Bytes
Aug 10 22:58:36 controller-1 Keepalived_vrrp: Using LinkWatch kernel netlink reflector...
Aug 10 22:58:36 controller-1 Keepalived_healthcheckers: Using LinkWatch kernel netlink reflector...
Aug 10 22:58:36 controller-1 Keepalived_healthcheckers: Activating healtchecker for service [192.168.56.200]:8080
Aug 10 22:58:37 controller-1 ntpd[1318]: Deleting interface #22 eth0, 192.168.111.1#123, interface stats: received=0, sent=0, dropped=0, active_time=76 secs
Aug 10 22:58:37 controller-1 ntpd[1318]: peers refreshed
Aug 10 22:58:37 controller-1 Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATE
Aug 10 22:58:38 controller-1 Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE
Aug 10 22:58:38 controller-1 avahi-daemon[775]: Joining mDNS multicast group on interface eth0.IPv4 with address 192.168.111.1.
Aug 10 22:58:38 controller-1 avahi-daemon[775]: New relevant interface eth0.IPv4 for mDNS.
Aug 10 22:58:38 controller-1 avahi-daemon[775]: Registering new address record for 192.168.111.1 on eth0.IPv4.
Aug 10 22:58:40 controller-1 ntpd[1318]: Listen normally on 23 eth0 192.168.111.1 UDP 123
Aug 10 22:58:40 controller-1 ntpd[1318]: peers refreshed
Aug 10 22:58:40 controller-1 ntpd[1318]: new interface(s) found: waking up resolver
Aug 10 22:58:42 controller-1 ntpd_intres[1320]: host name not found: 0.ubuntu.pool.ntp.org
Aug 10 22:58:42 controller-1 ntpd_intres[1320]: host name not found: 1.ubuntu.pool.ntp.org
Aug 10 22:58:42 controller-1 ntpd_intres[1320]: host name not found: 2.ubuntu.pool.ntp.org
Aug 10 22:58:42 controller-1 ntpd_intres[1320]: host name not found: 3.ubuntu.pool.ntp.org
Aug 10 22:58:42 controller-1 ntpd_intres[1320]: host name not found: ntp.ubuntu.com

In sysctl, ip_forward and ip_nonlocal_bind are enabled.

Solution

Your server application needs to listen on 0.0.0.0:8080, not just on 192.168.56.200:8080.

Use this to verify where the server is listening:

netstat -nlta | grep 8080

You'll likely find this:

tcp        0      0 192.168.56.200:8080           0.0.0.0:*               LISTEN

This indicates that the service is listening explicitly on that address, and not any other addresses that may be configured on the system. You can also use the notify_ directives to signal activity when the state changes (e.g., to restart the service so that it is aware of the new IP address(es)).

notify_master "/etc/init.d/service start"
notify_backup "/etc/init.d/service stop"
notify_fault  "/etc/init.d/service stop"

Comments

[jkff]

Hm, thanks - but why must my server listen on 0.0.0.0:8080? Isn't that what the IP address in "real_server" directive is for?

[bahamat]

No. The service application has no knowledge of what is going on with keepalived. If it is bound to a specific IP it will have no way of knowing that it needs to rebind on additional IP's unless you somehow tell it (i.e., using notify_*).