View Current State of Keepalived

Question

Is there a way to view the current state (Master/Backup) using a command line inside a server instance?

Cheers

Solution 1

You could use the notify command to write out a state file.

       # for ANY state transition.
       # "notify" script is called AFTER the
       # notify_* script(s) and is executed
       # with 3 arguments provided by keepalived
       # (ie don’t include parameters in the notify line).
       # arguments
       # $1 = "GROUP"|"INSTANCE"
       # $2 = name of group or instance
       # $3 = target state of transition
       #     ("MASTER"|"BACKUP"|"FAULT")
       notify /path/notify.sh

The create a notify script like:

#!/bin/bash
# notify.sh

echo $1 $2 is in $3 state > /var/run/keepalive.$1.$2.state

And a get state script like:

#!/bin/bash
# getstate.sh

cat /var/run/keepalive.*.*.state

Solution 2

Reading the current status through SNMP has proven the most reliable for me. To enable this you have to start keepalived with snmp support:

• add -x to the deamon options (see /etc/sysconfig/keepalived on RedHat based systems)

and install snmpd.

You can then reliably query the status via

snmpget -Oq -Ov -v2c -cpublic localhost KEEPALIVED-MIB::vrrpInstanceState.1

It can also be done throug the notify scripts, but these don't always fire, leaving the state file out of sync with reality.

Solution 3

Dumping current state can be done by sending USR2 signal to keepalived parent process:

kill -USR2 $(cat /var/run/keepalived.pid)

Look inside file /tmp/keepalived.stats. Here are examples of the result (for Keepalived v2.0.7):

• Master instance:

  VRRP Instance: Server1
    Advertisements:
      ...
    Became master: 15          <--- one more than the Released counter
    Released master: 14
    Packet Errors:
    ...
  

• Backup instance:

  VRRP Instance: Server2
    Advertisements:
      ...
    Became master: 33          <--- equal to Released counter
    Released master: 33
    Packet Errors:
    ...
  

Note: if SELinux is used (CentOS 7), this may prevent writing to this file. You can get around this with this:

touch /tmp/keepalived.stats
chmod go+w /tmp/keepalived.stats
chcon -t keepalived_var_run_t /tmp/keepalived.stats

Now try sending the signal again.

Solution 4

If you issue the command:

journalctl -u keepalived

it has the state listed:

Jul 12 13:45:52 vmt007 Keepalived_vrrp[14335]: VRRP_Instance(VI_INT) Entering MASTER STATE

on the slave side you can see:

Jul 12 13:45:51 vmt008 Keepalived_vrrp[3569]: VRRP_Instance(VI_INT) Entering BACKUP STATE

Solution 5

DBus

With version 1.3.0, Keepalived added a DBus interface^† to query the VRRP state, monitor VRRP events using signals and even modify some VRRP configuration at runtime. As of 2021, the interface is for VRRP functionality only and does not expose anything related to the IPVS subsystem of Keepalived. DBus support is enabled with the enable_dbus option in the global_defs block of the config file. Some additional options (not explained here) can be specified too.

The Interface

You can access the DBus interface with any DBus client/library (e.g. dbus-send, gdbus, qdbus). I'm using systemd's busctl as an example here, because it has a very nice interface:

# busctl tree org.keepalived.Vrrp1
└─/org
  └─/org/keepalived
    └─/org/keepalived/Vrrp1
      ├─/org/keepalived/Vrrp1/Instance
      │ └─/org/keepalived/Vrrp1/Instance/eth0
      │   └─/org/keepalived/Vrrp1/Instance/eth0/1
      │     └─/org/keepalived/Vrrp1/Instance/eth0/1/IPv4
      └─/org/keepalived/Vrrp1/Vrrp

Two interfaces are available, the global org.keepalived.Vrrp1.Vrrp on /org/keepalived/Vrrp1/Vrrp:

# busctl introspect org.keepalived.Vrrp1 /org/keepalived/Vrrp1/Vrrp org.keepalived.Vrrp1.Vrrp
NAME                      TYPE      SIGNATURE RESULT/VALUE FLAGS
.CreateInstance           method    ssuu      -            -
.DestroyInstance          method    s         -            -
.PrintData                method    -         -            -
.PrintStats               method    -         -            -
.ReloadConfig             method    -         -            -
.VrrpReloaded             signal    -         -            -
.VrrpStarted              signal    -         -            -
.VrrpStopped              signal    -         -            -

And foreach VRRP instance, the org.keepalived.Vrrp1.Instance interface on paths according to this template /org/keepalived/Vrrp1/Instance/<interface>/<virtual-router-id>/<ip-family>. For an IPv4 VRRP instance with id 1 on eth1 named my-instance, we get the following:

# busctl introspect org.keepalived.Vrrp1 /org/keepalived/Vrrp1/Instance/eth0/1/IPv4 org.keepalived.Vrrp1.Instance
NAME                          TYPE      SIGNATURE RESULT/VALUE   FLAGS
.SendGarp                     method    -         -              -
.Name                         property  s         "my-instance"  emits-change
.State                        property  (us)      2 "Master"     emits-change
.VrrpStatusChange             signal    u         -              -

If you're using multiple keepalived processes and set the instance option or if you're using the network namespace feature with the namespace option, the paths also include the instance and/or namespace.

Querying State

To get the state of a particular instance, we can use the following command

# busctl get-property org.keepalived.Vrrp1 /org/keepalived/Vrrp1/Instance/eth0/1/IPv4  org.keepalived.Vrrp1.Instance State
(us) 2 "Master"

The returned property is a STRUCT, with a state code and a human readable name. The important states are 0 "Init", 1 "Backup", 2 "Master", 3 "Fault". The first three are official RFC 2338 states and the last one is defined by Keepalived. Different versions of Keepalived have some additional states. As of version 2.2.4, there are for example also 97 "Stop" and 98 "Deleted". Until version 2.0.0 there was for example also 4 "Goto master" and code 98 had different semantics 98 "Goto fault", therefore your code should rely mainly on the four main ones and also handle new state codes.

Monitoring Signals

You can also monitor state changes using DBus signals. The following output shows DBus signals when starting, introducing a fault and stopping a Keepalived instance:

# busctl monitor --match="type='signal',sender='org.keepalived.Vrrp1'
‣ Type=signal  Endian=l  Flags=1  Version=1  Priority=0 Cookie=5
  Sender=:1.80  Path=/org/keepalived/Vrrp1/hades_root/Instance/br_vrrp/67/IPv4  Interface=org.keepalived.Vrrp1.Instance  Member=VrrpStatusChange
  UniqueName=:1.80
  MESSAGE "u" {
          UINT32 1;
  };

‣ Type=signal  Endian=l  Flags=1  Version=1  Priority=0 Cookie=6
  Sender=:1.80  Path=/org/keepalived/Vrrp1/Instance/eth0/1/IPv4  Interface=org.keepalived.Vrrp1.Instance  Member=VrrpStatusChange
  UniqueName=:1.80
  MESSAGE "u" {
          UINT32 2;
  };

‣ Type=signal  Endian=l  Flags=1  Version=1  Priority=0 Cookie=7
  Sender=:1.80  Path=/org/keepalived/Vrrp1/hades_root/Instance/br_vrrp/67/IPv4  Interface=org.keepalived.Vrrp1.Instance  Member=VrrpStatusChange
  UniqueName=:1.80
  MESSAGE "u" {
          UINT32 3;
  };

‣ Type=signal  Endian=l  Flags=1  Version=1  Priority=0 Cookie=8
  Sender=:1.80  Path=/org/keepalived/Vrrp1/hades_root/Instance/br_vrrp/67/IPv4  Interface=org.keepalived.Vrrp1.Instance  Member=VrrpStatusChange
  UniqueName=:1.80
  MESSAGE "u" {
          UINT32 1;
  };

‣ Type=signal  Endian=l  Flags=1  Version=1  Priority=0 Cookie=9
  Sender=:1.80  Path=/org/keepalived/Vrrp1/hades_root/Instance/br_vrrp/67/IPv4  Interface=org.keepalived.Vrrp1.Instance  Member=VrrpStatusChange
  UniqueName=:1.80
  MESSAGE "u" {
          UINT32 2;
  };

‣ Type=signal  Endian=l  Flags=1  Version=1  Priority=0 Cookie=7
  Sender=:1.80  Path=/org/keepalived/Vrrp1/Instance/eth0/1/IPv4  Interface=org.keepalived.Vrrp1.Instance  Member=VrrpStatusChange
  UniqueName=:1.80
  MESSAGE "u" {
          UINT32 98;
  };

‣ Type=signal  Endian=l  Flags=1  Version=1  Priority=0 Cookie=8
  Sender=:1.80  Path=/org/keepalived/Vrrp1/Vrrp  Interface=org.keepalived.Vrrp1.Vrrp  Member=VrrpStopped
  UniqueName=:1.80
  MESSAGE "" {
  };

We observe:

1. When Keepalived is started, no signal for the INIT state is emitted.
2. The instance transitions to BACKUP state (state BACKUP config was used).
3. The instance transitions to MASTER state.
4. A fault is detected and the instance transitions to FAULT state.
5. The instance recovers via going to BACKUP and then MASTER again.
6. Keepalived is stopped and the instance transitions to the STOP state.
7. Keepalived also emits a global VrrpStopped signal.

---

^† Keepalived must be built with the --enable-dbus configure option for this to work though. Most binary distributions however should have enabled it by now. Debian and Ubuntu for example do it since package version 1:1.3.9-1.

View more solutions

Comments

[Ben]

Nice solution! But what if you wanted to check the status without any state transition? Is it possible without using a patch?

[dmourati]

You mean if initial startup state is equal to current state? I guess you could just write out the /var/run/keepalive.*.*.stat file as part of your startup script.

[Ben]

I mean, if the keepalived is currently running. How can I determine the current running state without having a startup script?

[dmourati]

Try the above, I think it will work as is.

[jnovack]

As of this post, this does NOT work on CentOS 7 from the native repositories. Yum in the repo is 1.2.13. This has been solved in later versions if you want to compile it.

[gdanov]

on Ubuntu 20 this works out of the box, only need to enable dbus in the config as mentioned above

[Admin]

Can't find any documentation how to understand the data in this stats file

[Admin]

I found appropriate source code only, almost all counters stored in vrrp_stats struct.