Why is keepalived running two MASTER nodes in the Sticky VIP configuration

galerahaproxykeepalived

I have keepalived setup (floating VIP) in front of haproxy on each of my three-node galera cluster nodes. When I restart keepalived on any given node, sometimes I end up with two nodes running in MASTER (as evidenced by the /etc/keepalived/log_status.sh notify script):

# cat /etc/keepalived/log_status.sh 
#!/bin/bash
echo $1 $2 is in $3 state > /var/run/keepalive.$1.$2.state

From what I've read, the 'multiple masters' is due to Multicast being filtered on the switch but I can run tcpdump on any one of my galera nodes and see the MC traffic hitting the nic (these are KVM virtuals running). I can try changing to unicast, but would like to know if this is due to a bug, feature or my config.

# cat /etc/keepalived/keepalived.conf 
log "setting up keepalived"
global_defs {
  router_id    host1  # short hostname of each KA node (10.20.18.201-203)
}

vrrp_script check_haproxy {
   script      "pidof haproxy"
   interval    2
   weight      2
}

vrrp_instance 250 {
  virtual_router_id 250
  advert_int   1
  nopreempt
  priority     100
  state        BACKUP
  interface    eth0
  notify       /etc/keepalived/log_status.sh

  virtual_ipaddress {
    10.20.18.250 dev   eth0
  }

  track_script {
    check_haproxy
  }
}

tcpdump output:

09:44:00.934942 IP 10.20.18.203 > 224.0.0.18: VRRPv2, Advertisement, vrid 250, prio 102, authtype none, intvl 1s, length 20
09:44:01.936054 IP 10.20.18.203 > 224.0.0.18: VRRPv2, Advertisement, vrid 250, prio 102, authtype none, intvl 1s, length 20
09:44:02.937315 IP 10.20.18.203 > 224.0.0.18: VRRPv2, Advertisement, vrid 250, prio 102, authtype none, intvl 1s, length 20
09:44:03.938444 IP 10.20.18.203 > 224.0.0.18: VRRPv2, Advertisement, vrid 250, prio 102, authtype none, intvl 1s, length 20
09:44:04.942302 IP 10.20.18.203 > 224.0.0.18: VRRPv2, Advertisement, vrid 250, prio 102, authtype none, intvl 1s, length 20
09:44:05.373224 IP 10.20.18.201 > 224.0.0.22: igmp v3 report, 1 group record(s)
09:44:05.943936 IP 10.20.18.203 > 224.0.0.18: VRRPv2, Advertisement, vrid 250, prio 102, authtype none, intvl 1s, length 20
09:44:06.029216 IP 10.20.18.201 > 224.0.0.22: igmp v3 report, 1 group record(s)
09:44:06.385127 IP 10.20.18.201 > 224.0.0.22: igmp v3 report, 1 group record(s)
09:44:06.945303 IP 10.20.18.203 > 224.0.0.18: VRRPv2, Advertisement, vrid 250, prio 102, authtype none, intvl 1s, length 20
09:44:07.333210 IP 10.20.18.201 > 224.0.0.22: igmp v3 report, 1 group record(s)
09:44:07.946098 IP 10.20.18.203 > 224.0.0.18: VRRPv2, Advertisement, vrid 250, prio 102, authtype none, intvl 1s, length 20
09:44:08.947228 IP 10.20.18.203 > 224.0.0.18: VRRPv2, Advertisement, vrid 250, prio 102, authtype none, intvl 1s, length 20
09:44:09.948507 IP 10.20.18.203 > 224.0.0.18: VRRPv2, Advertisement, vrid 250, prio 102, authtype none, intvl 1s, length 20
09:44:10.548023 IP 10.20.18.202 > 224.0.0.22: igmp v3 report, 1 group record(s)
09:44:10.663961 IP 10.20.18.202 > 224.0.0.22: igmp v3 report, 1 group record(s)
09:44:10.949633 IP 10.20.18.203 > 224.0.0.18: VRRPv2, Advertisement, vrid 250, prio 102, authtype none, intvl 1s, length 20
09:44:11.559970 IP 10.20.18.202 > 224.0.0.22: igmp v3 report, 1 group record(s)
09:44:11.587980 IP 10.20.18.202 > 224.0.0.22: igmp v3 report, 1 group record(s)
09:44:11.950795 IP 10.20.18.203 > 224.0.0.18: VRRPv2, Advertisement, vrid 250, prio 102, authtype none, intvl 1s, length 20
09:44:12.952124 IP 10.20.18.203 > 224.0.0.18: VRRPv2, Advertisement, vrid 250, prio 102, authtype none, intvl 1s, length 20
09:44:13.953075 IP 10.20.18.203 > 224.0.0.18: VRRPv2, Advertisement, vrid 250, prio 102, authtype none, intvl 1s, length 20
09:44:14.953543 IP 10.20.18.203 > 224.0.0.18: VRRPv2, Advertisement, vrid 250, prio 102, authtype none, intvl 1s, length 20
09:44:15.954703 IP 10.20.18.203 > 224.0.0.18: VRRPv2, Advertisement, vrid 250, prio 102, authtype none, intvl 1s, length 20
09:44:15.987641 IP 10.20.18.203 > 224.0.0.18: VRRPv2, Advertisement, vrid 250, prio 0, authtype none, intvl 1s, length 20
09:44:15.992698 IP 10.20.18.203 > 224.0.0.22: igmp v3 report, 1 group record(s)
09:44:16.008817 IP 10.20.18.203 > 224.0.0.22: igmp v3 report, 1 group record(s)
09:44:17.008829 IP 10.20.18.203 > 224.0.0.22: igmp v3 report, 1 group record(s)
09:44:17.036879 IP 10.20.18.203 > 224.0.0.22: igmp v3 report, 1 group record(s)
09:44:20.613407 IP 10.20.18.203 > 224.0.0.18: VRRPv2, Advertisement, vrid 250, prio 102, authtype none, intvl 1s, length 20
09:44:21.615616 IP 10.20.18.203 > 224.0.0.18: VRRPv2, Advertisement, vrid 250, prio 102, authtype none, intvl 1s, length 20
09:44:22.616909 IP 10.20.18.203 > 224.0.0.18: VRRPv2, Advertisement, vrid 250, prio 102, authtype none, intvl 1s, length 20
09:44:23.618155 IP 10.20.18.203 > 224.0.0.18: VRRPv2, Advertisement, vrid 250, prio 102, authtype none, intvl 1s, length 20
09:44:24.619607 IP 10.20.18.203 > 224.0.0.18: VRRPv2, Advertisement, vrid 250, prio 102, authtype none, intvl 1s, length 20

Best Answer

Single word answer: iptables.

I was running two instances of keepalived - one to allow access from inside networks and the other to support external access.

I copied the internal configuration to create an external keepalived instance. While keepalived was working properly on the first interface ( the internal one, eth0 ) , my copied config was producing a VIP on both hosts.

My review of tcpdump showed that the bcast VRRP traffic was allowed in the network and visible to both keepalived instances. I reviewed tcp traffic on both the internal and external interfaces ( eth0 internal / eth1 external ) .

VRRP traffic must be allowed. I found that I could sniff the traffic successfully and saw VRRP traffic from both of my keepalived instances with the correct (and different) priorities. However, my iptables configuration was only allowing traffic on eth1.

The relevant lines in /etc/sysconfig/iptabes:

Before (problems on keepalived on eth1 but eth0 OK):

###Allow multicast for KeepAlived
-A INPUT -i eth0  -d 224.0.0.18/32 -p vrrp -j ACCEPT
-I OUTPUT -o eth0 -d 224.0.0.18/32 -p vrrp -j ACCEPT

After ( all good ) :

###Allow multicast for KeepAlived
-A INPUT -i eth0  -d 224.0.0.18/32 -p vrrp -j ACCEPT
-I OUTPUT -o eth0 -d 224.0.0.18/32 -p vrrp -j ACCEPT
-A INPUT -i eth1  -d 224.0.0.18/32 -p vrrp -j ACCEPT
-I OUTPUT -o eth1 -d 224.0.0.18/32 -p vrrp -j ACCEPT

Related Solutions

Both servers running keepalived become master

This can actually be caused a bug. I know because I've had to fix it myself.

According to the RFC, when priorities are equal on both nodes;

        If the Priority in the ADVERTISEMENT is equal to the local
        Priority and the primary IP Address of the sender is greater
        than the local primary IP Address, then:

         o Cancel Adver_Timer
         o Set Master_Down_Timer to Master_Down_Interval
         o Transition to the {Backup} state

So, he who has the biggest IP address will win.

In keepalived, the way this is done is basically wrong. Endianness is not considered properly when doing this comparison.

Lets imagine we have two routers, (A)10.1.1.200 and (B)10.1.1.201.

The code should perform the following comparison.

On A:

if (10.1.1.201 > 10.1.1.200) // True
   be_backup();

On B:

if (10.1.1.200 > 10.1.1.201) // False
  be_master();

However because the endianness is not incorrectly handled, the following comparison is made instead.

On A:

if (10.1.1.201 > 200.1.1.10) // False
  be_master();

On B:

if (10.1.1.200 > 201.1.1.10) // False
  be_master();

This patch should work, but i've remade it from my original patch and have not tested it. Not even tested it compiles! So no refunds!

--- vrrp/vrrp.c.old 2013-10-13 17:39:29.421000176 +0100
+++ vrrp/vrrp.c 2013-10-13 18:07:57.360000966 +0100
@@ -923,7 +923,7 @@
    } else if (vrrp->family == AF_INET) {
        if (hd->priority > vrrp->effective_priority ||
            (hd->priority == vrrp->effective_priority &&
-            ntohl(saddr) > ntohl(VRRP_PKT_SADDR(vrrp)))) {
+            ntohl(saddr) > VRRP_PKT_SADDR(vrrp))) {
            log_message(LOG_INFO, "VRRP_Instance(%s) Received higher prio advert"
                        , vrrp->iname);
            if (proto == IPPROTO_IPSEC_AH) {

Keepalived configuration – To set MASTER/BACKUP or EQUAL

If the cluster members are of the same specs (no machine is much better in terms of hardware over the other), I would prefer to keep the current master without failing back to the original master. At least, this will save the current sessions/transactions during the original master recovery and there will be no real benefit as the two machines are identical.

Best Answer

Related Solutions

Both servers running keepalived become master

Keepalived configuration – To set MASTER/BACKUP or EQUAL

Related Topic