Redis Sentinel Not Failing Over

failoverhigh-availabilityredis

We have three redis 3.2.7 servers. Each redis server runs redis itself and the redis sentinel (for failover). Here's the sentinel config for each node

Slave 1 (sentinel) 192.168.0.15

bind 192.168.0.15
port 26379
logfile "/var/log/sentinel.log"
sentinel myid 57ge72174330d523tcec8adcd1e039c7f51c298c
sentinel monitor mymaster 192.168.0.16 6379 2
sentinel down-after-milliseconds mymaster 5000
sentinel auth-pass mymaster super-secret-password
sentinel config-epoch mymaster 77
dir "/"
sentinel leader-epoch mymaster 0
sentinel known-sentinel mymaster 192.168.0.17 26379 2b72876887f0e2c8db75e6d407babb60eb1a7185
sentinel known-sentinel mymaster 192.168.0.16 26379 06b84b7e6e320797ef49681c2e0598364b673c15
sentinel current-epoch 78

Master (sentinel) – 192.168.0.16

bind 192.168.0.16
port 26379
logfile "/var/log/sentinel.log"
sentinel myid 06b84b7e6e320797ef49681c2e0598364b673c15
sentinel monitor mymaster 192.168.0.16 6379 2
sentinel down-after-milliseconds mymaster 5000
sentinel auth-pass mymaster super-secret-password
dir "/"
sentinel config-epoch mymaster 77
sentinel leader-epoch mymaster 0
sentinel known-sentinel mymaster 192.168.0.17 26379 2b72876887f0e2c8db75e6d407babb60eb1a7185
sentinel known-sentinel mymaster 192.168.0.15 26379 57ge72174330d523tcec8adcd1e039c7f51c298c
sentinel current-epoch 78

Slave S (sentinel) 192.168.0.17

bind 192.168.0.17
port 26379
logfile "/var/log/sentinel.log"
sentinel myid 2b72876887f0e2c8db75e6d407babb60eb1a7185
sentinel monitor mymaster 192.168.0.16 6379 2
sentinel down-after-milliseconds mymaster 5000
sentinel auth-pass mymaster super-secret-password
dir "/"
sentinel config-epoch mymaster 77
sentinel leader-epoch mymaster 0
sentinel known-sentinel mymaster 192.168.0.16 26379 06b84b7e6e320797ef49681c2e0598364b673c15
sentinel known-sentinel mymaster 192.168.0.15 26379 57ge72174330d523tcec8adcd1e039c7f51c298c
sentinel current-epoch 78

If I stop the redis-server and redis-sentinel on the Master node I see the following on the slaves:

2664:X 01 Mar 16:17:15.563 # +sdown master mymaster 192.168.0.16 6379

So the sentinels are seeing the Master as down. However no failover occurs. If I check the slaves using:

redis-cli -h 127.0.0.1 -p 26379 -a super-secret-password sentinel get-master-addr-by-name mymaster

I get:

1) "192.168.0.16"
2) "6379"

So they're seeing the old Master (which has been shutdown as still their Master. If they're seeing the Master as down why aren't they electing a new Master?

By the way I believe this configuration was working when we were running Redis 2.8.9. So I can only guess that redis sentinel might have changed in 3.2.7. I'm just not sure where to begin looking.

Any help or pointers would be appreciated.

Thanks
Brad

Best Answer

Looks like the following issue with Redis 3.2 https://github.com/redis/redis/issues/3416