You can run a query against your information_schema table to get my mysql stats on current usage by MEMORY tables, if you use them:
select sum(data_length+index_length) as memory from information_schema.tables where engine='memory';
Also if you use INNODB, you can run SHOW ENGINE INNODB STATUS granted you have the PROCESS privelage
At the server side, for a simple failover, you can do it by using Nagios and NRPE.
On the Nagios server:
define service{
use critical-service
host_name B
service_description redis:2302
check_command check_tcp!2302
event_handler promote_redis!C!2302
contact_groups admin-sms,admin
}
define command{
command_name promote_redis
command_line $USER1$/eventhandlers/promote_redis.sh $SERVICESTATE$ $SERVICESTATETYPE$ $SERVICEATTEMPT$ $ARG1$ $ARG2$
}
promote_redis.sh
#!/bin/bash
case "$1" in
OK)
;;
WARNING)
;;
UNKNOWN)
/usr/local/nagios/libexec/check_nrpe -H $4 -c promote_redis -a $5
;;
CRITICAL)
/usr/local/nagios/libexec/check_nrpe -H $4 -c promote_redis -a $5
;;
esac
exit 0
On the slave:
nrpe.cfg
command[promote_slave_redis]=/usr/lib64/nagios/plugins/promote_redis.sh $ARG1$
promote_redis.sh
#!/bin/bash
echo 'slaveof no one' | /usr/local/redis/bin/redis-cli -h C -p $1
Virtual IP can be set up by using keepalived, something like this:
/etc/keepalived/keepalived.conf
vrrp_script chk_redis {
script "killall -0 redis-server"
interval 2
weight 2
}
vrrp_instance VI_1 {
state MASTER
#nopreempt
interface eth0
lvs_sync_daemon_interface eth0
virtual_router_id 51
priority 101
advert_int 1
authentication {
auth_type PASS
auth_pass pa$$w0rd
}
virtual_ipaddress {
x.y.z.t
}
track_script {
chk_redis
}
}
But what happens if you want to run multiple instances?
Yves Trudeau and Percona wrote a great resource agent to failover MySQL. Based on that, Martin Walter rewrite for the Redis service. Give it a try!
crm configure show
node master_629
node svr200-632.localdomain
primitive VIP ocf:heartbeat:IPaddr2 \
params ip="192.168.6.179" cidr_netmask="24" \
meta target-role="Started"
primitive redis_6380 ocf:heartbeat:redis \
params config="/usr/local/redis/etc/redis.conf"
ms ms_redis_6380 redis_6380 \
meta master-max="1" master-node-max="1" clone-max="2" clone-node-max="1" notify="true" target-role="Started"
location REDIS-PREFER-master_629 ms_redis_12345 100: master_629
colocation vip_on_ms_redis_6380 inf: VIP:Started ms_redis_6380:Master
order vip_after_ms_redis_6380 inf: ms_redis_6380:promote VIP:start
property $id="cib-bootstrap-options" \
no-quorum-policy="ignore" \
default-action-timeout="60s" \
stonith-enabled="false" \
startup-fencing="false" \
dc-version="1.0.12-unknown" \
cluster-infrastructure="openais" \
expected-quorum-votes="2"
crm status
============
Last updated: Tue Aug 21 22:12:11 2012
Stack: openais
Current DC: master_629 - partition with quorum
Version: 1.0.12-unknown
2 Nodes configured, 2 expected votes
6 Resources configured.
============
Online: [ svr200-632.localdomain master_629 ]
Master/Slave Set: ms_redis_12345
Masters: [ master_629 ]
Slaves: [ svr200-632.localdomain ]
VIP (ocf::heartbeat:IPaddr2): Started master_629
Master/Slave Set: ms_redis_6380
Masters: [ master_629 ]
Slaves: [ svr200-632.localdomain ]
Best Answer
I was also having problems with redis CPU usage. My case was many redis connections (clients) and linux default ulimit of 1024 open files.
High CPU usage could be caused by some CPU intensive operations like smembers.
You can also use the SLOWLOG redis command to log the long running commands.
The best you can do to trace the problem, when no log and SLOWLOG results, is to strace the redis process and see what is going on. For my case it was 'Too many open files' error, but it was only seen in strace output, which was solved by increasing default nr of files limit over 1024.