Linux – Pacemaker complex resource colocation

heartbeathigh-availabilitylinuxpacemakerunix

I am working on a pacemaker project for our master and slave databases to perform IP based failover. There will be two IP resources, one for the master, and one for the slave that have to move together. I realize that what I have marked below is not a complete solution, but the requirements are as follows:

  1. ip_dbmaster can only run on db1 or db21
  2. ip_dbslave can only run on dbslave1 or dbslave2
  3. when ip_dbmaster is on db1, ip_dbslave must be on dbslave1. When ip_dbmaster is on db2, ip_dbslave must be on dbslave2
  4. before ip_dbmaster and ip_dbslave are run, do some "stuff" (shell scripty stuff, some extended health checks). Only migrate if "stuff" is successful
  5. same as above, except after the resource migrates

Here is my base configuration:

node $id="75463ec2-702c-427b-965b-b7ffb7814008" db1
node $id="a1f2d612-2d9f-4872-bf24-024f5bece3ce" dbslave2
node $id="d1d42f67-e4f2-4c71-950f-07d94ac01f8d" dbslave1
node $id="f243d865-c1a1-4d52-9100-b0d36a08207c" db2
primitive ip_dbmaster ocf:heartbeat:IPaddr2 \
  params ip="10.153.114.100" cidr_netmask="24"
primitive ip_dbslave ocf:heartbeat:IPaddr2 \
  params ip="10.153.114.101" cidr_netmask="24"
location loc-ip-dbmaster-1 ip_dbmaster \
  rule $id="loc-ip-dbmaster-1-rule" 200: #uname eq db1
location loc-ip-dbmaster-2 ip_dbmaster \
  rule $id="loc-ip-dbmaster-2-rule" 0: #uname eq db2
location loc-ip-dbmaster-3 ip_dbmaster \
  rule $id="loc-ip-dbmaster-3-rule" -inf: #uname eq dbslave2
location loc-ip-dbmaster-4 ip_dbmaster \
  rule $id="loc-ip-dbmaster-4-rule" -inf: #uname eq dbslave1
location loc-ip-dbslave-1 ip_dbslave \
  rule $id="loc-ip-dbslave-1-rule" 200: #uname eq dbslave1
location loc-ip-dbslave-2 ip_dbslave \
  rule $id="loc-ip-dbslave-2-rule" 0: #uname eq dbslave2
location loc-ip-dbslave-3 ip_dbslave \
  rule $id="loc-ip-dbslave-3-rule" -inf: #uname eq db1
location loc-ip-dbslave-4 ip_dbslave \
  rule $id="loc-ip-dbslave-4-rule" -inf: #uname eq db2
property $id="cib-bootstrap-options" \
  dc-version="1.0.8-042548a451fce8400660f6031f4da6f0223dd5dd" \
  cluster-infrastructure="Heartbeat" \
  expected-quorum-votes="2" \
  stonith-enabled="false" \
  symmetric-cluster="false"

So far, #1 and #2 work just great. #3 seems like it's sort of similar to colocation, but I can't figure out where to start or how to set the dependency. I see a rules engine, but I haven't quite figured it out yet. For 4-5, I have no idea if this can be done via pacemaker at all or whether I have to write an external script to handle this piece.

My goal would be able to do this:

crm resource move ip_dbmaster db1

and have everything just go. I'll settle for 

dbfailover db2

…if I have to code some of this external to Pacemaker, or whether this can be built-in.

Thanks for all your help!

Best Answer

First of all, let me say that I've set up quite a few clusters over the last decade, and I've never seen one where there was a dependency like you've described. Usually one would set it up so that the services provided don't depend on which host is active and which is standby and you don't care which host has the resource, as long as it's up on one of them.

The only way I can come up with to implement what you want is to implement the slave node as a resource that is initiated by the master node, for example by SSHing over to the slave node to run the IPaddr2 and other resources you need. Likely using SSH public key authentication with an identity file and authorized_keys entry so that the master can run the commands on the slave without requiring a password.

So this would require creating a "slaveIPaddr2" resource script, that would just wrap a command like:

HOST=`hostname`
exec ssh -i /path/to/ssh-identity dbslave$${HOST#db} /path/to/IPaddr2 "$@"

Then change the ip_dbslave resource to "slaveIPaddr2" instead of "IPaddr2" as the resource to run.

As far as scripts to run before and after migration, these mostly just sound like they would be the normal multiple resource scripts that make up a resource group and precedence using the "group" and "order" configuration items. For example, creating "master_pre" (the "before" script you want to run on the master), "slave_pre", "master_post", etc... resources, then using "order" to specify that they run in the appropriate order (master_pre, slave_pre, ip_dbmaster, ip_dbslave, master_post, slave_post). Here you'll also likely need to wrap the slave items with the SSH wrapper, to effectively treat them as a single host as I mentioned above.

It sounds like you want the "pre" script to be run before a migration is even attempted, rather than as a part of starting the resource? Pacemaker isn't going to migrate a service unless it's told to by you, or the node currently running the service is failing. In the case of a failing node, your service is down anyway, so no reason to check to try to avoid the migration. So if you are concerned with preventing the migration when you tell it to migrate, the best answer there may be to make a "migrate" script that runs your pre-service checks, and only goes on with the migration request if the tests succeed.

I don't know of a way in pacemaker to test the other hosts in the cluster before doing a migration, if that is what you are trying to achieve with #4, so it'll likely have to be an external check that enforces that.

Running other resources than just the IPaddr2 is easily done via the "group" and "order" directives.

Related Topic