So I am seeing a lot of conflicting viewpoints out there on using STONITH with a 2-node DRBD/Pacemaker/Corosync cluster for replicating MySQL data. The example I could find on the Pacemaker website seems to turn it off, but a lot of other places say you should keep it on….. My setup will be 2 nodes with 2 interfaces, one physically connected to the other machine, the other hooked up to a switch. In that case, if I have redundant communications is STONITH necessary? If a server loses both network connections it wont be receiving any MySQL data anyway, and when it comes back up I plan to set the stickyness to infinite so it (shouldn't) won't try to become the master. In this case is STONITH necessary or even advisable?
Mysql – STONITH with a DRBD/Pacemaker/Corosync 2 node cluster
drbdMySQLpacemaker
Related Solutions
This is a slightly older question but the problem presented here is based on a misconception on how and when failover in clusters, especially two-node clusters, works.
The gist is: You can not do failover testing by disabling communication between the two nodes. Doing so will result in exactly what you are seeing, a split-brain scenario with additional, mutual STONITH. If you want to test the fencing capabilities, a simple killall -9 corosync on the active node will do. Other ways are crm node fence or stonith_admin -F.
From the not quite complete description of your cluster (where is the output of crm configure show and cat /etc/corosync/corosync.conf?) it seems you are using the 10.10.10.xx addresses for messaging, i.e. Corosync/cluster communication. The 172.10.10.xx addresses are your regular/service network addresses and you would access a given node, for example using SSH, by its 172.10.10.xx address. DNS also seems to resolve a node hostname like node1 to 172.10.10.1.
You have STONITH configured to use SSH, which is not a very good idea in itself, but you are probably just testing. I haven't used it myself but I assume the SSH STONITH agent logs into the other node and issues a shutdown command, like ssh root@node2 "shutdown -h now" or something equivalent.
Now, what happens when you cut cluster communication between the nodes? The nodes no longer see each node as alive and well, because there is no more communication between them. Thus each node assumes it is the only survivor of some unfortunate event and tries to become (or remain) the active or primary node. This is the classic and dreaded split-brain scenario.
Part of this is to make sure the other, obviously and presumably failed node is down for good, which is where STONITH comes in. Keep in mind that both nodes are now playing the same game: trying to become (or stay) active and take over all cluster resources, as well as shooting the other node in the head.
You can probably guess what happens now. node1 does ssh root@node2 "shutdown -h now" and node2 does ssh root@node1 "shutdown -h now". This doesn't use the cluster communication network 10.10.10.xx but the service network 172.10.10.xx. Since both nodes are in fact alive and well, they have no problem issuing commands or receiving SSH connections, so both nodes shoot each other at the same time. This kills both nodes.
If you don't use STONITH then a split-brain could have even worse consequences, especially in case of DRBD, where you could end up with both nodes becoming Primary. Data corruption is likely to happen and the split-brain must be resolved manually.
I recommend reading the material on http://www.hastexo.com/resources/hints-and-kinks which is written and maintained by the guys who contributed (and still contribute) a large chunk of what we today call "the Linux HA stack".
TL;DR: If you are cutting cluster communication between your nodes in order to test your fencing setup, you are doing it wrong. Use killall -9 corosync, crm node fence or stonith_admin -F instead. Cutting cluster communication will only result in a split-brain scenario, which can and will lead to data corruption.
Red Hat Enterprise Linux 7.4 provides full support for the ability to configure a separate quorum device which acts as a third-party arbitration device for the cluster. Its primary use is to allow a cluster to sustain more node failures than standard quorum rules allow. A quorum device is recommended for clusters with an even number of nodes and highly recommended for two-node clusters.
Best Answer
The best thing to do is test what actually happens under different failure modes, to make sure there is no single failure that could cause both MySQL servers to try to become masters.
Test disabling the internet connection on one server. See what happens on both servers, and watch what happens when you bring it back up.
Do the same for any redundant connections. Then do the same with disabling ALL network connections at once.
One reason for not doing STONITH on a two node cluster is it is fairly easy to end up with both nodes trying to kill the other, and actually succeeding. You need to test your setup to make sure that they don't either both shutdown, or both keep running as masters and get your database out of sync.
One other thing I recommend, while you are testing it, before it goes into production: Intentionally break it. Do something that will cause mysql and drbd to get out of sync, and learn how to fix it. Write down what you needed to do to fix it. Because it's much better to know how to do that BEFORE you really need to.