How to make active/passive jboss resource in pacemaker

corosyncfailoverhigh-availabilityjbosspacemaker

I want to configure my pacemaker for jboss server. I have 2 nodes with mysql master/slave configured. The problem is I want to start jboss on node which is located master database. The one possible solution for this: Start jboss on 2 nodes, but deploy application only in one node where master is located. How can I solve this?

My current configuration is:

node oxygen \
        attributes standby="off" p_mysql_mysql_master_IP="10.3.1.4"
node oxygen2 \
        attributes standby="off" p_mysql_mysql_master_IP="10.3.1.5"
primitive VIRT_IP ocf:heartbeat:IPaddr2 \
        params ip="192.168.1.250" nic="eth1" \
        meta target-role="Started"
primitive p_mysql ocf:percona:mysql \
        params config="/etc/mysql/my.cnf" pid="/var/run/mysqld/mysqld.pid" socket="/var/run/mysqld/mysqld.sock" replication_user="repl_user" replication_passwd="ola5P1ZMU" max_slave_lag="60" evict_outdated_slaves="false" binary="/usr/bin/mysqld_safe" test_user="test_user" test_passwd="2JcXCxKF" \
        op monitor interval="5s" role="Master" OCF_CHECK_LEVEL="1" \
        op monitor interval="2s" role="Slave" OCF_CHECK_LEVEL="1" \
        op start interval="0" timeout="60s" \
        op stop interval="0" timeout="60s"
ms ms_MySQL p_mysql \
        meta master-max="1" master-node-max="1" clone-max="2" clone-node-max="1" notify="true" globally-unique="false" target-role="Started" is-managed="true"
property $id="cib-bootstrap-options" \
        dc-version="1.1.7-ee0730e13d124c3d58f00016c3376a1de5323cff" \
        cluster-infrastructure="openais" \
        expected-quorum-votes="2" \
        stonith-enabled="false" \
        no-quorum-policy="ignore" \
        stop-all-resources="false" \
        maintenance-mode="false"
property $id="mysql_replication" \
        p_mysql_REPL_INFO="10.3.1.4|mysql-bin.000030|107"

Best Answer

I have do similar configuration to make sure the virtual IP to live along with Mysql master server. For your case, I think the steps should be:

Add a primitive for two JBOSS intances (as your shareIP or MySQL servers)
Add colocation configure for JBOSS primitive to live along with MySQL Master as example below: colocation mysql_co_jboss inf: jboss ms_MySQL:Master

Related Solutions

How to setup STONITH in a 2-node active/passive linux HA pacemaker cluster

This is a slightly older question but the problem presented here is based on a misconception on how and when failover in clusters, especially two-node clusters, works.

The gist is: You can not do failover testing by disabling communication between the two nodes. Doing so will result in exactly what you are seeing, a split-brain scenario with additional, mutual STONITH. If you want to test the fencing capabilities, a simple killall -9 corosync on the active node will do. Other ways are crm node fence or stonith_admin -F.

From the not quite complete description of your cluster (where is the output of crm configure show and cat /etc/corosync/corosync.conf?) it seems you are using the 10.10.10.xx addresses for messaging, i.e. Corosync/cluster communication. The 172.10.10.xx addresses are your regular/service network addresses and you would access a given node, for example using SSH, by its 172.10.10.xx address. DNS also seems to resolve a node hostname like node1 to 172.10.10.1.

You have STONITH configured to use SSH, which is not a very good idea in itself, but you are probably just testing. I haven't used it myself but I assume the SSH STONITH agent logs into the other node and issues a shutdown command, like ssh root@node2 "shutdown -h now" or something equivalent.

Now, what happens when you cut cluster communication between the nodes? The nodes no longer see each node as alive and well, because there is no more communication between them. Thus each node assumes it is the only survivor of some unfortunate event and tries to become (or remain) the active or primary node. This is the classic and dreaded split-brain scenario.

Part of this is to make sure the other, obviously and presumably failed node is down for good, which is where STONITH comes in. Keep in mind that both nodes are now playing the same game: trying to become (or stay) active and take over all cluster resources, as well as shooting the other node in the head.

You can probably guess what happens now. node1 does ssh root@node2 "shutdown -h now" and node2 does ssh root@node1 "shutdown -h now". This doesn't use the cluster communication network 10.10.10.xx but the service network 172.10.10.xx. Since both nodes are in fact alive and well, they have no problem issuing commands or receiving SSH connections, so both nodes shoot each other at the same time. This kills both nodes.

If you don't use STONITH then a split-brain could have even worse consequences, especially in case of DRBD, where you could end up with both nodes becoming Primary. Data corruption is likely to happen and the split-brain must be resolved manually.

I recommend reading the material on http://www.hastexo.com/resources/hints-and-kinks which is written and maintained by the guys who contributed (and still contribute) a large chunk of what we today call "the Linux HA stack".

TL;DR: If you are cutting cluster communication between your nodes in order to test your fencing setup, you are doing it wrong. Use killall -9 corosync, crm node fence or stonith_admin -F instead. Cutting cluster communication will only result in a split-brain scenario, which can and will lead to data corruption.

Pacemaker/corosync timeout before resource transfers

It sounds like migration-threshold is what you're looking for.

Assumming that you have a resource which is setting op monitor interval to 20s:

primitive p_foo ocf:heartbeat:foo \
    params ...
    op monitor interval="20s"

and you want it doesn't migrate to a new node within 1 minute, so you should set migration-threshold to 3:

primitive p_foo ocf:heartbeat:foo \
    params ...
    op monitor interval="20s"
    meta target-role="Started" migration-threshold="3"

Best Answer

Related Solutions

How to setup STONITH in a 2-node active/passive linux HA pacemaker cluster

Pacemaker/corosync timeout before resource transfers

Related Topic