I figured it out, after a bit more trial and error. I was getting iSCSI errors going back up the stack too rapidly, as mentioned in this post on ServerFault.
In addition to changing the variables outlined in the post above, I also traced some network cables and discovered that node #2 was on a 100Mb link, while node #1 was on a Gig link, along with the SAN. After some careful shuffling, all the network connections are now running at Gig speeds.
Finally, I changed the MTU on the Linux interfaces to 9000 from 1500, which seems to also have sped things up a bit.
The final result is a working cluster with the domU booting even quicker than before on node #1.
Cheers,
Kendall
Your cluster architecture confuses me, as it seems you are running services that should be cluster-managed (like Varnish) standalone on two nodes at the same time and let the cluster resource manager (CRM) just juggle IP addresses around.
What is it you want to achieve with your cluster setup? Fault tolerance? Load balancing? Both? Mind you, I am talking about the cluster resources (Varnish, IP addresses, etc), not the backend servers to which Varnish distributes the load.
To me it sounds like you want an active-passive two-node cluster, which provides fault tolerance. One node is active and runs Varnish, the virtual IP addresses and possibly other resources, and the other node is passive and does nothing until the cluster resource manager moves resources over to the passive node, at which point it becomes active. This is a tried-and-true architecture that is as old as time itself. But for it to work you need to give the CRM full control over the resources. I recommend following Clusters from Scratch and modelling your cluster after that.
Edit after your updated question: your CIB looks good, and once you patched the Varnish init script so that repeated calls to "start" return 0 you should be able to add the following primitive (adjust the timeouts and intervals to your liking):
primitive p_varnish lsb:varnish \
op monitor interval="10s" timeout="15s" \
op start interval="0" timeout="10s" \
op stop interval="0" timeout="10s"
Don't forget to add it to the balancer group (the last element in the list):
group balancer eth0_gateway eth1_iceman_slider eth1_iceman_slider_ts \
eth1_iceman_slider_pm eth1_iceman_slider_jy eth1_iceman eth1_slider \
eth1_viper eth1_jester p_varnish
Edit 2: To decrease the migration threshold add a resource defaults section at the end of your CIB and set the migration-threshold property to a low number. Setting it to 1 means the resource will be migrated after a single failure. It is also a good idea to set resource stickiness so that a resource that has been migrated because of node failure (reboot or shutdown) does not automatically get migrated back later when the node is available again.
rsc_defaults $id="rsc-options" \
resource-stickiness="100" \
migration-threshold="1"
Best Answer
Definitely. Create a configuration file (named 'cib.txt', in our example) with the same syntax you've used in your example commands:
Then you can load that file using the following CRM shell command:
or completely replace the configuration:
NOTE: You can export the configuration from a cluster, for use on a new cluster or for backup purposes, with the following command:
WARN: Be sure to cut out anything specific to the original cluster if you intend to load it elsewhere (node id's, dc-version, last-lrm-refresh, etc).