Linux Graceful Shutdown

For what it is worth, I feel your pain. It seems that heartbeat considers the loss of the passive node the same as a takeover of the passive node, so it starts its services. When the start scripts failed, and there was no other node to which to failover, heartbeat stayed primary and shutdown all the services. The only way to get back up again is to re-start heartbeat when this happens.

We dealt with this problem by making a single script that starts all of the cluster services (IP, FS mount, ipvsadm, Apache, etc) only if they are not already running. We make sure the "all in one" init script only returns non-zero for actual startup failures (and not for warnings like "already running") to avoid problems like this.

If you use shutdown -h now, it will shut down immediatly regardless of "default settings" since you override those with the parameter.

That being said, last time we shut down our environment we did each VM seperately, waited until they all showed as powered off, and then shutdown the host (ESXi in this case). This allows you to check for errors during shutdown of the VM's, and in our case shutdown the webservers and THEN the database servers of a specific application to avoid any errors.

We do the same when starting up, and again make sure there's no errors.

I think it's important to figure out a proper shutdown and startup procedure in the case of production environments.

That being said, I think you will need to supply more information on what VM's they are. I think VMware Server won't be able to tell the difference between a shutdown with errors and one without.

P.S. why not use ESXi to begin with?