Change the UPS battery while it’s powered on

You should consider turning off the circuit breaker to the outlet running the rack in lieu of unplugging the cord from the wall. The UPS is losing its electrical ground when you unplug it from the wall. While it's unlikely that anything would go wrong, the UPS designers "expect" that path to ground to remain available at all times, and if something did short during your test you might see sparks (smoke, flame, etc) when the electricity takes another path to ground. I've unplugged UPSs from the wall for testing before, but seeing a flash of "lightning" and hearing a loud "bang" coming out of a UPS during one such test gave me "religion" about not doing that again. After talking to an electrician friend I decided that, from then on, I'd do UPS tests that didn't interrupt the ground to the UPS.

BTW: The PowerChute Network Shutdown software from APC is garbage. You might have a look at apcupsd. It runs under a variety of operating systems (Windows included) and is much easier to configure (and to replicate the configuration on multiple servers via copying files) than the APC alternative.

Currently when one of the two UPS dies, the doshutdown event is triggered, and executes the default script via apccontrol. The doshutdown script ignores the second UPS, as they are not event-connected, and proceed normally with the shutdown.

In order to have the doshutdown events somewhat connected, the two instances of apcupsd need a specifically customized configuration file. The difference will reside in the directory from which the events scripts have to be executed.

Main properties of first ups, in /etc/apcupsd/apcupsd.ups0.conf

SCRIPTDIR /etc/apcupsd/ups0
UPSNAME ups0
DEVICE /dev/ups0
PWRFAILDIR /etc/apcupsd/ups0
NOLOGINDIR /etc/apcupsd/ups0
NISPORT 3551
EVENTSFILE /var/log/apcupsd.0.events

And for the ups1, in /etc/apcupsd/apcupsd.ups1.conf

SCRIPTDIR /etc/apcupsd/ups1
UPSNAME ups1
DEVICE /dev/ups1
PWRFAILDIR /etc/apcupsd/ups1
NOLOGINDIR /etc/apcupsd/ups1
NISPORT 3552
EVENTSFILE /var/log/apcupsd.1.events

Each scriptdir should get a copy of the default scripts.
We want to customize the doshutdown script, which will not directly shutdown the machine, but has to check if the other UPS is still on, or is in shutdown mode.

At the top of the doshutdown script, we could add something like

for ups0

if [ ! -f /tmp/ups1.is.down ]
then
  touch /tmp/ups0.is.down
  exit 99
fi

for ups1

if [ ! -f /tmp/ups0.is.down ]
then
  touch /tmp/ups1.is.down
  exit 99
fi

the status 99 has a special meaning, that tells apccontrol to stop the action in progress. The five lines check if the other UPS-down-file has been created ; if no, the down-file is created for the ups being down, and exits. If yes, meaning the other UPS is down, this one is getting down as well, thus the script should continue and shutdown the machine.

The files /tmp/usp[01].is.down indicate if the ups[01] is currently down.

Important: the init.d start script of apcupsd should remove these files, if they have been created in a previous session:

rm -f /tmp/usp[01].is.down

Finally, the directories created above, /etc/apcupsd/ups[01] should be given access to the apcupsd user (or to whichever relevant user running the instances).

chown -R apcupsd /etc/apcupsd/ups[01]

Please have a look at the detailed documentation.

edit fixed the /tmp/ups[01].is.down names, the .is was missing.