How to detect APC UPS battery usage and run a script when on battery

Depending on the precise model of UPS, your best bet is to wire the EPO port on the back of those UPSes to the EPO switch on the fire panel.

Smart-UPS 2200 manual, see page 6.

Since these UPSes actually have an actual Emergency Power Off function built in, in order to make your fire-inspectors happy you need to wire those ports to the EPO function of the fire-panel itself. I'm not an electrician, so I don't know the specifics of how you'd wire 10's of these to a single EPO port on the fire panel, but that's a question for your fire panel system vendor.

Half-assing it with suicide software monitors on servers connected to each UPS is the kind of close-enough that gets fire-inspectors to shake their head in that special way that means you'll have to do it all over again, but right this time.

Currently when one of the two UPS dies, the doshutdown event is triggered, and executes the default script via apccontrol. The doshutdown script ignores the second UPS, as they are not event-connected, and proceed normally with the shutdown.

In order to have the doshutdown events somewhat connected, the two instances of apcupsd need a specifically customized configuration file. The difference will reside in the directory from which the events scripts have to be executed.

Main properties of first ups, in /etc/apcupsd/apcupsd.ups0.conf

SCRIPTDIR /etc/apcupsd/ups0
UPSNAME ups0
DEVICE /dev/ups0
PWRFAILDIR /etc/apcupsd/ups0
NOLOGINDIR /etc/apcupsd/ups0
NISPORT 3551
EVENTSFILE /var/log/apcupsd.0.events

And for the ups1, in /etc/apcupsd/apcupsd.ups1.conf

SCRIPTDIR /etc/apcupsd/ups1
UPSNAME ups1
DEVICE /dev/ups1
PWRFAILDIR /etc/apcupsd/ups1
NOLOGINDIR /etc/apcupsd/ups1
NISPORT 3552
EVENTSFILE /var/log/apcupsd.1.events

Each scriptdir should get a copy of the default scripts.
We want to customize the doshutdown script, which will not directly shutdown the machine, but has to check if the other UPS is still on, or is in shutdown mode.

At the top of the doshutdown script, we could add something like

for ups0

if [ ! -f /tmp/ups1.is.down ]
then
  touch /tmp/ups0.is.down
  exit 99
fi

for ups1

if [ ! -f /tmp/ups0.is.down ]
then
  touch /tmp/ups1.is.down
  exit 99
fi

the status 99 has a special meaning, that tells apccontrol to stop the action in progress. The five lines check if the other UPS-down-file has been created ; if no, the down-file is created for the ups being down, and exits. If yes, meaning the other UPS is down, this one is getting down as well, thus the script should continue and shutdown the machine.

The files /tmp/usp[01].is.down indicate if the ups[01] is currently down.

Important: the init.d start script of apcupsd should remove these files, if they have been created in a previous session:

rm -f /tmp/usp[01].is.down

Finally, the directories created above, /etc/apcupsd/ups[01] should be given access to the apcupsd user (or to whichever relevant user running the instances).

chown -R apcupsd /etc/apcupsd/ups[01]

Please have a look at the detailed documentation.

edit fixed the /tmp/ups[01].is.down names, the .is was missing.