I have a shutdown script that executes just fine after cli reboot
or shutdown
, but never executes when an instance is stopped or terminated from aws console.
[Unit]
Description=Gracefully shut down remnode to avoid database dirty flag
DefaultDependencies=no
After=poweroff.target shutdown.target reboot.target halt.target kexec.target
RequiresMountsFor=/data
Requires=network-online.target network.target data.mount
[Service]
Type=oneshot
ExecStop=/root/node_shutdown.sh
RemainAfterExit=yes
KillMode=none
[Install]
WantedBy=multi-user.target
What am I missing?
EDIT
Here is the journalctl
log for my service. The first two entries are reboot
via cli, the last 3 entries are 2 instance stops from AWS console. As you can see, there's not even a mention of winding down my service during instance stop. Rebooting from cli however outputs my echo and system logs startup as well as shutdown of the service.
-- Logs begin at Sun 2019-10-13 13:02:54 UTC, end at Mon 2019-10-14 19:41:01 UTC. --
Oct 13 13:03:24 ip-10-0-1-182 systemd[1]: Started Gracefully shut down remnode to avoid database dirty flag.
Oct 14 19:27:23 ip-10-0-1-182 systemd[1]: Stopping Gracefully shut down remnode to avoid database dirty flag...
Oct 14 19:27:23 ip-10-0-1-182 node_shutdown.sh[10635]: RUNNING SHUTDOWN SCRIPT
Oct 14 19:27:23 ip-10-0-1-182 systemd[1]: Stopped Gracefully shut down remnode to avoid database dirty flag.
-- Reboot --
Oct 14 19:27:37 ip-10-0-1-182 systemd[1]: Started Gracefully shut down remnode to avoid database dirty flag.
Oct 14 19:28:17 ip-10-0-1-182 systemd[1]: Stopping Gracefully shut down remnode to avoid database dirty flag...
Oct 14 19:28:17 ip-10-0-1-182 node_shutdown.sh[1712]: RUNNING SHUTDOWN SCRIPT
Oct 14 19:28:18 ip-10-0-1-182 systemd[1]: Stopped Gracefully shut down remnode to avoid database dirty flag.
-- Reboot --
Oct 14 19:28:32 ip-10-0-1-182 systemd[1]: Started Gracefully shut down remnode to avoid database dirty flag.
-- Reboot --
Oct 14 19:34:05 ip-10-0-1-182 systemd[1]: Started Gracefully shut down remnode to avoid database dirty flag.
-- Reboot --
Oct 14 19:40:26 ip-10-0-1-182 systemd[1]: Started Gracefully shut down remnode to avoid database dirty flag.
EDIT 2:
Because it seems like shutdown
trigger is basically never pulled, I've also tried setting kernel sysctl kernel.poweroff_cmd
to /bin/systemctl poweroff
following this thread. Hoping it would trigger the even, but no luck.
Best Answer
Just a hunch, but might be worth exploring:
Have you tried using
Before=
withExecStart=
instead ofAfter=
withExecStop=
? i.e. start the node_shutdown.sh script before the final state of a shutdown/halt/reboot is reached. There are answers to similar questions here and here that allude to this.That said, the accepted answer for the first linked question uses neither option! And why it works on a manual shutdown but not a console shutdown I'm not sure. I know a console shutdown registers as a "power button pressed" event - maybe this event is treated differently by the OS?
A power button press triggers an ACPI event which (I think) runs
/etc/acpi/events/hibinit-power
=>/etc/acpi/actions/hibinit-power.sh
, which itself sends a poweroff command to the message bus. It might be worth checking the contents of that .sh in case there's anything funky in there. My vanilla Ubuntu 18.04 looks like this:As asktyagi suggests it's probably worth determining if the script is actually running - maybe try writing to a log at the top of the node_shutdown script?
Edit - here are the contents of the following files:
/etc/acpi/events/hibinit-power
/etc/acpi/events/hibinit-sleep
/etc/acpi/actions/sleep.sh
One thing to also check is
/etc/systemd/logind.conf
to see how the system handles the power key being pressed. Mine appears to have all the defaults set, notably the ones below:I double checked this with a fresh Ubuntu instance built from the AWS stock image (ami-02df9ea15c1778c9c in EU-West-1)