AWS instance Stop and Terminate don’t execute systemd stop script

amazon-web-servicessystemdubuntu-18.04

I have a shutdown script that executes just fine after cli reboot or shutdown, but never executes when an instance is stopped or terminated from aws console.

[Unit]
Description=Gracefully shut down remnode to avoid database dirty flag
DefaultDependencies=no
After=poweroff.target shutdown.target reboot.target halt.target kexec.target
RequiresMountsFor=/data
Requires=network-online.target network.target data.mount

[Service]
Type=oneshot
ExecStop=/root/node_shutdown.sh
RemainAfterExit=yes
KillMode=none

[Install]
WantedBy=multi-user.target

What am I missing?

EDIT

Here is the journalctl log for my service. The first two entries are reboot via cli, the last 3 entries are 2 instance stops from AWS console. As you can see, there's not even a mention of winding down my service during instance stop. Rebooting from cli however outputs my echo and system logs startup as well as shutdown of the service.

-- Logs begin at Sun 2019-10-13 13:02:54 UTC, end at Mon 2019-10-14 19:41:01 UTC. --
Oct 13 13:03:24 ip-10-0-1-182 systemd[1]: Started Gracefully shut down remnode to avoid database dirty flag.
Oct 14 19:27:23 ip-10-0-1-182 systemd[1]: Stopping Gracefully shut down remnode to avoid database dirty flag...
Oct 14 19:27:23 ip-10-0-1-182 node_shutdown.sh[10635]: RUNNING SHUTDOWN SCRIPT
Oct 14 19:27:23 ip-10-0-1-182 systemd[1]: Stopped Gracefully shut down remnode to avoid database dirty flag.
-- Reboot --
Oct 14 19:27:37 ip-10-0-1-182 systemd[1]: Started Gracefully shut down remnode to avoid database dirty flag.
Oct 14 19:28:17 ip-10-0-1-182 systemd[1]: Stopping Gracefully shut down remnode to avoid database dirty flag...
Oct 14 19:28:17 ip-10-0-1-182 node_shutdown.sh[1712]: RUNNING SHUTDOWN SCRIPT
Oct 14 19:28:18 ip-10-0-1-182 systemd[1]: Stopped Gracefully shut down remnode to avoid database dirty flag.
-- Reboot --
Oct 14 19:28:32 ip-10-0-1-182 systemd[1]: Started Gracefully shut down remnode to avoid database dirty flag.
-- Reboot --
Oct 14 19:34:05 ip-10-0-1-182 systemd[1]: Started Gracefully shut down remnode to avoid database dirty flag.
-- Reboot --
Oct 14 19:40:26 ip-10-0-1-182 systemd[1]: Started Gracefully shut down remnode to avoid database dirty flag.

EDIT 2:

Because it seems like shutdown trigger is basically never pulled, I've also tried setting kernel sysctl kernel.poweroff_cmd to /bin/systemctl poweroff following this thread. Hoping it would trigger the even, but no luck.

Best Answer

Just a hunch, but might be worth exploring:

Have you tried using Before= with ExecStart= instead of After= with ExecStop=? i.e. start the node_shutdown.sh script before the final state of a shutdown/halt/reboot is reached. There are answers to similar questions here and here that allude to this.

That said, the accepted answer for the first linked question uses neither option! And why it works on a manual shutdown but not a console shutdown I'm not sure. I know a console shutdown registers as a "power button pressed" event - maybe this event is treated differently by the OS?

A power button press triggers an ACPI event which (I think) runs /etc/acpi/events/hibinit-power => /etc/acpi/actions/hibinit-power.sh, which itself sends a poweroff command to the message bus. It might be worth checking the contents of that .sh in case there's anything funky in there. My vanilla Ubuntu 18.04 looks like this:

#!/bin/sh

# shut down system in a way that respects inhibitors
# see: https://github.com/systemd/systemd/pull/9356
dbus-send --system --print-reply --dest=org.freedesktop.login1 /org/freedesktop/login1 org.freedesktop.login1.Manager.PowerOff boolean:false

As asktyagi suggests it's probably worth determining if the script is actually running - maybe try writing to a log at the top of the node_shutdown script?

Edit - here are the contents of the following files:

/etc/acpi/events/hibinit-power

event=button/power
action=/etc/acpi/actions/hibinit-power.sh "%e"

/etc/acpi/events/hibinit-sleep

# ACPID config to power down machine if powerbutton is pressed, but only if
# no gnome-power-manager is running

event=button/sleep.*
action=/etc/acpi/actions/sleep.sh %e

/etc/acpi/actions/sleep.sh

#!/bin/sh

#PATH=/sbin:/bin:/usr/bin

do_hibernate() {
    if [ -d /run/systemd/system ]; then
        systemctl hibernate
    else
        pm-hibernate
        swapoff /swap-hibinit
    fi
}


case "$2" in
    SBTN)
        swapon /swap-hibinit && do_hibernate
        ;;
    *)
        logger "ACPI action undefined: $2" ;;
esac

One thing to also check is /etc/systemd/logind.conf to see how the system handles the power key being pressed. Mine appears to have all the defaults set, notably the ones below:

[Login]
...
#HandlePowerKey=poweroff
#HandleSuspendKey=suspend
#HandleHibernateKey=hibernate
...
#PowerKeyIgnoreInhibited=no
#SuspendKeyIgnoreInhibited=no
#HibernateKeyIgnoreInhibited=no

I double checked this with a fresh Ubuntu instance built from the AWS stock image (ami-02df9ea15c1778c9c in EU-West-1)

Related Topic