Linux Systemd – Configure Buggy Service to Terminate via SIGKILL

linuxservicesignalssystemd

Background

I've been asked to create a systemd script for a new service, foo_daemon, that sometimes gets into a "bad state", and won't die via SIGTERM (likely due to custom signal handler). This is problematic for developers, as they are instructed to start/stop/restart the service via:

systemctl start foo_daemon.service
systemctl stop foo_daemon.service
systemctl restart foo_daemon.service

Problem

Sometimes, due to foo_daemon getting into a bad state, we have to forcibly kill it via:

systemctl kill -s KILL foo_daemon.service

Question

How can I setup my systemd script for foo_daemon so that, whenever a user attempts to stop/restart the service, systemd will:

Attempt a graceful shutdown of foo_daemon via SIGTERM.
Give up to 2 seconds for shutdown/termination of foo_daemon to complete.
Attempt a forced shutdown of foo_daemon via SIGKILL if the process is still alive (so we don't have a risk of the PID being recycled and systemd issues SIGKILL against the wrong PID). The device we're testing spawns/forks numerous processes rapidly, so there is a rare but very real concern about PID recycling causing a problem.
If, in practise, I'm just being paranoid about PID recycling, I'm OK with the script just issuing SIGKILL against the process' PID without being concerned about killing a recycled PID.

Best Answer

systemd already supports this out of the box, and it is enabled by default.

The only thing you might want to customize is the timeout, which you can do with TimeoutStopSec=. For example:

[Service]
TimeoutStopSec=2

Now, systemd will send a SIGTERM, wait two seconds for the service to exit, and if it doesn't, it will send a SIGKILL.

If your service is not systemd-aware, you may need to provide the path to its PID file with PIDFile=.

Finally, you mentioned that your daemon spawns many processes. In this case, you might wish to set KillMode=control-group and systemd will send signals to all of the processes in the cgroup.

Related Solutions

Systemd – How to Set Environment Variable in Service

Times change and so do best practices.

The current best way to do this is to run systemctl edit myservice, which will create an override file for you or let you edit an existing one.

In normal installations this will create a directory /etc/systemd/system/myservice.service.d, and inside that directory create a file whose name ends in .conf (typically, override.conf), and in this file you can add to or override any part of the unit shipped by the distribution.

For instance, in a file /etc/systemd/system/myservice.service.d/myenv.conf:

[Service]
Environment="SECRET=pGNqduRFkB4K9C2vijOmUDa2kPtUhArN"
Environment="ANOTHER_SECRET=JP8YLOc2bsNlrGuD6LVTq7L36obpjzxd"

Also note that if the directory exists and is empty, your service will be disabled! If you don't intend to put something in the directory, ensure that it does not exist.

For reference, the old way was:

The recommended way to do this is to create a file /etc/sysconfig/myservice which contains your variables, and then load them with EnvironmentFile.

For complete details, see Fedora's documentation on how to write a systemd script.

Corosync systemd resource does not reflect service status

The pacemaker resource couldn't get the current status of the service because I missed the monitor operation.

op monitor interval=20

Unfortunately I did not find useful documentation on it nor the systemd resource agent implementation. I would appreciate any links.

The working primitive configuration from crm configure show is:

 primitive service_jira systemd:jira \
    op start interval=0 timeout=60 \
    op stop interval=0 timeout=90 \
    op monitor interval=10 \
    meta target-role=Started