Linux – get notification when systemd-monitored service enters failed state

bashlinuxservicesystemd

I need to have network messages sent when a systemd service I have crashes or is hung (i.e., enters failed state; I monitor for hung by using WatchdogSec=). I noticed that newer systemd have FailureAction=, but then saw that this doesn't allow arbitrary commands, but just rebooting/shutdown.

Specifically, I need a way to have one network message sent when systemd detects the program has crashed, and another when it detects it has hung.

I'm hoping for a better answer than "parse the logs", and I need something that has a near-instant response time, so I don't think a polling approach is good; it should be something triggered by the event occurring.

Best Answer

systemd units support OnFailure that will activate a unit (or more) when the unit goes to failed. You can put something like

 OnFailure=notify-failed@%n

And then create the notify-failed@.service service where you can use the required specifier (you probably will want at least %i) to launch the script or command that will send notification.

You can see a practical example in http://northernlightlabs.se/systemd.status.mail.on.unit.failure

Related Solutions

Systemd – How to Set Environment Variable in Service

Times change and so do best practices.

The current best way to do this is to run systemctl edit myservice, which will create an override file for you or let you edit an existing one.

In normal installations this will create a directory /etc/systemd/system/myservice.service.d, and inside that directory create a file whose name ends in .conf (typically, override.conf), and in this file you can add to or override any part of the unit shipped by the distribution.

For instance, in a file /etc/systemd/system/myservice.service.d/myenv.conf:

[Service]
Environment="SECRET=pGNqduRFkB4K9C2vijOmUDa2kPtUhArN"
Environment="ANOTHER_SECRET=JP8YLOc2bsNlrGuD6LVTq7L36obpjzxd"

Also note that if the directory exists and is empty, your service will be disabled! If you don't intend to put something in the directory, ensure that it does not exist.

For reference, the old way was:

The recommended way to do this is to create a file /etc/sysconfig/myservice which contains your variables, and then load them with EnvironmentFile.

For complete details, see Fedora's documentation on how to write a systemd script.

Find the location of a systemd unit file service

For units that are defined in actual, static files, this can be seen in systemctl status:

$ systemctl status halt-local.service
● halt-local.service - /usr/sbin/halt.local Compatibility
   Loaded: loaded (/lib/systemd/system/halt-local.service; static)
   Active: inactive (dead)

But there are units that are not defined by files, e.g. with systemd-cron installed. These have no useful location listed with status:

$ systemctl status cron-jojo-0.timer
● cron-jojo-0.timer - [Cron] "*/10 * * * * ..."
   Loaded: loaded (/var/spool/cron/crontabs/jojo)
   Active: active (waiting) since Mon 2015-05-18 14:53:01 UTC; 9min ago

In either case, though, the FragmentPath field is educating:

$ systemctl show -p FragmentPath cron-daily.service
FragmentPath=/lib/systemd/system/cron-daily.service
$ systemctl show -p FragmentPath cron-jojo-0.service
FragmentPath=/run/systemd/generator/cron-jojo-0.service
$ systemctl show -p FragmentPath halt-local.service
FragmentPath=/lib/systemd/system/halt-local.service

Best Answer

Related Solutions

Systemd – How to Set Environment Variable in Service

Find the location of a systemd unit file service

Related Topic