Systemd does not restart service, although Restart=always

fault-tolerancesystemd

Here is my unit file of a systemd service:

[Unit]
Description=Tunnel For %i
After=network.target

[Service]
User=autossh
ExecStart=/usr/bin/autossh -M 0 -N -o "ExitOnForwardFailure yes" -o "ConnectTimeout=1" -o "ServerAliveInterval 60" -o "ServerAliveCountMax 3" -R 40443:installserver:40443 -R 8080:installserver:8080 tunnel@%i
Restart=always

[Install]
WantedBy=multi-user.target

The unit failed 15 days ago and systemd did not restart it, although "Restart=always" is in above unit file.

Here the status output of this service:

salt:/srv # systemctl status autossh@eins-work
autossh@eins-work.service - Tunnel For eins-work
      Loaded: loaded (/etc/systemd/system/autossh@.service; enabled)
      Active: failed (Result: start-limit) since Wed, 2016-02-10 14:33:34 CET; 2 weeks and 1 days ago
    Main PID: 17980 (code=exited, status=0/SUCCESS)
      CGroup: name=systemd:/system/autossh@.service/eins-work

Feb 10 14:33:34 salt systemd[1]: Stopping Tunnel For eins-work...
Feb 10 14:33:34 salt systemd[1]: Starting Tunnel For eins-work...
Feb 10 14:33:34 salt systemd[1]: Failed to start Tunnel For eins-work.
Feb 10 14:33:34 salt systemd[1]: Unit autossh@eins-work.service entered failed state

How to configure a systemd service to always restart if something fails?

Best Answer

The service failed to start too many times in a row, and so systemd gave up trying. This is what Result: start-limit means.

Note that units which are configured for Restart= and which reach the start limit are not attempted to be restarted anymore; however, they may still be restarted manually at a later point, from which point on, the restart logic is again activated. Note that systemctl reset-failed will cause the restart rate counter for a service to be flushed, which is useful if the administrator wants to manually start a unit and the start limit interferes with that.

You should resolve the problem that is causing the service to fail to start.

If you need to tune how many times and how quickly systemd attempts to restart a failing service, see the StartLimitInterval=, StartLimitBurst= and StartLimitAction= options.

Related Solutions

How to set environment variable in systemd service

Times change and so do best practices.

The current best way to do this is to run systemctl edit myservice, which will create an override file for you or let you edit an existing one.

In normal installations this will create a directory /etc/systemd/system/myservice.service.d, and inside that directory create a file whose name ends in .conf (typically, override.conf), and in this file you can add to or override any part of the unit shipped by the distribution.

For instance, in a file /etc/systemd/system/myservice.service.d/myenv.conf:

[Service]
Environment="SECRET=pGNqduRFkB4K9C2vijOmUDa2kPtUhArN"
Environment="ANOTHER_SECRET=JP8YLOc2bsNlrGuD6LVTq7L36obpjzxd"

Also note that if the directory exists and is empty, your service will be disabled! If you don't intend to put something in the directory, ensure that it does not exist.

For reference, the old way was:

The recommended way to do this is to create a file /etc/sysconfig/myservice which contains your variables, and then load them with EnvironmentFile.

For complete details, see Fedora's documentation on how to write a systemd script.

Find the location of a systemd unit file service

For units that are defined in actual, static files, this can be seen in systemctl status:

$ systemctl status halt-local.service
● halt-local.service - /usr/sbin/halt.local Compatibility
   Loaded: loaded (/lib/systemd/system/halt-local.service; static)
   Active: inactive (dead)

But there are units that are not defined by files, e.g. with systemd-cron installed. These have no useful location listed with status:

$ systemctl status cron-jojo-0.timer
● cron-jojo-0.timer - [Cron] "*/10 * * * * ..."
   Loaded: loaded (/var/spool/cron/crontabs/jojo)
   Active: active (waiting) since Mon 2015-05-18 14:53:01 UTC; 9min ago

In either case, though, the FragmentPath field is educating:

$ systemctl show -p FragmentPath cron-daily.service
FragmentPath=/lib/systemd/system/cron-daily.service
$ systemctl show -p FragmentPath cron-jojo-0.service
FragmentPath=/run/systemd/generator/cron-jojo-0.service
$ systemctl show -p FragmentPath halt-local.service
FragmentPath=/lib/systemd/system/halt-local.service

Best Answer

Related Solutions

How to set environment variable in systemd service

Find the location of a systemd unit file service

Related Topic