Systemd-Run – Running systemd-run –user Inside a Systemd Service

dbussystemdsystemd-service

I have a web app that handles user input, and as part of that, runs some commands with systemd-run --user --scope ... to limit memory and CPU usage.

The application runs fine when run normally, but when run as a systemd service, I get:

Failed to connect to bus: No medium found

What do I need to do in the unit for the service to enable this?

Best Answer

--user contacts the per-user instance of systemd for that UID, which is only started when there is an actual login session for that UID. When there aren't any, you need to loginctl enable-linger <user> to have the service manager be always running for that user.

The webapp needs to have the correct XDG_RUNTIME_DIR= in its environment so that it could find the bus socket; it's always at the same location /run/user/<uid>. (The directory will be created by systemd when linger is enabled.)

You will probably need to use --service instead of --scope. The latter won't be able to migrate your process between cgroups due to lack of "common ancestor" permissions, so it becomes necessary to have systemd itself start the process.

(In a user session, your own per-UID user-1234.slice cgroup is the common ancestor of both your interactive terminal cgroup and the new systemd scope cgroup, but when trying to do this from a service, the only common ancestor would be -.slice which you do not have permissions to.)

Configuring the webapp .service with a PAMName= so that the webapp actually runs in a login session would avoid some of the above problems, but I also have the feeling it'll cause new problems, so I'm not suggesting it as the first option.

Related Solutions

Systemd – How to Set Environment Variable in Service

Times change and so do best practices.

The current best way to do this is to run systemctl edit myservice, which will create an override file for you or let you edit an existing one.

In normal installations this will create a directory /etc/systemd/system/myservice.service.d, and inside that directory create a file whose name ends in .conf (typically, override.conf), and in this file you can add to or override any part of the unit shipped by the distribution.

For instance, in a file /etc/systemd/system/myservice.service.d/myenv.conf:

[Service]
Environment="SECRET=pGNqduRFkB4K9C2vijOmUDa2kPtUhArN"
Environment="ANOTHER_SECRET=JP8YLOc2bsNlrGuD6LVTq7L36obpjzxd"

Also note that if the directory exists and is empty, your service will be disabled! If you don't intend to put something in the directory, ensure that it does not exist.

For reference, the old way was:

The recommended way to do this is to create a file /etc/sysconfig/myservice which contains your variables, and then load them with EnvironmentFile.

For complete details, see Fedora's documentation on how to write a systemd script.

Linux – Use of CPUQuota in systemd

The Good

Your solution is the correct and should actually be quite future-proof; by using systemd to control the services cgroup settings, eg. CPUQota.

[Unit]
Description=Virtual Distributed Ethernet

[Service]
ExecStart=/usr/bin/ddcommand
CPUQuota=10%

[Install]
WantedBy=multi-user.target

See the man systemd.resource-control for more useful cgroup settings in systemd.

The Bad

There are two caveats to this though, which I (and possibly a few others) stumpled upon. Those caveats are really difficult to track down as there does not seem to be much easily findable information about this, which is the main reason for this answer.

Caveat 1:

The CPUQuota setting is only available since systemd 213, see https://github.com/systemd/systemd/blob/master/NEWS

    * The CFS CPU quota cgroup attribute is now exposed for
      services. The new CPUQuota= switch has been added for this
      which takes a percentage value. Setting this will have the
      result that a service may never get more CPU time than the
      specified percentage, even if the machine is otherwise idle.

This is for example an issue with Debian Jessie which only comes with systemd 208. As an alternative one could configure cpu.cfs_period_us and cpu.cfs_quota_us manually using cgcreate and cgset from the cgroup-bin package, eg.

sudo cgcreate -g cpu:/cpulimited
sudo cgset -r cpu.cfs_period_us=50000 cpulimited
sudo cgset -r cpu.cfs_quota_us=10000 cpulimited
sudo cgexec -g cpu:cpulimited /usr/bin/ddcommand

Caveat 2

For the settings cpu.cfs_period_us and cpu.cfs_quota_us to be available the Kernel needs to be compiled with config-flag CONFIG_CFS_BANDWIDTH. Sadly the 3.16.x Kernel for Debian Jessie is not compiled with this flag by default, see this feature request.

This will be available in Debian Stretch though. One could also use the kernel from jessie-backports, which should have the flag enabled.

I hope this answer helps a few people with the same issue as me...

PS: An easy way to test wether CPUquota is working in your environment is:

$ apt-get install stress
$ systemd-run -p CPUQuota=25% --slice=stress -- stress -c <your cpu count>

and watch with top or htop, the load should be spread (evenly) accross all cpus/cores, summing up to 25%.

Alternative

As an alternative tool one could use cpu-limit which should be available in most distros, eg.

$ apt-get install cpulimit
$ cpulimit -l 10 -P /usr/bin/ddcommand

It works by sending SIGSTOP and SIGCONT to the attached command to pause and resume its operation.

AFAIK it was difficult to control multiple separate/stand-alone processes simultaneously with this, in like group them together, but there might also be solution for this...