Systemd – Troubleshooting ‘systemctl list-units’ for Type Target

systemctlsystemd

Summary: 'systemctl list-units –type=target –all' does not always list targets that I've added

Detail:

I have a project that consists of several related daemons; let's call this project 'Foo.' The daemons are managed by systemd service units, and the services are grouped together for easier management in a target. I have 'production' and 'testing' deployments of projcet Foo. So, I've created foo_testing.target and foo_production.target under /etc/systemd/system. In fact, I have this arrangement on a variety of physical hosts and VMs — some have both production and testing, some have only one or the other.
I therefore need to be able to interrogate a given host as to what is installed and what is running.

Having reviewed the docs for systemd and systemctl I hit upon

systemctl list-units --type=target --all

This works fine for targets that are running. If production and/or testing is running it will reliably show up in the output of list-units.

The problem is stopped targets. If I 'systemctl stop foo_testing.target' and immediately ask for the list of units, then foo_testing.target will be in the list (marked inactive). But, if I leave the testing deployment stopped for a day or two, and then come back and do 'systemctl list-units –type=target –all' then foo_testing won't be in the list at all.

So, why does systemd 'forget' about my targets if they haven't been active for a while?

Here's the system info for the most problematic host:

$ lsb_release -a
Distributor ID: Ubuntu
Description:    Ubuntu 21.04
Release:    21.04
Codename:   hirsute
$ uname -a
Linux <redacted> 5.11.0-49-generic #55-Ubuntu SMP Wed Jan 12 17:36:34 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
$systemctl --version
systemd 247 (247.3-3ubuntu3.7)
+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +ZSTD +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=hybrid

[Edit]
As requested, here's the output of 'systemctl cat'

$ systemctl cat "foo*.target"
# /etc/systemd/system/foo_testing.target
[Unit]
Description=Foo Testing Environment
AllowIsolate=no
Wants=serv1_testing.service serv2_testing.service serv3_testing.service

# /etc/systemd/system/foo_production.target
[Unit]
Description=Foo Production Environment
AllowIsolate=no
Wants=serv1_prod.service serv2_prod.service serv3_prod.service

Note, though, that this output does not change. It is the behavior of 'systemctl list-units' that changes.

[Edit 2] Q: Any non-target systemd units with matching base name installed on the system?

A: such units do exist — owing to my naming scheme. Is this potentially part of the explanation?

Best Answer

list-units cannot be expected to report consistently on inactive units:

list-units [PATTERN…]

List units that systemd currently has in memory. This includes units that are either referenced directly or through a dependency, units that are pinned by applications programmatically, or units that were active in the past and have failed.

from man systemctl, emphasis mine.

The command seems more geared towards letting a human see the unit state at a glance, and must be expected to forget about units no longer needed, e.g. after some unspecified time, or on daemon reload or reexec.

If in a script you need the running status of a unit, use systemctl is-active (for state is-enabled, for file existence: list-unit-files, ..) which produce consistent results, no matter the in-memory status.

Related Topic