Linux – How to run a systemd service after network link has been established

linuxsystemdsystemd-service

A while ago I set up a systemd service to run after network comes online per the documentation.
This is the unit file:

[Unit]
Description=Refresh Pacman mirrorlist with Reflector.
Documentation=https://wiki.archlinux.org/index.php/Reflector
Wants=network-online.target
After=network-online.target nss-lookup.target

[Service]
Type=oneshot
ExecStart=/usr/bin/reflector @/etc/xdg/reflector/reflector.conf
CacheDirectory=reflector
CapabilityBoundingSet=~CAP_SETUID CAP_SETGID CAP_SETPCAP CAP_SYS_ADMIN CAP_SYS_PTRACE CAP_CHOWN CAP_FSETID CAP_SETFCAP CAP_DAC_OVERRIDE CAP_DAC_READ_SEARCH CAP_FOWNER CAP_IPC_OWNER CAP_NET_ADMIN CAP_SYS_TIME CAP_AUDIT_CONTROL CAP_AUDIT_READ CAP_AUDIT_WRITE CAP_KILL CAP_NET_BIND_SERVICE CAP_NET_BROADCAST CAP_NET_RAW CAP_SYS_NICE CAP_SYS_RESOURCE CAP_MAC_ADMIN CAP_MAC_OVERRIDE CAP_SYS_BOOT CAP_LINUX_IMMUTABLE CAP_IPC_LOCK CAP_SYS_CHROOT CAP_BLOCK_SUSPEND CAP_LEASE CAP_SYS_PACCT CAP_SYS_TTY_CONFIG CAP_WAKE_ALARM
Environment=XDG_CACHE_HOME=/var/cache/reflector
LockPersonality=true
MemoryDenyWriteExecute=true
NoNewPrivileges=true
PrivateDevices=true
PrivateTmp=true
PrivateUsers=true
ProtectClock=true
ProtectControlGroups=true
ProtectHome=true
ProtectHostname=true
ProtectKernelTunables=true
ProtectKernelLogs=true
ProtectKernelModules=true
ProtectSystem=strict
ReadOnlyPaths=/etc/xdg/reflector/reflector.conf
ReadWritePaths=/etc/pacman.d/mirrorlist
RemoveIPC=true
RestrictAddressFamilies=~AF_AX25 AF_IPX AF_APPLETALK AF_X25 AF_DECnet AF_KEY AF_NETLINK AF_PACKET AF_RDS AF_PPPOX AF_LLC AF_IB AF_MPLS AF_CAN AF_TIPC AF_BLUETOOTH AF_ALG AF_VSOCK AF_KCM AF_UNIX AF_XDP
RestrictNamespaces=true
RestrictRealtime=true
RestrictSUIDSGID=true
SystemCallArchitectures=native
SystemCallFilter=@system-service
SystemCallFilter=~@resources @privileged
UMask=177

[Install]
WantedBy=multi-user.target

Per the documentation, the service should already be set up correctly to wait for network connectivity to be established (i.e.: I'm logged in, NetworkManager has connected to WiFi and the system has been assigned an IP).

network-online.target is a target that actively waits until the nework is "up", where the definition of "up" is defined by the network management software. Usually it indicates a configured, routable IP address of some kind. Its primary purpose is to actively delay activation of services until the network is set up.

I've also made sure that NetworkManager-wait-online.service is enabled.

$ systemctl is-enabled NetworkManager-wait-online.service systemd-networkd-wait-online.service
enabled
disabled

Instead of the service waiting to start until an internet connection has been established, I see an error on the system console almost as soon as I see my login prompt.

Jul 05 07:40:02 gadget systemd[1]: Starting Refresh Pacman mirrorlist with Reflector....
Jul 05 07:40:03 gadget reflector[564]: error: failed to retrieve mirrorstatus data: URLError: <urlopen error [Errno -3] Temporary failure in name resolution>
Jul 05 07:40:03 gadget systemd[1]: reflector.service: Main process exited, code=exited, status=1/FAILURE
Jul 05 07:40:03 gadget systemd[1]: reflector.service: Failed with result 'exit-code'.
Jul 05 07:40:03 gadget systemd[1]: Failed to start Refresh Pacman mirrorlist with Reflector..

Since it should wait for network to be online and DNS to be available, I also checked the status of NetworkManager-wait-online.service, network-online.target and nss-lookup.target:

$ systemctl status NetworkManager-wait-online.service
● NetworkManager-wait-online.service - Network Manager Wait Online
     Loaded: loaded (/usr/lib/systemd/system/NetworkManager-wait-online.service; enabled; vendor preset: disabled)
     Active: active (exited) since Mon 2021-07-05 07:40:02 CEST; 44min ago
       Docs: man:nm-online(1)
   Main PID: 544 (code=exited, status=0/SUCCESS)
      Tasks: 0 (limit: 38321)
     Memory: 0B
        CPU: 0
     CGroup: /system.slice/NetworkManager-wait-online.service

Jul 05 07:40:02 gadget systemd[1]: Starting Network Manager Wait Online...
Jul 05 07:40:02 gadget systemd[1]: Finished Network Manager Wait Online.
$ systemctl status network-online.target
● network-online.target - Network is Online
     Loaded: loaded (/usr/lib/systemd/system/network-online.target; static)
     Active: active since Mon 2021-07-05 07:40:02 CEST; 1h 5min ago
       Docs: man:systemd.special(7)
             https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget

Jul 05 07:40:02 gadget systemd[1]: Reached target Network is Online.
$ systemctl status nss-lookup.target
○ nss-lookup.target - Host and Network Name Lookups
     Loaded: loaded (/usr/lib/systemd/system/nss-lookup.target; static)
     Active: inactive (dead)
       Docs: man:systemd.special(7)

It strikes me as odd that NetworkManager-wait-online.service reports finished right after starting. Is there something else I need to configure? Is this a bug in NetworkManager and/or nm-online?

I've looked for similar topics and found these questions, but they don't answer my issue:

Best Answer

The meaning of "network online" is nebulous and network-online.target only fufills the most basic of possible meanings of "network online". See https://freedesktop.org/wiki/Software/systemd/NetworkTarget for a more complete explanation.

You need to determine precisely what you need to be up in the network and what services need to be working for your service to function correctly and either add those as dependencies, or add tests in your service startup sequence to wait for them.

As the page above states at the end, best would be to adjust your service to be able to handle dynamic changes to network configuration rather than failing if it isn't totally as expected.

Related Topic