Docker – ExecStartPost in a systemd unit file does not start docker service

dockerfirewalldsystemd

I am trying to establish a connection between firewalld and docker services. I extended the default firewalld.service unit file by adding the following:

[Service]
ExecStartPost=-/bin/bash -c '/usr/bin/test -f /etc/default/docker && /bin/systemctl stop docker && /bin/systemctl start docker'

to /etc/systemd/system/firewalld.service.d/docker.conf.

But this did not work, firewalld service timed out on start:

root@ubuntu-xenial:~# systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/firewalld.service.d
           └─docker.conf
   Active: failed (Result: timeout) since Sat 2017-04-08 14:39:45 UTC; 1min 35s ago
  Process: 26050 ExecStartPost=/bin/bash -c /usr/bin/test -f /etc/default/docker && /bin/systemctl stop docker && /bin/systemctl start docker (code=killed, signal=TERM)
  Process: 26000 ExecStart=/usr/sbin/firewalld --nofork --nopid (code=exited, status=0/SUCCESS)
 Main PID: 26000 (code=exited, status=0/SUCCESS)

Apr 08 14:38:10 ubuntu-xenial systemd[1]: Starting firewalld - dynamic firewall daemon...
Apr 08 14:39:41 ubuntu-xenial systemd[1]: firewalld.service: Start-post operation timed out. Stopping.
Apr 08 14:39:45 ubuntu-xenial systemd[1]: Failed to start firewalld - dynamic firewall daemon.
Apr 08 14:39:45 ubuntu-xenial systemd[1]: firewalld.service: Unit entered failed state.
Apr 08 14:39:45 ubuntu-xenial systemd[1]: firewalld.service: Failed with result 'timeout'.

I am aware of BindTo and Requires but I would like to restart the docker service on every state change of firewalld so these directives do not solve my problem.

Upd.: strace shows that systemctl start is waiting on a unix socket:

root@ubuntu-xenial:~# lsof -p 27898
COMMAND     PID USER   FD   TYPE             DEVICE SIZE/OFF   NODE NAME
systemctl 27898 root  cwd    DIR                8,1     4096      2 /
systemctl 27898 root  rtd    DIR                8,1     4096      2 /
systemctl 27898 root  txt    REG                8,1   659848    124 /bin/systemctl
systemctl 27898 root  mem    REG                8,1    80496   2057 /lib/x86_64-linux-gnu/libgpg-error.so.0.17.0
systemctl 27898 root  mem    REG                8,1    14608   2060 /lib/x86_64-linux-gnu/libdl-2.23.so
systemctl 27898 root  mem    REG                8,1   456632   2144 /lib/x86_64-linux-gnu/libpcre.so.3.13.2
systemctl 27898 root  mem    REG                8,1  1864888   2066 /lib/x86_64-linux-gnu/libc-2.23.so
systemctl 27898 root  mem    REG                8,1   138696   2082 /lib/x86_64-linux-gnu/libpthread-2.23.so
systemctl 27898 root  mem    REG                8,1   919168   2037 /lib/x86_64-linux-gnu/libgcrypt.so.20.0.5
systemctl 27898 root  mem    REG                8,1   137400   2067 /lib/x86_64-linux-gnu/liblzma.so.5.0.0
systemctl 27898 root  mem    REG                8,1    31712   2077 /lib/x86_64-linux-gnu/librt-2.23.so
systemctl 27898 root  mem    REG                8,1   130224   2154 /lib/x86_64-linux-gnu/libselinux.so.1
systemctl 27898 root  mem    REG                8,1   162632   2069 /lib/x86_64-linux-gnu/ld-2.23.so
systemctl 27898 root    0r   CHR                1,3      0t0      6 /dev/null
systemctl 27898 root    1w   CHR                1,3      0t0      6 /dev/null
systemctl 27898 root    2w   CHR                1,3      0t0      6 /dev/null
systemctl 27898 root    3u  unix 0xffff880003155400      0t0 177152 type=STREAM
root@ubuntu-xenial:~# strace -p 27898
strace: Process 27898 attached
ppoll([{fd=3, events=POLLIN}], 1, NULL, NULL, 8 <unfinished ...>
+++ killed by SIGKILL +++
root@ubuntu-xenial:~#

Upd. 2: according to gdb, systemctl is blocked on a DBus call:

root@ubuntu-xenial:~# gdb -ex "set pagination 0" -ex "thread apply all bt" --batch -p 18145
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007f99b3edebcd in __GI_ppoll (fds=0x7ffc098bb800, nfds=1, timeout=<optimized out>, sigmask=0x0) at ../sysdeps/unix/sysv/linux/ppoll.c:50
50  ../sysdeps/unix/sysv/linux/ppoll.c: No such file or directory.

Thread 1 (Thread 0x7f99b4f0d880 (LWP 18145)):
#0  0x00007f99b3edebcd in __GI_ppoll (fds=0x7ffc098bb800, nfds=1, timeout=<optimized out>, sigmask=0x0) at ../sysdeps/unix/sysv/linux/ppoll.c:50
#1  0x000056094741272e in ppoll () at /usr/include/x86_64-linux-gnu/bits/poll2.h:71
#2  bus_poll.lto_priv.198 (bus=0x560948d86040, need_more=<optimized out>, timeout_usec=<optimized out>) at ../src/libsystemd/sd-bus/sd-bus.c:2874
#3  0x00005609473b435e in bus_process_wait (bus=0x560948d86040) at ../src/shared/bus-util.c:1993
#4  bus_wait_for_jobs (extra_args=0x0, quiet=false, d=0x560948d86910) at ../src/shared/bus-util.c:2136
#5  start_unit.lto_priv.362 (argc=<optimized out>, argv=<optimized out>, userdata=<optimized out>) at ../src/systemctl/systemctl.c:2787
#6  0x00005609473b0716 in dispatch_verb (verbs=0x56094743f760 <verbs>, userdata=0x0, argv=0x7ffc098bbb38, argc=<optimized out>) at ../src/basic/verbs.c:92
#7  systemctl_main () at ../src/systemctl/systemctl.c:7447
#8  main (argc=<optimized out>, argv=0x7ffc098bbb38) at ../src/systemctl/systemctl.c:7713
root@ubuntu-xenial:~#

Best Answer

I was able to solve this problem by adding --no-block option to the systemctl command like this:

[Service]
ExecStartPost=-/bin/bash -c '/usr/bin/test -f /etc/default/docker && /bin/systemctl --no-block restart docker'
ExecStopPost=-/bin/bash -c '/usr/bin/test -f /etc/default/docker && /bin/systemctl --no-block restart docker'