Systemd causing multiple supervisor processes

init.dsupervisordsystemctlsystemd

I'm having a problem narrowing down where the issue is for my supervisor configuration that causes a program to be duplicated.

I'm running a Python Tornado Web Server through an Nginx reverse proxy. I think the problem is more related to supervisor, though.

/etc/supervisor/supervisord.conf

[unix_http_server]
file=/var/run/supervisor.sock
chmod=0700

[supervisord]
logfile=/var/log/supervisor/supervisord.log
pidfile=/var/run/supervisord.pid
childlogdir=/var/log/supervisor           
logfile_maxbytes=50MB                           
logfile_backups=5                               
loglevel=error                                  

[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

[supervisorctl]
serverurl=unix:///var/run/supervisor.sock

[include]
files = /etc/supervisor/conf.d/*.conf

/etc/supervisor/conf.d/tornado.conf

[program:tornado]
numprocs = 4
numprocs_start = 8000
process_name = %(program_name)s_%(process_num)02d
directory=/home/username/webapp
environment=PATH="/home/username/.virtualenvs/tornado_env/bin"
command = python server.py --port=%(process_num)s
user=www-data
redirect_stderr=true
stdout_logfile=/var/log/webapp/%(program_name)s_%(process_num)02d.log
loglevel=info

As you can see numprocs = 4, but my confusion comes from this after just restarting the server.

ps -aux | egrep "supervisor|python"

root       399  0.1  2.6  14496  9584 ?        Ss   15:48   0:03 /usr/bin/python /usr/bin/supervisord -c /etc/supervisor/supervisord.conf
www-data   410  0.1  3.9  16872 14200 ?        S    15:48   0:02 python server.py --port=8002
www-data   411  0.1  3.9  16872 14180 ?        S    15:48   0:02 python server.py --port=8003
www-data   412  0.1  3.9  16872 14124 ?        S    15:48   0:02 python server.py --port=8000
www-data   413  0.1  3.9  16872 14180 ?        S    15:48   0:02 python server.py --port=8001
www-data   489  0.0  3.0  16872 11024 ?        S    15:49   0:00 python server.py --port=8003
www-data   490  0.0  3.1  16872 11228 ?        S    15:49   0:00 python server.py --port=8000
www-data   491  0.0  3.1  16872 11244 ?        S    15:49   0:00 python server.py --port=8002
www-data   492  0.0  3.0  16872 11192 ?        S    15:49   0:00 python server.py --port=8001

There are 8 Python processes, but supervisor only knows about 4. What else do I look for?

sudo supervisorctl status

tornado:tornado_8000             RUNNING    pid 412, uptime 0:36:03
tornado:tornado_8001             RUNNING    pid 413, uptime 0:36:03
tornado:tornado_8002             RUNNING    pid 410, uptime 0:36:05
tornado:tornado_8003             RUNNING    pid 411, uptime 0:36:04

UPDATE: Somehow running supervisor and systemd is spawning a duplicate configuration. So now my question is do I disable the supervisor service and if so, then what else is causing it to start?

sudo systemctl status supervisor

● supervisor.service - LSB: Start/stop supervisor
   Loaded: loaded (/etc/init.d/supervisor)
   Active: active (running) since Wed 2015-12-23 15:48:45 CST; 1h 31min ago
  Process: 305 ExecStart=/etc/init.d/supervisor start (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/supervisor.service
           ├─399 /usr/bin/python /usr/bin/supervisord -c /etc/supervisor/supervisord.conf
           ├─489 python server.py --port=8003
           ├─490 python server.py --port=8000
           ├─491 python server.py --port=8002
           └─492 python server.py --port=8001

Best Answer

Upon further investigation, the problem was actually my Python Tornado Server. It was forking one process per CPU, which confuses me on a single core Raspberry Pi, but whatever.

So, the output of the commands above are valid; there are no "duplicates" from supervisor or Systemd. The problem was more related to lifecycle management of these processes by supervisor, i.e. start, stop, reload.

When I would stop it before, the forked processes would remain active, so when I went to restart supervisor, I would then get a total of 4 + 8 = 12 processes.

The fix was to add this line to my /etc/supervisor/conf.d/tornado.conf. 

stopasgroup=true

For which the documentation reads

If true, the flag causes supervisor to send the stop signal to the whole process group and implies killasgroup is true. This is useful for programs, such as Flask in debug mode, that do not propagate stop signals to their children, leaving them orphaned.

Related Topic