Ubuntu – Supervisord leaking child processes launched through shell script

pythonsupervisordUbuntu

I'm using supervisor 3 to daemonize a python process, which is launched through a bash script that sets up some configuration for python. The bash script runs python program in the foreground, FWIW.

This setup has the following problem: when supervisord tries to kill the process, it only kills the bash script, not the python process. So I end up leaking python processes running old code and all sorts of problems ensue. How can I fix this?

Is there some configuration option for supervisor to tell it to kill child processes as well?

Would it help to have the bash script trap the TERM signal and explicitly pass it on to the child? This seems unnecessary since when I run it from an interactive shell, the child python process is killed just fine.

I see that supervisor lets me pick which signal to kill with, defaulting to TERM right now. Might changing that help?

Best Answer

Here's a similar question on Stackoverflow:

Tornado doesn't restart cleanly in supervisor

Solution that worked for me was to use stopasgroup = true in the relevant [program:x] group.

Related Solutions

Linux – Redirect output of Python program to /dev/null

The program may be writing to the tty instead of stdout. You may be able to use the unbuffer script that comes with expect or some other technique using expect or something similar.

Here's a demonstration. Create this script and call it "writetty". Then do chmod u+x writetty:

#!/bin/bash
date > /dev/tty

Now try it out:

$ ./writetty
Tue Jun  8 12:24:36 CDT 2010
$ ./writetty > /dev/null
Tue Jun  8 12:24:46 CDT 2010
$ var=$(./writetty)
Tue Jun  8 12:24:56 CDT 2010
$ unbuffer ./writetty > /tmp/writetty.out
$ cat /tmp/writetty.out
Tue Jun  8 12:25:06 CDT 2010
$ var=$(unbuffer ./writetty)
$ echo $var
Tue Jun  8 12:25:16 CDT 2010

You could use expect to respond to the program's request for input instead of killing it.

Python – Django + gunicorn + virtualenv + Supervisord issue

I tracked this one down with the help of davisp, a gunicorn developer. Thank you !

It was an environnement issue, caused by an invalid HOME setting in the environnement of the supervisor gunicorn subprocess.

I did not get anything on stderr until I put an "import psycopg2" in my settings.py file, which produced the following message on supervisor's stderr files for gunicorn.

Error: Can't extract file(s) to egg cache

The following error occurred while trying to extract file(s) to the Python egg cache:

  [Errno 13] Permission denied: '/root/.python-eggs'"

I added the following line in my supervisor configuration file for gunicorn, and now Python finds its way to a u+writable egg cache. Everything is OK.

environment=HOME='/home/hc'

Best Answer

Related Solutions

Linux – Redirect output of Python program to /dev/null

Python – Django + gunicorn + virtualenv + Supervisord issue

Related Topic