So I have a shell script that daemonizes celery, and creates a bunch of workers running as a daemon. I would like to have a way to restart the celery task when the underlying source gets changed since the --autoreload option does not work.
According to my reading of the celery docs, kill -HUP $pid will kill the daemonized celery process and then make a new one with the same arguments. However, when I try it, celery goes down and doesn't come back up. Is something wrong with my command? Could celery be failing silently in the background when it starts (and if this is the case, where would I go to figure out what's wrong and see log output)?
Literal command is kill -HUP \`cat /var/run/celery/w1.pid\`. Checking ps aux | grep celery returns nothing. There is no logfile output at all after I send the kill signal.
Any ideas?
Best Answer
The HUP handler in Celery simply starts the shutdown procedure and calls execv with the same arguments as the process started with when it completes. This is a pretty naive way, since it cannot know if it will come back up, but we haven't found a better solution that works in a signal handler yet.
If you use
celery multithen it's better to use thecelery multi restartcommand, which will also wait for the old worker to stop first (the generic-init.d script uses this for its restart command).Here's the SIGHUP handler code: https://github.com/celery/celery/blob/master/celery/apps/worker.py#L280-L295 As you may see this requires
sys.executableandsys.argvto be set so that it can be used to restart the worker (not sure why this isn't documented)I don't know who started this trend, but people have come to expect
SIGHUPto either reread configuration files or restart itself, but it's not always easy to do correctly and I think it was irresponsible to choose a signal that's sent when the terminal window closes ;) We have discussed removing the Celery HUP handler on several occasions.