I have a box with celery workers, some of which perform reasonably long-running tasks (a few minutes). When this box updates, I'd like to shutdown the celery workers gracefully, so that ideally:
- New tasks stop being sent to workers
- Existing tasks are allowed to complete (with some timeout)
- The workers are finally killed, and the new workers started
I'm using systemd to manage services on my Ubuntu box. Here's my conf file:
[Unit]
Description=Celery Service
After=network.target
[Service]
User=celery
Type=forking
WorkingDirectory=/var/app/celery/
ExecStart=/var/app/celery/run_celery.sh
ExecStop=/var/app/celery/stop_celery.sh
Nice=1
Restart=always
RestartSec=5
[Install]
WantedBy=multi-user.target
The run_celery.sh script contains (in brief):
celery multi start 2 -A my_app --pidfile=/tmp/celery-%n.pid
And stop_celery.sh contains:
celery multi stopwait 2 --pidfile=/tmp/celery-%n.pid
However when I stop the script using systemctl, tasks are killed, rather than allowing workers to finish. What am I doing wrong?
Best Answer
You probably want to set
TimeoutStopSec=
to what you want the timeout to be. See thesystemd.service
man page for more in-depth information.