[Django]-How to restart Celery gracefully without delaying tasks

celeryd has –autoreload option. If enabled, celery worker (main process) will detect changes in celery modules and restart all worker processes. In contrast to SIGHUP signal, autoreload restarts each process independently when the current executing task finishes. It means while one worker process is restarting the remaining processes can execute tasks.

http://celery.readthedocs.org/en/latest/userguide/workers.html#autoreloading

I’ve recently fixed the bug with SIGHUP: https://github.com/celery/celery/pull/662

rm *.pyc

This causes the updated tasks to be reloaded. I discovered this trick recently, I just hope there are no nasty side effects.

Well you using SIGHUP (1) for warm shutdown of celery. I am not sure if it actually causes a warm shutdown. But SIGINT (2) would cause a warm shutdown. Try SIGINT in place of SIGHUP and then start celery manually in your script (I guess).

Can you launch it with a custom pid file name. Possibly timestamped, and key off of that to know which PID to kill?

CELERYD_PID_FILE="/var/run/celery/%n_{timestamp}.pid"

^I dont know the timestamp syntax but maybe you do or you can find it?

then use the current system time to kill off any old pids and launch a new one?

A little late, but that can fixed by deleting the file called celerybeat.pid.

Worked for me.

I think you can try this:

kill -s HUP ``cat /var/run/celeryd.pid`` 
python manage.py celeryd --pidfile=/var/run/celeryd.pid

HUP may recycle every free worker and leave executing workers keep running and HUP will let these workers be trusted. Then you can safely restart a new celery worker main process and workers. Old workers may be killed itself when task has been finished.

I’ve use this way in our production and it seems safe now. Hope this can help you!