3👍
After spending some time and actually implementing Celery, I was able to figure out what’s actually going on. This is perhaps a short coming of my own ways of learning or the documentation overlooks it, either way, I want to leave it here in case others are also wondering the same.
The following is completely not obvious IMO:
-
Django app: Celery tasks are defined here. It has Celery installed as dependency.
-
Message broker: A message broker gets this task from 1. and queues it.
-
Celery Worker: This is actually not a separate python app that only contains Celery. It is a full blown instance of 1.. So you launch Django twice, one is producing tasks and the other is working on them. Both containing the same
@shared_task
function.
Caveats that the documentation should explain:
- If you update your main Django app and not restart Celery worker, your workers are working on a stale version of the
@shared_task
function. - The app in 1. is launched using Gunicorn (or whatever you want), but App in 3. is launched using
celery -A "your_app" workers -l INFO
. It’s not running Gunicorn, but still contains all the code in 1.
I think right off the bat in the introduction of Celery, they should explain that both the producer and worker instances contain your source code.