[Django]-How does Celery worker run the code defined elsewhere in a task?

3👍

After spending some time and actually implementing Celery, I was able to figure out what’s actually going on. This is perhaps a short coming of my own ways of learning or the documentation overlooks it, either way, I want to leave it here in case others are also wondering the same.

The following is completely not obvious IMO:

  1. Django app: Celery tasks are defined here. It has Celery installed as dependency.

  2. Message broker: A message broker gets this task from 1. and queues it.

  3. Celery Worker: This is actually not a separate python app that only contains Celery. It is a full blown instance of 1.. So you launch Django twice, one is producing tasks and the other is working on them. Both containing the same @shared_task function.

Caveats that the documentation should explain:

  • If you update your main Django app and not restart Celery worker, your workers are working on a stale version of the @shared_task function.
  • The app in 1. is launched using Gunicorn (or whatever you want), but App in 3. is launched using celery -A "your_app" workers -l INFO. It’s not running Gunicorn, but still contains all the code in 1.

I think right off the bat in the introduction of Celery, they should explain that both the producer and worker instances contain your source code.

👤Neil

Leave a comment