[Django]-Celery – Tasks that need to run in priority

46👍

Celery does not support task priority. (v3.0)

http://docs.celeryproject.org/en/master/faq.html#does-celery-support-task-priorities

You may solve this problem by routing tasks.

http://docs.celeryproject.org/en/latest/userguide/routing.html

Prepare default and priority_high Queue.

from kombu import Queue
CELERY_DEFAULT_QUEUE = 'default'
CELERY_QUEUES = (
    Queue('default'),
    Queue('priority_high'),
)

Run two daemon.

user@x:/$ celery worker -Q priority_high
user@y:/$ celery worker -Q default,priority_high

And route task.

your_task.apply_async(args=['...'], queue='priority_high')

20👍

If you use RabbitMQ transport then configure your queues the following way:
settings.py

from kombu import Queue
...
CELERY_TASK_QUEUES = (
    Queue('default', routing_key='task_default.#', max_priority=10), 
    ...)

Then run your tasks:

my_low_prio_task.apply_async(args=(...), priority=1)
my_high_prio_task.apply_async(args=(...), priority=10)

Presently this code works for kombu==4.6.11, celery==4.4.6.

Leave a comment