[Answer]-Multiple Celery instances consuming a single queue

by

1👍

Yes it’s possible, they just have to use the same broker. For instance, if you are using AMQP, the configs on your servers must share the same

BROKER_URL = 'amqp://user:password@localhost:5672//'

See the routing page for more details. For instance let’s say you want to have a common queue for two servers, then one specific to each of them, you could do

On server 1:

CELERY_ROUTES = {'your_app.your_specific_tasks1': {'queue': 'server1'}}
user@server1:/$ celery -A your_celery_app worker -Q server1, default

On server 2:

CELERY_ROUTES = {'your_app.your_specific_tasks2': {'queue': 'server2'}}
user@server2:/$ celery -A your_celery_app worker -Q server2, default

Of course it’s optional, by default all the tasks will be routed to the queue named celery.

👤Seb D.

Leave a comment