[Django]-Python Celery Threads, Workers and vCPUs

4👍

Short answer: You will need to understand what you are doing, and probably measure it yourself

Longer:

The main question is if your tasks are CPU bound or I/O (network/disk) bound. If your tasks are CPU bound (probably stuff like generating templates, images), you won’t get any improvements by adding workers. However most chances are you are on I/O bound (network) tasks, and if you are waiting for network acknowledgments, and there is no bottleneck in the mail server etc, you will probably be able to gain higher results by using more workers.

To understand this much better, I highly recommend walking slowly through David Beazley’s eye opening presentation here: An Introduction to Python Concurrency. This does not cover Celery and Tornado, but gives an excellent overview of the underlying technology and problems, and lays out the solutions (with examples) as well.

👤Udi

Leave a comment