[Django]-Celery worker will not pick up a new task after the current one is finished

It sounds like the issue is celery’s default pre-fetch behavior. Each worker will reserve a set number of tasks ahead of time while it is currently at maximum capacity and this is known as the Prefetch Multiplier.

The reason it does that is so that when you have a high volume of short tasks, your overall throughput will be much higher if the tasks are already prefetched and ready to execute immediately.

The problem is that when you have a lot of long running tasks or a mixture of long and short tasks, then tasks can get reserved and blocked up by a busy worker even though other workers are available to handle it.

Therefore, in your case you may need to lower the prefetch multiplier to 1.