[Fixed]-Difference between usage of Django celery and Django cron-jobs?

The two things can be used for the same goal (background execution). However, if you are going to choose wisely, you should really understand that they are actually completely different things.

Here’s what I wish someone had told me back when I was a noob (instead of the novice level that I have achieved today :)).

cron

The concept of a cron job is that we want a command / process to be executed on some schedule. Furthermore, we want that process to receive x,y,z parameters, run with a,b,c environment variables, and as user id 123.

Some cron systems may facilitate a few extra features, such as:

• catching up on missed tasks (e.g. the server was off for a power outage all night and as soon as we turn it on, it runs the 8 instances of the command we normally run hourly).
• might help you with the type of locking you normally do using a pid file in order to avoid parallel runs of the same command.

For the most part, cron systems are meant to be dumb: “just run this command at this time, thanks!”.

Celery

The concept of Celery is much more sophisticated. It works with tasks, chains & chords of tasks, error handling, and (in most cases) collection of work result. It has a queue (or many queues) of work and a worker (or many). When a task (really just a message describing requested work) enters the queue it waits there until a worker is available to handle it. Much the same way as 1 or more employees at the DMV service a room full of waiting customers.

Furthermore, Celery can facilitate distributed work. That’s a bit like (if I may torture the analogy a bit) – the difference between a DMV office where every worker shares the same phone, computer, copier, etc and a DMV where workers have dedicated resources and are never blocked by other workers.

Celery for web apps

In web applications, Celery is often used when a bit of web access results in a thing to be done that should be handled out of band of the conversation with the web browser. For example:

• the web user just did something which should result in an email being sent. In order to send an email, your web server will need to contact a mail server. This could take time, the server could be busy, etc – we cant make the web user just wait, seeing nothing on their browser while we do this. Well, you can but it won’t work reliably. So, we do that email send as a bit of work in the queue. That way, it can happen “whenever” and the web server can get back to communicating with the browser.

• the user just submitted a credit card as payment. You’re going to need to contact the card processor, but that might take several seconds. You might even have to contact them multiple times (e.g. they are really busy there right now). Again, you don’t want your user’s web browser to just sit blankly and you don’t want a web server process or thread of execution tied up. Instead, you use Celery to create a job, you tell the browser to check back in a few seconds (or use a “web socket”), and your web server moves on and talks to other web users. When the browser checks back later, you lookup the task id and find out from celery whether it is finished and what the outcome was (card declined, etc).

Using Celery as cron

When you use Celery as a “cron system” all you are really doing is saying: “hey, can someone please generate work of X type on Y schedule”. A process is created that runs continuously which sleeps most of the time and wakes up occasionally to inject a bit of work into the queue on the schedule you requested.

Usually the “hey someone” that you ask to do that for you is: celery beat and beat gets the schedule you want from somewhere in the database or from your settings file.