[Fixed]-Long running tasks with Django

11đź‘Ť

âś…

Basically you need to have a process that runs outside of the request. The absolute simplest way to do this (on a Unix-like operating system, at least) is to fork():

if os.fork() == 0:
    do_long_thing()
    sys.exit(0)
… continue with request …

This has some downsides, though (ex, if the server crashes, the “long thing” will be lost)… Which is where, ex, Celery can come in handy. It will keep track of the jobs that need to be done, the results of jobs (success/failure/whatever) and make it easy to run the jobs on other machines.

Using Celery with a Redis backend (see Kombu’s Redis transport) is very simple, so I would recommend looking there first.

👤David Wolever

6đź‘Ť

You might need to have a process outside the request / response cycle. If that is the case, Celery with a Redis backend is what I would suggest looking into, as that integrates nicely with Django (as David Wolever suggested).

Another option is to create Django management commands, and then use cron to execute them at scheduled intervals.

👤Brian Neal

Leave a comment