[Solved]-How to best launch an asynchronous job request in Django view?

10đź‘Ť

There are a couple of solutions to this problem, and the best one depends a bit on how heavy your workload will be.

If you have a light workload you can use the approach used by django-mailer which is to define a “jobs” model, save new jobs into the database, then have cron run a stand-alone script every so often to process the jobs stored in the database (deleting them when done). You can use something like django-chronograph to manage the job scheduling easier

If you need help understanding how to write a script to process the job see James Bennett’s article Standalone Django Scripts for help.

If you have a very high workload, meaning you’ll need more than a single server to process the jobs, then you want to use a real distribute task queue. There is a lot of competition here so I can’t really detail all the options, but a good one to use with for Django apps is celery.

👤Van Gale

1đź‘Ť

Why not simply start a thread to do the processing and then go on to send the response?

1đź‘Ť

Before you select a solution, you need to determine how the process will be run. I.e is it the same process for every single user, the data is the same and can be scheduled regularly? or does each user request something and the results are slightly different ?

As an example, if the data will be the same for every single user and can be run on a schedule you could use cron.

See: http://www.b-list.org/weblog/2007/sep/22/standalone-django-scripts/
or
http://docs.djangoproject.com/en/dev/howto/custom-management-commands/

However if the requests will be adhoc and you need something scalable that can handle high load and is asynchronous: what you are actually looking for is a message queing system. Your view will add a request to the queue which will then get acted upon.

There are a few options to implement this in Django:
Django Queue service is purely django & python and simple, though the last commit was in April and it seems the project has been abandoned.
http://code.google.com/p/django-queue-service/

The second option if you need something that scales, is distributed and makes use of open source message queing servers: celery is what you need

http://ask.github.com/celery/introduction.html
http://github.com/ask/celery/tree

👤ismail

Leave a comment