[Fixed]-How to run a custom script in parallel with django

by

1👍

✅

I end up parallelizing django scripts with ipython ipyparallel package, here is how to do it!

first you need to install ipyparallel: pip install ipyparallel -U

on the default profile from ipython (or any profile of your preference) we need to add this startup imports to load django:

from MyProject import settings
import django
django.setup()

This should be added to a path like this: ~/.ipython/profile_default/startup/00-load-django.py

This will load django on the engines you need to start so ipython can parallelize your functions.

Now, lets start the engines that will be able to parallelize your django scripts coded on the fly, be sure to be in the main folder from your django project (where the manage.py file is): ipcluster start -n X where X will be the number of engines wanted (IMHO it would be the number of cores in the current computer + 1)

Please let the ipcluster be fully operational before entering into ipython.

Now, lets parallelize that django script, enter into ipython:

import ipyparallel as ipp
rc = ipp.Client()  # Create the client that will connect to the ipython engines
lview = rc.load_balanced_view()

@lview.parallel()
def show_polls(user_range):
    from poll.models import Poll
    return list(Poll.objects.filter(user_id__gte=user_range, user_id__lt=user_range+100))

for res in show_polls.map(range(0, 1000, 100)):
    print res

and there we go, a django script parallelized! notice that I convert the QuerySet into a list, that’s because anything returned needs to be pickable.

👤Hassek

Leave a comment