[Django]-Profiling Django

Just type “django-profiling” on google, you’ll get these links (and more):

http://code.djangoproject.com/wiki/ProfilingDjango

http://code.google.com/p/django-profiling/

http://www.rkblog.rk.edu.pl/w/p/django-profiling-hotshot-and-kcachegrind/

Personally I’m using the middleware approach – i.e. each user can toggle a “profiling” flag stored in a session, and if my profiling middleware notices that a flag has been set, it uses Python’s hotshot module like this:

def process_view(self, request, view_func, view_args, view_kwargs):

     # setup things here, along with: settings.DEBUG=True 
     # to get a SQL dump in connection.queries

     profiler = hotshot.Profile(fname)
     response = profiler.runcall(view_func, request, *view_args, **view_kwargs)
     profiler.close()

     # process results

     return response

EDIT: For profiling SQL queries http://github.com/robhudson/django-debug-toolbar mentioned by Konstantin is a nice thing – but if your queries are really slow (probably because there are hundreds or thousands of them), then you’ll be waiting insane amount of time until it gets loaded into a browser – and then it’ll be hard to browse due to slowness. Also, django-debug-toolbar is by design unable to give useful insight into the internals of AJAX requests.

EDIT2: django-extensions has a great profiling command built in:

https://github.com/django-extensions/django-extensions/blob/master/docs/runprofileserver.rst

Just do this and voila:

$ mkdir /tmp/my-profile-data
$ ./manage.py runprofileserver --kcachegrind --prof-path=/tmp/my-profile-data

Shameless plug here, but I recently made https://github.com/django-silk/silk for this purpose. It’s somewhat similar to django toolbar but with history, code profiling and more fine grained control over everything.

For profiling data access (which is where the bottleneck is most of the time) check out django-live-profiler. Unlike Django Debug Toolbar it collects data across all requests simultaneously and you can run it in production without too much performance overhead or exposing your app internals.

I needed to profile a Django app recently and tried many of these suggestions. I ended up using pyinstrument instead, which can be added to a Django app using a single update to the middleware list and provides a stack-based view of the timings.

Quick summary of my experience with some other tools:

• Django Debug Toolbar is great if you the issue is due to SQL queries and works well in combination with pyinstrument
• django-silk works well, but requires adding a context manager or decorator to each part of the stack where you want sub-request timings. It also provides an easy way to access cProfile timings and automatically displays ajax timings, both of which can be really helpful.
• djdt-flamegraph looked promising, but the page never actually rendered on my system.

Compared to the other tools I tried, pyinstrument was dramatically easier to install and to use.

For all you KCacheGrind fans, I find it’s very easy to use the shell in tandem with Django’s fantastic test Client for generating profile logs on-the-fly, especially in production. I’ve used this technique now on several occasions because it has a light touch — no pesky middleware or third-party Django applications are required!

For example, to profile a particular view that seems to be running slow, you could crack open the shell and type this code:

from django.test import Client
import hotshot

c = Client()
profiler = hotshot.Profile("yourprofile.prof")  # saves a logfile to your pwd
profiler.runcall(c.get, "/pattern/matching/your/view/")
profiler.close()

To visualize the resulting log, I’ve used hotshot2cachegrind:

• http://kcachegrind.sourceforge.net/html/ContribPython.html

But there are other options as well:

• http://www.vrplumber.com/programming/runsnakerun/
• https://code.djangoproject.com/wiki/ProfilingDjango

When the views are not HTML, for example JSON, use simple middleware methods for profiling.

Here are a couple examples:

https://gist.github.com/1229685 – capture all sql calls went into the view

https://gist.github.com/1229681 – profile all method calls used to create the view

You can use line_profiler.

It allows to display a line-by-line analysis of your code with the time alongside of each line (When a line is hit several times, the time is summed up also).

It’s used with not-Django python code but there’s a little trick to use it on Django in fact: https://stackoverflow.com/a/68163807/1937033

I am using silk for live profiling and inspection of Django application. This is a great tool. You can have a look on it.

https://github.com/jazzband/django-silk