[Django]-How to log memory usage of an Django app per request

by

24👍

✅

A Django middleware for tracking memory usage and generating a usable result immediately, needs to hook both process request and process response. In other words, look at difference between start and finish of request and log a warning if exceeds some threshold.

A complete middleware example is:

import os
import psutil
import sys

THRESHOLD = 2*1024*1024

class MemoryUsageMiddleware(object):

    def process_request(self, request):
        request._mem = psutil.Process(os.getpid()).memory_info()

   def process_response(self, request, response):
        mem = psutil.Process(os.getpid()).memory_info()
        diff = mem.rss - request._mem.rss
        if diff > THRESHOLD:
            print >> sys.stderr, 'MEMORY USAGE %r' % ((diff, request.path),)
        return response

This requires the ‘psutil’ module to be installed for doing memory calculation.

Is brute force and can lead to false positives in a multithread system. Because of lazy loading, you will also see it trigger on first few requests against new process as stuff loads up.

👤Graham Dumpleton

1👍

This may not fully cover your question, but I recommend trying nginx+uwsgi instead of apache2+mod_wsgi. In my tests it turned out to be much more stable (mod_wsgi choked at some point completely), much faster and uses a lot less memory (it may just fix all your issues altogether).

About tracking memory usage, you can create a simple middleware:

class SaveMemoryUsageMiddleware(object):
    def process_response(self, request, response):
        # track memory usage here and append to file or db
        return response

and add it to your middlewares.

For memory tracking code I recommend checking out:
Total memory used by Python process?

However, it would probably be better if you could avoid doing this on production. Just for dev and tests to track down real problem.

👤arkens

Leave a comment