[Django]-How to disable Django query cache?

by

46👍

✅

I came across behavior that I thought was some kind of caching, but it turned out to be database transactions fooling me.

I had the problem where in another process, items were get added to the database, and I wanted to monitor progress of the other process, so I opened up a django shell and issued the following:

>>> MyData.objects.count()
74674

>>> MyData.objects.count()
74674

The value wasn’t changing, even though it actually was in the database. I realized that at least with the way I had MySQL & django setup that I was in a transaction and would only see a “snapshot” of the database at the time I opened the transaction.

Since with views in django, I had autocommit behavior defined, this was fine for each view to only see a snapshot, as the next time a view was called it would be in a different transaction. But for a piece of code that was not automatically committing, it would not see any changes in the db except those that were made in this transaction.

Just thought I would toss this answer in for anyone who may come upon this situation.

To solve, commit your transaction, which can be manually done like so:

>> from django.db import transaction
>> transaction.enter_transaction_management()
>> transaction.commit() # Whenever you want to see new data
👤Kekoa

9👍

Query caching only applies within a QuerySet. In other words, if you evaluate the same queryset object twice, query caching will operate. But if you are doing a query every 10 seconds, presumably this is via a cron that spawns a new process each time, so there is no way Django will cache anything.

It is possible that your database’s own cache will come into operation if you’re repeatedly performing exactly the same query. You should look at the documentation for your DBMS to see how to manage that properly.

👤Daniel Roseman

6👍

The link you provide to the Django Documentation implies that the following:

>>> print [e.headline for e in Entry.objects.all()]
>>> print [e.pub_date for e in Entry.objects.all()]

creates two queries to the database, whilst:

>>> queryset = Poll.objects.all()
>>> print [p.headline for p in queryset] # Evaluate the query set.
>>> print [p.pub_date for p in queryset] # Re-use the cache from the evaluation.

uses the query cache, as you are accessing the same evaluation results.

👤Amos

1👍

Thank you very much for your answers, your replies made me take a few steps back and rethink.

In order to test caching on a DBMS level, I went away from Django and used a shell script I anyway had handy to periodically query data from a SQLite db, while I added data in a second shell session. The new data showed up in the periodic queries right after I added them, so no query caching here.

This narrowed it down to the Django part. The code involved is not very complex and a little log output and a code review revealed the problem: The query used to obtain the queryset used in turn to create the MD5 sum had an error and was always empty. Therefore, the MD5 sum was always the same. Did indeed look like a cached result – data is changing, but the queryset stays the same.
The problem did not show in the application, as a different query was used to obtain data displayed there.

Lesson learned: If you’re completely puzzled, take a step back and re-think your assumptions.

Thanks again! 🙂

👤ssc

0👍

I met this problem on django version 1.8. There is not direct way to do it, but there are some ways to make the queryset re-evaluated and executed by accessing db instead of cache. I found it in Django Queryset Documentation

I used one of them to handle my problem. It is exists() function of querysets. len() and repr() can also be used. They worked for me too.

Example

queryset = ModelClass.objects.filter(....)
queryset.exists()

#or len(queryset)
#or repr(queryset)

#Now queryset is re-evaluated.
👤Ahmet DAL

0👍

You can use iterator() to skip the cache and use the underlying database cursor:

querySet = m.DataPoint.objects.all()
for dp in querySet.iterator():
   # do the job
👤János Roden

Leave a comment