[Django]-How does django handle multiple memcached servers?

I tested part of this and found some interesting stuff with django 1.1 and python-memcached 1.44.

On django using 2 memcache servers

cache.set('a', 1, 1000)

cache.get('a') # returned 1

I looked up which memcache server ‘a’ was sharded to using 2 other django setups each pointing at one of the memcache servers. I simulated a connectivity outage by putting up a firewall between the original django instance and the memcache server that ‘a’ was stored in.

cache.get('a') # paused for a few seconds and then returned None

cache.set('a', 2, 1000)

cache.get('a') # returned 2 right away

The memcache client library does update its sharding strategy if a server goes down.

Then I removed the firewall.

cache.get('a') # returned 2 for a bit until it detected the server back up then returned 1!

You can read stale data when a memcache server drops and comes back! Memcache doesn’t do anything clever to try to prevent this.

This can really mess things up if you’re using a caching strategy that puts things in memcache for a long time and depends on cache invalidation to handle updates. An old value can be written to the “normal” cache server for that key and if you loose connectivity and an invalidation is made during that window, when the server becomes accessible again, you’ll read stale data that you shouldn’t be able to.

One more note: I’ve been reading about some object/query caching libraries and I think johnny-cache should be immune to this problem. It doesn’t explicitly invalidate entries; instead, it changes the key at which a query is cached when a table changes. So it would never accidentally read old values.

Edit: I think my note about johnny-cache working ok is crap. http://jmoiron.net/blog/is-johnny-cache-for-you/ says “there are extra cache reads on every request to load the current generations”. If the generations are stored in the cache itself, the above scenario can cause a stale generation to be read.

Thought to add this answer two years after the question was asked, since it ranks very highly in search and because we did find a situation where django was talking to only one of the memcached servers.

With a site running on django 1.4.3, python-memcached 1.51 talking to four memcached instances, we found that the database was being queried far more often than expected. Digging futher, we found that cache.get() was returning None for keys that were knew to be present in at least one of the memcached instances. When memcached was started with the -vv option it showed that the question was asked only of one server!

After a lot of hair had been pulled, we switched the backend to django.core.cache.backends.memcached.PyLibMCCache (pylibmc) and the problem went away.

If using two distinct memcache’s is ideal, django’s default implementation allows for this behavior.

First you’ll want to update your settings.py:

CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache',
        'LOCATION': '127.0.0.1:11211',
    },
    'rusty': {
        'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache',
        'LOCATION': '127.0.0.1:11222',
    }
}

Inside your django code, the default method for accessing memcache hasn’t changed. You can now use the other cache interface as follows:

from django.core.cache import cache, caches

cache.set("activity", 'great stuff', 15 ) # Default cache
caches["rusty"].set("activity", "A great time}", 32) # New rusty cache interface

The Django documentation has a great write up covering this topic: https://docs.djangoproject.com/en/dev/topics/cache/