[Django]-Speeding up Django Testing

Using an in-memory SQLite database during testing definitely speeds things up.

I don’t like the idea of using a different database (SQLite) for testing, so my unit tests use the same database as the production application – postgres.

Out of the box, this makes creating/destroying the database the slowest step in running tests.

Django 1.8 will solve this problem with the –keepdb flag

But we’re still not there yet, so we must make do with other means.

Solution 1) Use pytest-django

You can use that to make your tests run without re-creating the database. It works.
If you only care about running tests on the command line, I suggest this.

In my case, I like to use the PyCharm IDE, and being able to run tests by right clicking files/methods is definitely a plus for me, so I had to go with…

Solution 2) The TEST_MIRROR trick.

In your settings.py file, configure your database like:

if os.getenv('USE_TEST_DB') == '1':
    DATABASES = {
        'default': {
            'ENGINE': 'django.db.backends.postgresql_psycopg2',
            'NAME': 'mydbtesting',
            'USER': 'mydb',
            'PASSWORD': 'mydb',
            'HOST': 'localhost',
            'PORT': '5432',
            'TEST_MIRROR': 'default',
        }
    }
else:
    DATABASES = {
        'default': {
            'ENGINE': 'django.db.backends.postgresql_psycopg2',
            'NAME': 'mydb',
            'USER': 'mydb',
            'PASSWORD': 'mydb',
            'HOST': 'localhost',
            'PORT': '5432',
        }
    }

So, “mydb” is the database that will be used for normal execution and “mydbtesting” is for tests.

The TEST_MIRROR setting is not actually meant for this, but the fact is that if you run tests with a database configured like that, Django will not re-create/destroy which is what we want.

But first we have to create that database with something like:

export USE_TEST_DB=1
./manage.py syncdb --migrate

Then whenever you want to run tests fast, just set the USE_TEST_DB environment variable to ‘1’.
To get same benefit on Pycharm, you can go to Run/Debug Configurations, Defaults / Django tests, then on Environment variables, add USE_TEST_DB = 1

UPDATE:

Sample application is on Github: https://github.com/freedomsponsors/www.freedomsponsors.org/blob/099ec1a7a1c404eba287d4c93d58c8cf600b2769

I have found another way to speed up the testing. If your test models are auth users (User model), and you set a password for them, the hashing function takes a decent number of milliseconds to finish. What I do is add this to my test settings:

PASSWORD_HASHERS = (
    'django.contrib.auth.hashers.MD5PasswordHasher',
)

This enforces MD5 hashing for password which is much faster than the default one.
In my case, this improved 12 tests, each creates 7 users, from 4.5 seconds to 500 ms.

Be careful not to add this to your production settings!

You can run only tests that interest you specyfically, look here: http://docs.djangoproject.com/en/dev/topics/testing/?from=olddocs#running-tests

Like in this example – run just specyfic TestCase:

$ ./manage.py test animals.AnimalTest

As for the test database – it is created and destroyed each time the test runs 🙁
Also for testing you could use sqlite database if it is possible in your workflow.

As of Django 1.8, you can keep the test database around so that you don’t rebuild it every time you test. Just add the –keepdb flag.

python manage.py test --keepdb

Exclude the –keepdb flag to rebuild the test database from scratch.

I have found another way to speed up testing. The most time consuming operation is writing to/reading from hard drive (I’m using sqlite for testing). The solution is to create ramdisk, and put the sqlite database file there. I have reduced testing time by factor of 10.

Creating ramdisk:

#!/bin/sh

mkdir -p /tmp/ramdisk; chmod 777 /tmp/ramdisk
mount -t tmpfs -o size=256M tmpfs /tmp/ramdisk/

Changing db file path:

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': '/tmp/ramdisk/test.db',
        'TEST_NAME': '/tmp/ramdisk/test.db',
    }
}

Here is simple test tools that provides no-reload database along with signals so you don’t need to care about test database https://github.com/plus500s/django-test-tools