[Django]-Django Unit Testing taking a very long time to create test database

Summary

Use pytest !

Operations

1. pip install pytest-django
2. pytest --nomigrations instead of ./manage.py test

Result

• ./manage.py test costs 2 min 11.86 sec
• pytest --nomigrations costs 2.18 sec

Hints

• You can create a file called pytest.ini in your project root directory, and specify default command line options and/or Django settings there.

  # content of pytest.ini
  [pytest]
  addopts = --nomigrations
  DJANGO_SETTINGS_MODULE = yourproject.settings
  

  Now you can simply run tests with pytest and save you a bit of typing.

• You can speed up the subsequent tests even further by adding --reuse-db to the default command line options.

  [pytest]
  addopts = --nomigrations --reuse-db
  

  However, as soon as your database model is changed, you must run pytest --create-db once to force re-creation of the test database.

• If you need to enable gevent monkey patching during testing, you can create a file called pytest in your project root directory with the following content, cast the execution bit to it (chmod +x pytest) and run ./pytest for testing instead of pytest:

  #!/usr/bin/env python
  # -*- coding: utf-8 -*-
  # content of pytest
  from gevent import monkey
  
  monkey.patch_all()
  
  import os
  
  os.environ.setdefault("DJANGO_SETTINGS_MODULE", "yourproject.settings")
  
  from django.db import connection
  
  connection.allow_thread_sharing = True
  
  import re
  import sys
  
  from pytest import main
  
  if __name__ == '__main__':
      sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
      sys.exit(main())
  

  You can create a test_gevent.py file for testing whether gevent monkey patching is successful:

  # -*- coding: utf-8 -*-
  # content of test_gevent.py
  import time
  from django.test import TestCase
  from django.db import connection
  import gevent
  
  
  def f(n):
      cur = connection.cursor()
      cur.execute("SELECT SLEEP(%s)", (n,))
      cur.execute("SELECT %s", (n,))
      cur.fetchall()
      connection.close()
  
  
  class GeventTestCase(TestCase):
      longMessage = True
  
      def test_gevent_spawn(self):
          timer = time.time()
          d1, d2, d3 = 1, 2, 3
          t1 = gevent.spawn(f, d1)
          t2 = gevent.spawn(f, d2)
          t3 = gevent.spawn(f, d3)
          gevent.joinall([t1, t2, t3])
          cost = time.time() - timer
          self.assertAlmostEqual(cost, max(d1, d2, d3), delta=1.0,
                                 msg='gevent spawn not working as expected')
  

References

• pytest-django documentation
• pytest documentation

use ./manage.py test –keepdb when there are no changes in the migration files

Database initialization indeed takes too long…

I have a project with about the same number of models/tables (about 77), and approximately 350 tests and takes 1 minute total to run everything. Deving in a vagrant machine with 2 cpus allocated and 2GB of ram. Also I use py.test with pytest-xdist plugin for running multiple tests in parallel.

Another thing you can do is tell django reuse the test database and only re-create it when you have schema changes. Also you can use SQLite so that the tests will use an in-memory database. Both approaches explained here:
https://docs.djangoproject.com/en/dev/topics/testing/overview/#the-test-database

EDIT: In case none of the options above work, one more option is to have your unit tests inherit from django SimpleTestCase or use a custom test runner that doesn’t create a database as explained in this answer here: django unit tests without a db.

Then you can just mock django calls to the database using a library like this one (which admittingly I wrote): https://github.com/stphivos/django-mock-queries

This way you can run your unit tests locally fast and let your CI server worry about running integration tests that require a database, before merging your code to some stable dev/master branch that isn’t the production one.

If you’re using Postgres, use a Postgres template to store a copy of an otherwise empty database with all migrations applied. Then configure Django to use that template during test database creation.

One way to do this:

1. Run Django’s test command with the --keepdb argument.
2. On Postgres, rename the created test database using the query

alter database "test_your_db_name" rename to "test_your_db_name_template";

3. On Postgres, make the database a template using the query

alter database "test_your_db_name_template" IS_TEMPLATE = true

4. Tweak your test settings using something like:

DATABASES["default"]["TEST"] = {}
DATABASES["default"]["TEST"]["TEMPLATE"] = "test_your_db_name_template"

5. Run the Django tests again without the --keepdb argument. Since the migrations were already applied to the template database, they will not have to run again, so the tests will start pretty much immediately.

See also the documentation:

• Django TEMPLATE database setting

I also run into issue, One solution what I did is to subclass the Django.TestCase –> create subclass of Django.TestCase

and overwritten the method like :

@classmethod
def _databases_support_transactions(cls):
return True

the backend DB is apache cassandra ..