1👍
Well, it took me a lot of debugging, but I figured out the problem…
Our 10,000 tests take a long, long time to run, unless we run them in parallel processes. So we use a tool that splits up the nose tests into 20 parallel processes and runs the tests in groups. (This still takes roughly 20 minutes to complete, but it’s better than nearly two hours.)
We use the FastFixtureTestCase
, which extends the TransactionTestCase
. At the start of each test case, FastFixtureTestCase
calls django.test.testcases: connections_support_transactions()
. That function loops over all the DATABASES
connections and calls supports_transactions
on each one. My mistake was assuming that supports_transactions
should be an inherently safe operation. It’s not.
supports_transactions
does the following things:
- Creates a new table
- Commits
- Inserts a value into that table
- Rolls back
- Selects the number of rows in that table
- Drops the table
- Commits
- Returns
True
if the number of rows in the table was 0 (meaning the rollback succeeded, so transactions must be supported).
This is not safe. This is very dangerous. No two processes or servers can run this at the same time against the same database. If two or more processes run this function at the same time, at best, one will return True
and the other(s) will raise an exception. At worst, all will raise an exception.
In my case, because we have so many test cases, most of the time the processes were avoiding executing connections_support_transactions
at the same time, but when they did, it resulted in a small handful of random failures that were different each time.
One possible solution is to use SimpleTestCase
instead of FastFixtureTestCase
, as @kmmbvnr pointed out. However, this is not an option for us, as our entire test infrastructure depends on what FastFixtureTestCase
does to the rest of our databases. So, instead, I overrode supports_transactions
just for the static, shared database with the following line of code, and the errors went away:
connections['geo'].features.supports_transactions = True