[Fixed]-Django test runner fails in virtualenv on Ubuntu

Short answer

The fix in my code was to use os.path.realpath to get a canonicalized version of the installed package path, and pass this value on the command line that invokes the django-admin utility. In my case it looks something like this:

approot = os.path.dirname(os.path.dirname(os.path.realpath(__file__)))

and:

with ChangeCurrentDir(approot):
    subprocess_command = (
        "django-admin test --pythonpath=%s --settings=%s --top-level-directory=%s"%
            (approot, settings_module_name, approot)
        )
    status = subprocess.call(subprocess_command.split())

(where ChangeCurrentDir is a context handler that runs the enclosed code with a specified current working directory).

More details

Some further experimentation showed that I could “fix” the problem by strategic replacement of os.path.abspath with os.path.realpath in the Python and/or Django library code.

The core problem I found was in:

/usr/lib/python2.7/unittest/loader.py

Specifically:

File "/usr/lib/python2.7/unittest/loader.py", line 267, in _find_tests
    raise ImportError(msg % (mod_name, module_dir, expected_dir))
ImportError: 'test_entity' module incorrectly imported from ...

The offending code in loader.pyleading up to this is:

        if realpath.lower() != fullpath_noext.lower():
            module_dir = os.path.dirname(realpath)
            mod_name = os.path.splitext(os.path.basename(full_path))[0]
            expected_dir = os.path.dirname(full_path)
            msg = ("%r module incorrectly imported from %r. Expected %r. "
                   "Is this module globally installed?")
            raise ImportError(msg % (mod_name, module_dir, expected_dir))

If I replace the if statement with this:

        if os.path.realpath(realpath).lower() != fullpath_noext.lower():

Then everything is happy. This confirms that it is a symlink aliasing problem, as os.path.realpath() resolves any symlinks to the underlying path. But this is not a solution for an installable software package, as it involves modifying the underlying Python installation. So, having probed the underlying problem, I need for something more accessible to attack.

Next port of call was the Django test runner library, which is installed in the virtual environment.

<base-env>/local/lib/python2.7/site-packages/django/test/runner.py

In particular, focusing on this part of the stack trace:

  File "/home/annalist/anenv/local/lib/python2.7/site-packages/django/test/runner.py", line 96, in build_suite
    tests = self.test_loader.discover(start_dir=label, **kwargs)

Digging around this code, I was able to identify the problem was related to the label parameter, which defaults to '.' (i.e. current directory). No easy fix is apparent here, but it suggests the problem may be related to the current directory and/or path used when running django-admin. This led to the above solution (which may be overkill – I’m not sure that the --pythonpath= option is needed, but it is working for me).