I’ve set up code coverage tools for python many times before, but have always
found the process rather cumbersome and difficult to understand with fragmented
and frequently incorrect information out there. In an effort to improve this
situation, I decided to write this article focusing on producing coverage with
my usual tools of choice:
pytest-cov (which uses
Your mileage my vary with different test runners, but the issues flagged up here will help debug coverage collection issues for any test runner.
One of the main problems arising in collecting code coverage in python is
not knowing which version of the code you’re running your tests against. Now you
might think this is ridiculous, and how could you not know which version of code
you’re running tests against, but with tools like
tox that install your package into a
virtualenv per python interpreter version, you’ll not be testing against the
code in the root of your project folder, but rather that contained in
.tox/pyXX/lib/pythonX.X/site-packages/mypackage. This is quite easy to
overlook and can result in frustration when you end up 0% coverage metrics.
To aid your experimentation/debugging I’ve also put together a
repo with example
project layouts showing the results of running
pytest in different ways with
different coverage configurations
The most common set up runs tests against the source code present in your project root. For this, the tools work pretty well and you’ll only encounter a few hiccups.
We’ll assume you have a package structure like this:
├── LICENSE ├── MANIFEST.in ├── README.rst ├── setup.py ├── tests │ ├── test_blah.py │ └── test_foo.py └── mypackage ├── __init__.py ├── package_a ├── package_b └── module_c.py
Running your tests would be accomplished by executing
PYTHONPATH=. pytest. The
reason for having to use
PYTHONPATH=. is that be default
pytest will remove
sys.path to avoid testing against the in-src package. This
behaviour holds, unless your
tests directory is a package, i.e. it contains a
__init__.py, quoting the
… But now this introduces a subtle problem: in order to load the test modules from the tests directory, pytest prepends the root of the repository to sys.path, which adds the side-effect that now mypkg is also importable. This is problematic if you are using a tool like tox to test your package in a virtual environment, because you want to test the installed version of your package, not the local code from the repository.
Coverage can be collected by invoking
pytest --cov=mypackage tests.
Whilst these approaches are suitable if you’re happy with collecting coverage against
your development install, you should probably be running your tests against an
installed version of your package to pick up any issues present in the packaging
phase of a build (e.g. missing files). We cover this approach in the next
section, but before moving on, a quick note about using a
src project layout
which can help mitigate some of the confusing behaviour when attempting to test
against the installed package.
The benefit of using a
src based layout is that your package is no longer
importable from the root project directory by default. This makes diagnosing
coverage collection issues considerably easier, especially when collecting
coverage against installed versions of your package.
If we modified the repository structure from the previous section to use a
layout, we’d end with this:
├── LICENSE ├── MANIFEST.in ├── README.rst ├── setup.py ├── tests │ ├── test_blah.py │ └── test_foo.py └── src └── mypackage ├── __init__.py ├── package_a ├── package_b └── module_c.py
Now running tests would explicitly require the setting of
PYTHONPATH if we
wish to test against the development version we run
PYTHONPATH=src pytest, and
to collect coverage:
PYTHONPATH=src pytest --cov mypackage
Cristin Maries has a
good blog post
on the benefits of using a
project layout which is worth reading. I personally use
src layouts for all my
new python projects as I’ve found it results in fewer headaches debugging
Whilst I’ve advocated testing against installed packages, there are a few issues
to be aware of in collecting coverage from an installed package.
The coverage paths (those reported by
coverage report and present in the
.coverage file) will be those to the installed version of package.
To get nice reports that are like
package/module.py rather than
you’ll need to tell
coverage.py that these paths are equivalent, and then run
coverage combine command after running tests with coverage collection to
rewrite the long paths to short paths. To do
this we leverage the
[paths] configuration section of
[paths] source = mypackage **/site-packages/mypackage
Collecting coverage now and generating a report will go something like this:
$ pytest --cov mypackage $ coverage combine $ coverage report
The coverage report produced by
pytest will have the long paths, but once
coverage report will have the short paths.
If you’re having issues with collecting coverage, I urge you to check out the accompanying repository to this post that has example projects with multiple layouts and shows common mistakes in configuring coverage and their fixes.