Skip to content

Tests

  • Tests are written using Pytest. Tests are split at the top folder level into unit, e2e_integration and migration tests (alongside some shared fixtures).
  • The distinction between unit and e2e_integration here is any test that requires some/all of the docker compose testing stack to be running is considered an end-to-end/integration test.
  • Note that there maybe "unit like" tests inside the e2e_integration dir, primarily for convenience/legacy reasons (before this top level split was made).
  • Unit tests should follow the pattern tests/unit/<package>/<module>/<test_file_name>.py (i.e. match the module structure of the packages in the codebase).
  • migration tests cover the database migration scripts located in the Alembic migrations directory and are autogenerated by the pytest-alembic plugin. They check for things like if the database can be rolled back and forth cleanly through all migration scripts and if all db models in the code correctly match the state after applying all migration scripts (aka you haven't created a migration script for your latest model changes).
  • coverage.py can be used to analyse the code coverage of the tests, with some special setup to help capture the Python code that is run in Docker containers (fastAPI and Celery workers, specifically). For details, see the To run tests together with code coverage analysis section below.

As we run tests for the frontend with playwright, you may need to install the browsers used by playwright for testing. You can do this by running:

playwright install

To run all tests

pytest -s

Why the -s flag?

Some tests that rely on typer's CliRunner (which is essentially Click's CliRunner) will fail without the -s flag because they rely on the output of the CLI commands to be printed to a terminal. By default, pytest captures the output of tests and only shows it if a test fails. The -s flag tells pytest to not capture the output and instead print it directly to the terminal, which is necessary for these tests to work.

You can achieve the same effect as -s using: 

FORCE_TERMINAL=1 COLUMNS=1000 pytest --run-slow

To run only unit, e2e_integration or migration tests:

pytest -s tests/unit
pytest -s tests/e2e_integration
pytest -s tests/migration

The e2e_integration tests will be slower the first time you run them as the docker images will need to be downloaded and built. If you use "-s" you'll see the status of the docker compose building steps.

To run tests together with code coverage analysis

The testing suite can be run with coverage.py through the pytest-cov plugin. By default, the code coverage analysis is performed on the Python code that is executed on the local machine. DivBase, however, consists of several containerized services that run Python code upon request from the CLI client that coverage.py does not capture out-of-the-box.

The custom solution for the DivBase testing suite is based on starting coverage.py listeners on the host and inside the containers, capturing the executed lines of code in separate coverage result files, and then merging them to a single deduplicated report with coverage combine.

There is some complexity to this setup so the reccomended way to run the coverage analysis is with this wrapper script:

scripts/run_tests_with_coverage.sh

# The wrapper script accepts pytest args
# Example: ./scripts/run_tests_with_coverage.sh --run-slow

This runs pytest -s tests/ --coverage-docker --cov --cov-branch --cov-context=test --cov-report=term-missing (where --coverage-docker is a custom option implemented in the DivBase testing suite to apply the docker/divbase_compose.tests.coverage.yaml overlay), ensures that the docker compose stack stops gracefully to trigger coverage results collection before terminating the containers, ensures all intermediate coverage results files are collected in docker/coverage-data/, combines them with coverage combine into a single results file, and builds an HTML report with per-test context.

Important

The wrapper script will print coverage results to the terminal after the pytest run has finished. This is the code coverage of the local machine Python process. For a complete report that includes code coverage of the containers too, please refer to the HTML report created by the wrapper script.

The coverage analysis report is set up to track which tests trigger which line of code. This is displayed to the far right of a code line in the HTML report. However, this feature only works for code that ran on the host machine. All code that ran inside a container will say (empty). This is a limitation of this custom container coverage setup.

Note

It is also possible to analyse the code coverage of just the code that ran on the host machine (e.g. unit tests and CLI functions), i.e. ommiting the code that ran inside the Docker compose stack during the tests: 

# Run all tests, but only measure coverage on code lines that run on the host machine
pytest -s --cov

# Or just the unit tests (faster)
pytest -s tests/unit --cov