Tests¶

• Tests are written using Pytest. Tests are split at the top folder level into unit, api, e2e_integration and migration tests (alongside some shared fixtures).
• The distinction between unit and e2e_integration here is any test that requires some/all of the docker compose testing stack to be running is considered an end-to-end/integration test.
• Note that there maybe "unit like" tests inside the e2e_integration dir, primarily for convenience/legacy reasons (before this top level split was made).
• Unit tests should follow the pattern tests/unit/<package>/<module>/<test_file_name>.py (i.e. match the module structure of the packages in the codebase).
• API tests are located in tests/api and test some of DivBase's API endpoints. The API tests are run against a test database (postgresql docker). The tests do not have access to the full Docker compose setup, so no Celery workers, no S3 etc.... It therefore does not make sense to test some endpoints here, as easier to just test with the full docker compose stack (e2e_integration) running instead. - and test it via the CLI commands that call these API endpoints.
• migration tests cover the database migration scripts located in the Alembic migrations directory and are autogenerated by the pytest-alembic plugin. They check for things like if the database can be rolled back and forth cleanly through all migration scripts and if all db models in the code correctly match the state after applying all migration scripts (aka you haven't created a migration script for your latest model changes).
• Some tests are prefixed with test_regression_*. These tests protect hard system contracts from accidental change during refactoring. If one fails, treat it as a potential design-level break, not just an update of a broken test. See each test docstring for details on the specific regressions they guard.
• coverage.py can be used to analyse the code coverage of the tests, with some special setup to help capture the Python code that is run in Docker containers (fastAPI and Celery workers, specifically). For details, see the To run tests together with code coverage analysis section below.

As we run tests for the frontend with playwright, you may need to install the browsers used by playwright for testing. You can do this by running:

uv run playwright install

To run all tests¶

uv run pytest
# include slow tests (these currently only test pagination is handled properly)
uv run pytest --run-slow

To only run a subset of the 4 different groups of tests we have:

uv run pytest tests/unit
uv run pytest tests/e2e_integration
uv run pytest tests/migrations
uv run pytest tests/api

The e2e_integration, migration and api tests will be slower the first time you run them as the docker images will need to be downloaded and built. If you use "-s" you'll see the status of the docker building steps.

Tips¶

1. Testing typer.Badparameter rasies

   If a divbase-cli command is expected to raise typer.BadParameter, (because the user provided invalid input or input combinations), the test should check for an exit code of 2 and that the output contains the usage message. Trying to check for specific substrings in the output message will likely fail in the GH actions runner. If you really want to test for the exact error message phrasing, consider a unit test instead. See existing tests in the codebase for how to do this.

2. Checking for long blocks of text in divbase-cli command output.

   If you want to check for a long block of text in the output of a divbase-cli command (e.g. for a test inside the tests/e2e_integration/cli_commands), you'll need to handle that the output can be split onto multiple lines by the typer.testing.CliRunner, and the exact split points depend on the width of the terminal. This leads to flaky tests as (depending on the terminal width) the assert statement will break.

   To handle this, you can cleanup the result.output (or .stdout or .stderr) first. For example:

   result = runner.invoke(app, "my divbase command")
   assert result.exit_code == 0
   output = result.output.strip().replace("\n", "")
   assert "my long block of text" in output
   

To run tests together with code coverage analysis¶

The testing suite can be run with coverage.py through the pytest-cov plugin. By default, the code coverage analysis is performed on the Python code that is executed on the local machine. DivBase, however, consists of several containerized services that run Python code upon request from the CLI client that coverage.py does not capture out-of-the-box.

The custom solution for the DivBase testing suite is based on starting coverage.py listeners on the host and inside the containers, capturing the executed lines of code in separate coverage result files, and then merging them to a single deduplicated report with coverage combine.

There is some complexity to this setup so the reccomended way to run the coverage analysis is with this wrapper script:

scripts/run_tests_with_coverage.sh

# The wrapper script accepts pytest args
# Example: ./scripts/run_tests_with_coverage.sh --run-slow

This runs pytest -s tests/ --coverage-docker --cov --cov-branch --cov-context=test --cov-report=term-missing (where --coverage-docker is a custom option implemented in the DivBase testing suite to apply the docker/divbase_compose.tests.coverage.yaml overlay), ensures that the docker compose stack stops gracefully to trigger coverage results collection before terminating the containers, ensures all intermediate coverage results files are collected in docker/coverage-data/, combines them with coverage combine into a single results file, and builds an HTML report with per-test context.

The coverage analysis report is set up to track which tests trigger which line of code. This is displayed to the far right of a code line in the HTML report. However, this feature only works for code that ran on the host machine. All code that ran inside a container will say (empty). This is a limitation of this custom container coverage setup.

# Run all tests, but only measure coverage on code lines that run on the host machine
pytest -s --cov

# Or just the unit tests (faster)
pytest -s tests/unit --cov