CI that keeps up
how-toby Kyle Penfound

How do I run pytest the same way locally and in CI?

To run pytest the same way on a laptop and in CI, meet the same requirements in both places. A good setup has five requirements: the same pinned environment, a result for each test, the test services, the same test selection, and parallel runs. By hand, teams meet them with uv, a CI job, and CI service containers. Then the Python version and the services live in more than one place. Dagger meets the same requirements in one place. Its official pytest module runs the suite in a container with uv, and Cloud Checks run the same check on each push.

What a good pytest setup needs

A pytest setup that runs the same on a laptop and in CI has five requirements. Each one comes from the docs of pytest or of the tools around it:

  1. The same pinned environment. Each machine runs the same Python, pytest, and dependency versions, from a lockfile. The pytest docs recommend a virtual environment, because it “ensures your code and dependencies are isolated from your system Python installation”. The tests import the project as an installed package, as with the src layout.
  2. A result for each test. CI shows which test failed, not only that the job failed. pytest writes JUnit XML for that purpose, as “result files which can be read by Jenkins or other Continuous integration servers”.
  3. The test services. A test that needs a database gets one, at an address that it reads from the environment. Where the database is not available, the test skips with skipif and does not fail. A skip means that “you expect your test to pass only if some conditions are met”.
  4. The same test selection. A developer and CI select tests with the same options: a file, a keyword with -k, or a marker with -m. “Pytest supports several ways to run and select tests” from the command line.
  5. Parallel runs. A large suite uses all the CPUs of the machine. With the pytest-xdist plugin and -n auto, “pytest-xdist will use as many processes as your computer has physical CPU cores”.

Meet the requirements by hand with uv

On a laptop, uv meets “the same pinned environment”. It reads the Python version from a .python-version file and installs the versions in uv.lock. With --locked, “if the lockfile is not up-to-date, uv will raise an error instead of updating the lockfile”. uv installs the project itself only when it declares a build system, so pyproject.toml needs a [build-system] table. Install the environment and run the tests:

uv sync --locked
uv run pytest -q -rs
Resolved 13 packages in 11ms
Audited 10 packages in 0.16ms
..s                                                                      [100%]
=========================== short test summary info ============================
SKIPPED [1] tests/test_db.py:8: DATABASE_URL not set
2 passed, 1 skipped in 2.00s

-rs prints the reason for each skip. test_database skips, because DATABASE_URL is not set. To give it a database, a developer starts Postgres, for example with Docker Compose, and sets the variable. How do I run integration tests against real databases and Kafka without mocks? compares Compose, Testcontainers, and CI service containers for that step.

Select tests with pytest’s own options. This command skips the tests with the slow marker:

uv run pytest -q -m "not slow"
.s                                                                       [100%]
1 passed, 1 skipped, 1 deselected in 0.00s

For parallel runs, add pytest-xdist to the dev dependencies and give -n auto:

uv run pytest -q -n auto
bringing up nodes...
bringing up nodes...

.s.                                                                      [100%]
2 passed, 1 skipped in 2.62s

For a result for each test, add --junit-xml=report.xml. pytest then writes a JUnit file that a CI system can read.

Run pytest in a CI job

A CI job runs the same uv commands. On GitHub Actions, the uv docs install uv with the astral-sh/setup-uv action, then run uv sync --locked and uv run pytest (Using uv in GitHub Actions). The action pins the uv version with its version input. For the database, a services: block in the job starts Postgres before the steps (service containers). The GitHub guide for Python writes the JUnit file with --junitxml and keeps it with actions/upload-artifact (Building and testing Python).

Where the hand-built setup strains

The commands work, but the setup grows harder to keep the same in both places:

  • The same pinned environment. The Python version can live in .python-version and in the workflow. A matrix version in setup-uv “will override the Python version specification in the pyproject.toml or .python-version files”. Each laptop runs the uv version that its developer installed.
  • A result for each test. The JUnit file reaches the reader as a workflow artifact or through another report step. On a laptop, the result is the terminal output.
  • The test services. CI service containers run only in CI. A laptop needs a second definition of each service, such as a Compose file, and the two can drift apart.
  • Parallel runs. pytest-xdist uses the CPUs of one machine. To use more machines, the workflow needs a matrix and a way to divide the test files between the jobs.

The same pinned environment, with the Dagger pytest module

The Dagger pytest module runs pytest in a container that has Python and uv. A Dagger module is a package of functions and checks that Dagger runs in containers. The project must be a Git repository with a Dagger workspace. A workspace is the dagger.toml file that lists the modules a repository uses, and dagger init creates it. Install the official module for pytest:

dagger module install dagger.io/python/pytest
dagger.io/python/pytest ➡️ https://github.com/dagger/python#v0.2.0:pytest
Installed module "pytest" in /work/dagger.toml

The command adds a [modules.pytest] entry to dagger.toml. It also writes dagger.lock, which pins the module and the images the workspace resolves to exact versions.

The module looks for pytest projects. A project is a directory that holds pytest.ini, pyproject.toml, tox.ini, or setup.cfg, and that has test files of its own. A test file is named test_*.py or *_test.py and defines a test function or a Test class.

The container has Python 3.14 by default. Set the version setting to match the project:

dagger module settings pytest version 3.13

A project that does not pin pytest gets pytest 9.1.1, from the defaultVersion setting. If the project has a uv.lock, the module installs from it with --locked. A lockfile that is out of date then fails the environment (module README).

The module uses uv, so the build system rule from the hand-built setup applies here too. If the tests import the project’s own package, as with a src/ layout, pyproject.toml needs a [build-system] table. Without one, collection fails:

E   ModuleNotFoundError: No module named 'shop'

A result for each test, from the module’s trace

dagger check reports each test by name, on a laptop and in CI. The module adds one check, test. A check is a function that passes or fails, and dagger check runs every check in the workspace. List the checks:

dagger check -l
MODULE  PYTEST-PROJECT  PYTEST-TEST-FILE  CHECK  DESCRIPTION
pytest  *               *                 test   Run this test file.
# Use --all to expand collections and list each item.

Run the tests:

dagger check
== CHECKS ==  ✔ 1 passed
✔ dag://?check=test&pytest-project=. 16.0s OK
  == TESTS ==
    ∅ tests/test_db.py › test_database SKIP
          Skipped: DATABASE_URL not set

    ✔ tests/test_cart.py  2 passed

    ∅ 1 skipped
    ✔ 2 passed

The module runs one pytest command per project. It adds a bundled OpenTelemetry plugin, so each test shows as its own result, with no change to the project. The tracing setting loads the plugin, and it is on by default. Dagger Cloud shows the same tests from the trace (pytest module). A failure names the test and shows the pytest assertion. It also gives a command that runs the same selection again:

== CHECKS ==  ✘ 1 failed
✘ dag://?check=test&pytest-project=. 3.1s ERROR
  == TESTS ==
    ✘ tests/test_cart.py › test_total FAIL
          def test_total():
          >       assert add(2, 2) == 5
          E       assert 4 == 5
          E        +  where 4 = add(2, 2)
          
          tests/test_cart.py:9: AssertionError

    ∅ tests/test_db.py › test_database SKIP
          Skipped: DATABASE_URL not set

    ✘ 1 failed
    ∅ 1 skipped
    ✔ 1 passed

== RUN LOCALLY ==
dagger check "dag://?check=test&pytest-project=."

The test services, with a test runtime

If the tests need Postgres, give the module a test container that has the database bound to it. The base setting takes that container. A function in a module of your own supplies it, and Dagger gives the function’s result to the pytest module. The docs call this module wiring. This is the Compose model: one definition of each service, outside the tests.

The tests keep the conventions from the hand-built setup, so they have no Dagger code:

  • The tests read the database address from an environment variable, such as DATABASE_URL.
  • The tests skip with skipif when the variable is not set, as in the SKIP result above.

Install the Dang SDK and create the project’s dev module. Dang is Dagger’s own language for modules:

dagger module install dagger.io/sdk/dang
dagger module init dang -y

dagger module init creates the module at .dagger/modules/<project>-dev and makes it the workspace main module. Replace the contents of .dagger/modules/<project>-dev/main.dang with this code. ShopDev stands in for the type name that dagger module init generated, so keep your generated name.

type ShopDev {
  """
  A Postgres database for development and tests.
  """
  postgres: Service! {
    container
      .from("postgres:17")
      .withEnvVariable("POSTGRES_PASSWORD", "test")
      .withExposedPort(5432)
      .asService
  }

  """
  The container pytest runs in, with Python, uv, and Postgres.
  """
  testRuntime: Container! {
    container
      .from("ghcr.io/astral-sh/uv:python3.13-trixie-slim")
      .withServiceBinding("db", postgres)
      .withEnvVariable("DATABASE_URL", "postgresql://postgres:test@db:5432/postgres")
  }
}

testRuntime starts from an official uv image with Python. It binds Postgres under the hostname db and sets the variable that the tests read. Each container that the pytest module builds from this base keeps the binding, so Postgres runs whenever pytest runs. postgres returns a Service, so dagger start can also start it for local development:

dagger start -l
MODULE    SERVICE   DESCRIPTION
work-dev  postgres  A Postgres database for development and tests.

The MODULE column shows the name of your dev module. This guide’s scratch project is named work, so its module is work-dev.

The image in base sets the Python version, so you cannot set both version and base. With both set, dagger check fails:

! evaluating new() for Pyproject: version and base are mutually exclusive

Remove version if you set it, then wire the function into base:

dagger module settings -u pytest version
dagger module settings pytest base test-runtime

The command accepts a bare function name from the workspace main module and stores its full address. dagger.toml now reads:

[modules.pytest]
source = "dagger.io/python/pytest"

[modules.pytest.settings]
base = "dag://<project>-dev/test-runtime"

[modules.dang-sdk]
source = "dagger.io/sdk/dang"

[sdks.dang]
module = "dang-sdk"

[modules.<project>-dev]
main = true
source = ".dagger/modules/<project>-dev"

[sdks.dang.scopes.".dagger/modules/<project>-dev"]
is-module = true

Run the database test again. It passes now, where it skipped before:

dagger check --pytest-test-file=tests/test_db.py
== CHECKS ==  ✔ 1 passed
✔ dag://?check=test&pytest-project=.&pytest-test-file=tests/test_db.py 14.7s OK
  == TESTS ==
    ✔ tests/test_db.py  1 passed

    ✔ 1 passed

== SERVICES ==
✔ q5q0otklmb4uk.8j704dlocoubk.dagger.local exec docker-entrypoint.sh postgres 4.2s EXITED span=5ee7d0524e043670

The SERVICES section lists the Postgres service that the check started.

Some tests start their own containers, as with Testcontainers. Dagger supports that model too. The tests call the Dagger API, and the nesting setting lets them reach a nested Dagger engine. It is on by default. Start services from test code with a Dagger client shows this model.

The same test selection, with check flags

The pytest module turns projects and test files into flags for dagger check, and it passes pytest’s own options through the args setting. Projects and test files are collections. Add --all to list one row per test file:

dagger check -l --all
MODULE    PYTEST-PROJECT  PYTEST-TEST-FILE    CHECK  DESCRIPTION
dang-sdk                                      stale  staleness check: re-generate modules and clients managed by the Dang SDK
pytest    .               tests/test_cart.py  test   Run this test file.
pytest    .               tests/test_db.py    test   Run this test file.

The stale check comes from the Dang SDK, which you installed for the dev module.

Select tests by test file:

dagger check --pytest-test-file=tests/test_db.py
== CHECKS ==  ✔ 1 passed
✔ dag://?check=test&pytest-project=.&pytest-test-file=tests/test_db.py 14.7s OK
  == TESTS ==
    ✔ tests/test_db.py  1 passed

    ✔ 1 passed

== SERVICES ==
✔ q5q0otklmb4uk.8j704dlocoubk.dagger.local exec docker-entrypoint.sh postgres 4.2s EXITED span=5ee7d0524e043670

With every file of a project selected, the module runs pytest on the whole project. With some files selected, it gives only those files to pytest. Repeat the flag to select more than one file.

In a repository with several projects, each project root is a key. List the keys:

dagger list pytest-projects -a
MODULE  PYTEST-PROJECT   DESCRIPTION
pytest  services/api     The pytest project rooted at path.
pytest  services/worker  The pytest project rooted at path.

Run one project:

dagger check --pytest-project=services/api
== CHECKS ==  ✔ 1 passed
✔ dag://?check=test&pytest-project=services/api 9.2s OK
  == TESTS ==
    ✔ tests/test_api.py  2 passed

    ✔ 2 passed

The flags select whole files. To select tests by marker or keyword, give pytest the same options as by hand through args. The default is ["-v"].

The scope setting selects project roots, and its default is ["**"]. A ! pattern excludes a project, and an exclusion always wins. A list value is one JSON argument. This example skips tests with the slow marker and excludes a docs project:

dagger module settings pytest args '["-v", "-m", "not slow"]'
dagger module settings pytest scope '["**", "!docs"]'

dagger.toml then holds the settings, so every machine selects the same tests:

[modules.pytest.settings]
args = ["-v", "-m", "not slow"]
scope = ["**", "!docs"]

Select collection items covers every selection form.

Parallel runs, across projects and engines

The pytest module tests each selected project in parallel, and it reports every project that fails. Inside a project, the module runs one pytest command. If the project depends on pytest-xdist, give -n auto through args, as by hand. pytest-xdist then starts one worker for each physical CPU core:

dagger module settings pytest args '["-v", "-n", "auto"]'

For a large suite, Dagger splits the tests across engines. It balances the parts with the measured resource use, input cost, and cache state of each test (test splitting). You write no CI matrix for it.

Run pytest locally and with Cloud Checks

dagger check runs the same check on a laptop and in Dagger Cloud. Dagger caches each step by its inputs. In this guide’s project, the first dagger check took 12.7 seconds. A rerun with no changes took 2.5 seconds. A rerun after an edit to one test file took 6.8 seconds, because the dependency install stayed in the cache. These numbers come from one run on one machine, not from a benchmark.

Cloud Checks run your checks on Dagger’s Cloud Engines for each push and pull request to a GitHub repository, with no CI workflow file. Commit the workspace configuration, the lock file, and the dev module:

git add dagger.toml dagger.lock .dagger
git commit -m "Run pytest with Dagger"

Turn on Cloud Checks from the project root:

dagger cloud checks on

The command finds the repository from its Git origin remote. In a terminal, it offers to create a Dagger Cloud account and to connect GitHub if those steps are not done. Push to start the checks. The results show as GitHub check statuses and in Dagger Cloud, and dagger workspace activity lists the recent runs.

Bottom line: To run pytest the same way locally and in CI, pin Python, pytest, and the dependencies, report each test, and give the tests their services. Select tests with the same options everywhere, and run them in parallel. uv, a CI job, and CI service containers do this by hand, with the settings in several files. The Dagger pytest module does it from one dagger.toml, and Cloud Checks run the same dagger check on each push.