CI that keeps up
how-toby Kyle Penfound

How do I split Go integration tests across CI runners?

To shard Go integration tests safely, put each test package in exactly one shard. Give each shard its own fresh Postgres, and start the tests only after the database accepts connections. Keep the Go module cache and build cache between runs, and run the same shards on a laptop and in CI. A CI matrix with service containers meets most of these requirements. Dagger meets all of them in one module. Each shard is a check with its own Postgres service, and dagger check runs the shards in parallel the same way locally and in CI.

What a sharded integration test setup needs

A good setup for sharded Go integration tests meets five requirements. The sections that follow use these names:

  • Every package in exactly one shard. The split must be deterministic, with no overlap and no skipped package. Depot describes its own splitter this way: each shard gets “a different, non-overlapping subset of the candidates” (Split tests by timing).
  • A database for each shard. Each shard gets a new Postgres server, so tests in one shard cannot see the tables of another shard. Packages in one shard share that server. By default, go test runs packages in parallel. The -p n flag sets “the number of programs, such as build commands or test binaries, that can be run in parallel”, and the default is GOMAXPROCS (cmd/go). Run the packages of a shard one at a time with -p 1, or give each package its own database name.
  • Healthy before the tests start. The tests start only after Postgres accepts connections. GitHub’s Postgres example sets health checks “to wait until postgres has started” (Creating PostgreSQL service containers).
  • Cached Go modules and builds. Each shard reuses downloaded modules and compiled packages from earlier runs, as the GitHub action setup-go does when it caches “Go modules and build outputs” (actions/setup-go).
  • The same shards locally and in CI. When shard 3 fails in CI, a developer can run shard 3 on a laptop, with the same database.

Shard the tests by hand with a CI matrix and service containers

A CI matrix job with a services: block meets the first four requirements. GitHub Actions creates one job for each matrix value. It “creates a fresh Docker container for each service configured in the workflow, and destroys the service container when the job completes” (Communicating with Docker service containers). Thus each shard gets its own Postgres.

Depot CI runs the same YAML. Its compatibility table marks jobs.<job_id>.strategy and jobs.<job_id>.services as supported. Depot CI treats labels that are not Depot labels as depot-ubuntu-latest (Depot CI compatibility). Put this workflow in .github/workflows/ for GitHub Actions, or in .depot/workflows/ for Depot CI (Depot CI quickstart):

name: Integration tests
on: [push, pull_request]

jobs:
  integration:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        shard: [0, 1, 2, 3]
    services:
      postgres:
        image: postgres:17-alpine
        env:
          POSTGRES_PASSWORD: test
        ports:
          - 5432:5432
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5
    env:
      DATABASE_URL: postgres://postgres:test@localhost:5432/postgres
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-go@v7
        with:
          go-version-file: go.mod
      - name: Run shard ${{ matrix.shard }}
        run: |
          go list -tags=integration -f '{{if or .TestGoFiles .XTestGoFiles}}{{.ImportPath}}{{end}}' ./... \
            | awk -v i=${{ matrix.shard }} -v n=4 'NR % n == i' \
            | xargs -r go test -tags=integration -count=1 -p 1

Each part of the workflow meets a requirement:

  • Every package in exactly one shard. go list prints the packages that have tests, in a stable order. awk keeps every fourth line, and each shard starts at a different line.
  • A database for each shard. Each matrix job gets its own service container. -p 1 runs the packages of the shard one at a time against that database.
  • Healthy before the tests start. The --health-* options hold the steps until pg_isready passes.
  • Cached Go modules and builds. setup-go caches modules and build outputs, and caching is on by default (actions/setup-go).

On Depot CI and Depot runners, depot tests run can replace the awk split. It balances the shards by the time that each package took in earlier runs (Split tests by timing).

Where the hand-built shards strain

The matrix meets four requirements in CI, but the same shards locally and in CI is harder. A service container belongs to a CI job. To run shard 3 with plain go test on a laptop, a developer needs a second definition of Postgres, such as a Compose file. To run several shards on one machine, each copy needs its own Compose project name, which Compose uses to “prevent interference between builds” (Specify a project name). Then the image tag, the password, and the health check live in two files that must change together.

Every package in exactly one shard also gets harder as the suite grows. A split by count ignores how long each package takes, so one slow package makes its shard the last to finish. Depot’s timing-based split solves that. Its commands are “supported only in Depot CI and Depot GitHub Actions runner jobs” (Split tests by timing). Depot users can test a workflow against the local working tree without a push, with depot ci run (Depot CI quickstart).

Shard the tests with Dagger

Dagger meets the same five requirements in one place. The setup is a Dagger module, a package of functions, checks, and services that the project can run. The steps need a Dagger workspace, the dagger.toml file that holds the configuration of the project. If the project has no dagger.toml, do the Quickstart first. The project must be a Git repository.

First, install the official Go module, dagger.io/go. It runs go test on each package as a separate check:

dagger module install dagger.io/go
dagger check --go --check=test

The integration tests need a database for each shard, so they go in a check of your own. The Go module’s base setting gives every package one test container. That fits tests that share one database. Shards that each need their own Postgres get a small module instead.

Write the check in Dang, Dagger’s native language for modules. Install the Dang SDK and create the dev module of the project:

dagger module install dagger.io/sdk/dang
dagger module init dang

dagger module init creates the module at .dagger/modules/<project>-dev and makes it the workspace main. You can call the functions of the workspace main without the module name. The command returns a changeset, a diff that Dagger shows for review before it writes files. Pass -y in a headless run.

The shard module in Dang

The dev module finds the integration test packages, splits them into shards, and gives each shard a check with its own Postgres. The project in this guide is named orders, so the module is orders-dev and its type is OrdersDev. Keep the type name that dagger module init generated. Replace .dagger/modules/<project>-dev/main.dang with this code:

"""
Integration tests for the orders project, split into shards.
"""
type OrdersDev {
  pub source: Directory!

  """
  The number of integration test shards.
  """
  pub shards: Int!

  new(ws: Workspace!, shards: Int! = 4) {
    self.source = ws.directory("/", exclude: [".git", ".dagger"])
    self.shards = shards
    self
  }

  """
  Integration test shards, keyed by shard number.
  """
  pub integration: Shards! {
    let numbers = [] :: [Int!]!
    let n = 1
    loop {
      if (n > shards) { break }
      numbers += [n]
      n += 1
    }
    let packages = source
      .glob("**/*_integration_test.go")
      .map { f => "./" + Path(f).parent.string }
      .uniq
    Shards(numbers: numbers, total: shards, packages: packages, source: source)
  }
}

"""
Integration test shards, keyed by shard number.
"""
type Shards @collection {
  pub numbers: [Int!]! @keys
  let total: Int!
  let packages: [String!]!
  let source: Directory!

  """
  The shard with this number.
  """
  pub shard(number: Int!): Shard! @get {
    Shard(number: number, total: total, packages: packages, source: source)
  }
}

"""
One shard of the integration tests, with its own Postgres.
"""
type Shard {
  pub number: Int!
  let total: Int!
  let packages: [String!]!
  let source: Directory!

  """
  The packages in this shard.
  """
  pub assigned: [String!]! {
    packages
      .map { p, i => {{ p: p, keep: i % total == number - 1 }} }
      .filter { x => x.keep }
      .map { x => x.p }
  }

  """
  A fresh Postgres for this shard only.
  """
  pub postgres: Service! {
    container
      .from("postgres:17-alpine")
      .withEnvVariable("POSTGRES_PASSWORD", "test")
      .withEnvVariable("SHARD", toString(number))
      .withExposedPort(5432)
      .asService
  }

  """
  Run the integration tests of this shard against its own Postgres.
  """
  pub test: Void @check {
    if (assigned.length == 0) {
      null
    } else {
      container
        .from("golang:1.26-alpine")
        .withMountedCache("/go/pkg/mod", cacheVolume("go-mod"))
        .withMountedCache("/root/.cache/go-build", cacheVolume("go-build"))
        .withDirectory("/src", source)
        .withWorkdir("/src")
        .withExec(["go", "mod", "download"])
        .withServiceBinding("db", postgres)
        .withEnvVariable("DATABASE_URL", "postgres://postgres:test@db:5432/postgres")
        .withExec(["go", "test", "-tags=integration", "-p", "1", "-count=1"] + assigned)
        .sync
      null
    }
  }
}

A check is a function that validates the project, and dagger check runs every check. The next sections show how this code meets each requirement.

Every package in exactly one shard, with a Dagger collection

The shards are a collection, a set of items that a module finds at run time, each with a unique key. Dagger adds a flag for each collection, so you can select one shard. Each shard takes the packages whose position, modulo the shard count, matches its number. This is the same split as the awk command. A shard with no packages passes at once.

The shard count is a constructor argument, so it is a module setting. Set it in dagger.toml for everyone who uses the repository:

dagger module settings orders-dev shards 4

The command writes shards = 4 under [modules.orders-dev.settings]. List the shards:

dagger list orders-dev-shards -a
MODULE      ORDERS-DEV-SHARD  DESCRIPTION
orders-dev  1                 The shard with this number.
orders-dev  2                 The shard with this number.
orders-dev  3                 The shard with this number.
orders-dev  4                 The shard with this number.

Show the packages in shard 2:

dagger -c 'integration | get 2 | assigned'
./catalog
./shipping

A database for each shard, with Dagger services

Each shard binds its own Postgres as a Dagger service, a container that runs for as long as a client needs it. The test container reaches it at the hostname db. Dagger starts the service when the shard needs it, and stops it when nothing references it.

The SHARD variable gives each shard a different service definition. Dagger gives each service a hostname from its definition, and clients that request the same service “share one instance” (Running services). Without the variable, the four definitions are the same, so the four shards share one Postgres.

In each shard, -p 1 runs the packages one at a time against the database of the shard, as in the hand-built version. -count=1 turns off the Go test result cache. Go calls it “the idiomatic way to disable test caching explicitly” (cmd/go). Without it, go test can show a cached result and not test against the new database.

Healthy before the tests start, with Dagger health checks

Dagger health-checks a service before any client can connect, “so there is no race between service startup and test execution” (Running services). Thus the go test command of a shard can connect to Postgres as soon as it starts. The module needs no pg_isready loop and no wait script.

Cached Go modules and builds, with cache volumes

The two cache volumes, go-mod and go-build, keep the Go module cache and the build cache between runs. Dagger also caches each step by its inputs, so a run with no changes reuses the result of each shard.

In this guide’s project, eight packages each have an integration test that waits 3 seconds. One script measured these times:

  • With shards set to 1, the first run took 36 seconds.
  • With shards set to 4, the first run took 8.6 seconds.
  • A rerun with no changes took 0.2 seconds.
  • After a one-line change in billing/billing.go, the run took 7.2 seconds.

Every shard ran again after the change, because each shard mounts the whole source directory. These numbers come from one run on one machine. They are an example, not a benchmark.

The same shards locally and in CI, with Cloud Checks

dagger check runs the same shards, with the same databases, on a laptop and in CI. Run all four shards:

dagger check dag://orders-dev/integration
CHECKS  ✔ 4 passed
✔ dag://?check=test&orders-dev-shard=1 7.1s OK
✔ dag://?check=test&orders-dev-shard=2 7.1s OK
✔ dag://?check=test&orders-dev-shard=3 7.1s OK
✔ dag://?check=test&orders-dev-shard=4 7.1s OK

If shard 3 fails in CI, run only shard 3. In this run, Dagger reused the cached result of the earlier run:

dagger check --orders-dev-shard=3
CHECKS  ✔ 1 passed
✔ dag://?check=test&orders-dev-shard=3 0.0s OK

Commit dagger.toml, dagger.lock, and .dagger. dagger.lock pins each module and image to an exact version, so CI uses the same Postgres image that you tested with.

In CI, Cloud Checks run the same checks after each push and pull request, on Dagger’s Cloud Engines, with no CI workflow file. The repository must be on GitHub, and its origin remote must point there. Run this command from the project root:

dagger cloud checks on

The split in this guide is by count. Dagger’s test splitting splits a large test suite across engines. It balances the work with data from earlier runs, such as the CPU, memory, and transfers that each test used.

Bottom line: Give each shard a deterministic set of packages and its own fresh Postgres. Wait until the database is healthy, cache Go modules and builds, and run the same shards locally and in CI. A CI matrix with service containers, in GitHub Actions or Depot CI, meets the first four requirements in CI. A Dagger module meets all five: each shard is a check with its own Postgres service, and dagger check runs the shards in parallel on a laptop and in Cloud Checks.