CI that keeps up
how-toby Kyle Penfound

How do I run a Buildkite pipeline locally?

The current Buildkite CLI does not run a pipeline on your laptop. To test a change before you push, do two things. First, validate pipeline.yml locally with bk pipeline validate. Second, make the pipeline portable. Put the commands, the image, and the Postgres service of each step in one place that your laptop can also run, and make pipeline.yml only call it. Dagger does this. Its functions hold the steps and the Postgres service, and dagger check runs them the same way on a laptop and on a Buildkite agent. You can then run a failed Buildkite step again on your machine, with no push.

The requirements for a pipeline you can run before you push

A Buildkite pipeline that you can test before you push meets five requirements:

  1. Valid YAML before the push. The pipeline file parses and matches the Buildkite schema, so a typo does not cost a build.
  2. One definition of the steps. The commands of each step live in one place. Your laptop and the agent both call that place.
  3. The same environment. The tests use the same Go image and the same Postgres image on the laptop and on the agent.
  4. A healthy Postgres before the tests. The integration tests start only after Postgres accepts connections.
  5. A thin pipeline file. pipeline.yml says which steps run. It does not hold the test logic, so a change to the tests does not change the YAML.

What Buildkite offers for local runs

Buildkite has tools for the first requirement and for remote test builds. None of them runs the steps of a pipeline on your laptop.

  • bk pipeline validate checks a pipeline file against the Buildkite pipeline schema. The CLI reference says that it “does not require an API token since validation is done locally.”
  • buildkite-agent pipeline upload --dry-run parses and interpolates the pipeline, then prints it instead of an upload (agent reference). It shows the result of interpolation, such as $$VAR, which the schema check does not show.
  • bk local run was the local runner of the old CLI. A Buildkite maintainer wrote that “The local run isn’t in 3.0.0” because it fell out of sync with the agent (issue #79). The community tool bkl continues it, and its README lists input steps, if conditionals, and vendored plugins as not working.
  • bk preflight is an experimental command. It pushes your uncommitted changes to a bk/preflight/ branch on origin, starts a build, and reports failures in your terminal (Preflight). The build runs on your Buildkite agents, not on your laptop.
  • buildkite-agent bootstrap runs one job locally. Its documented example first loads the environment of a job from the Buildkite API, so it needs a job that already exists.

Validate pipeline.yml before you push

Run the schema check from the repository root. It reads .buildkite/pipeline.yml by default:

bk pipeline validate
Validating 1 pipeline file(s)...

✅ Pipeline file is valid: .buildkite/pipeline.yml

All pipeline files passed validation successfully!

A misspelled key fails the check with exit code 1. For a step with comand: instead of command: in bad.yml, bk pipeline validate -f bad.yml printed:

Validating 1 pipeline file(s)...

❌ Pipeline file is invalid: bad.yml

- steps > 0: Must validate at least one schema (anyOf) (field: steps.0)
- steps > 0: Additional property comand is not allowed (field: steps.0)

1 of 1 file(s) failed validation.

To see the pipeline after interpolation, use the dry run of the agent. In this guide’s test, agent v4.8.1 stopped with Missing agent-access-token when no token was set, also with --dry-run. A placeholder value worked:

BUILDKITE_AGENT_ACCESS_TOKEN=placeholder buildkite-agent pipeline upload --dry-run --format yaml

Both checks look only at the YAML. Neither one runs a command of the pipeline.

Run the steps by hand with Docker Compose

Teams usually run the steps on a laptop with Docker Compose. The Buildkite Docker Compose plugin runs the command of a step in a Compose service. Buildkite’s docs say that a step with run: app is the equivalent of docker-compose run app test.sh. Put Postgres and the Go container in docker-compose.yml:

services:
  db:
    image: postgres:16
    environment:
      POSTGRES_PASSWORD: postgres
    healthcheck:
      test: ["CMD", "pg_isready", "-U", "postgres"]
      interval: 2s
      retries: 15

  app:
    image: golang:1.27-alpine
    working_dir: /app
    volumes:
      - .:/app
    environment:
      DATABASE_URL: postgres://postgres:postgres@db:5432/postgres?sslmode=disable
    depends_on:
      db:
        condition: service_healthy

The healthcheck and the service_healthy condition make Compose start app only after Postgres is ready. In .buildkite/pipeline.yml, the integration step uses the plugin. The plugin’s wait option also waits for healthy dependencies:

steps:
  - label: ":go: unit tests"
    command: go test .
    plugins:
      - docker#v5.14.0:
          image: "golang:1.27-alpine"

  - label: ":postgres: integration tests"
    command: go test ./store/...
    plugins:
      - docker-compose#v5.16.0:
          run: app
          wait: true

On your laptop, run the same command as the integration step:

docker compose run --rm app go test ./store/...

This meets the third and fourth requirements for the integration step. The Postgres image and the health check are in one file that the laptop and the agent both read.

Pipeline steps need a second copy for laptops

The Compose file covers one step. The other requirements still depend on the YAML:

  • The steps have two definitions. go test ./store/... is in pipeline.yml, and the laptop command is in a README or a script. The two copies must change together.
  • The Go image is in two places. The unit test step uses the Docker plugin, and the integration step uses Compose. Each one names golang:1.27-alpine.
  • The plugins do work that a local run does not do. The Docker Compose plugin can propagate pipeline environment variables into the container, and can give prebuilt images to later steps through build metadata. A local docker compose run does neither.
  • Each new service or test package adds YAML. You can test that YAML only on an agent.

Meet the requirements with Dagger

Dagger runs each step in containers that the project defines. The Dagger engine runs these containers on your laptop, on a Buildkite agent, or on a remote server. A module is a package of checks and functions for a tool. A check is a function that dagger check runs, and that passes or fails (Checking your code). The official Go module finds every Go package and test, and runs go test as a check.

Install the Dagger CLI. From the repository root, create the workspace. Then install the Go module and the Dang SDK:

dagger init
dagger module install dagger.io/go
dagger module install dagger.io/sdk/dang

dagger init creates dagger.toml, the workspace configuration for everyone who uses the repository. Dagger also writes dagger.lock, which pins each module and image to an exact version. Commit both files.

The integration tests need Postgres. The Go module’s base setting takes the container that runs the tests, with a Go toolchain. For tests that need a database, the docs say to “wire a container with that service bound into the base setting” (Go module). A small module supplies that container. Dang is Dagger’s native language for modules. Create the dev module of the project:

dagger module init dang -y

The command creates the module in .dagger/modules/, named after the project directory, and installs it as the workspace main module. For a project in a shop directory, the module is shop-dev. In the snippets, shop-dev and ShopDev stand in for the names that dagger module init generated. Replace .dagger/modules/shop-dev/main.dang with this code:

"""
Test services for the shop project: the Postgres database of the integration tests.
"""
type ShopDev {
  """
  Postgres, with the image and password from docker-compose.yml.
  """
  postgres: Service! {
    container
      .from("postgres:16")
      .withEnvVariable("POSTGRES_PASSWORD", "postgres")
      .withExposedPort(5432)
      .asService
  }

  """
  The Go test container: the Go toolchain, with Postgres bound to it.
  """
  goTestBase: Container! {
    container
      .from("golang:1.27-alpine")
      .withServiceBinding("postgres", postgres)
      .withEnvVariable("DATABASE_URL", "postgres://postgres:postgres@postgres:5432/postgres?sslmode=disable")
  }
}

Give the Go module this container as its base:

dagger module settings go base go-test-base

The command writes a module reference to dagger.toml:

[modules.go.settings]
base = "dag://shop-dev/go-test-base"

One definition of the steps

dagger.toml and the dev module now hold the unit tests, the integration tests, and the check for stale go generate output. List the checks, with one row for each item:

dagger check -l --all
MODULE    GO-PACKAGE  GO-TEST         CHECK  DESCRIPTION
dang-sdk                              stale  staleness check: re-generate modules and clients managed by the Dang SDK
go        .                           stale  staleness check: run go generate in this package
go        store                       stale  staleness check: run go generate in this package
go        .           TestHello       test   Run this test.
go        store       TestDatabase    test   Run this test.
go        store       TestMigrations  test   Run this test.

Run every check on your laptop before you push:

dagger check
CHECKS  ✔ 4 passed
✔ dag://?check=dang-sdk/generate/stale 1.2s OK
✔ dag://?check=go/packages/generate/stale 1.4s OK
✔ dag://?check=test&go-package=. 1.1s OK
✔ dag://?check=test&go-package=store 1.2s OK

In this guide’s project, an edit to a test file that was not committed changed the result of the next dagger check. You do not need a commit to test a change.

The same environment

The tests run in golang:1.27-alpine on every machine, not in the Go installation of the host. Postgres is postgres:16, the same image as in docker-compose.yml.

A healthy Postgres before the tests

withServiceBinding makes Postgres reachable from the test container by the name postgres. Dagger health-checks a service before any client can connect, so the module needs no pg_isready loop. To use the same database with an app on your laptop, list the services, and start one with dagger start postgres:

dagger start -l
MODULE    SERVICE   DESCRIPTION
shop-dev  postgres  Postgres, with the image and password from docker-compose.yml.

Call the checks from pipeline.yml

The Buildkite pipeline now only selects which checks each step runs. Replace .buildkite/pipeline.yml:

steps:
  - label: ":go: unit tests"
    command: dagger check --check=test --go-package=.

  - label: ":postgres: integration tests"
    command: dagger check --check=test --go-package=store

  - label: ":recycle: generated files"
    command: dagger check --check=stale

This file passed bk pipeline validate. It holds no image, plugin, or service, so it does not change when the tests change. The agents need the Dagger CLI and a container runtime. Install the CLI on the agents at the same version as on your laptop, with the install command.

On each agent, the CLI starts an engine by default. On an agent that is replaced after each job, the cache of that engine is lost (Self-Hosting CI with Dagger). To keep the cache, set DAGGER_ENGINE in the pipeline env. Point it to a self-hosted engine with a persistent disk, or set it to cloud with a DAGGER_CLOUD_TOKEN to use Dagger’s Cloud Engines.

Reproduce a Buildkite failure on your laptop

When a step fails on Buildkite, its log names the failed test. The log ends with the command that runs the same selection again. For the integration step, the report was:

CHECKS  ✘ 1 failed
✘ dag://?check=test&go-package=store 1.4s ERROR
  TESTS
    ✘ example.com/shop/store FAIL

    ✘ example.com/shop/store › TestMigrations FAIL
              store_test.go:24: latest migration = "0007_orders", want "0008_orders"

    ✘ 1 failed
    ✔ 1 passed

RUN LOCALLY
  dagger check "dag://?check=test&go-package=store"

Copy the RUN LOCALLY command into a terminal on your laptop. It runs the same tests, in the same Go image, against the same Postgres. To run only the failed test, select it by name:

dagger check --go-package=store --go-test=TestMigrations

Run the checks with no pipeline file

If the repository is on GitHub, Cloud Checks can run the same checks without a Buildkite pipeline. Cloud Checks runs your checks on Dagger’s Cloud Engines for each push and pull request, with no CI configuration file. Turn it on from the repository root:

dagger cloud checks on

Each push then runs the checks that dagger check runs on your laptop.

Bottom line: The current Buildkite CLI does not run pipeline steps on your laptop. bk pipeline validate and buildkite-agent pipeline upload --dry-run check the YAML, and bk preflight runs your changes on your agents. To run the steps before you push, put the steps, the Go image, and Postgres in Dagger. dagger check then runs them on your laptop, and pipeline.yml calls the same command on Buildkite.