CI that keeps up
migrationby Kyle PenfoundTested with Dagger v1.0.0-beta.21

How do I migrate from Buildkite to Dagger?

You do not rewrite your Buildkite pipelines as Dagger code. Dagger’s official modules find the tests, linters, and builds in your repository and run them as checks. Steps that no official module covers, such as a release, become functions in a small module. Move in five steps: install the modules, run the checks on your laptop, choose what starts them, run both systems side by side, then remove each pipeline step that a check replaces.

After the move, two setups can start the checks. If the repository is on GitHub, Cloud Checks run them after each push and pull request, with no CI file. If your team keeps its Buildkite agents, Buildkite stays as the trigger, and pipeline.yml only calls dagger check.

Do I have to rewrite my Buildkite pipelines?

No. You write code only for the steps that no official module does. A Dagger module is a package of checks and functions for a tool, and the Dagger team maintains official modules for common tools. A check is a command that passes or fails, such as a test run or a linter (Checking your code). Each official module finds the files of its tool. For example, the Go module finds each go.mod, and makes each Go package and each test a check.

Today the official modules cover Go, golangci-lint, Staticcheck, Deno, Jest, Vitest, Playwright, ESLint, Prettier, Biome, pytest, ShellCheck, PSScriptAnalyzer, and Helm (module reference). A Buildkite step that runs one of these tools becomes a check with no code.

You do not copy the structure of pipeline.yml. Its steps, plugins, and wait steps tell Buildkite how to schedule jobs on agents. dagger check runs all the checks in your workspace in parallel (Checking your code).

Install the modules and run the checks locally

Install the Dagger CLI. Then, from the repository root, create the workspace:

dagger init

The workspace is the Dagger configuration of the repository, in a file named dagger.toml (Quickstart). Without a terminal, dagger init prints the next commands:

Workspace configuration initialized at ./dagger.toml

To continue setup, run these commands in order:

#!/bin/sh

# 1. Find and install suitable modules
dagger module recommend

# 2. Enable cloud checks
dagger cloud checks on

Install the modules that Dagger recommends. Add -y to install all of them with no prompt:

dagger module recommend -y

In a Go repository, the command installs the Go module and adds it to dagger.toml:

[modules.go]
source = "dagger.io/go"

The command also writes dagger.lock, which pins the versions of the modules and images (Workspaces). Commit it with dagger.toml. If Dagger recommends no module for one of your tools, search by the tool name, such as dagger module search eslint.

List the checks that the modules found:

dagger check -l
MODULE  GO-PACKAGE  GO-TEST  CHECK  DESCRIPTION
go      *                    stale  staleness check: run go generate in this package
go      *           *        test   Run this test.

A * stands for each package or test that the module found. Run all the checks:

dagger check
CHECKS  ✔ 3 passed
✔ dag://?check=go/packages/generate/stale 0.9s OK
✔ dag://?check=test&go-package=. 0.9s OK
  TESTS
    ✔ example.com/shop  1 passed

    ✔ 1 passed
✔ dag://?check=test&go-package=store 1.0s OK
  TESTS
    ✔ example.com/shop/store  1 passed

    ✔ 1 passed

When a test fails, the report names the test. It ends with the command that runs the same selection again:

CHECKS  ✘ 1 failed  ✔ 2 passed
✘ dag://?check=test&go-package=store 0.8s ERROR
  TESTS
    ✘ example.com/shop/store FAIL

    ✘ example.com/shop/store › TestGet FAIL
              store_test.go:7: Get() returned the wrong value

    ✘ 1 failed
✔ dag://?check=go/packages/generate/stale 0.7s OK
✔ dag://?check=test&go-package=. 0.7s OK
  TESTS
    ✔ example.com/shop  1 passed

    ✔ 1 passed

RUN LOCALLY
  dagger check "dag://?check=test&go-package=store"

Compare the list of checks with the steps in pipeline.yml. Each step that a check replaces is a step that you can remove later.

How Buildkite concepts map to Dagger

Most Buildkite concepts have a Dagger counterpart. If you keep Buildkite as the trigger, some of them stay in Buildkite. The sections after this table explain each row.

Buildkite Dagger
Command steps in pipeline.yml Checks from official modules, plus functions in your own module
Docker and Docker Compose plugins Containers and services that modules define
Cache plugin The engine caches each operation, and cache volumes hold package caches
Agents and queues Engines: local, self-hosted, or Cloud Engines
parallelism and bktec test splitting Checks run in parallel, and test splitting divides large suites across engines
Dynamic pipelines with pipeline upload Modules find packages and tests at run time
Artifacts Functions pass files and containers to each other in the engine
Buildkite secrets and secrets plugins Secret providers, such as Vault, 1Password, and AWS Secrets Manager

Pipeline steps and plugins become checks

A Buildkite pipeline is a list of steps in YAML, in pipeline.yml or in the YAML steps editor (defining steps). Each command step runs on an agent with a clean checkout of the repository. Plugins add common tasks to a step (using plugins).

In Dagger, a step that runs a test or a linter becomes a check of an official module. The common plugins map like this:

  • Docker plugin. A step that runs a command in an image becomes a check that runs in a container. The Go module tests each Go module in golang:<version>-alpine, for the version in its go.mod (Go module).
  • Docker Compose plugin. The plugin runs a step in a Compose service (plugin README). For Go tests that need a database, give the Go module a base container with the database bound to it. How do I run a Buildkite pipeline locally? shows this with Postgres. To build images and start services from a Compose file, see How do I run Docker builds and Compose in CI?.
  • Cache plugin. The plugin saves a directory, such as node_modules, to a backend such as S3, and restores it while a manifest file stays the same (plugin README). The Dagger engine caches all operations by default (Self-Hosting CI with Dagger). When the inputs of an operation do not change, Dagger uses the cached result, with no key to maintain. A module can also mount a named cache volume for a package cache (Dang SDK).
  • Registry login plugins. A function adds registry credentials to the container that it pushes. The release module later in this guide shows this.

Agents, queues, and parallelism become engines

Buildkite agents are build runners. You host them on your own machines, or Buildkite hosts them (architecture). Queues send each step to a set of agents, through the agents attribute of the step (queues). The parallelism attribute splits one step into many jobs. Buildkite recommends its Test Engine Client, bktec, to divide the tests between those jobs (parallel builds).

In Dagger, an engine runs the checks. The Dagger Engine is open source, and runs on any system that can run Linux containers. On a laptop, the CLI starts a local engine. In CI, the DAGGER_ENGINE variable selects the engine (Self-Hosting CI with Dagger):

  • Self-hosted engines. You run the engine on your own machines, for example with the Dagger Helm chart or with Docker. Give each engine a persistent disk, because the cache is on that disk.
  • Cloud Engines. Dagger Cloud runs the engines for you. Set DAGGER_ENGINE=cloud and DAGGER_CLOUD_TOKEN.

You do not divide the tests by job index. For a large test suite, Dagger splits the tests across engines. It balances the parts by the resource use, input cost, and cache state of each test (test splitting).

If you keep Buildkite as the trigger, queues still decide which agent starts each job. The engine is a separate choice. To spread the load of self-hosted engines, assign each check to an engine, and run each selection from its own job, such as dagger check --go-package=store (Self-Hosting CI with Dagger).

Dynamic pipelines and artifacts move into modules

Teams use dynamic pipelines to generate steps at build time. A script writes YAML, and the step pipes it to buildkite-agent pipeline upload (dynamic pipelines). A common script adds one test step for each directory, so a new directory needs no YAML change.

Dagger modules find this work themselves. The Go module finds each package and each test at run time, so a new package becomes a check with no change to a CI file.

Buildkite artifacts pass files from one step to another, and store build outputs such as reports and archives (build artifacts). Steps need them because each step can run on a different agent. In Dagger, a function returns a file, a directory, or a container, and another function takes it as an argument in the engine (Dang SDK). To get a file onto the host, export it. This command builds the root Go package with the Go module and writes the binary to dist/shop:

dagger -c 'go | packages | get . | binary | export ./dist/shop'

A Buildkite step can run this command and keep artifact_paths to store the binary in Buildkite.

Move releases, secrets, and deployments into your own module

Write a module for each step that no official module covers, such as a release. Dang is Dagger’s own language for modules. Install the Dang SDK, then create the dev module of the project:

dagger module install dagger.io/sdk/dang
dagger module init dang -y

The second command creates the module in .dagger/modules/ and installs it as the main module of the workspace. For a project in a shop directory, the module is shop-dev. In the code, ShopDev stands in for the type name that dagger module init generated. Replace .dagger/modules/shop-dev/main.dang with this code:

"""
Release steps of the shop project that no official module covers.
"""
type ShopDev {
  """
  Build the image from the Dockerfile at the repository root.
  """
  image(ws: Workspace!): Container! {
    ws.directory("/", exclude: [".git", ".dagger"]).dockerBuild
  }

  """
  Build the image and push it. Returns the image reference with its digest.
  """
  publish(
    ws: Workspace!,
    """
    Image reference to push, such as registry.example.com/shop:1.2.0.
    """
    ref: String!,
    """
    Registry user name.
    """
    username: String!,
    """
    Registry password or token.
    """
    password: Secret!,
  ): String! {
    image(ws)
      .withRegistryAuth(ref, username, password)
      .publish(ref)
  }
}

You call the functions of the main module with no module name. To push to a test registry from your laptop, read the token from a file:

dagger api call publish --ref=ttl.sh/shop-demo:1h --username=ci --password=file:$HOME/.registry-token
ttl.sh/shop-demo:1h@sha256:daf1bbf6d0d5ff352add59655af1d1029afc9c34dde9bc4cf6aff7b344a9f56f

Keep your secrets where they are. Buildkite recommends a secrets storage service, such as AWS Secrets Manager or HashiCorp Vault, and has plugins that read from them (managing secrets). Dagger reads a secret from its provider when a function needs it. Providers include environment variables, files, command output, Vault, 1Password, AWS Secrets Manager, and Google Cloud Secret Manager (Secrets). The secrets attribute of a step puts a Buildkite secret into the job environment, so Dagger reads it as env:NAME. Dagger redacts secret values from logs, never writes them to container layers, and leaves them out of the cache.

A module can find out if it runs on the main branch. Use that to deploy to production from main and to a preview environment from a pull request (branch filtering).

Run the checks with Cloud Checks or from Buildkite

If the repository is on GitHub, Cloud Checks can run the checks. Cloud Checks run your checks on Cloud Engines after each push and pull request, with no CI file. The origin remote of the repository must point to GitHub. From the repository root, run:

dagger cloud checks on

In a terminal, the command helps you sign in to Dagger Cloud and connect your GitHub organization. Commit dagger.toml, dagger.lock, and the module, then push. The results show as GitHub check statuses and in Dagger Cloud (Cloud Checks).

If you keep Buildkite as the trigger, .buildkite/pipeline.yml only calls Dagger. This version uses Cloud Engines and publishes the image from the default branch:

env:
  DAGGER_ENGINE: "cloud"

secrets:
  - DAGGER_CLOUD_TOKEN

steps:
  - label: ":dagger: checks"
    key: checks
    command: dagger check

  - label: ":rocket: publish"
    depends_on: checks
    if: build.branch == pipeline.default_branch
    command: >-
      dagger api call publish
      --ref=registry.example.com/shop:$BUILDKITE_COMMIT
      --username=ci
      --password=env:REGISTRY_TOKEN
    secrets:
      - REGISTRY_TOKEN

Store DAGGER_CLOUD_TOKEN and REGISTRY_TOKEN as Buildkite secrets. The if attribute uses the pipeline.default_branch variable of Buildkite conditionals.

To use a self-hosted engine instead, set DAGGER_ENGINE: "tcp://dagger-engine.internal:1234" and remove DAGGER_CLOUD_TOKEN. A tcp:// address has no authentication. Make the port available only on the private network of your agents (Self-Hosting CI with Dagger). Install the Dagger CLI on the agents at the same version as the engine.

Move one pipeline step at a time

Keep the Buildkite pipeline as it is while you set up Dagger. Add a trial step that cannot fail the build. The soft_fail attribute lets a step exit with a non-zero status without a failed build:

steps:
  - label: ":go: tests"
    command: go test ./...
    plugins:
      - docker#v5.14.0:
          image: "golang:1.24-alpine"

  - label: ":dagger: checks (trial)"
    command: dagger check
    soft_fail: true

Then do these steps:

  1. Compare the results of each old step with the checks that replace it, over several builds.
  2. Remove soft_fail from the Dagger step when the results match.
  3. Remove one old step and its plugins from pipeline.yml.
  4. Do steps 1 to 3 again for each step that a check replaces.

If you move to Cloud Checks, change the required checks too. Make the Dagger checks required status checks in your GitHub branch protection rules. Then remove the Buildkite status from the list, and turn off the Buildkite pipeline.

When to keep Buildkite as the trigger

Buildkite does some jobs that Dagger leaves to the system that starts the checks. Keep Buildkite in front of dagger check in these cases:

  • Some builds need a macOS or Windows host. The Buildkite agent has installers for macOS and Windows (agent install). The Dagger Engine runs on systems that can run Linux containers (Self-Hosting CI with Dagger).
  • Your repository is not on GitHub. Cloud Checks need a GitHub repository (Set up Cloud Checks). Buildkite starts builds from GitLab, Bitbucket, and other Git servers (source control), and each build can run dagger check.
  • You manage a large agent fleet with clusters and queues, and your platform team wants one place that starts every build (queues).
  • You approve deploys by hand. A block step stops a build until a team member unblocks it.
  • You quarantine flaky tests with Test Engine, so that they stop blocking builds (Test Engine).
  • Your code must stay on your machines. Cloud Checks run on Dagger’s Cloud Engines. Buildkite agents with a self-hosted engine keep the code and the cache in your environment.

In each case, the work still moves into Dagger, and the steps run the same on a laptop and on the agents. For a full comparison, see Dagger vs Buildkite.

Your Git host, tests, and secret managers stay

Your Git host stays the home for code and review. Your tests stay as they are, because official modules run them with no change to the test code (test-native CI). Your Dockerfiles stay, and a module builds them. Your secret managers stay the source of your secrets. If you keep Buildkite as the trigger, your agents, queues, and Buildkite build pages stay too.

Bottom line: You migrate from Buildkite to Dagger without a rewrite. Install the official modules that dagger module recommend finds, run dagger check locally, and move release steps into a small Dang module. Then run the checks with Cloud Checks or from a thin pipeline.yml, and remove each old step when its check gives the same result.