CI that keeps up
how-toby Kyle Penfound

How do I build Docker images and run Compose services the same way locally and in CI?

A Docker setup that works the same on a laptop and in CI meets four requirements. It lints each Dockerfile, builds every image and every test stage, validates each Compose file, and starts each service after the services it depends on. Teams meet them with hadolint, docker build, and docker compose in a CI job and in local scripts. Dagger meets the same four requirements in one place. Its official Docker module finds every Dockerfile and Compose file, and runs the same checks on a laptop and in Dagger Cloud Checks.

What a good Docker and Compose setup needs

A Docker and Compose setup for CI has four requirements. Each one comes from the docs of Docker and hadolint, and this guide judges each approach by them:

  1. Lint each Dockerfile. hadolint is a “Dockerfile linter, validate inline bash”. It finds problems such as unpinned package versions. Docker’s own build checks also lint a Dockerfile. But “check violations for builds are reported as warnings, with exit code 0, by default.” A lint that must fail CI needs a tool that exits with an error.
  2. Build every image and every test stage. Docker recommends that you build and test your images in CI for each change and pull request. Many Dockerfiles have a test or lint stage that the final image does not use. BuildKit “only builds the stages that the target stage depends on” (multi-stage builds). Thus a normal build skips those stages.
  3. Validate each Compose file. The --quiet flag of docker compose config tells it to “Only validate the configuration, don’t print anything”. It finds an error in the Compose file before any service starts.
  4. Start each service after the services it depends on. “On startup, Compose does not wait until a container is “ready”, only until it’s running“ (startup order). A healthcheck on a service and the service_healthy condition in depends_on make Compose wait until the dependency is healthy.

Meet the requirements with Docker commands

Teams meet the four requirements with the Docker CLI and hadolint. The examples come from a project with a Dockerfile in app/ and a Compose file at the root. The Dockerfile has a test stage that no other stage uses. The Compose file is compose.yaml:

services:
  web:
    build: ./app
    ports:
      - "8080:8080"
    environment:
      REDIS_URL: redis://redis:6379
    depends_on:
      - redis
  redis:
    image: redis:7-alpine
    ports:
      - "6379:6379"

Lint each Dockerfile. Run hadolint from its image, as the hadolint README shows. NO_COLOR=1 turns off color in the output:

docker run --rm -i -e NO_COLOR=1 hadolint/hadolint:v2.14.0-alpine < app/Dockerfile
-:9 DL3018 warning: Pin versions in apk add. Instead of `apk add <package>` use `apk add <package>=<version>`

hadolint exits with code 1, so the CI job fails. To ignore a rule, list it under ignored in a .hadolint.yaml file.

Build every image and every test stage. Build the final image, then build the test stage with --target:

docker build -t web app
docker build --target test app

In this guide’s project, the first command built the deps stage and the final stage. It did not run the test stage. Only the second command ran it.

Validate each Compose file. Run docker compose config --quiet in the directory of the Compose file. It prints nothing and exits with code 0 when the file is valid.

Start each service after the services it depends on. Start the services, and wait until they run:

docker compose up --wait
 Network compose_default  Creating
 Network compose_default  Created
 Container compose-redis-1  Creating
 Container compose-redis-1  Created
 Container compose-web-1  Creating
 Container compose-web-1  Created
 Container compose-redis-1  Starting
 Container compose-redis-1  Started
 Container compose-web-1  Starting
 Container compose-web-1  Started
 Container compose-redis-1  Waiting
 Container compose-web-1  Waiting
 Container compose-redis-1  Healthy
 Container compose-web-1  Healthy

The --wait flag waits “for services to be running|healthy”. Neither service has a healthcheck, so Compose waited only until each container ran. After the run, docker compose down stops and removes the containers and the network.

In CI, a job runs the same commands on a runner with Docker. On GitHub Actions, Docker’s official actions set up Buildx and Compose and build images. hadolint has a GitHub Action and a GitLab CI example.

Where the hand-built setup strains

Each requirement becomes one or more steps that someone keeps up to date:

  • Lint each Dockerfile. The hadolint image tag is in each CI job and each local script. If the copies differ, a laptop and CI apply different rules.
  • Build every image and every test stage. Each Dockerfile and each test stage needs its own docker build command. A new test stage that nobody adds to the CI job never builds, because BuildKit skips it.
  • Validate each Compose file. Each directory with a Compose file needs its own docker compose config step.
  • Start each service after the services it depends on. The runner needs a Docker daemon and free host ports, such as 8080 and 6379. The job must also run docker compose down when the tests end.

Meet the requirements with the Dagger Docker module

Dagger is a CI orchestration engine that runs the same pipelines on a laptop and in CI. A module is a package of checks, services, and functions that a project installs. The official Docker module finds each Dockerfile and each Compose file in the repository. It meets each requirement with the same tools as the Docker commands above. You write no Dagger code.

Complete the Quickstart first. Then, from the repository root, install the Docker module from its main branch, because it has no tagged release:

dagger module install dagger.io/docker@main
Created workspace config in /work
Installed module "docker" in /work/dagger.toml

The command adds the module to dagger.toml, the workspace configuration, and creates the file if the project has none. It also writes dagger.lock, which pins the Git commit of main that Dagger used. Commit both files. To move to a newer commit of the module, run dagger lock update.

A check is a function that validates the project and passes or fails, such as a lint or a build. List the checks of the Docker module:

dagger check --docker -l
MODULE  DOCKER-COMPOSE-PROJECT  DOCKER-COMPOSE-SERVICE  DOCKER-PROJECT  DOCKER-STAGE  CHECK          DESCRIPTION
docker  *                                                                             lint           Validate this project's Compose file with 'docker compose config'.
docker  *                       *                                                     build-service  Build this Compose service's image, if it declares a build.
docker                                                  *                             build          Build this project's Dockerfile.
docker                                                  *                             lint           Lint this project's Dockerfile with hadolint.
docker                                                  *               *             build-stage    Build this Dockerfile stage.
# Use --all to expand collections and list each item.

A * stands for every item that the module found. Add --all to list one row for each item:

dagger check --docker -l --all
MODULE  DOCKER-PROJECT  DOCKER-COMPOSE-PROJECT  DOCKER-COMPOSE-SERVICE  DOCKER-STAGE  CHECK          DESCRIPTION
docker  app                                                                           build          Build this project's Dockerfile.
docker                  .                       redis                                 build-service  Build this Compose service's image, if it declares a build.
docker                  .                       web                                   build-service  Build this Compose service's image, if it declares a build.
docker  app                                                             test          build-stage    Build this Dockerfile stage.
docker                  .                                                             lint           Validate this project's Compose file with 'docker compose config'.
docker  app                                                                           lint           Lint this project's Dockerfile with hadolint.

The module works on four collections. Each collection adds a column and a flag that selects its items:

Collection One item is Flag
Docker projects a directory with a Dockerfile, such as app --docker-project
Dockerfile stages a named stage that the project build does not reach, such as test --docker-stage
Compose projects a directory with a Compose file, . for the root --docker-compose-project
Compose services a service name in a Compose file --docker-compose-service

Lint each Dockerfile with Dagger

The lint check of each Docker project runs hadolint on its Dockerfile. The module pins the hadolint image in its lintImage setting. The default is docker.io/hadolint/hadolint:v2.14.0-alpine (module README). Thus a laptop and CI run the same hadolint release. Run every check in the workspace:

dagger check

In this guide’s project, the first run failed on the same hadolint rule as the docker run command:

== CHECKS ==  ✘ 1 failed  ✔ 4 passed
✘ dag://?check=docker/projects/lint 4.3s ERROR
  ┃ [hadolint] Dockerfile:9 DL3018 warning: Pin versions in apk add. Instead of `apk add <package>` use `apk add <package>=<version>`
  ! hadolint failed in 1 of 1 projects:
    - app: hadolint failed (exit 1):
    Dockerfile:9 DL3018 warning: Pin versions in apk add. Instead of `apk add <package>` use `apk add <package>=<version>`
  ✘ exec.processRun 0.0s ERROR
  ! /usr/local/bin/crun did not terminate successfully: exit status 1
✔ dag://?check=build-service&docker-compose-project=. 9.2s OK
✔ dag://?check=build-stage&docker-project=app 2.5s OK
✔ dag://?check=docker/compose/projects/lint 8.6s OK
✔ dag://?check=docker/projects/build 4.0s OK

== RUN LOCALLY ==
dagger check "dag://?check=docker/projects/lint"

The RUN LOCALLY command runs only the failed check again. To run hadolint on every Dockerfile and nothing else, select the Docker projects and the check name:

dagger check --docker-projects --check lint

The module gives hadolint the nearest .hadolint.yaml or .hadolint.yml, from the directory of the Dockerfile up to the repository root (module README). It is the same hadolint configuration file that you use without Dagger. To ignore a rule, create .hadolint.yaml at the root:

ignored:
  - DL3018

To give hadolint extra arguments, use the lintArgs setting. For example, make hadolint fail only on errors, not on warnings:

dagger module settings docker lintArgs '["--failure-threshold", "error"]'

The command writes lintArgs = ["--failure-threshold", "error"] under [modules.docker.settings] in dagger.toml.

Build every image and every test stage with Dagger

The build check builds the final stage of each Dockerfile. The build-stage check builds each named stage that no other stage uses and that is not the final stage (module README). The module reads the stages from the Dockerfile. Thus a new test stage gets a check without a change to any CI file. After the .hadolint.yaml from the previous section, all five checks passed:

== CHECKS ==  ✔ 5 passed
✔ dag://?check=build-service&docker-compose-project=. 1.1s OK
✔ dag://?check=build-stage&docker-project=app 1.1s OK
✔ dag://?check=docker/compose/projects/lint 0.7s OK
✔ dag://?check=docker/projects/build 1.1s OK
✔ dag://?check=docker/projects/lint 0.6s OK

When a RUN step fails, the report gives the project and the exit code. For a Dockerfile with RUN exit 3, the report showed:

  ! docker build failed in 1 of 1 projects:
    - app: docker build failed (exit code 3 in a RUN step; see the trace for its output)

The output of the step is in the trace, under the failed withExec.

Some Dockerfiles end in a development stage, after the production stage. If yours do, set buildTarget to the stage that the build check must build:

dagger module settings docker buildTarget production

The build check then builds that stage in each Dockerfile that declares it. A Dockerfile without the stage still builds its final stage. The stage that buildTarget names leaves the stages collection, so build-stage does not build it a second time. To go back to the default, remove the setting:

dagger module settings -u docker buildTarget

Validate each Compose file with Dagger

The lint check of each Compose project runs docker compose config --quiet in the directory of the Compose file. The build-service check builds each service that has a build section, with its build args, target, and Dockerfile. A service that only runs an image, such as redis, has nothing to build, so its check passes. To build only the web service:

dagger check --docker-compose-service=web

The module looks for compose.yaml, compose.yml, docker-compose.yml, and docker-compose.yaml, in that order. It uses the first file that it finds in each directory, as Compose does. Compose also merges a standard override file, such as compose.override.yaml. The module reads service names from the text of the file. Thus, it does not see services from include: or from files that -f or COMPOSE_FILE names. It also does not see services from YAML merge keys at the services level (module README).

Start each service after the services it depends on, with Dagger

The module starts Compose services as Dagger services. A Dagger service is a container that Dagger starts so that other containers or your machine can reach it. List the services:

dagger start -l -a
MODULE  DOCKER-COMPOSE-PROJECT  DOCKER-COMPOSE-SERVICE  SERVICE        DESCRIPTION
docker  .                                               batch/up       Run the selected services and the services they depend on, behind one
docker                                                  cli/engine     The Docker Engine this client is wired to.
docker                                                  docker/engine  Spawn an ephemeral Docker Engine in a container.
docker  .                       redis                   up             Run this service and the services it depends on, behind one proxy that
docker  .                       web                     up             Run this service and the services it depends on, behind one proxy that

Start the web service:

dagger start --docker-compose-service=web

Dagger builds web from app/Dockerfile and starts redis, because web depends on it. It publishes the ports of both services on your machine. In this guide’s project, curl localhost:8080 returned the page, and Redis answered PING on port 6379. To stop the services, press Ctrl+C. To select one Compose project, add --docker-compose-project. Always select a project or a service, because a dagger start with no filter also starts the Docker engines of the module.

Dagger applies the Compose file as far as it can (module README):

  • The current Docker module does not translate Compose health checks. A service starts after the ports of its dependencies accept connections. The module does not run healthcheck tests or the depends_on conditions that Compose uses for startup order. Dagger itself supports Docker health checks on services, with withDockerHealthcheck in the Container API. Until the module translates them, make a dependent service retry its connection if a dependency opens its port before it is ready.
  • Services reach each other by name. web connects to redis:6379, as on the default Compose network.
  • Bind mounts are copies. Dagger mounts files from the repository as copies, so writes do not change your files. Dagger does not mount a path from outside the repository, such as /var/run/docker.sock.
  • Named volumes persist. Each named volume becomes a cache volume that keeps its data between runs.
  • Profiles are off. Dagger does not start services that have a Compose profile.

For each setting that Dagger cannot apply, such as cap_add, network_mode, or a UDP port, it prints a warning in the trace. In a check, Dagger stops each service when nothing references it, so there is no down step. Other modules can use a Compose service as a setting. The guide to Playwright tests against a Docker Compose app shows how.

Run the same checks locally and with Cloud Checks

Commit dagger.toml and dagger.lock. If you created .hadolint.yaml, commit it too. The project has no Dagger module code to commit.

In CI, Cloud Checks run the checks. Cloud Checks run your checks on Cloud Engines, Dagger’s managed engines, after each push and pull request. They need no CI workflow file. The repository must be on GitHub, and its origin remote must point to it. Turn them on:

dagger cloud checks on

In an interactive terminal, the command offers to sign you in to Dagger Cloud and to connect GitHub. After that, each push runs hadolint, the Dockerfile builds, and the Compose checks on Cloud Engines, as dagger check does on a laptop. The hadolint release and the list of Dockerfiles, stages, and Compose files come from the same dagger.toml and the same repository on both. To see the runs, use dagger workspace activity.

Bottom line: Lint each Dockerfile with hadolint, build every image and every test stage, and validate each Compose file. Start each service after its dependencies. The Docker CLI meets these requirements one command at a time. Dagger’s Docker module finds every Dockerfile, stage, and Compose file, and runs the same checks on a laptop and with Cloud Checks.