CI that keeps up
how-toby Kyle PenfoundTested with Dagger v1.0.0-beta.21

How do I reproduce a Jenkins build locally?

To reproduce a Jenkins build on your laptop, run the build in a container image that Jenkins and your laptop share. Pin the base image by digest, and put the JDK and system libraries in that image instead of on the agent. Then run the same build commands inside it in Jenkins and on your machine, and open a shell in it when a step fails. You can do this by hand with a Dockerfile, the Jenkins docker or dockerfile agent, and docker run. Dagger meets the same requirements in one place. Its official Maven module runs the build in the same image, and a short Jenkinsfile and your laptop both run the same dagger check.

What a reproducible Jenkins build environment needs

A “works on my machine” failure comes from a difference between the CI environment and yours. Common examples are a different JDK patch version, a missing system library, or a different time zone. A setup that removes these differences meets four requirements:

  1. Pin the base image by digest. A tag such as maven:3.9.9-eclipse-temurin-17 can point to a new image later. Docker says that a digest “pins” an image “to a specific version in time” (docker image pull).
  2. Put the system libraries in the image, not on the agent. A package that someone installed on one Jenkins agent is not on your laptop. A Dockerfile in the repository records every package and setting.
  3. Run the same build commands in that image, in Jenkins and locally. The image is only half of the environment. The commands, flags, and environment variables must match too.
  4. Debug inside the same environment. When a step fails, you need a shell in the container where it failed, with the same files and variables.

Meet the requirements with a Dockerfile and the Jenkins docker agent

Start with a Dockerfile that pins the base image and installs the libraries that the build needs. The Docker Pipeline plugin can build it and run your stages inside it. This example is Dockerfile.build at the root of the repository:

FROM maven:3.9.9-eclipse-temurin-17@sha256:f58d59b6273e785ac0a4477f6e9b5ba1d7731c75b906c0f7b34076f1851318cc
RUN apt-get update \
 && apt-get install -y --no-install-recommends fontconfig fonts-dejavu-core \
 && rm -rf /var/lib/apt/lists/*
ENV TZ=UTC LANG=C.UTF-8

The FROM line meets requirement 1. The RUN and ENV lines meet requirement 2. To find the digest of a tag, pull the tag once. docker pull prints the digest after the pull (docker image pull).

In the Jenkinsfile, the dockerfile agent builds this image and runs the stage steps inside it. Its filename option selects a file with another name, and additionalBuildArgs passes arguments to docker build (Pipeline Syntax: agent):

pipeline {
    agent {
        dockerfile {
            filename 'Dockerfile.build'
        }
    }
    stages {
        stage('Test') {
            steps {
                sh 'mvn -B verify'
            }
        }
    }
}

The docker agent runs an image that is already built. Its args parameter passes arguments to docker run. For an image in a private registry, set registryUrl and registryCredentialsId (Pipeline Syntax: agent). The Jenkins docs use args to mount a Maven cache, such as -v $HOME/.m2:/root/.m2 (Using Docker with Pipeline).

If one stage uses a container and other stages use the agent, set reuseNode true on the container stage. Without it, Jenkins can run the stage on another agent, in a new workspace. With it, Jenkins mounts the current workspace into the container on the same node (Workspace synchronization).

To meet requirement 3 on your laptop, build the same file and run the same command in it:

docker build -f Dockerfile.build -t orders-build .
docker run --rm -v "$PWD":/src -w /src orders-build mvn -B verify

For requirement 4, run the same docker run command with -it, and replace mvn -B verify with bash. Then repeat the failed step by hand.

Jenkinsfile Runner is another tool. It packages the Jenkins pipeline engine as a command-line tool, so you can run a Jenkinsfile on your machine. It needs the plugins that the pipeline uses, and you assemble them in a plugin directory first.

Local runs copy the Jenkinsfile by hand

The Dockerfile keeps the image in one file, but the rest of the environment lives in the Jenkinsfile. Requirement 3 then depends on people who keep two copies the same:

  • The commands live in Groovy. The sh 'mvn -B verify' step is in the Jenkinsfile. The local docker run line is a copy of it, in a README or a script, and the copy drifts when a flag changes.
  • The container options live in Groovy too. The args, additionalBuildArgs, and reuseNode settings change how Jenkins runs the container. Your local command must repeat each of them by hand.
  • Each agent needs Docker. The docker and dockerfile agents need the Docker Pipeline plugin and a node “pre-configured to accept Docker-based Pipelines” (Pipeline Syntax: agent).

Meet the requirements with Dagger

Dagger runs build steps in containers, and you define those steps in code that Jenkins and your laptop both call. A Dagger module is a package of functions that build and test the project. A check is a function that validates the project, and dagger check runs every check. The functions run on the Dagger engine. On a laptop, the CLI starts a local engine.

The official Maven module, dagger.io/java/maven, runs Maven in a container and makes mvn verify a check. Install the Dagger CLI first. Then, from the project root, install the module:

dagger module install dagger.io/java/maven
Created workspace config in /work
Installed module "maven" in /work/dagger.toml

The command creates dagger.toml, the workspace configuration, and records the module in it. Dagger also writes dagger.lock, which pins the module to an exact version. Commit both files.

By default, the Maven module builds in its own maven:3.9.16-eclipse-temurin-21 image. To build in the image from Dockerfile.build, give that image to the module’s base setting. A small project module builds the image. Write it in Dang, Dagger’s native language for modules. Install the Dang SDK and create the module:

dagger module install dagger.io/sdk/dang
dagger module init dang -y

dagger module init with no name creates .dagger/modules/orders-dev, named after the project directory, and installs it as the workspace main module. In the snippet, OrdersDev stands in for the type name that dagger module init generated. Replace .dagger/modules/orders-dev/main.dang with this code:

"""
The CI build image of the orders service, for the Maven module.
"""
type OrdersDev {
  """
  The image from Dockerfile.build: the pinned JDK and Maven, the system libraries, and the time zone.
  """
  pub buildImage(ws: Workspace!): Container! {
    ws.directory("/", include: ["Dockerfile.build"])
      .dockerBuild(dockerfile: "Dockerfile.build")
  }
}

The function reads only Dockerfile.build, so a change to the source does not build the image again. Give the image to the Maven module:

dagger module settings maven base build-image

The command writes a module reference to dagger.toml:

[modules.maven.settings]
base = "dag://orders-dev/build-image"

Pin the base image by digest

Dagger uses the same pin as the hand-built version. dockerBuild builds Dockerfile.build, with the digest in its FROM line. When a module starts from an image tag with container.from instead, Dagger records the digest that it resolved in dagger.lock. Later runs use that digest until you run dagger lock update (dagger.lock).

Put the system libraries in the image

The libraries and the time zone stay in Dockerfile.build, as in the hand-built version. The Maven module keeps the environment of the base image. It adds the source and a cache volume for the Maven repository. A cache volume is a directory that keeps its content between runs, so the agent needs no -v mount. The Maven and Java versions come from the image:

dagger -c 'maven | project --path=. | run --goals=--version | stdout'
Apache Maven 3.9.9 (8e8579a9e76f7d015ee5ec7bfcdc97d260186937)
Maven home: /usr/share/maven
Java version: 17.0.15, vendor: Eclipse Adoptium, runtime: /opt/java/openjdk

If the project has a Maven Wrapper, the module runs ./mvnw instead, and the wrapper selects the Maven version.

Run the same build commands in Jenkins and locally

The Maven commands now live in the Maven module, not in the Jenkinsfile. List the checks:

dagger check -l
MODULE    MAVEN-PROJECT  CHECK   DESCRIPTION
dang-sdk                 stale   staleness check: re-generate modules and clients managed by the Dang SDK
maven     *              verify  Run through verify: unit tests, packaging, integration tests, and quality gates.
# Use --all to expand collections and list each item.

Run them:

dagger check
CHECKS  ✔ 2 passed
✔ dag://?check=dang-sdk/generate/stale 1.0s OK
✔ dag://?check=maven/projects/verify 41.0s OK

The verify check runs mvn verify, the same lifecycle phase as the Jenkins stage. In this guide’s project, the first verify took 41 seconds, including the image build and the dependency downloads. A rerun with no changes took 0.1 seconds, because the engine cached the result. After a one-line change to a source file, the check took 1.5 seconds. These numbers come from one run on one machine.

To build the jar and save it on the host, call the build function. It runs mvn package:

dagger -c 'maven | project --path=. | build | file target/orders-1.0.0.jar | export target/orders.jar'

Debug inside the same environment

A failed check prints the Maven output and the command that runs that check again. In this example, a test expects a different time zone from the TZ=UTC that the image sets:

CHECKS  ✘ 1 failed  ✔ 1 passed
✘ dag://?check=maven/projects/verify 1.4s ERROR
  ┃ [ERROR] Failures: 
  ┃ [ERROR]   OrdersTest.zone:16 expected: <Europe/Paris> but was: <UTC>
  ┃ [INFO] 
  ┃ [ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0
  ┃ [INFO] 
  ┃ [INFO] ------------------------------------------------------------------------
  ┃ [INFO] BUILD FAILURE
✔ dag://?check=dang-sdk/generate/stale 0.5s OK

RUN LOCALLY
  dagger check "dag://?check=maven/projects/verify"

To open a shell in the build environment, run dagger -c 'maven | project --path=. | container | terminal'. The shell has the image, the source, and the variables of the check, such as TZ=UTC. To open a shell in the exact state where a command failed, run dagger check -i (Glossary: Dev shell).

Call the same checks from a thin Jenkinsfile

The Jenkinsfile keeps the triggers, stages, and artifact archive, and each step calls Dagger. The agent needs the Dagger CLI, not the Docker Pipeline plugin. Set DAGGER_ENGINE in every job, so the CLI connects to a self-hosted engine that keeps its cache. Without it, the CLI starts an engine on the agent, and an ephemeral agent loses that cache after each job (Self-Hosting CI with Dagger).

pipeline {
    agent { label 'dagger' }
    environment {
        DAGGER_ENGINE = 'tcp://dagger-engine.internal:1234'
    }
    stages {
        stage('Test') {
            steps {
                sh 'dagger check'
            }
        }
        stage('Package') {
            steps {
                sh "dagger -c 'maven | project --path=. | build | file target/orders-1.0.0.jar | export target/orders.jar'"
                archiveArtifacts artifacts: 'target/orders.jar', fingerprint: true
            }
        }
    }
}

Keep the engine port on the private network of the agents. Install the CLI on the agent at the same version as the engine. Cloud Engines are managed remote engines that Dagger Cloud runs. To run the checks on them, set DAGGER_ENGINE to cloud and give the job a DAGGER_CLOUD_TOKEN. Store the token as a Jenkins credential (Handling credentials).

If the repository is on GitHub, Cloud Checks can run the same checks on Cloud Engines on every push and pull request, with no CI configuration. Run dagger cloud checks on to connect the repository to Dagger Cloud.

Bottom line: To reproduce a Jenkins build locally, pin the base image by digest in a Dockerfile, put the system libraries in it, and run the same commands in it in Jenkins and with docker run. That works, but the commands and container options stay in the Jenkinsfile, and local copies of them drift. With Dagger, the official Maven module runs mvn verify in the image from the same Dockerfile, and dagger check runs it the same way on a laptop and from a thin Jenkinsfile.