We’re always interested in hearing how our community is using Dagger – their use cases, their challenges, and their experiences with deploying Dagger in different environments. Our [Discord](https://discord.com/invite/dagger-io) is a great place to find these stories, and to benefit from the knowledge and experience of the Dagger community.

In this blog post, we’ll share the story of [Flipt](https://www.flipt.io/), an open source, self-hosted feature flag solution. Flipt uses Dagger as an integral part of its CI/CD pipeline. With Dagger, Flipt has seen its build pipeline with subsequent integration tests go from 10 minutes with an empty cache to 17 seconds on a subsequent run!

> "Dagger feels like an actual innovation in CI/CD. There's a real attention to getting feedback from everyone and feeding that back into the design and building something that people need." - George MacRorie, Flipt

### The problem: Networking and language limitations for CI testing

Flipt stores its source code in a Go multi-module repository and uses a Go workspace to bring the modules together. Initially, Flipt's CI/CD pipeline used GitHub Actions for both unit and integration tests. The pipeline used a matrix strategy to test different configurations of feature flags and configurations.

There were two main challenges with this approach.

* Test scripts were written in Bash. Flipt, however, is written entirely in Go. This created an inconsistency, wherein application code was written in one language and application tests in a different language.
* Test scripts were executed in separate VMs by Github Actions, which was cumbersome. They had brittle little bash scripts for backgrounding the Flipt process and then running the tests against them.

The Flipt team had previously experimented with Dagger's CUE-based implementation in other projects. Dagger's release of a Go SDK gave them a reason to rethink their CI/CD strategy and consider re-implementing it in Go with Dagger. Dagger being open-source also aligned closely with Flipt's own values.

> "Open source is a value that's super important to us. Flipt being 100% open source and the value alignment with Dagger in that respect just made a lot of sense to us." - Yoofi Quansah, Flipt

### The solution: Fast, full-featured native-language CI testing with Dagger

The Flipt team currently uses Dagger for both build and test phases of its pipeline, handing off to GoReleaser for the release phase.

* In the build phase, the Dagger pipeline downloads the application source code, adds required dependencies and build tools, and builds the application. The end result of the build phase is two artifacts: a "test" image with the source code plus build and test dependencies, and a "production" image optimized for distribution.
* In the test phase, the Dagger pipeline runs Flipt's complete suite of unit and integration tests using the Go SDK, validating the test image against a common set of Flipt operations. In addition to performing complete end-to-end API testing, the pipeline also performs additional tests related to importing, validating and exporting feature flag configurations. These tests are performed against a variety of databases (including Postgres, MySQL and CockroachDB).

Flipt has seen several benefits after switching to Dagger:

* Native Go support makes it easier for the team to expand the scope of the test suite with additional configurations and permutations -for example, tests against the HTTP or gRPC API, tests with or without authentication enabled, and so on. Since tests are now written in Go, it's much easier for anyone in the team to add new tests or improve existing tests.
* Dagger's built-in support for container networking enables testing to be performed against an isolated, containerized instance of the Flipt API instead of the live API.
* Dagger's built-in support for service containers makes it possible to test against multiple database engines, and to quickly add test support for a new database engine.
* The same Dagger pipeline works consistently both locally and remotely, reducing the feedback loop when modifying the pipeline - for example, when adding tests for a new database engine or a new feature flag.

Perhaps the biggest benefit, though, is in application build and test times. Flipt's pipeline is fairly large and complex. However, by leveraging Dagger's engine caching and mounted cache volumes, Flipt is able to avoid unnecessary re-downloading or re-computing of assets. When Flipt is compiled with an empty build cache, a fresh run will produce 1 GB of build-cache entries and take approximately 10 minutes. This drops to approximately 17 seconds on subsequent runs, thanks to Dagger's caching...an improvement of ~97%!

> "We've got a distributable that can be configured in many ways. We want to test that it still works even when we change the configuration, or when we configure it in different ways. Being able to encode those requirements in Go is a very nice experience." - George MacRorie, Flipt

### The future: Release packaging, profiling and remote caching with Dagger

Flipt currently uses GoReleaser to package and distribute the artifacts generated by the Dagger pipeline. In the near future, the team plans to completely integrate Dagger into the CI pipeline, including for the release phase. Dagger's multi-platform support will be key here, to build and release the Flipt distributable for different architectures.

Flipt is also experimenting with importing and merging Go profiling data from the Dagger pipeline to get a deeper understanding of test coverage and performance for integration tests. And the team is keen to improve the CI pipeline further - for example, by using Dagger to connect to various remote storage engines for caching.

Do you have a Dagger story you’d like us to feature? Tell us all about it in [Discord](https://discord.com/invite/dagger-io)!

Learn how Dagger became an integral part of Flipt's CI/CD pipeline. 

Dagger in Action: How Flipt Improved Coverage and Build Times with Dagger

AWS CodeBuild and AWS CodePipeline are managed continuous integration/continuous delivery services from AWS to build and release software in a predictable and efficient manner. Dagger integrates with both these services, enabling developers and DevOps teams to run Dagger pipelines on AWS infrastructure easily.

If this interest you, check out our new tutorial on [using Dagger to continuously build and publish a Node.js application with AWS CodeBuild and AWS CodePipeline](https://docs.dagger.io/183109/aws-codebuild-codepipeline). You will learn how to:

* Create an AWS CodeBuild project and connect it to an AWS CodeCommit repository
* Create a Dagger pipeline using a Dagger SDK
* Integrate the Dagger pipeline with AWS CodePipeline to automatically build and publish the application on every repository commit.

If you don't use AWS, fear not - you can also easily integrate Dagger with other CI/CD services. We already have detailed walkthroughs for using Dagger with [Azure DevOps](https://docs.dagger.io/620301/azure-pipelines-container-instances), [GitHub Actions](https://docs.dagger.io/759201/gitlab-google-cloud) and [GitLab CI](https://docs.dagger.io/759201/gitlab-google-cloud), and we'll be adding more guides for other services soon!

 Discover the simplicity and efficiency of AWS CodeBuild and CodePipeline with Dagger. Dive into our comprehensive tutorial where we explore Dagger's seamless integration with these services, empowering developers and DevOps teams to effortlessly run Dagger pipelines on AWS infrastructure.

Use Dagger with AWS CodeBuild and AWS CodePipeline


Dagger 0.6 updates the `dagger run` CLI command, which you can now use to run your Dagger code with a shiny new terminal UI (TUI):

<video controls autoplay loop width="90%" style="display: block; margin: 0 auto 32px">
  <source src="/vid/tui.mp4" type="video/mp4">
  <source src="/vid/tui.webm" type="video/webm">
</video>

#### Why?

Dagger works by building up a DAG of operations and evaluating them, often in parallel. By nature, this is a difficult thing to display in the terminal. It's easy to show the output of one operation at a time, but as soon as you have parallelism you'll get jumbled output that's hard to grok.

The current `LogOutput` format solves this by alternating between active operations and showing chunks of output at a time. It will print a header showing the output source and then print its output, with a small cooldown that allows it to get "bored" and swap to another active operation.

For example, here's how `node ./build.js` currently looks for the [Node.js services example](https://github.com/dagger/examples/tree/main/nodejs/db-service):

```
Connected to engine c1b34068aa26
#1 resolve image config for docker.io/library/postgres:15.2
#1 DONE 1.2s

#2 resolve image config for docker.io/library/node:16
#2 DONE 0.3s

#3 service 85BR3ID518VOQ
#3 resolve docker.io/library/postgres:15.2@sha256:78a275d4c891f7b3a33d3f1a78eda9f1d744954d9e20122bfdc97cdda25cddaf done
#3 0.211 The files belonging to this database system will be owned by user "postgres".
<snipped>
#3 1.568 2023-05-18 18:18:14.071 UTC [15] LOG:  database system is ready to accept connections
#3 ...

#5 from node:16
#5 resolve docker.io/library/node:16@sha256:550f484fc5f314b575f5e397c9e2c71d7f218e59729fcda9ffa7ea1fc825dce7 0.0s done
#5 CACHED

#6 
#6 DONE 0.0s

#6 
#6 4.953 
#6 4.953 added 341 packages, and audited 342 packages in 5s
<snipped>
#6 DONE 5.2s

#3 service 85BR3ID518VOQ
#3 ...

#6 
#6 0.531 
<snipped>
#6 1.556 Ran all test suites.
#6 DONE 7.0s

#3 service 85BR3ID518VOQ
#3 CANCELED
```

This format is much better than directly printing output in parallel, because you can at least see which operation is producing each chunk of output. 

When multiple commands are actively printing output, however, it becomes more difficult to follow. It can either stick to one command for too long, not showing the concurrent output at all, or hop back and forth between them with a lot of extra noise from the headers in the output. This output format comes directly from Buildkit's plaintext console renderer, which also limits what we can show to the user: only things that go through Buildkit's `Solve()` call.

But the biggest problem with the `LogOutput` format is that it's not very pretty. Using Dagger should spark joy. If users are going to be running CI/CD builds all day, it needs to be easy on the eyes!

#### Enter the new terminal UI

The new terminal UI is implemented with a new progress toolkit that we can use to display anything we want: service health-checks, GraphQL API calls, debug logs, etc. It shows a full-blown DAG in a style similar to `git log --graph`, with the current state of your DAG's evaluation and a full snapshot of all output at the end. 

This new terminal UI is designed for rapid iteration, in the same way regular scripting works: you run a command, it shows live progress, and then leaves all the output in your terminal scrollback. You've already seen it at the top of this post, but here it is again:

<video controls width="90%" style="display: block; margin: 0 auto 32px">
  <source src="/vid/tui.mp4" type="video/mp4">
  <source src="/vid/tui.webm" type="video/webm">
</video>

There's a lot of information being conveyed in this UI, so here's a quick primer:

* Parallel pipelines are represented by vertical columns (┃ for active, │ for inactive).
* Operations within a pipeline are represented by block characters (█).
* Sub-tasks of an operation are listed beneath it (┣).
* Actively running operations blink until they finish (█▓▒░█▓▒░).
* Each pipeline forks from its parent pipeline and show its name in bold after a caret (▼).
* Dependencies across pipelines are shown as a column that forks beneath the output operation and connects to each input, with the name of the output in grey (`pull docker.io/library/node:16`).
* Actively running operations are always shown at the bottom of the screen.
* Failed operations are also always shown at the bottom of the screen.

It's a lot to take in, and you're not meant to digest it all the time. The idea is for it to act as passive feedback as you refactor your DAG. If you don't care about every single detail, you can just keep your eyes on the bottom of the screen to see anything important.

#### Debugging

With `dagger run --debug`, you can see all of the underlying GraphQL API calls. This reveals what's actually happening if things appear to pause for a moment, as with the `import()` API call in the demo below.

<video controls width="90%" style="display: block; margin: 0 auto 32px">
  <source src="/vid/tui-debug.mp4" type="video/mp4">
  <source src="/vid/tui-debug.webm" type="video/webm">
</video>

Admittedly, the output appears pretty cluttered owing to the small screen size. We expect to fine-tune things some more in the future!

#### The "other" terminal UI

Astute observers may have noticed this looks nothing like the terminal UI [demonstrated](https://www.youtube.com/watch?v=oFRx_QxzET8) in previous community calls.

This "other" UI is called the interactive, or tree-style, TUI. The interactive TUI is still available for the moment while we experiment. You can enable it by setting the `_EXPERIMENTAL_DAGGER_INTERACTIVE_TUI` environment variable. Here's what it looks like:

<video controls width="90%" style="display: block; margin: 0 auto 32px">
  <source src="/vid/tui-interactive.mp4" type="video/mp4">
  <source src="/vid/tui-interactive.webm" type="video/webm">
</video>

After [asking the community](https://discord.com/channels/707636530424053791/1105631313198841956/1105706970050199564), we decided not to make this one the default. The interactive tree-style UI is better for interactive debugging as opposed to rapid iteration.

#### SDK Updates

We've also released new versions of our SDKs with various SDK-specific bug fixes and improvements.

For a complete list of improvements, refer to the changelog for each SDK:

- Go SDK v0.7.0 [changelog](https://github.com/dagger/dagger/releases/tag/sdk%2Fgo%2Fv0.7.0) and [documentation](https://pkg.go.dev/dagger.io/dagger)
- Node.js SDK v0.6.0 [changelog](https://github.com/dagger/dagger/releases/tag/sdk%2Fnodejs%2Fv0.6.0) and [documentation](https://docs.dagger.io/current/sdk/nodejs/reference/modules)
- Python SDK v0.6.0 [changelog](https://github.com/dagger/dagger/releases/tag/sdk%2Fpython%2Fv0.6.0) and [documentation](https://dagger-io.readthedocs.org/)

#### New Dagger Cookbook and Documentation

We've been continuously improving and adding to our documentation. Here are a few highlights:

- Our documentation now includes a [Dagger cookbook](https://docs.dagger.io/7442989/cookbook). As the name suggests, this provides ready-made, focused "recipes" to help you quickly accomplish specific tasks with Dagger. It includes code samples for common operations related to the host filesystem, builds, build inputs and outputs, secrets and caching, and we're always adding more to it. [Try the Dagger cookbook](https://docs.dagger.io/7442989/cookbook).

- Are you interested in using Dagger with other OCI runtimes, such as `podman` or `nerdctl`? We've added a guide on using [using Dagger with alternative OCI runtimes](https://docs.dagger.io/541047/alternative-runtimes).
 
- We don't (yet!) have a PHP SDK for Dagger...but you can still [build, test and publish a Laravel Web application with Dagger](https://docs.dagger.io/128409/build-test-publish-php) using our GraphQL API.
 
- Are you interested in creating a Dagger SDK for your favourite language? Check out our [SDK contribution guide](https://github.com/dagger/dagger/blob/v0.6.0/CONTRIBUTING.md).

Have comments on the new terminal UI? An interesting recipe to add to our cookbook? An idea for new documentation? Let us know in [Discord](https://discord.com/invite/dagger-io) or create a [GitHub issue](https://github.com/dagger/dagger/issues/new/choose).


Learn about the new terminal UI shipping with Dagger 0.6, plus other SDK and documentation news

Dagger 0.6: New Terminal UI

Most organizations have Dockerfiles somewhere. Some have a lot of them. They might be managed by another team, and maybe your team only interacts with the built images, but chances are that if you’re thinking about CI, you’ve thought about writing, maintaining, simplifying, or getting rid of your Dockerfiles.

If you’re using Dagger to turn your pipelines into code via a Dagger SDK in Go, Python, or Node.js (or via the GraphQL API directly), then you have a couple of options on how to handle your Dockerfiles.

* Keep your existing Dockerfiles and drive Docker builds from Dagger
* Turn your Dockerfiles into Dagger code

Let’s examine each of these.

#### Build from your existing Dockerfiles

You might want to do this if you have a deep investment in Dockerfiles and they are an important interface for developers or platform engineers. You might also want to focus on orchestrating the rest of your pipeline first and tackling your working Dockerfiles later.

It’s easy to read in a directory of content that includes a Dockerfile and produce an image with Dagger. You can also initialize a new Container in Dagger from a Directory with a Dockerfile in it.

##### Python:

```
import anyio
import sys
import dagger

async def main():
    async with dagger.Connection(dagger.Config(log_output=sys.stderr)) as client:
        build = client.host().directory(".").docker_build()
        await build.publish("jeremyatdockerhub/myexample:latest")

anyio.run(main)
```

Read more about [building Dockerfiles](https://dagger-io.readthedocs.io/en/latest/api.html#dagger.api.gen.Directory.docker_build) in the [Python SDK Reference](https://dagger-io.readthedocs.io/en/latest/).

##### Go:

```
package main


import (
 "context"
 "os"
 "dagger.io/dagger"
)


func main() {
 ctx := context.Background()
 client, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stderr))
 if err != nil {
   panic(err)
 }
 defer client.Close()


 _, err = client.Host().Directory(".").
   DockerBuild().
   Publish(ctx, "jeremyatdockerhub/myexample:latest")
 if err != nil {
   panic(err)
 }
}
```

Read more about [building Dockerfiles](https://pkg.go.dev/dagger.io/dagger#Directory.DockerBuild) in the [Go SDK Reference](https://pkg.go.dev/dagger.io/dagger). 

##### Node.js:

```
import { connect } from "@dagger.io/dagger"

connect(async (client) => {

  const imageRef = await client.host().directory(".").
    dockerBuild().
    publish("jeremyatdockerhub/myexample:latest")

}, { LogOutput: process.stderr })
```

Read more about [building Dockerfiles](https://docs.dagger.io/current/sdk/nodejs/reference/classes/api_client_gen.Directory#dockerbuild) in the [Node.js SDK Reference](https://docs.dagger.io/current/sdk/nodejs/reference/modules/api_client_gen).

##### GraphQL API:

```
query {
  host {
    directory(path: ".") {
      dockerBuild(dockerfile: "./Dockerfile") {
    	 publish(address: "jeremyatdockerhub/myexample:latest")
      }
    }
  }
```

Read more about [building Dockerfiles](https://docs.dagger.io/api/reference/#Directory-dockerBuild) in the [API Reference](https://docs.dagger.io/api/reference/).

#### Turn your Dockerfiles into Dagger code

Whether your Dockerfiles are super simple (and thus trivial to turn into Dagger SDK code), or super gnarly, converting your existing Dockerfiles into Dagger code means you can take full advantage of both Dagger and your language of choice to create testable, extensible builds and pipelines. It might be that Dockerfile syntax is a mystery to many of your engineers and thus folks are reluctant to touch them, or that you’ve outgrown the Dockerfile format and need more control that a full programming language can offer including types, functions, loops, and conditionals. 

If you’re familiar with Dockerfiles or the concepts behind them, you’ll find the Dagger API implemented by the SDKs has familiar constructs for working with Containers, Directories, Files, Git, Environment Variables, Secrets, etc.

Luckily we have an excellent [blog post](https://dagger.io/blog/replace-dockerfile-with-go) and [technical guide](https://docs.dagger.io/205271/replace-dockerfile) that illustrates the approach in Go, Python and Node.js. It shows you how to

* Understand and break down a Dockerfile into steps
* Create a Dagger client
* Write a Dagger pipeline to replicate the Dockerfile build process, taking advantage of functions and the Dagger API implemented by your SDK of choice to:
* Configure a container with all required dependencies and environment variables for an application
  * Download and build the application source code in the container
  * Set the container entrypoint
  * Publish the built container image to Docker Hub
* Test the Dagger pipeline locally

[Read the guide](https://docs.dagger.io/205271/replace-dockerfile).

#### Mapping Dockerfiles to Dagger

To help with mapping your Dockerfile to Dagger, here’s a handy table of some of the common Dockerfile commands and their Dagger equivalents. It actually helps to think of Dagger as much more than a Dockerfile equivalent, because it is! Some things are more akin to what you’d do with docker build or docker run or buildx. For example mounting directories, cache mounts, and build args. A lot of this functionality can be found in the docs under the Container type, so that’s a great place to browse.

| Dockerfile    | Python SDK                        | Go SDK                        | Node.js SDK                   |
| ------------- | --------------------------------- | ----------------------------- | ----------------------------- |
| FROM alpine:3 | container().from_(“alpine:3”)     | Container.From(“alpine:3”)    | container.from(“alpine:3”)    |
| COPY          | .with_file() or .with_directory() | WithFile() or WithDirectory() | withFile() or withDirectory() |
| RUN           | .with_exec()                      | WithExec()                    | withExec()                    |
| ENV           | .with_env_variable()              | WithEnvVariable()             | withEnvVariable()             |
| WORKDIR       | .with_workdir()                   | WithWorkdir()                 | withWorkdir()                 |
| ENTRYPOINT    | .with_entrypoint()                | WithEntryPoint()              | withEntryPoint()              |
| CMD           | .with_default_args()              | WithDefaultArgs()             | withDefaultArgs()             |
| USER          | .with_user()                      | WithUser()                    | withUser()                    |
| LABEL         | .with_label()                     | WithLabel()                   | withLabel()                   |


#### Conclusion

Whether you need to keep your existing Dockerfiles around for a while or are ready to turn them into Dagger SDK code, you’ve now got the tools to incorporate your existing Dockerfiles into your Dagger pipelines. As always, we’re available in [Discord](https://discord.com/invite/dagger-io) if you need help!

Whether you need to keep your existing Dockerfiles around for a while or are ready to turn them into Dagger SDK code, this blog post will help you navigate how to do it. 

Using Dockerfiles with Dagger

We’re always interested in hearing how our community is using Dagger – their use cases, their challenges, and their experiences with deploying Dagger in different environments. Our [Discord](https://discord.gg/dagger-io) is a great place to find these stories, and to benefit from the knowledge and experience of the Dagger community.

In this blog post, we’ll share the story of Dagger community member Kevin Minehart (aka @Kevin on Discord). Kevin, a software engineer at Grafana Labs, uses Dagger to create bespoke Grafana packages for end-users. Previously, Kevin built every package using a remote CI engine, which was a time-consuming process. With Dagger, Kevin is able to build packages locally in 8 minutes or less...a build-time improvement of ~75%!

> "With Dagger, we were able to go from something that takes an hour to something that takes 8 minutes."

#### The problem: Time-consuming, remote-only builds

Kevin and his team are often asked to deliver bespoke, pre-packaged versions of Grafana to potential customers. These packages share the same application codebase, but are customized to end-user requirements - for example, packages with support for multiple architectures, specific feature flags or pre-activated plugins. Before switching to Dagger, they built these packages using a standalone Go program (migrated from Bash scripts) containing multiple sub-commands.

There were a few challenges with this approach:

* Building an application package required multiple Git operations to be performed in a specific sequence. This was time-consuming and tedious, requiring manual coordination and control by an engineer.
* The Go program and sub-commands were themselves opaque, with limited documentation and visibility into how they worked. Engineers had to inspect the code to understand what each command did.
* Although the packaging code was written in Go, it could not run locally due to highly-specific environment requirements. In practice, the team could only run it in a customized Docker container on their CI engine.

> "Whenever I had to build Grafana for one or two different versions, it consumed most of my day."

#### The solution: Fast, consistent local and remote builds with Dagger

Kevin had heard about Dagger previously and had experimented with it in some personal projects. As he was already familiar with it, he decided to try and automate the Grafana packaging process using Dagger's Go SDK. With Dagger, he was able to consolidate all the packaging tasks into a standalone Dagger pipeline that could be invoked with a single command.

> "The caching has already proven to be a really big performance increase for us, especially whenever we're doing a lot of the same things locally."

After trying the new approach for a month, the team concluded that using Dagger offered numerous advantages over the previous system. Key benefits are:

* Dagger's built-in caching produces a significant increase in performance. The previous packaging process would, very often, take an hour or more; the Dagger pipeline typically delivers its results in 8 minutes.
* Dagger's multi-platform support allows the team to quickly and consistently build Grafana packages for multiple architectures and platforms: Linux (5), Mac OS (2) and Windows. 
* The same Dagger pipeline works consistently locally and in remote CI runners, eliminating much of the FUD of the previous process. With Dagger, every member of the team can build and test packages locally, despite having diverse development environments.
* The Dagger pipeline is invoked as a single process which contains all the business logic (including Git operations) necessary to produce a final package. This means that an engineer can trigger it and come back later to collect the result - no manual intervention or ongoing attention required.
* Dagger's Go SDK makes it easier for the team of mostly Go engineers to continuously improve the pipeline by adding tests, reusing Go modules and adopting existing best practices.

> "Dagger being open source was very helpful. I discovered quite a few functions and things that I should have been using in different places just by inspecting the code or looking at the Go docs."

#### The future: Using Dagger for all Grafana packaging processes

Going forward, Kevin plans to add service containers to his team's Dagger pipeline for integration testing. He is also hoping to use Dagger's support for remote caching when it becomes available. In the near future, he plans to start using Dagger to build Grafana automatically, and then replace the existing packaging processes with the new Dagger pipeline.

> “I don't think I can emphasize enough the value that Dagger has given us. We were not able to even build a Grafana package locally before this. So Dagger was a huge win for us.”

Do you have a Dagger story you’d like us to feature? Tell us all about it in [Discord](https://discord.gg/dagger-io)!

With Dagger, Kevin is able to build packages locally in 8 minutes or less...a build-time improvement of ~75%!

Dagger in Action: Building Grafana for Multiple Architectures in 8 Minutes or Less


Dagger allows you to use confidential information, such as passwords, API keys, SSH keys and so on, when running your Dagger pipelines. Keeping these secrets secure while also ensuring they are accessible to the right people is a common challenge.

At Dagger, we understand this challenge and are always looking for ways to make secrets management easier and more secure. That's why we're excited to announce the launch of a new dynamic secrets API in Dagger 0.5.

> “Build pipelines and deployments that don’t need secrets are about as useful as deploying to /dev/null. So with the new secrets API available with Dagger, my pipelines have gone from “fun idea” to “oh crap, we’re shipping to prod” and I couldn’t be more excited. Finally I have a programmatic and composable build pipeline I can run anywhere. And I really mean anywhere.” – [David Flanagan](https://twitter.com/rawkode), Founder, Rawkode Academy

#### The challenges of Dynamic Secrets

##### Before

![before dynamic secrets](/img/mermaid-diagram-2023-04-06-165223.png)

Until this release, Dagger could securely load secrets from a _fixed_ snapshot of the client machine’s environment variables. This works well for simple workflows, where all secrets are static and known in advance. However, many CI/CD workflows now require retrieving secrets _dynamically_, for example by querying a secret store such as [Hashicorp Vault](https://www.hashicorp.com/products/vault), executing a [cloud-specific authentication helper](https://github.com/awslabs/amazon-ecr-credential-helper), or [integrating with a password manager](https://1password.com). Implementing these workflows with Dagger was only possible with unreliable and possibly insecure hacks - until now.

##### After

![after dynamic secrets](/img/mermaid-diagram-2023-04-06-165535.png)

Our new secrets API enables you to retrieve secrets however you like, including from external sources such as Vault, Scaleway, GCP, Azure, AWS, and others (using their pre-defined systems, SDKs or tools).Then, you set them in your Dagger pipeline either as secret environment variables, or as mounted secret files.

Here’s a quick example, which reads a plaintext GitHub personal access token from a host environment variable named `GH_SECRET` and sets it as a secret for a Dagger pipeline:

```
package main

import (
   "context"
   "fmt"
   "os"

   "dagger.io/dagger"
)

func main() {
   // initialize Dagger client
   ctx := context.Background()
   client, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stderr))
   if err != nil {
       panic(err)
   }
   defer client.Close()

   // read secret from host variable
   secret := client.SetSecret("gh-secret", os.Getenv("GH_SECRET"))

   // use secret in container environment
   out, err := client.
       Container().
       From("alpine:3.17").
       WithSecretVariable("GITHUB_API_TOKEN", secret).
       WithExec([]string{"apk", "add", "curl"}).
       WithExec([]string{"sh", "-c", `curl "https://api.github.com/repos/dagger/dagger/issues" --header "Accept: application/vnd.github+json" --header "Authorization: Bearer $GITHUB_API_TOKEN"`}).
       Stdout(ctx)
   if err != nil {
       panic(err)
   }

   fmt.Println(out)
}
```

Notes:

- File secrets are mounted on a special mount, and if mounted elsewhere, they are not part of the image, nor the cache. This is the most secure option.
- Environment secrets are not part of the build environment (like `build-args` can be), either, so they don't leak in the build history (you can't access them using `docker history`, for example).

##### Security

As of [Dagger 0.4](https://dagger.io/blog/dagger-engine-0-4-0), Dagger scrubs secrets from its logs to ensure that sensitive information is protected, and it’s now better than before - our scrubbing strategy can now handle multi-line secrets such as SSH keys or TLS certificates. This applies to secrets stored in both environment variables and file mounts.

#### What's Next?

We have more improvements planned to the secrets API. For example, we're working on [improving our scrubbing strategy](https://github.com/dagger/dagger/issues/4864) to further reduce the risk of leaks. We're also exploring ways to give you [more control over secret mounts](https://github.com/dagger/dagger/issues/3323), such as UID/GID and Linux filesystem rights. If you're using a Dockerfile and want to pass a secret during the build, we don't support that yet, but have [an issue open](https://github.com/dagger/dagger/issues/4289) to fix it.

To learn more about our new secrets API, refer to our [documentation](https://docs.dagger.io/723462/use-secrets), which includes several code samples to get you started quickly. If you have feedback, please share your comments with us in [Discord](https://discord.com/invite/dagger-io) or create a [GitHub issue](https://github.com/dagger/dagger/issues/new/choose).

Want to help us pick the next improvements to the secrets API? Vote [on our latest discussion post](https://github.com/dagger/dagger/discussions/4910).


Dagger in Action: Restoring Developer Confidence in CI

Dagger in Action: How Flipt Improved Coverage and Build Times with Dagger

Use Dagger with AWS CodeBuild and AWS CodePipeline

Dagger 0.6: New Terminal UI

Using Dockerfiles with Dagger

Dagger in Action: Building Grafana for Multiple Architectures in 8 Minutes or Less

Dagger 0.5: New Dynamic Secrets API and more secure scrubbing