At Dagger, CI/CD is something that we think about a lot.

While you can [read our new guides](https://docs.dagger.io/278912/guides), watch [community call videos](https://www.youtube.com/playlist?list=PLyHqb4A5ee1tEgcr7KsNFzQSPN-R3fPs2), or even look at how our [various](https://github.com/dagger/examples) [repositories](https://github.com/dagger/registry-redirect/blob/3d9ea2214f7b2de88bda354a0035fa77f6cd3e48/magefiles/dagger.go) [use Dagger](https://github.com/dagger/dagger/tree/v0.3.13/internal/mage), this post aims to go beyond all that.

We want to share how we think about our own releases, what goes into a release & how they are produced. We will wrap-up this post with the release improvements that we are currently working on.

[This is a high-level sequence diagram of how all the pieces fit together](https://mermaid.live/edit#pako:eNqFVN2K00AUfpVhLsouNNNtkm2TyC6CFRF_EOqVxos0mSbpJpkhP67Z0jtvBVFwWUEEL7zyBXwdn8BH8MxM_rpaDLRzZs53vvOdMz9b7LOAYgevE3bpR15eoucLN0Pw5ZSzo5cu_v31_Q0K4zKqVsRn6STwwpDmzeDiV8cKHtDXAfMLGfDxi5jShIgVopAkZgBusB3wAzoAEdmRRrTzlhiNJFR51X9RrcLc4xHi1SqJi4jUaaI84qNZGGf0SKT5fIPuy5mLj3uAn8TKe43uPX6IRtmq4Hc6BM2CYS6p5-zsvKEFNRA-BCwNWdH1WxQkfT19mxRqldNLrfT44b5GLKUtqguDXEiDXiwNOXT-rBUXxkWZ16qn37v5QAdt6m8Iw8jPOw1g91JvI5t6RVoBlEbL3-YPmeL6iR4wtFw86tMwTTTuf6dIA1wRXAzD-EXYch5spmRftzCIICEjcFwm_4y4tY8hUzUpha0NJLC1ini_t-Iwyv7DALEjxOsyYhkYGdygTdEmUMtK1Dv0TIGGPeE1j1u3sAnLwwnP2Yb6ZduPvYvgBWVE-yvzCXUgMnD-dXlEkY1GUZxIBWIHEa1ipR_If_34hp7ChGyKPcUZTxsvWJtC7t_dg7vSJVe8MjnEKSce45TmqRcH8OZsxZqLQU4K580BM6Brr0pKF7vZDqBeVbJlnfnYKfOKjnHFA6-ki9iDa59iZ-0lBaxyL3vB2N4cO1v8BjvmlJzMLd04nRonFvwMc4xr7Ogzm1hze25atmUY5qk9243xlaSYEsM2dVPXrdnM0qe6PcY0iEuWP1HPpHwtd38AFWu-EA):

![release diagram](/img/pako_enqfvn2k00aufpvhlsounnntkm2tyc6cfrf_eoqvxos0msbpjpkhp67z0jtvbvfwwueel7zybxwdn8bh8mxm_rpadlrzzs53vvodmz9b7loaygeve3bpr15eoucln0pw5zszo5cu_v31_q0k4zkqvsrn6stwwpdmzediv8ckhtdxafmlgfdxi5jshigvopakzgbusb3wazoaedmrrrtzlhinjfr51x9rrclc4xhi1sq.png "release diagram")

#### How do we think of Dagger releases?

Our aim is to release as often as necessary, always in small batches - think one or two weeks worth of changes at most. Fixes and small improvements should be available to our users within a handful of days. Few things show that we care about our users' experience more than addressing their issues within days.

We tend to release larger features behind an experimental environment variable flag, e.g. `_EXPERIMENTAL_DAGGER_SERVICES_DNS`. This enables us to get real-world feedback, work on the feature together with our users, and reduce the risk of breaking things for the unsuspecting. As soon as we have sufficient feedback from users that care the most about a specific feature, we make it available to everyone else by default, without any experimental opt-in. The biggest advantage is that there is no last-minute integration effort: the feature has been available for many weeks already, just not the default option.

The last significant feature that we have released using this approach is [service containers](https://dagger.io/blog/dagger-engine-0-4-0).

#### What goes into a Dagger release?

We currently release five different artifacts:

1. Dagger Engine
2. Dagger CLI
3. Go SDK
4. Python SDK
5. Node.js SDK

The source of truth for all these releases lives in [github.com/dagger/dagger](https://github.com/dagger/dagger). We like this [monorepo](https://en.wikipedia.org/wiki/Monorepo) approach because it keeps things simple.

#### 🚙 Dagger Engine & 🚗 CLI

The Dagger Engine & CLI are always produced together, as a single release. This is what the v0.3.13 release looks like:

![release view](/img/screenshot_2023-03-21_at_17.02.50.png "release view")

Under **Assets**, there is a Dagger CLI for each platform that we support. The Engine is not listed here. It is published as a container image & available via [registry.dagger.io](https://registry.dagger.io). Users don't download this image directly. They set up their preferred SDK as a package dependency in their language of choice, which internally references the dependent CLI version. The Dagger CLI has all the platform-specific code necessary to download and set up the required Engine container image. This is transparent to the end-user.

*It is worth mentioning that every merge to main publishes a development version of the Engine & CLI. The CLI is available via `dl.dagger.io/dagger/main/<GIT_SHA>/dagger_<GIT_SHA>_<OS>_<PLATFORM>.tar.gz`. This is helpful for anyone that needs to test a fix or feature before a new version with that change is published.*

#### 🐹 Dagger Go SDK

All GitHub releases that start with the `sdk/go` tag are for the Go SDK. There are no assets, since Go packages are referenced via a URL. [These releases](https://github.com/dagger/dagger/releases?q=tag%3Asdk%2Fgo%2Fv0) are the source of truth for our Go SDK.

Our users are encouraged to use the `dagger.io/dagger` package which resolves to the [dagger/dagger-go-sdk](https://github.com/dagger/dagger-go-sdk) read-only repository replica - a limitation of how GitHub serves Go packages.

Dagger Go SDK is also published to [pkg.go.dev/dagger.io/dagger](https://pkg.go.dev/dagger.io/dagger). This is the best place to check the reference documentation. For all other Go SDK documentation, check out [docs.dagger.io/sdk/go.](https://docs.dagger.io/sdk/go)

#### 🐍 Dagger Python SDK

Python SDK releases on GitHub start with the `sdk/python` tag. This has the added benefit that [all release notes can be read as a single list](https://github.com/dagger/dagger/releases?q=tag%3Asdk%2Fpython%2Fv0).

We publish the Dagger Python SDK to [pypi.org](https://pypi.org), as <https://pypi.org/project/dagger-io/>. 

The reference documentation is published to [dagger-io.readthedocs.org](https://dagger-io.readthedocs.org/). As with all our SDKs, we release the main documentation to [docs.dagger.io/sdk/python](https://docs.dagger.io/sdk/python).

#### ⬢ Dagger Node.js SDK

Node.js SDK releases are prefixed with the `sdk/nodejs` tag. See the [changelog for all releases](https://github.com/dagger/dagger/releases?q=tag%3Asdk%2Fnodejs%2Fv0).

The Dagger Node.js SDK is published to [npmjs.com/@dagger.io/dagger](https://www.npmjs.com/package/@dagger.io/dagger).

All documentation - including the reference - is available via [docs.dagger.io/sdk/nodejs](https://docs.dagger.io/sdk/nodejs).

#### 📒 Docs

Documentation is a cross-cutting artifact, meaning that it is closely coupled to all the other artifacts mentioned above. Documentation has a different release cadence. Every merge to the `main` branch of [github.com/dagger/dagger](https://github.com/dagger/dagger) automatically triggers a build on Netlify for two separate websites:

1. <https://devel.docs.dagger.io/>
2. <https://docs.dagger.io/>

The important difference between the two websites is that the first one gets automatically published, while the second one - our live docs - requires a human decision. We manually check that all docs are there and everything looks good before we publish as <https://docs.dagger.io>

#### How are Dagger releases produced?

We use Dagger to release Dagger. The runtime context is GitHub Actions, since our code is already on GitHub. All workflows used for releasing start with `publish-`, e.g. `publish-sdk-go.yml`, `publish.yml`, etc. These workflows are triggered by specific git tags, e.g. `sdk/go/**`, `v**`, etc.

For example, if we look at [`.github/workflows/publish-sdk-go.yml`](https://github.com/dagger/dagger/blob/ad722627391f3e7d5bf51534f846913afc95d555/.github/workflows/publish-sdk-go.yml), we notice surprisingly little YAML - just 17 lines. The most important line is `./hack/make sdk:go:publish ${{ github.ref_name }}`. This kicks off the Dagger Go SDK pipeline, which publishes the Dagger Go SDK. That's right: we use the Go SDK to publish the Go SDK. First class dogfooding 🐶 

If you want to see the rest of the code, check out the `Publish` function in [`dagger/internal/mage/sdk/go.go`](https://github.com/dagger/dagger/blob/275a7135e4f95f0ba1d13a7f48cdfb467809b1e7/internal/mage/sdk/go.go#L95-L160). That Publish function represents the entire Dagger pipeline for publishing the Go SDK.

You can check the code that publishes the other SDKs by looking at the files next to `go.go`, i.e. `python.go` & `nodejs.go`. If it helps, the Go interface for all SDK pipelines is declared in the `all.go` file.

[The Dagger Engine & CLI GitHub Actions publish workflow](https://github.com/dagger/dagger/blob/ad722627391f3e7d5bf51534f846913afc95d555/.github/workflows/publish.yml) is slightly more involved: 292 lines of YAML. Almost all of this YAML is necessary since we are replicating what will happen on hosts that run Dagger for the first time. Running this inside Dagger would defeat the purpose: we are testing the correctness of the code that provisions Dagger. There is a specific line of interest: `./hack/make dagger:publish ${{ github.ref_name }}`. The Go code for this starts in `dagger/internal/mage/dagger.go`. From this file, follow the CLI & Engine references to see the rest. Noteworthy facts:

* We use the GoReleaser Pro container to publish the Dagger CLI
* We publish the Dagger Engine image to [ghcr.io](https://ghcr.io), but serve via [registry.dagger.io](https://registry.dagger.io) (first step towards adding more resiliency & lower global latency)

The other interesting fact is that a new Dagger Engine & CLI release will open a new PR that sets the Engine version across all SDKs to the one that was just released. Once this PR gets merged, new SDK versions are ready to be released.

If you are curious to find out more about how Dagger releases are produced, check out [github.com/dagger/dagger/RELEASING.md](https://github.com/dagger/dagger/blob/a3b647bad3eaae53074703541dd7cbe4f71288a2/RELEASING.md)

To summarize all the above, [this sequence diagram captures it all](https://mermaid.live/edit#pako:eNqdlc9u00AQxl9ltAdOyTqO4zbxoUKtUYqgVdRwQrls7Kmzqb1r1uuUEPXGFQmBBOKExIETL8Dr8AQ8Ams7bhvFJW198b_vN_vNeDy7IoEMkXgkwzc5igB9ziLFkokAc7BcS5EnU1QTUT1JmdI84CkTGobHwDL4-_3jN4i4nuVTGsjEClkUoVqftqGzivn8ExRGPNNqSSsp5dJCEXGB29DYqaiv7yGMb_RNll79x9NMJjhVeNnWLG1ihzvzaUeynYUXDfAa_Q3pRUQjSUNcWDeJ3VWN0Rr7AOky5VSqyEqVnGOg6wWbshz56yJ-gWsVVchCPcNQBlljaU4L5s-vHyDSZJ6VqT3dbbBe6ROUgW-VvtL60G4fHIDvwZEUmotc5lm8hBDTWC5LplYGipuoLAa_jAHPym8NT-Do5fNKURymp4p4Zx6M8mnMsxkIvKy1gVmCmQsFPGERblFjZxNrimw6ZFN0bJri0DQFXDfFhtokZiqrsdQqjJFl2AKkEYVFhzrUdhqhwzxJa9sLVBmXArgAFscw9l9kFYMivKs6Q1noHubHNKYVScvY6tG9BnTowQlXSiqTarT9frMsKQsuTI1r8zsNj5Z6ZnJ8lOm0ZK2yntvlHN3D2G25f6NXeI6qmGo77Z-aKUjn2eP8CwPPs8q_u4WfPsz_vez711IlE2mMxcZdpssfrv75tISYL5C0SIIqYTw0U35V4BNi5kSCE-KZyxDPWR7rCZmIKyMtJv54KQLiaZVji-RpaCKvN4X6oZkMr6W8fUu8FXlLPLtj0353sN9z7e5gr9_vd1tkSbyBTTt2p9N1-71e33WuWuRdidt033Fsx92zO-7Acd1Br0Uw5Fqqk2pTKvemq3-nLSFl):

![](/img/release-.png " sequence diagram")

#### Upcoming improvements to the Dagger release process

Now that we understand how all the pieces fit together, and what goes into producing a Dagger release, we can talk about what aspects we are thinking of improving next.

![](/img/screenshot_2023-03-21_at_17.04.43.png "release discord channel ")

New SDK releases are already announced in Discord, in the [\#releases channel](https://discord.com/channels/707636530424053791/1080642696265342996). We are yet to configure our release bot to announce Engine & CLI releases.

Publishing <https://docs.dagger.io> currently requires a manual step. In order to make releases flow smoother, new documentation should be automatically published when a new SDK release goes out. If you are interested, we started discussing this aspect in this [public GitHub issue](https://github.com/dagger/dagger.io/issues/1805).

The last & most significant improvement that we are thinking about are our release notes. Currently, these are generated manually, with the `gh` CLI. This is a simple first step, but we think that friendlier release notes are going to increase the chances of our users reading them. Pull request titles can only take us so far.

One idea is to define a simple pull request interface. For starters, each pull request should be clearly labeled in terms of the area that it impacts, e.g. *sdk/python*, *area/documentation*, *area/engine* etc. The other label is the type of change, e.g. *Feature*, *Fix*, *Docs*, etc. Also, when a pull request implements changes that warrant a minor version bump, it should be labeled with *release/minor*. Lastly, each pull request should include a RELEASE NOTES summary in the top comment that will be used for the final release notes, e.g. 

<details><summary>RELEASE NOTES</summary>  Sevices now has an Opt-in option. Follow the instructions in the PR to enable it. </details>

The other idea is for each pull request to add release notes in a *CHANGELOG* file. This is based on the [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) idea and is known to work well with <https://changie.dev>. We want to thank our community member @sheldonhull for mentioning this back in November 2022 in the context of [dsv-github-action](https://github.com/DelineaXPM/dsv-github-action/blob/main/RELEASE.md).

Before we wrap this blog post up, is there something that we could improve in our release process that you would benefit from? Let us know via [Discord](https://discord.com/invite/dagger-io).

How we release Dagger

> “Replacing our makefiles with Dagger has greatly simplified our process for deployments since it’s much faster (1-3 hours to 3 minutes)...The power, flexibility and capability behind Dagger is something that gets me excited.”

We’re always interested in hearing how our community is using Dagger – their use cases, their challenges, and their experiences with deploying Dagger in different environments. Our [Discord](https://discord.gg/dagger-io) is a great place to find these stories, and to benefit from the knowledge and experience of the Dagger community.

In this blog post, we’ll share the story of Dagger community member, [Andy Gonzalez](https://www.linkedin.com/in/-andrew-gonzalez/) (aka @duchi on Discord). Andy, an SRE team leader, used Dagger to speed up his production AWS Lambda deployments from a few hours to just a few minutes…a 93% reduction in processing time!

#### **The problem: Manual, time-consuming deployments and complex YAML**

Andy and his team deploy their application using a CircleCI pipeline. This pipeline is a 3000+-line YAML file that performs multiple tasks for their core application. Besides those core tasks, there is a need to deploy 15 different AWS Lambdas (serverless functions) to support the primary application across multiple environments and using different accounts. To begin with, this was a totally manual process, and incorporating it into the existing CI YAML was daunting. Nevertheless, something had to be done since the manual Lambda deployments took between 1 and 3 hours every time!

Each Lambda is unique – it performs a different task and has its own set of dependencies. Some examples include antivirus scanning of file uploads by customers, RDS snapshot backups in multiple regions, client side logging and more. These functions are updated and re-deployed once per week.

There were a few challenges with this approach:

* All the Terraform commands to deploy the Lambdas were defined in Makefiles. As the number of environments increased, these Makefiles started to become increasingly complex and hard to maintain.
* The entire YAML pipeline – all 3000+ lines of it – was heavily tied to CircleCI. This effectively locked the team into a specific platform and made it difficult to consider shifting to a different CI vendor in future.
* The Lambda deployment process had zero automation in place; it was manual and extremely time-consuming. Typically, it would take between 1-3 hours.

> “Before we started to use Dagger, we deployed our Lambdas one at a time between all of our environments. We have four accounts with each containing one or more ‘environments’, so this would take a long time. Here is what our old manual steps would look like: (1) Get credentials and specify environment; (2) Deploy or update Lambda using Terraform CLI; (3)  Verify Lambdas are behaving correctly; (4) Move on to the next environment.”

#### **The solution: Fast, parallelized deployment with Dagger and a matrix**

Andy had heard about Dagger and, as an initial proof-of-concept, he decided to try and automate the AWS Lambda deployment sub-process using Dagger. With Dagger, he was able to consolidate all the tasks related to Lambda deployment in a separate Dagger pipeline and run them alongside matrix jobs in CircleCI.

Under the new approach, the AWS Lambda deployment process is handled in a separate Dagger pipeline outside the CircleCI pipeline. When triggered, this pipeline performs the following tasks:

* It creates a Terraform plan and puts it in the comment of a PR.
* The Terraform plan is executed manually for each Lambda.
* Control then passes to CircleCI, which executes the Dagger pipeline for each environment using a matrix job.

**Key benefits of this new approach are:**

* Using a matrix job allows the Dagger pipelines for the different environments to run in parallel, significantly speeding up the deployment process. Lambdas are now deployed in roughly 3-4 minutes instead of the 1-3 hours required earlier.
* Deployment business logic is encapsulated in the Dagger pipelines, with CircleCI only providing the matrix job infrastructure and triggering the pipelines. This reduces the dependency on CircleCI and makes it easier to migrate to a different platform in future.
* Testing the pipeline logic is now much easier, as pipelines can be run locally. Deployments can also be performed either via local runs or from CircleCI.

> “Dagger has allowed our engineers to deploy at a higher velocity and focus on other items instead of the manual workflow of deploying Lambdas. Our new Dagger process looks like this: (1) Run Dagger pipeline in CircleCI; (2) Verify Lambdas are behaving correctly.” 

#### The future: Eliminating YAML and improving secrets management

Going forward, Andy plans to combine the separate Lambda deployment pipeline with the existing CircleCI pipeline and simultaneously begin migrating the CircleCI pipeline to Dagger. He expects this will make the deployment process easier to test and maintain, and less reliant on the CircleCI platform.

In the CircleCI matrix, each job currently uses OIDC to connect to AWS and pass credentials to Dagger via environment variables. Andy will be looking for a better way to pass secrets through to executions – for example, using Dagger to connect to AWS directly for secrets.

> “Dagger is an extremely powerful tool which allows users to upshift from any CI/CD platform without hesitation. I believe that Dagger is the future for CI/CD since the tool allows users to be so decoupled by the actual platform itself.”

Do you have a Dagger story you’d like us to feature? Tell us all about it in [Discord](https://discord.gg/dagger-io)!

Dagger in Action: Cutting Deployment Times from 3 Hours to 3 Minutes

Today we are delighted to introduce Dagger 0.4.0, featuring service containers, secrets scrubbing, and more.

##### Service containers

Service containers have been one of the most requested features. They are now finally available. This feature enables you to run network services - like databases or webapps - as ephemeral containers, directly inside a Dagger pipeline.

Some common use cases for this new feature are:

* Run a test database
* Run end-to-end integration tests
* Run sidecar services

Service containers come with the following built-in features:

* Service containers are started just-in-time, de-duplicated, and stopped when no longer needed
* Service containers are health checked prior to running clients
* Service containers are given an alias for the client container to use as its hostname

###### Using service containers

Binding a service to a container expresses a dependency: the service container needs to be running when the client container runs. The bound service container is started automatically whenever its client container runs.

Here's an example of creating an HTTP service and fetching from it:

<iframe src="https://play.dagger.cloud/embed/pQoE-5_0Ghg" className="embed"> </iframe>

Here, the `WithExposedPort()` method sets the ports that the container will listen on. Dagger checks the health of each exposed port prior to running any clients that use the service, so that clients don't have to implement their own polling logic.

Want to learn more about how services work in Dagger? [Check out our documentation](https://docs.dagger.io/757394/use-service-containers).

##### Secrets scrubbing

Dagger now automatically scrubs secrets from its various logs and output streams. This ensures that sensitive data does not leak - for example, in the event of a crash. This applies to secrets stored in both environment variables and file mounts.

Here's an example that demonstrates this in action:

```go
package main

import (
	"context"
	"fmt"
	"os"

	"dagger.io/dagger"
)

func main() {
	cleanupEnv := setDemoHostEnv("my_secret_file")
	defer cleanupEnv()

	ctx := context.Background()
	// create Dagger client
	client, err := dagger.Connect(ctx)
	if err != nil {
		panic(err)
	}
	defer client.Close()

	secretEnv := client.Host().EnvVariable("MY_SECRET_ENV").Secret()
	secretFile := client.Host().Directory(".").File("my_secret_file").Secret()

	output, err := client.Container().
		From("alpine@sha256:e2e16842c9b54d985bf1ef9242a313f36b856181f188de21313820e177002501").
		WithSecretVariable("MY_SECRET_ENV", secretEnv).
		WithMountedSecret("/my_secret_file", secretFile).
		WithExec([]string{"sh", "-c", `echo -e "env secret data: $MY_SECRET_ENV || secret file data: "; cat /my_secret_file`}).
		Stdout(ctx)

	if err != nil {
		panic(err)
	}

	fmt.Println(output)
}

func setDemoHostEnv(hostFilePath string) (cleanup func() error) {
	// Setting a test host environment variable
	const envName = "MY_SECRET_ENV"
	err := os.Setenv(envName, "Secret Env Content")
	if err != nil {
		panic(err)
	}

	// Setting a test host file
	const hostFileContent = "Secret File Content"
	err = os.WriteFile(hostFilePath, []byte(hostFileContent), 0o644)
	if err != nil {
		panic(err)
	}

	return func() error {
		return os.Remove(hostFilePath)
	}
}

```

The output will be:

```
env secret data: *** || secret file data: 
***
```

Instead of

```
env secret data: Secret Env Content || secret file data: 
Secret File Content
```

##### SDK Updates

We've also released new versions of our SDKs with support for all the new features in Dagger 0.4.0, plus various SDK-specific bug fixes and improvements.

For a complete list of improvements, refer to the changelog for each SDK:

* Go SDK 0.5.0 [changelog](https://github.com/dagger/dagger/releases/tag/sdk%2Fgo%2Fv0.5.0) and [documentation](https://pkg.go.dev/dagger.io/dagger)
* Node.js SDK 0.4.0 [changelog](https://github.com/dagger/dagger/releases/tag/sdk%2Fnodejs%2Fv0.4.0) and [documentation](https://docs.dagger.io/current/sdk/nodejs/reference/modules)
* Python SDK 0.4.0 [changelog](https://github.com/dagger/dagger/releases/tag/sdk%2Fpython%2Fv0.4.0) and [documentation](https://dagger-io.readthedocs.org/)

We look forward to releasing more features soon! Meanwhile, if you have feedback or would like to suggest new features or documentation, let us know on [Discord](https://discord.com/invite/dagger-io) or create a [GitHub issue](https://github.com/dagger/dagger/issues/new/choose).


Dagger 0.4: service containers, secrets scrubbing, and more

The term "reproducible build" refers to a build process that produces identical results on every run. With a reproducible build process, the build outputs of each run behave identically and they can be verified (via checksum) to be bit-by-bit identical.

#### The Importance of Being Reproducible

Reproducible builds are important for a number of reasons:

* They provide a way to reliably connect build artifacts to source code. The same source code always produces the same build output, regardless of the build environment.
* They provide a way to verify build artifacts. Different people can independently build a piece of source code and verify that their checksums are identical and/or match those provided by the original developer.
* They improve software supply chain security. When builds are always consistent, externally-injected changes (such as malware or spyware) will be immediately visible in a bit-wise comparison of build outputs.

#### Different Inputs = Different Outputs

As you might expect, creating reproducible builds is harder than it sounds, because of the large number of variables involved in the usual build process. These variables include:

* the host operating system
* the host timezone and locale
* the host environment configuration
* the compiler and its configuration
* the versions and configurations of build dependencies, such as system or third-party libraries
* file timestamps
* [and](https://reproducible-builds.org/docs/archives/) [many](https://reproducible-builds.org/docs/randomness/) [more](https://reproducible-builds.org/docs/build-path/)...

Implementing a reproducible build strategy thus implies extreme and strict control over all the variables involved in the build...not an easy task!

**Dagger and Reproducible Builds**

While Dagger doesn't claim to solve reproducible builds, we are working towards making them easier to implement. 

* Dagger executes your pipelines entirely as [standard OCI containers](https://opencontainers.org/). Containerization helps to reduce build variance by isolating the build environment from the host filesystem and environment. Containerization also makes it easier to use deterministic [build paths](https://reproducible-builds.org/docs/build-path/) and provides mechanisms to audit the build environment.
* For builds that depend on external resources, Dagger supports mounting pinned versions of those external resources. For example, you can use Dagger’s support for Git [repositories](https://docs.dagger.io/api/reference/#definition-GitRepository) and [references](https://docs.dagger.io/api/reference/#definition-GitRef) to easily retrieve a snapshot of a source code repository at an exact commit.
* Dagger also addresses part of what the [Reproducible Builds website](https://reproducible-builds.org/) calls “the [biggest source of reproducibility issues](https://reproducible-builds.org/docs/timestamps/)”: file timestamps. With Dagger, you can force all entries within a directory to have the same creation/modification timestamps. This eliminates one of the most common causes of variation in the build process.

Here's an example of this in action:

```
c, _ := dagger.Connect(ctx, dagger.WithLogOutput(os.Stdout))

   gitTag := "v0.3.10"
   repo := c.Git("https://github.com/dagger/dagger.git").Tag(gitTag).Tree()
   daggerBinary := c.Container().From("golang:1.19-alpine3.17").
   	  WithMountedDirectory("/src", repo).
   	  WithEnvVariable("CGO_ENABLED", "0").
   	  WithWorkdir("/src").
   	  WithExec([]string{"go", "build", "-o", "dagger", "./cmd/dagger"}).
   	  File("dagger")

    defaultTimestamp, _ := c.Container().From("alpine:3.17").
   	  WithMountedFile("/dagger", daggerBinary).
   	  WithExec([]string{"stat", "-c", "%y %s", "/dagger"}).
   	  Stdout(ctx)

    fixedTimestamp, _ := c.Container().From("alpine:3.17").
   	  WithMountedFile("/dagger", daggerBinary.WithTimestamps(0)).
   	  WithExec([]string{"stat", "-c", "%y %s", "/dagger"}).
   	  Stdout(ctx)

    fmt.Println("defaultTimestamp:", defaultTimestamp)
    fmt.Println("fixedTimestamp:", fixedTimestamp)

```

**The Future**

Containerization, resource pinning and timestamp fixation are only a few aspects of a reproducible build strategy. Other areas where Dagger could potentially be helpful are:

* Support for the special [SOURCE_DATE_EPOCH](https://reproducible-builds.org/docs/source-date-epoch/) variable ([recently added to buildkit](https://github.com/dagger/dagger/issues/4680)) 
* Support for disabling internet access during container execution, thereby forcing the build to be more "hermetic"
* Support for verifying if files downloaded via HTTP match a given checksum 
* Support for fine-grained source policies or “source pinning” ([also recently added to buildkit](https://github.com/dagger/dagger/issues/4679))
* Support for [tools that help you identify what about your build isn't deterministic](https://reproducible-builds.org/tools/)

Do you need support for one of the issues mentioned above? Leave us a comment in the issue to let us know.  We look forward to continuing to improve the reproducible builds experience with Dagger.

Reproducible Builds with Dagger


A few months ago, we released a [Python SDK for Dagger](https://dagger.io/blog/python-sdk). This SDK makes it easy to develop CI/CD pipelines in Python and run them on any OCI-compatible container runtime. Since the release, we’ve been continuously updating the SDK by adding new features and making changes to improve the code quality. In this blog post, I’ll discuss the “why” and “how” of one of our most recent improvements: adopting Ruff as the linter for our Python code.

Ruff is a new Python linter that is very fast because it’s built in Rust. In its inaugural post [“Python tooling could be much, much faster”](https://notes.crmarsh.com/python-tooling-could-be-much-much-faster), [Charlie Marsh](https://twitter.com/charliermarsh) (Ruff’s creator) goes into more detail over how it works, tradeoffs, and implications, which I think is worth a read.

### Why Ruff?

Charlie was one of our early Python SDK testers before we [launched](https://dagger.io/blog/python-sdk). At the time I noticed he built a new tool and [checked it out](https://notes.crmarsh.com/python-tooling-could-be-much-much-faster). I found it interesting but, as an early project, it certainly didn’t cross my mind to use it in Dagger then.

A few weeks ago I came across a [tweet from Charlie](https://twitter.com/charliermarsh/status/1613198373594578946) noting that [Pandas](https://pandas.pydata.org/) was moving to Ruff and that caught my attention because Pandas is a very well known and popular Python project. I took another [look](https://github.com/charliermarsh/ruff) and saw that several other popular open-source projects were using it as well. Some of which I’ve used, like [FastAPI](https://github.com/tiangolo/fastapi), [Pydantic](https://github.com/pydantic/pydantic), [Sphinx](https://github.com/sphinx-doc/sphinx) and [Hatch](https://github.com/pypa/hatch).

There were other signs that  pointed to Ruff’s growing popularity, including a tweet where [PyPI](https://pypi.org/) (the Python packages repository) reported that the [project had become critical](https://twitter.com/charliermarsh/status/1618809642079105025) (i.e., a project in the top one percent of downloads over the previous six months).

I thought it was time to give Ruff a try in Dagger, not because I was looking to solve a specific problem, but because I was curious. During experimentation, I found enough benefits to make the switch.

### Adopting Ruff

The [Adopt Ruff PR](https://github.com/dagger/dagger/pull/4399/files) demonstrates both a reduction in dependencies and numerous code improvements to satisfy the linter. Below, I’ll expand a bit more on what I found were the main benefits.

#### Speed

The main advantage of Ruff is its speed, but it wasn’t the deciding factor because our [codebase](https://github.com/dagger/dagger/tree/main/sdk/python) is small, so linting time wasn’t a problem for us. Linting approximately 6500 lines of code in 40+ files used to take over 3 seconds. With Ruff, it takes approximately 0.03 seconds. That’s 100x faster.

The speed bump unexpectedly improved my productivity, because Ruff is fast enough to lint while editing code and provide immediate feedback. I used to run the linter in the terminal before committing, but Ruff has good documentation on supporting multiple environments and IDEs, which led me to set up [ruff-lsp](https://github.com/charliermarsh/ruff-lsp). Now, I get the warnings directly in my IDE, in real-time.

#### Dependency reduction

<div style="text-align:center;margin:40px 0px;">
    <img width="355" height="386" src="/img/219827583-695b3735-a8e5-48f9-a3fd-d448e0c2db23.png"></img>
    <p style="font-size:1.5rem;">diff in lint dependencies and commands in the <a style="font-size:1.5rem;" href="https://github.com/dagger/dagger/pull/4399/files">Adopt ruff PR</a></p>
</div>

Reducing dependencies always feels good. This was a major advantage for us.

Like many Python projects, Dagger was using [flake8](https://github.com/pycqa/flake8) and [isort](https://github.com/PyCQA/isort) to lint our codebase, as well as [autoflake](https://github.com/PyCQA/autoflake) to automatically fix a few linting violations during formatting. Flake8 has a pluggable architecture, which means you need to install quite a few plugins if you want to cover more ground, adding to your dependencies.

Ruff has near feature parity with these tools and is implementing many popular flake8 plugins in its codebase, making them “batteries included”. So, your project has fewer dependencies, you get many more linting rules for free, and it all still runs faster.

#### Code improvements

With flake8, and even Ruff, you usually specifically select which rules you want to use for linting. With Dagger, I decided to take the opposite approach - excluding the ones I didn’t want but allowing everything else.

This means that when Ruff is updated (which usually results in more checks due to the implementation of more flake8 plugins), linting is likely to fail. But I like it that way since I get notified by GitHub’s [dependabot](https://docs.github.com/en/code-security/dependabot) automatically that there’s code that could probably be improved by following some best practice or avoiding a security or potential bug issue. That’s what linters are for.

Compared to the stagnation we had before, having only a short set of linter rules enabled, this is an easy win for continually improving the codebase.

### Future steps

- Ruff is released frequently. This is good but puts a burden on maintenance. I get a new [dependabot PR](https://github.com/dagger/dagger/pull/4546) for it almost daily but can’t review them everyday. Dependabot closes the previous PR and opens a new one so there’s only ever one to deal with at any time, but I don’t like to ignore it in the meantime. I’m thinking about changing the schedule frequency from daily to weekly and reserving a little bit of time each week to review all updates.
- We still use [black](https://github.com/psf/black) to format code. It’s also tied to the lint step because I consider it a linting violation if black has changes to make. So in both the lint and format steps, we’re only using Ruff and black now. It would be great to only use Ruff for both steps. It seems [others want it too](https://github.com/charliermarsh/ruff/issues/1904) and that some experimentation has started on this, so it may become a reality soon.

### Conclusion

Why not use a more performant language to write tooling for Python? Ruff started as a proof-of-concept to [question the status quo](https://notes.crmarsh.com/python-tooling-could-be-much-much-faster). On one hand, it limits contribution by pythonistas but on the other hand, as a user of the tool, I do appreciate the fast execution and reduction of dependencies while still getting many more rules in return.

I’m looking forward to seeing how this project evolves and if it inspires even more performant tools to emerge as well.


Adopting Ruff for Dagger’s Python SDK

It's the start of a new year and to celebrate, we've been hard at work adding improvements to Dagger. Our latest release of the Dagger is v0.3.10, which incorporates two significant new features for status reporting and container metadata management.

### Dagger 0.3.10 Highlights

#### New Pipeline API

Dagger now provides a [Pipeline API](https://docs.dagger.io/api/reference/#query-pipeline), which enables developers and operators to programmatically group related operations together into "pipelines" for status reporting and visualization purposes (e.g. logs). This new feature makes it easier to cut through the "noise" of your CI's daily operations and improves visibility of specific operations or events.

Pipelines are hierarchical and can contain any number of child pipelines. Here's an example of using this new API in Go:

```go
// `c` belongs to the root pipeline at this point
c, _ := dagger.Connect(ctx)

// everything using `c` is nested under the `sdk` pipeline from now on
c = c.Pipeline("sdk")

nodejs :=
c.Pipeline("nodejs").Container().From("node:16").WithExec([]string{"yarn install"}) // pipeline is `sdk/nodejs`
lint := nodejs.Pipeline("lint").WithExec([]string{"yarn lint"}) // 
pipeline is `sdk/nodejs/lint`
test := nodejs.Pipeline("test").WithExec([]string{"yarn test"}) // 
pipeline is `sdk/nodejs/test`
```

#### Support for Container Metadata

Dagger now allows developers to add and manage container metadata via support for the [Dockerfile LABEL instruction](https://docs.docker.com/engine/reference/builder/#label). For any given container, you can now add a label (withLabel), remove a label (withoutLabel), print the value of a label (label) or print all the labels (labels).

Here's an example of using these new methods in Go:

```go
c, _ := dagger.Connect(ctx, dagger.WithLogOutput(os.Stdout))

gitTag := "v0.3.10"
repo :=
c.Git("https://github.com/dagger/dagger.git").Tag(gitTag).Tree()
daggerBinary := c.Container().From("golang:1.19-alpine3.17").
    WithMountedDirectory("/src", repo).
    WithEnvVariable("CGO_ENABLED", "0").
    WithWorkdir("/src").
    WithExec([]string{"go", "build", "-o", "dagger", "./cmd/dagger"}).
    File("dagger")

publishTarget := "fake.registry.invalid/test/publish:latest"
labelName := "dagger.version"
_, err := c.Container().From("gcr.io/distroless/static-debian11")
     .WithFile("/bin/dagger", daggerBinary).
     WithLabel(labelName, gitTag).
     Publish(ctx, publishTarget)
if err != nil {
    return err
}

labelVal, err := c.Container().From(publishTarget).Label(ctx, labelName)
if err != nil {
    return err
}
fmt.Println(labelName, "=", labelVal)
```

### SDK Highlights

We've also released new versions of our SDKs with support for all the new features in Dagger 0.3.10, plus various SDK-specific bug fixes and improvements.

#### Targeted Builds

All SDKs now have an improved Build() method that allows users to build a particular target in a multi-stage Dockerfile. This was a frequently requested feature from the community. Here's an example using the Nodejs SDK:

```javascript
import { connect } from "@dagger.io/dagger";

connect(async (client) => {
    const src = client
    .directory()
    .withNewFile("Dockerfile",
      `FROM node:16-slim AS base
       CMD echo "base"
       FROM base AS stage1
       CMD echo "stage1"
       FROM base AS stage2
       CMD echo "stage2"
       `
    )

      const output = await client.container().build(src, { target: 
  "stage1" }).withExec([]).stdout()

    // This will output "stage1"
    console.log(output)
  
  }, { LogOutput: process.stdout })
```

#### Improved Error Messages

The Python SDK now provides more verbose error messages whenever a Dagger session connection cannot be established. This makes it quicker and easier to debug failed sessions. Here's an example:

Before

```
dagger.exceptions.ProvisionError: Dagger engine failed to start
```

After

```
dagger.exceptions.SessionError: Dagger engine failed to start:Command 
'dagger/dagger-0.3.9 session' returned non-zero exit status 1.Error: 
failed to run container: docker: Cannot connect to the Docker daemon 
at unix:///var/run/docker.sock.Is the docker daemon running?.
```

#### Other Improvements

For a complete list of improvements, refer to the changelog for each SDK:

* Go SDK 0.4.4 [changelog](https://github.com/dagger/dagger/releases/tag/sdk%2Fgo%2Fv0.4.4) and [documentation](https://pkg.go.dev/dagger.io/dagger)
* Node.js SDK 0.3.2 [changelog](https://github.com/dagger/dagger/releases/tag/sdk%2Fnodejs%2Fv0.3.2) and [documentation](https://docs.dagger.io/current/sdk/nodejs/reference/modules)
* Python SDK 0.3.1 [changelog](https://github.com/dagger/dagger/releases/tag/sdk%2Fpython%2Fv0.3.1) and [documentation](https://dagger-io.readthedocs.org/)

### New Documentation

Interested in using Dagger with GitHub? We've added a new end-to-end tutorial that teaches you how to use a Dagger pipeline to continuously build and deploy a Node.js application with GitHub Actions.

[Read the tutorial](https://docs.dagger.io/sdk/nodejs/620941/github-google-cloud/)

\
We look forward to releasing more features and documentation soon! In the meanwhile, if you have feedback or would like to suggest new features or documentation, let us know on [Discord](https://discord.com/invite/dagger-io) or create a [GitHub issue](https://github.com/dagger/dagger/issues/new/choose).

Dagger 0.3.10: New pipeline API, support for container metadata, and more

Dagger's approach to building CI/CD pipelines is unique. Our [SDKs](https://docs.dagger.io/#which-sdk-should-i-use) let you develop your pipelines as code, in a programming language you already know, and then execute those pipelines as standard [OCI containers](https://opencontainers.org/). This approach is both supremely portable and consistent (because your pipeline runs the same way locally and in a remote CI environment) and it has the added bonus of compatibility with the existing Docker ecosystem.

One of the most common questions users ask when first encountering Dagger is, "I already have a Dockerfile, why should I use Dagger?" And the answer (like most good answers) depends very much on the intended use case.

### Usage Scenarios

It's true that for most small and medium size projects without complex CI/CD needs, a Dockerfile can (and does) get the job done. But there are a number of scenarios where replacing a Dockerfile with a Dagger pipeline can yield significant benefits. Here are a few:

* Replacing monolithic CI dependency images: Sometimes, teams maintain one giant everything-including-the-kitchen-sink Dockerfile containing multiple dependencies so they only have to reference one image (like `myorg/ci`) in all their CI pipelines. Building and pushing these Dockerfiles becomes quite a chore, and teams must also deal with registry rate limits (if using a public registry) or maintain a private registry instead. Dagger lets you maintain images purely as code, so you never have to worry about building and pushing gigantic monolithic CI images.
* Sharing data between CI pipelines and application code: Since your Dagger pipeline is usually written in the same language as your application, it's much easier to share data between the two. For example, if you have a constant (like an image name) that you use in your actual production codebase, you can import it directly into your CI code rather than attempting to pass it as a Dockerfile argument.
* Greater re-usability and abstraction: Dagger lets you use general purpose programming languages and therefore supports more re-usability and abstraction. You can access native control structures like conditionals and loops, benefit from the language's existing testing tools, and add new functionality or integrate with third-party services using community extensions or packages. These operations are either extremely awkward or unsupported in the standard Dockerfile model, but very easy with Dagger. By the same token, Dagger pipelines also benefit from static typing and easier refactoring.
* Gradually rewriting a legacy Dockerfile: Sometimes you are stuck with a Dockerfile that is both very important, and painful to use. Everyone knows it should be replaced with something more powerful, ideally using a real programming language - but rewriting it all at once, without breaking compatibility, is too hard, so it never happens. Since Dagger can natively run Dockerfiles with full compatibility, it’s easy to wrap your existing Dockerfile in a Dagger pipeline, and gradually refactor it over time, without breaking your team’s workflow.

### Getting Started

If you've struggled with one or more of these scenarios, or if you're just curious about how it all works, we've put together a brief [guide on replacing a Dockerfile with a Dagger pipeline](https://docs.dagger.io/205271/replace-dockerfile). This guide explains how to use the Dagger Go SDK to perform all the same operations that you would typically perform with a Dockerfile. 

It illustrates the process by replacing the Dockerfile for the popular open source Memcached caching system with a Dagger Go pipeline and covers the following common tasks:

* Create a Dagger client in Go
* Write a Dagger pipeline in Go to:
* Configure a container with all required dependencies and environment variables
* Download and build the application source code in the container
* Set the container entrypoint
* Publish the built container image to Docker Hub
* Test the Dagger pipeline locally

We hope you find this guide useful and look forward to releasing more such guides soon! In the meanwhile, if you have feedback or would like to suggest new features or documentation, let us know in [Discord](https://discord.com/invite/dagger-io) or create a [GitHub issue](https://github.com/dagger/dagger/issues/new/choose).

Replacing your Dockerfile with Go code

It's been three weeks since we released the [Dagger Node.js SDK 0.1.0](https://dagger.io/blog/nodejs-sdk). Since then, we've received many questions and suggestions from the community and we've incorporated several of them in our latest release, the Dagger Node.js SDK 0.3.0.

This version introduces two breaking API changes. However, we believe the improved API allows you to write much shorter code, which is significantly better in the long run.

[Get Started](https://docs.dagger.io/sdk/nodejs/783645/get-started)

### Dagger Node.js SDK 0.3.0 Highlights

#### Optional parameters as objects

You can now use objects for optional parameters. 

* Required parameters can be provided directly to functions as positional arguments.
* Optional parameters can be provided through an object that lets you mix and match the options you want. 

B﻿efore: 

```javascript
const source = await 
client.container().from(“alpine:16”).withExec([“echo”, “foo”], 
undefined, undefined, undefined, true)
```

A﻿fter: 

```
const source = await 
client.container().from('alpine').withExec(['echo', 'foo'], 
{experimentalPrivilegedNesting: true})
```

[Read the SDK documentation](https://docs.dagger.io/current/sdk/nodejs/reference/modules)

#### Less boilerplate

You no longer need to pass IDs, which makes the code you need to write much shorter. 

B﻿efore: 

```
const source = await client.host().directory(".").id()
const c = await client
    .container()
    .from("node:16")
    .withMountedDirectory("/src", source.id)
```

A﻿fter: 

```
const source = client.host().directory(“.”)
const c = await client
    .container()
    .from("node:16")
    .withMountedDirectory("/src", source)
```

[Read the SDK documentation ](https://docs.dagger.io/current/sdk/nodejs/reference/modules)

#### Centralized error handling

Debugging and error handling is faster and easier with new, more precise error messages and codes. 

B﻿efore: 

```
file:///home/user/src/demos/node_modules/@dagger.io/dagger/dist/provisioning/docker-provision/image.js:255
        	throw new Error("failed to read port from engine session");
              	^
Error: failed to read port from engine session
	at DockerImage.<anonymous> (file:///home/dolanor/src/demos/node_modules/@dagger.io/dagger/dist/provisioning/docker-provision/image.js:255:19)
	at Generator.next (<anonymous>)
	at fulfilled
```

A﻿fter: 

```
Error: buildkit failed to respond
file:///home/user/src/demos/node_modules/@dagger.io/dagger/dist/provisioning/docker-provision/image.js:262
        	throw new EngineSessionPortParseError("No line was found to parse the engine port");
              	^
EngineSessionPortParseError [EngineSessionPortError]: No line was found to parse the engine port
	at DockerImage.<anonymous> (file:///home/dolanor/src/demos/node_modules/@dagger.io/dagger/dist/provisioning/docker-provision/image.js:262:19)
	at Generator.next (<anonymous>)
	at fulfilled (file:///home/dolanor/src/demos/node_modules/@dagger.io/dagger/dist/provisioning/docker-provision/image.js:4:58)
	at processTicksAndRejections (node:internal/process/task_queues:95:5) {
  cause: undefined,
  code: 'D103'
}
```

[Learn more about the new error types](https://docs.dagger.io/current/sdk/nodejs/reference/modules/common_errors)

#### Improved filesystem API

We've added new methods to make it easier to work with the container filesystem

* `Container.WithDirectory`: returns the container plus a directory written at the given path.
* `Container.WithFile`: returns the container plus the contents of the given file copied to the given path.
* `Container.WithNewFile`: returns the container plus a new file written at the given path.

##### W﻿ithNewFile

B﻿efore: 

```
let textFile = fs.openSync("myTextFile", "w")
fs.writeFileSync(textFile, "my text content")

let packageFile = await client
    .host()
    .workdir(undefined, ["myTextFile"])
    .id()

await client
    .container()
    .from("node")
    .withMountedDirectory("/tmp/filedir", packageFile.id)
    .exec([ "cat", "/tmp/filedir/myTextFile" ])
    .stdout()
    .contents()
```

A﻿fter: 

```
await client
    .container()
    .from("node")
    .withNewFile("/tmp/filedir/myTextFile", { contents: "my text content" })
    .withExec([ "cat", "/tmp/filedir/myTextFile" ])
    .stdout()
```

[Read the SDK documentation](https://docs.dagger.io/current/sdk/nodejs/reference/classes/api_client_gen.Container)

We believe these changes bring the overall experience of using the SDK closer to what Node.js developers expect and use elsewhere, and also result in shorter and more readable code.

Thank you to the Dagger Community for your continuous and quick feedback. We look forward to releasing more features and documentation soon! In the meanwhile, if you have feedback or would like to suggest new features or documentation, let us know on [Discord](https://discord.com/invite/dagger-io) or create a [GitHub issue](https://github.com/dagger/dagger/issues/new/choose).

Dagger Node.js SDK 0.3: API improvements, better error handling and less boilerplate


Here at Dagger, we are on a mission to make your CI/CD pipelines programmable in the language of your choice. In the last few weeks, we have made great progress towards that goal by releasing SDKs for [Go](https://dagger.io/blog/go-sdk), [Python](https://dagger.io/blog/python-sdk), [Javascript and Typescrip](https://dagger.io/blog/nodejs-sdk)t in rapid succession. So rapid in fact, that some of you wondered how exactly we managed to pull it off!

[![Tweets](/img/screen-shot-2022-11-30-at-1.15.40-pm.png "Tweet References")](https://twitter.com/felipecruz/status/1593297610919628800)

Here’s the answer: we were cheating. Our SDKs don’t actually run your pipelines! Instead they send pipeline definitions to the Dagger Engine, which runs them and returns the result in a language-agnostic way.

Under the hood, we use a powerful API technology called [GraphQL](https://graphql.org/). And as of today, **so can you**. We are excited to publicly release the Dagger GraphQL API: a unified interface for programming the Dagger Engine using any standard GraphQL client.

[![GraphQL diagram](/img/dagger_graphql-diagram_01.png "Where the GraphQL API fits in the Dagger architecture")](https://docs.google.com/document/d/1QOG3ydXjfj9zA6W09ENecvAQvUiTEuDRVPPpT-PEOrA/edit#)

This release includes:

- [API documentation](https://docs.dagger.io/api), including guides, examples and a complete reference.
- An [API playground](https://play.dagger.cloud/) to run pipelines straight from your browser!
- A [new CLI](https://docs.dagger.io/cli) to run pipelines directly from the command-line

![playground gif](/img/solomon-playground-short.gif "Run CI/CD pipelines straight from your browser with the Dagger API playground!")

#### What does this mean for me?

First, and most importantly, if you are currently using a Dagger SDK, you can safely ignore this announcement and continue using Dagger exactly as before.

However, this release creates exciting new possibilities for the Dagger platform:

1. **Developers** can now use Dagger to program their CI/CD pipelines in over 20 languages, thanks to [GraphQL’s large and growing list of client implementations](https://graphql.org/code/#language-support). Note that the experience of using a standard GraphQL client is not the same as an official Dagger SDK. This is because “vanilla” clients follow the current state of the art in GraphQL development, which requires developers to write their queries manually (ideally with the help of an [interactive playground](https://play.dagger.cloud)). Dagger SDKs on the other hand automatically generate client libraries to hide the underlying GraphQL language, for a more native and polished experience. Developers that already know and love GraphQL may prefer the “vanilla” method; others will prefer official SDKs when they are available. Besides the aesthetic preferences, both are equally robust and full-featured. Going forward, we will continue to expand the list of available SDKs, and make it easier for the community to contribute new SDKs.
2. **Devops engineers** can use the new CLI to [run pipelines on-the-fly from the terminal or a shell script](https://docs.dagger.io/cli/389936/run-pipelines-cli). This can be used for experimentation, troubleshooting, and to retrofit Dagger pipelines into any system that can run shell commands, including: Makefiles, Jenkinsfiles, Github workflows, Gitlab pipelines, CircleCI configurations, Ansible playbooks, and many more.
3. There are many ways to develop a Dagger pipeline, but only one way to run them: by sending GraphQL queries to the Dagger Engine. This gives **infrastructure engineers** a natural insertion point to control every aspect of how pipelines are run, regardless of how they were developed. Whether it’s engine provisioning, logging, monitoring, authentication, authorization, networking, artifact management, or any other aspect of a robust and secure CI infrastructure: Dagger’s architecture allows for all of it to be configured or overridden by the operator. This will be the primary focus of our roadmap in the coming weeks.

#### Why GraphQL?

Engineers are generally not known for agreeing on things - we like to argue! But one thing we all seem to agree on is that the CI/CD experience is often frustrating, and should be improved. But how?

![yaml test](/img/ga-actions.png "yaml test")

We believe the answer is to decouple pipeline _logic_ from pipeline _infrastructure_, which in turn allows decoupling the concerns of application and infrastructure teams. This can be achieved with a powerful architectural pattern called the [hourglass model](https://arxiv.org/pdf/1607.07183.pdf): the same pattern that enabled the Internet, Unix, and more recently, Docker.

To apply the hourglass pattern to CI/CD we need an API expressive enough to connect the two parts of the hourlgass: run any pipeline logic, on any pipeline infrastructure - and do it well.

That is where GraphQL comes in. GraphQL is a perfect fit for the CI/CD “hourglass”, for three reasons:

1. **Technical**: GraphQL is fundamentally a graph traversal language. And it turns out that pipeline execution is fundamentally a graph traversal problem - it’s no coincidence that “Dagger” starts with “[DAG](https://en.wikipedia.org/wiki/Directed_acyclic_graph)”! Our early prototypes ended up looking so much like GraphQL, that it quickly became obvious that we should just build on top of it instead of reinventing the wheel.
2. **Human:** GraphQL was designed to decouple concerns between frontend and backend teams, by enabling frontend teams to compose queries themselves from building blocks provided by the backend team. Meanwhile, Dagger seeks to decouple concerns between application and infrastructure teams, by enabling application teams to compose pipelines themselves from building blocks provided by the infrastructure team. In both case, decoupling of concerns allows everyone to be happier and more productive.
3. **Ecosystem**: no one company can solve this problem alone. We are already standing on the shoulders of giants by leveraging the container ecosystem. Thanks to the GraphQL ecosystem we are now standing on the shoulders of TWO GIANTS! Every time anyone in the GraphQL and container ecosystem solves a problem for themselves, they are also solving it for everyone in the Dagger ecosystem.

Ready to give it a try? You can follow our [introduction guide](https://docs.dagger.io/api/254103/build-custom-client), browse the [API reference](https://docs.dagger.io/api/reference/), or [experiment with the API playground](https://play.dagger.cloud).

As always, we love hearing from you, let us know how you enjoy using the API, and what cool things you build with it.

Happy coding!


Introducing the Dagger GraphQL API

Today we are proud to introduce the [Node.js SDK](https://docs.dagger.io/sdk/nodejs): a new way to develop your CI/CD pipelines as code and run them anywhere.\
​

**The Dagger Node.js SDK may be a good fit if you are...**

* A front-end or back-end end developer wishing your CI pipelines were TypeScript or JavaScript code that you can run locally in seconds, instead of YAML that can only run on a proprietary service
* A developer who needs CI/CD, and is looking for an excuse to learn TypeScript or JavaScript
* Your team's "designated devops person", hoping to replace a pile of artisanal scripts with something more powerful
* A platform engineer writing custom TypeScript tooling, with the goal unifying continuous delivery across organizational silos

This is our third SDK release in the past month, after [Go](https://dagger.io/blog/go-sdk) and [Python](https://dagger.io/blog/python-sdk). Our ultimate goal is to ensure all developers can develop their CI/CD pipelines in the same way they develop the rest of their code. If you are looking for a different SDK, [let us know](https://airtable.com/shrzABOn1wCk5yBF4). You can also subscribe to **[our newsletter](http://eepurl.com/hzIma5)** to be notified of future SDK launches.

**What is Dagger again?**

Dagger is a programmable CI/CD engine that runs your pipelines in containers.

**Programmable:** develop your CI/CD pipelines as code, in the same programming language as your application.

**Runs your pipelines in containers:** Dagger executes your pipelines entirely as [standard OCI containers](https://opencontainers.org/). This has several benefits:

* Instant local testing
* Portability: the same pipeline can run on your local machine, a CI runner, a dedicated server, or any container hosting service.
* Superior caching: every operation is cached by default, and caching works the same everywhere
* Compatibility with the Docker ecosystem: if it runs in a container, you can add it to your pipeline
* Cross-language instrumentation: teams can use each other’s tools without learning each other’s language

![diagram](/img/screen-shot-2022-10-25-at-9.22.41-am.png "diagram")

### **Dagger Node.js in Action**

In this demo, we’ll show you how to create a CI tool to test your Node.js application against multiple Node.js versions.

<iframe width="90%" height="400px" src="https://www.youtube.com/embed/cuqmq_aTNfY" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

**Want to try it yourself? Check out our [Getting Started Guide](https://docs.dagger.io/sdk/nodejs/783645/get-started).** 

##### **Want to Get Involved?**

We have lots of ways to contribute and stay engaged with Dagger updates. We look forward to seeing you in our community forum:

* Join us on [Discord](https://discord.gg/dagger-io)
* [Report an issue or tell us about your use case](https://github.com/dagger/dagger/issues/new/choose)
* Join our weekly community call or meet up with us at an event. Check out the [events page](https://dagger.io/events) to see the latest!

Introducing the Dagger Node.js SDK

Today we are proud to introduce the [Dagger Python SDK](https://docs.dagger.io/sdk/python): a new way to develop your CI/CD pipelines as code and run them anywhere.

​
**The Dagger Python SDK may be a good fit if you are...**

* A Python developer wishing your CI pipelines were Python code that you can run locally in seconds, instead of YAML that can only run on a proprietary service
* A developer who needs CI/CD, and is looking for an excuse to learn Python
* Your team's "designated devops person", hoping to replace a pile of artisanal scripts with something more powerful
* A platform engineer writing custom Python tooling, with the goal of unifying continuous delivery across organizational silos
* A data engineer looking to better integrate with your organization's CI/CD pipelines.

Just two weeks ago we announced the [Dagger Go SDK](https://docs.dagger.io/sdk/go), and will continue to release more SDKs in the future. Our ultimate goal is to ensure all developers can develop their CI/CD pipelines with the language of their choice, and enjoy all the benefits of running them with Dagger.

If you are looking for a different SDK, let us know by [filling this form](https://airtable.com/shrzABOn1wCk5yBF4). We will notify you when your SDK is ready.

**What is Dagger again?**

Dagger is a programmable CI/CD engine that runs your pipelines in containers.

**Programmable:** develop your CI/CD pipelines as code, in the same programming language as your application.

**Runs your pipelines in containers:** Dagger executes your pipelines entirely as [standard OCI containers](https://opencontainers.org/). This has several benefits:

* Instant local testing
* Portability: the same pipeline can run on your local machine, a CI runner, a dedicated server, or any container hosting service.
* Superior caching: every operation is cached by default, and caching works the same everywhere
* Compatibility with the Docker ecosystem: if it runs in a container, you can add it to your pipeline
* Cross-language instrumentation: teams can use each other’s tools without learning each other’s language

![Diagram](/img/screen-shot-2022-10-25-at-9.22.41-am.png "Diagram ")

### **Dagger Python SDK in Action**

In this demo, we’ll show you how to create a CI tool to test your Python application against multiple Python versions.

<iframe width="90%" height="400px" src="https://www.youtube.com/embed/c0bLWmi2B-4" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

**Want to try it yourself? Check out our [Getting Started Guide](https://docs.dagger.io/sdk/python/628797/get-started).** 

##### **Want to Get Involved?**

We have lots of ways to contribute and stay engaged with Dagger updates. We look forward to seeing you in our community forum:

* Join us on [Discord](https://discord.gg/dagger-io)
* [Report an issue or tell us about your use case](https://github.com/dagger/dagger/issues/new/choose)
* Join our weekly community call or meet up with us at an event. Check out the [events page](https://dagger.io/events) to see the latest!

Develop your pipelines as code with the Dagger Python SDK

As you might have seen, we released the [Dagger Go SDK 0.3.0](https://dagger.io/blog/go-sdk) early last week. Thanks to your many questions and suggestions, we were able to ship several additional improvements, which we are releasing today as the Dagger Go SDK 0.4.0.

Note that this version introduces a breaking API change. However, we believe the improved API allows you to write much shorter code, which is significantly better in the long run.

**[Get Started](https://docs.dagger.io/sdk/go)** 

### Dagger Go SDK 0.4 Highlights:

#### Multi-platform pipelines

* Dagger now supports pulling container images and executing commands on platforms that differ from the underlying host.
* For example, you can use Dagger to compile binaries that target different CPU architectures, test those binaries and push them to a registry bundled as a multi-platform container image, all on a single host. 

**[Learn more about Dagger’s multi-platform support](https://docs.dagger.io/sdk/go/406009/multiplatform-support/)**

#### Improved filesystem API

* Import any directory from the host filesystem with `Host.Directory.` This replaces the more cumbersome LocalDirs, `Host.Workdir` and `HostDirectory.Read`
* Export any directory to the host filesystem with `Directory.Export`. This replaces `HostDirectory.Write` 
* Export any file to the host filesystem with `Directory.Export`
* Export any container to an OCI archive on the host filesystem with `Container.Export`. 
* `Host.Directory`, `Host.Workdir`, and `Directory.WithDirectory` now support `include:/exclude:` params for filtering the set of files to select.

**[See more examples of working with the host filesystem](https://docs.dagger.io/421437/work-with-host-filesystem/)**

#### Less boilerplate

You no longer need to pass IDs, which makes the code you need to write much shorter. See the example below: 

Before: 

```
modules := c.Directory()
for _, f := range []string{"go.mod", "go.sum", "sdk/go/go.mod", "sdk/go/go.sum"} {
	fileID, err := workdir.File(f).ID(ctx)
	if err != nil {
		return err
	}

	modules = modules.WithCopiedFile(f, fileID)
}
```

After: 

```
modules := c.Directory()
for _, f := range []string{"go.mod", "go.sum", "sdk/go/go.mod", "sdk/go/go.sum"} {
	modules = modules.WithCopiedFile(f, workdir.File(f))
}
```

**[Read the API documentation](https://pkg.go.dev/dagger.io/dagger)**

### New Documentation

Along with the new release, we’ve added four [guides](https://docs.dagger.io/sdk/go/275922/guides/) to our [documentation](https://docs.dagger.io/) that answer the most common questions we’ve heard in the days following the initial release. We hope these guides will help you make the most of the Go SDK.

* [Work with the Host Filesystem](https://docs.dagger.io/421437/work-with-host-filesystem/): This guide explains how to interact with the host filesystem using the Go SDK. It includes code examples of setting the working directory on the host, mounting a host directory in a container, exporting a directory from a container to the host, and more.
* [Understand Multi-Platform Support](https://docs.dagger.io/sdk/go/406009/multiplatform-support/): This guide demonstrates different techniques to build applications for multiple platforms using the Go SDK. It includes code examples of building multi-platform applications using emulation and cross-compilation.
* [Copy Embedded Directories into a Container](https://docs.dagger.io/110632/embed-directories/): This guide explains how to copy an entire directory tree from a host to a container.
* [Use Dagger with Private Git Repositories](https://docs.dagger.io/sdk/go/710884/private-repositories/): You can use `ssh-add` to securely authenticate your Dagger Go CI pipeline against private Git repositories.This guide shows you how. 

Thank you to the Dagger Community for your continuous and quick feedback through our **[Discord channel](https://discord.com/invite/dagger-io)** and **[GitHub issues](https://github.com/dagger/dagger/issues/new/choose)**. We look forward to releasing more features and documentation soon! 

Are you looking for updates on a different SDK? Fill out this **[form](https://blocklayer.typeform.com/to/a6m5gKSS?typeform-source=dagger.io)**, and we will notify you when your SDK is ready. 

Do you have feedback on our documentation? Would you like to suggest a new guide? Send us your feedback and ideas in our **[Discord channel](https://discord.com/invite/dagger-io)** or **[open a new issue on GitHub](https://github.com/dagger/dagger/issues/new?assignees=&labels=area%2Fdocumentation&template=documentation.yaml).**

Dagger Go SDK 0.4: multi-platform, improved filesystem API and more

Today we are proud to introduce the [Dagger Go SDK](https://docs.dagger.io/sdk/go): a new way to develop your CI/CD pipelines as code and run them anywhere.[​](https://devel.docs.dagger.io/8g34z/get-started-sdk-go/#who-is-it-for)

**The Dagger Go SDK may be a good fit if you are...**

* A Go developer wishing your CI pipelines were Go code that you can run locally in seconds, instead of YAML that can only run on a proprietary service
* A developer who needs CI/CD, and is looking for an excuse to learn Go
* Your team's "designated devops person", hoping to replace a pile of artisanal scripts with something more powerful
* A platform engineer writing custom Go tooling, with the goal unifying continuous delivery across organizational silos
* A cloud-native developer advocate or solutions engineer, looking to demonstrate a complex integration on short notice

**The Dagger Go SDK may not be a good fit if you are...**

* A developer who doesn't know Go, and is not interested in learning it. If you are looking for another SDK, let us know [here](https://airtable.com/shrzABOn1wCk5yBF4). We will notify you when your SDK is ready.
* A "designated devops person" who doesn't think of themselves as a developer - nothing makes you happier than a battle-hardened shell script.
* Someone who loves programming in YAML, thank you very much.
* A container skeptic: the less containers are involved, the happier you are.

In the future, we will release more SDKs. Our ultimate goal is to ensure all developers can develop their CI/CD pipelines in the same way they develop the rest of their code. If you are looking for a different SDK, [let us know](https://dagger-io.typeform.com/to/a6m5gKSS). If you fill out the form, we will notify you when your SDK is ready!

**What is Dagger again?**

Dagger is a programmable CI/CD engine that runs your pipelines in containers.

**Programmable:** develop your CI/CD pipelines as code, in the same programming language as your application.

**Runs your pipelines in containers:** Dagger executes your pipelines entirely as [standard OCI containers](https://opencontainers.org/). This has several benefits:

* Instant local testing
* Portability: the same pipeline can run on your local machine, a CI runner, a dedicated server, or any container hosting service.
* Superior caching: every operation is cached by default, and caching works the same everywhere
* Compatibility with the Docker ecosystem: if it runs in a container, you can add it to your pipeline
* Cross-language instrumentation: teams can use each other’s tools without learning each other’s language

![Dagger Diagram ](/img/screen-shot-2022-10-25-at-9.22.41-am.png "Dagger Diagram ")

#### Dagger Go SDK in Action

In this demo, you will see how quickly you can create a Go CI tool that builds a Go application with multiple architectures and Go versions using the Dagger Go SDK.

<iframe width="90%" height="400px" src="https://www.youtube.com/embed/GgMskf-znh4" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

**Want to try it yourself? Check out our [Getting Started Guide](https://docs.dagger.io/sdk/go/).**

##### **Want to Get Involved?**

We have lots of ways to contribute and stay engaged with Dagger updates. We look forward to seeing you in our community forum:

* Join us on [Discord](https://discord.gg/dagger-io)
* [Report an issue or tell us about your use case](https://github.com/dagger/dagger/issues/new/choose)
* Join our weekly community call or meet up with us at an event. Check out the [events page](https://dagger.io/events) to see the latest!

Your CI pipelines should be code: introducing the Dagger Go SDK

A few days after [launching the public beta of Dagger](https://dagger.io/blog/public-launch-announcement), we asked the community a simple question: [how can we improve our developer experience](https://github.com/dagger/dagger/discussions/2052)? The most popular request was for a [VS Code](https://en.wikipedia.org/wiki/Visual_Studio_Code) extension with proper [Language Server](https://en.wikipedia.org/wiki/Language_Server_Protocol) integration. So we took the community's advice and got to work.

Today, we're happy to introduce the first version of [Dagger for VS Code](https://marketplace.visualstudio.com/items?itemName=Dagger.dagger), bringing a first-class Dagger development experience straight to one of the world's most popular IDE. We're also open-sourcing [our LSP implementation for CUE](https://github.com/dagger/cuelsp), so that every other IDE and text editor can benefit from an improved CUE development experience as well. For example, here is a [guide for adding CUE LSP support to vim](https://github.com/dagger/cuelsp/blob/main/docs/vim.md).

#### Features

For the first release, Dagger for VS Code includes three features: *jump to definition*; *hover*; and *syntax highlighting*. All three features can be used from other IDEs, using the [cuelsp implementation](https://github.com/dagger/cuelsp).

##### Jump to Definition

![Jump to Definition Example](/img/jump-to-def.gif)

Now, we have a simple way to explore the [Dagger Universe](https://universe.dagger.io) and all its definitions. The Dagger Universe is a collection of reusable components, so we can use the “go to definition” feature to pick the one you need quickly.

##### Hover

![Hover Example](/img/hover.gif)

You will need a basic understanding of [CUE definitions](https://cuetorials.com/overview/foundations/#definitions) to get the most out of Dagger. To decrease the learning curve, we have added the ability to show references inline. This provides an easy way to see the overall description of an action and its internal type.

When using “go to definition” and “show definitions”, you will have a complete overview of all CUE definitions used in your Dagger project.

##### Syntax highlighting

We took inspiration from the Golang syntax to create our LSP theme so you can find the essential information quickly.

![lsp theme](/img/lsp-theme.png)

#### What’s an LSP?

A Language Server Protocol (LSP) is a standard used between an editor and a language server to provide language-specific features like auto(completion), go to definition, show references, etc. An LSP is a developer's best friend since it enables an easier way of exploring and understanding code libraries, directly from your code editor:

![LSP Example](/img/buildkit-schema-2-1-.png)

#### The CUE LSP challenge

Creating a CUE language server differs from standard languages like Golang, Python, or C++. In a typical language server, you work directly with the files and interpret them to provide language features.

With CUE, it’s the same, but you also need to work with the CUE engine to compile CUE values to simplify the implementation further.

When creating the Dagger CUE LSP, this constraint added a layer of complexity that we addressed by building a multi-component architecture. We decided to split the responsibility of file loading, jump to definition, hover, etc., into small pieces of code so we can extend it with new features later.

This resulted in the architecture below:

![Dagger CUE LSP Architecture](/img/lip-architecture-1-.png)

#### What’s next?

If you're using VS Code, give [the Dagger extension](https://marketplace.visualstudio.com/items?itemName=Dagger.dagger) a try! If you are using vim, [follow our CUE LSP setup guide](https://github.com/dagger/cuelsp/blob/main/docs/vim.md).

We love hearing from you! To give us feedback, you can [start a discussion](https://github.com/dagger/dagger/discussions), [create an issue](https://github.com/dagger/dagger/issues/new/choose), or [join our Discord server](https://discord.gg/ufnyBtc8uY).


Introducing Dagger for VS Code with CUE LSP support


There’s a running joke in the developer community that the physical supply chain–the one moving goods around the globe– is far more efficient than the software supply chain. And with essentially all computing moving to the cloud, software supply chains are growing in size and complexity. Devops engineers tasked with automating them are under increasing pressure to break through the bottleneck.

My co-founders [Sam Alba](https://twitter.com/sam_alba), [Andrea Luzzardi](https://twitter.com/aluzzardi) and I had seen the same problem time and time again: burned out devops teams trying their hardest to cobble together deployment pipelines that are nearly impossible to scale or port to other cloud platforms. The app deployment experience is painful, and solving this problem is the holy grail of devops.

This is the problem that we aspire to solve with Dagger. Devops engineers using Dagger get a standardized set of building blocks that they can assemble any way they want, knowing that the result will run reliably on any cloud. We’ve been working with some of the best engineers in the world to develop the platform and are [excited to launch the public beta version today](https://dagger.io/blog/public-launch-announcement).

And we’re just getting started. Today, we also announce the closing of a $20 million Series A investment led by Redpoint Ventures, with participation from Y Combinator, Nat Friedman (former CEO, Github), Brian Stevens (former CTO, Google Cloud and former CTO, Red Hat), Idit Levine (founder and CEO, solo.io), Julius Volz (creator of Prometheus), Ellen Pao (former CEO, Reddit), Daniel Lopez (co-founder, Bitnami), and others. Erica Brescia at Redpoint will be an invaluable asset as the newest addition to our board. Erica has been a leader in the devops space for over a decade, as the former co-founder of Bitnami (acquired by VMWare) and former COO of GitHub. We couldn’t imagine a better partner to build an industry-defining platform business.

We would also like to take the opportunity to thank [New Wave](https://newwave.vc), who led our $7 million seed funding round last year. As European founders we are proud to count a European fund as one of our major investors, proof that excellence in venture capital has no borders. Our total funding raised now amounts to $30 million, and we’re thrilled to keep growing our team and developing the Dagger platform with the open source community.

If you or anyone you know wants to join the team building the next stage of the devops revolution, [we would love to chat](https://dagger.io/careers).


Announcing our $20M Series A from Redpoint Ventures


Hi! We’re the creators of Docker, and today we’re proud to introduce our new project, [Dagger](https://dagger.io). Dagger is a portable devkit for CI/CD pipelines. It allows devops engineers to build powerful CI/CD pipelines quickly, then run them anywhere. Benefits include: unifying dev and CI environments; testing and debugging pipelines locally; and avoiding CI lock-in. Most importantly, it introduces an ecosystem of truly composable software components: a “lego system” which we hope will help the devops community solve its fragmentation problem. Under the hood, Dagger is powered by two of the most exciting technologies in the cloud-native ecosystem: [Buildkit](https://github.com/moby/buildkit) and [CUE](https://cuelang.org).

We’ve been developing Dagger together with a community of early adopters. Now we are ready to continue developing it [in the open](https://github.com/dagger/dagger). If you’d like to participate, [join us on Discord!](https://discord.gg/dagger-io).

To our alpha and private beta testers: _thank you_ for investing your time and energy to help us test, review and develop Dagger. We dedicate this launch to you, and we cannot wait to grow this community together.

We’re also hiring! If you’d like to help us improve the lives of devops engineers, [we’d love to talk to you](https://dagger.io/careers).

#### Why Dagger?

##### A developer’s dream is a devops engineer’s nightmare

![engineer automating delivery](/img/cicd-fragmented.png)

There’s never been a better time to be a developer. Thanks to an ever-expanding selection of specialized tools and cloud infrastructure, a dedicated team of developers can ship a competitive product in almost any industry within months. The result is unprecedented growth and innovation at every level of the stack.

This developer’s dream creates a nightmare for devops engineers, whose job is to automate the delivery of increasingly complex applications to increasingly complex cloud infrastructure. In order to continuously build, test and deploy their software, developers need a CI/CD pipeline that glues together an ever-expanding collection of specialized tools. The result is a mess of homemade scripts that are painful to write, difficult to debug and test, and must be rewritten for each new development and CI environment. Component reuse between pipelines is low, which causes fragmentation and a constant reinvention of the wheel.

##### Modernizing the CI/CD development experience

![astronaut writing a plan](/img/unifed-automated-resilient.png)

Dagger solves the devops engineer’s nightmare by telling them: “your pipelines are software. This makes you a developer, and you deserve a proper developer experience”. For example:

- Instead of gluing pipelines together with throwaway scripts, they can “play lego” by composing reusable actions, using an intuitive declarative language and API.
- Actions can safely be shared and reused thanks to a complete package management system, and there is a growing catalog of open-source actions developed by the community.
- Pipelines can be tested and debugged locally, then run on any CI system
- All actions are automatically cached: no manual tweaking necessary

This allows devops engineers to continuously improve their pipelines, with the confidence afforded by modern development practices.

##### Write your pipeline once, run anywhere

![astronaut deploying](/img/write-pipeline-once.png)

Because Dagger runs equally well anywhere a Docker-compatible runtime is available, it eliminates two major problems in the devops experience: dev/CI drift, and CI lock-in.

- _Dev/CI drift_: instead of writing the same automation twice - once in a CI-specific configuration, and again in a Docker compose file, shell script or Makefile - simply write a Dagger plan once, then run it in all environments.
- _CI lock-in_: migrating from one CI system to another, or supporting teams with multiple CI systems, should not require rewriting entire pipelines from scratch! The same Dagger plan can be run without modification on all major CI systems.

#### What’s next?

Dagger is still in the early stages of development. Thanks to a [new round of funding](https://dagger.io/blog/series-a) we will continue to expand our team, and continue building the ultimate CI/CD experience: more portable, more productive… and more fun!

If you share our definition of fun, [come say hi](https://discord.gg/dagger-io)! Our community is open to all, from beginners to seasoned devops veterans. We look forward to building something great together.

Happy hacking!


Introducing Dagger: a new way to create CI/CD pipelines


#### Today we are proud to introduce dagger 0.2, codename Europa.

This is our first major release since the beginning of our private beta last year, and it is based entirely on your feedback.

To all of you who have invested the time to test our nearly 40 releases; to report issues, suggest improvements, contribute and review code and documentation: thank you! This release would not be possible without you.

#### Migrating from 0.1 to 0.2

Please note that, unfortunately, this release is not backwards compatible: plans written for dagger 0.1 will not run on version 0.2. Automatic migration is not supported, but manual migration is relatively straightforward. Going forward, we plan to make backwards compatibility a priority, either by not breaking APIs, or providing seamless migration if we do. If you need help migrating your plan, we will be happy to assist you!

[Learn more about migrating to 0.2](https://docs.dagger.io/1214/migrate-from-dagger-0.1).

#### What's new in dagger 0.2?

This release has one focus: improve the developer experience. It features a redesigned CLI and powerful new APIs.

The result is a completely new way to develop CI/CD pipelines: more productive, more portable… and way more fun!

##### **One simple configuration for everything: the Plan**

We have radically simplified the process of setting up and configuring a pipeline. From having to learn half a dozen different commands and configuration files... to just one step: write the plan.

In Dagger 0.2, every aspect of your pipeline is controlled, declaratively, in your plan. In addition to making Dagger simpler to understand and use, it makes it natively compatible with declarative patterns such as gitops.
[Learn more about the Dagger Plan](https://docs.dagger.io/1202/plan).

##### **Lego for your pipelines: the Action API**

Dagger’s superpower is its lego-like composition model: simple actions can be composed to form a more complex one; which in turn can be composed again; and so on, until even the most complex pipelines can be assembled at a fraction of the usual cost, and at higher quality.

The foundation of this model is the Action API: a complete framework for composing arbitrarily complex automations out of standardized primitives, declaratively, using CUE and Buildkit.
[Learn more about the Action API](https://docs.dagger.io/1213/api).

##### **One simple command: dagger do**

We have simplified the dagger CLI so that anyone can intuitively understand what your pipelines do and how to use them, even if they’ve never used Dagger. No knowledge of CUE required!

Running a pipeline is as simple as _dagger do ACTION_. For example, _dagger do build_, _dagger do test_, _dagger do deploy_. Actions are loaded directly from your plan, so you can declaratively control your pipeline's command-line experience. Best of all, _dagger do --help_ dynamically loads available actions and presents them to the user.

##### **Integrate dagger anywhere: the Client API**

In dagger 0.2, a plan may use the Client API to interact with the client machine in a variety of ways: read or write to the filesystem; access environment variables; open network connections; query platform information; and even execute commands. This allows seamless integrations into your existing tooling and infrastructure.

For example, your development machine may have Docker credentials stored in the MacOS keychain; your CI runner may be configured to request ephemeral tokens for a specific profile before each AWS operation; or you may want to run tests against a local source code checkout. These integrations cannot be sandboxed in a container: they require access to the client machine. The Client API makes them possible in just a few lines of CUE.

[Learn more about the Client API](https://docs.dagger.io/1203/client)

##### **Secure by default: the Secrets API**

Secrets are a crucial part of any CI/CD pipeline, which is why Dagger introduced native support for secrets from the very beginning. Dagger 0.2 expands and improves this feature in several ways:

- Dagger secrets are never merged into the plan, which helps prevent credential leaks. And just in case you accidentally print a secret’s contents in your action logs… Dagger will automatically detect it and scrub it. You’re welcome!
- Using the new Client API, secrets can now be loaded from virtually any source: a file, an environment variable, a proprietary KMS, a vault service, a local keychain, etc.
- Actions can now dynamically create their own secrets, using the [\#NewSecret core action](https://github.com/dagger/dagger/blob/main/pkg/dagger.io/dagger/secrets.cue#L18).
- Actions can now perform common operations against the contents of a secret, securely, without accessing the contents directly (to avoid leaks). Operations include [extracting a secret from a JSON- or YAML-encoded envelope](https://github.com/dagger/dagger/blob/main/pkg/dagger.io/dagger/secrets.cue#L5), and [trimming space characters](https://github.com/dagger/dagger/blob/main/pkg/dagger.io/dagger/secrets.cue#L32).
- Secrets can now be injected into containers as environment variables, in addition to temporary files

[Learn more about the Secrets API](https://docs.dagger.io/1204/secrets)

---

#### Get involved!

Dagger is a community-driven project. We develop in the open, and anyone is welcome to join the fun, no matter how experienced. Whether you’re a seasoned engineer, or looking to make your first open-source contribution, there are many ways you can contribute. Come say hello [on our Discord server](https://dagger.us20.list-manage.com/track/click?u=78ddebf129e125c5037fd6f4d&id=bda77f8e8d&e=174ebdaff6 'https://dagger.us20.list-manage.com/track/click?u=78ddebf129e125c5037fd6f4d&id=bda77f8e8d&e=174ebdaff6')!


Dagger 0.4: service containers, secrets scrubbing, and more

How we release Dagger

Dagger in Action: Cutting Deployment Times from 3 Hours to 3 Minutes

Dagger 0.4: service containers, secrets scrubbing, and more

Reproducible Builds with Dagger

Adopting Ruff for Dagger’s Python SDK

Dagger 0.3.10: New pipeline API, support for container metadata, and more

Replacing your Dockerfile with Go code

Dagger Node.js SDK 0.3: API improvements, better error handling and less boilerplate

Introducing the Dagger GraphQL API

Introducing the Dagger Node.js SDK

Develop your pipelines as code with the Dagger Python SDK

Dagger Go SDK 0.4: multi-platform, improved filesystem API and more

Your CI pipelines should be code: introducing the Dagger Go SDK

Introducing Dagger for VS Code with CUE LSP support

Announcing our $20M Series A from Redpoint Ventures

Introducing Dagger: a new way to create CI/CD pipelines

Europa Rising: Introducing Dagger 0.2