More and more applications are running on Kubernetes, but application delivery is still a challenge. CI/CD operations are typically expressed in YAML or shell scripts; the more complex they are, the harder they are to decipher and debug. Delivery pipelines often perform unnecessary rebuilds and test reruns even when nothing has changed, resulting in long wait times and slow deployments.

Dagger solves these problems by giving platform engineers a rich API and native-language SDKs in which to express CI/CD operations, and intelligent caching to significantly accelerate delivery pipelines. When these features are combined with Kubernetes' scalability and orchestration capabilities, they not only make the delivery platform more efficient, but they also improve the overall experience for both development and platform teams.

If you're interested in running Dagger on Kubernetes, this blog post provides important background information to help you get started. It also explains the recommended architecture pattern and components, together with links to more detailed documentation.

TIP: Before going deeper, this may be a good time to remind yourself of the [basics
of Dagger](https://docs.dagger.io/quickstart/319191/basics/). 

### How does Dagger run locally?

By default, any Dagger SDK (seamlessly) uses the CLI to auto-provision the Dagger Engine as a Docker container. This is automatic and only requires that Docker is available locally, where the SDK runs. While this default behavior optimizes for developer convenience, and it's a good starting point on all CI/CD platforms where Docker is available, taking control of where the Dagger Engine runs has numerous benefits, starting with removing the Docker dependency.

### How does Dagger run in CI/CD?

By default, the same as when run locally: a Dagger SDK uses the CLI to auto-provision the Dagger Engine as a Docker container. If Docker is not available, auto-provisioning fails.

### Why run Dagger on Kubernetes?

One of the most important reasons to use Dagger is persistent caching that works out of the
box.

A Dagger pipeline is composed of discrete operations. The result of each operation is persisted locally, as a file on disk. These files are effectively OCI image layers. Data from mounts is also stored on a local path where the Dagger Engine runs. When running locally, this is great since caching is fast and reliable (most of us have SSDs, some even NVMEs). However, when running on a CI/CD platform, each run starts with a clean state (because the runners are ephemeral).

Re-using the same Dagger Engine state across CI/CD runs yields the same great caching benefits that are the norm when running pipelines locally, on workstations. And when you combine the Dagger Engine's caching powers with everything Kubernetes provides in terms of auto-scaling and orchestration, you end up with a fast, scalable and resource-efficient CI system!

### How to set up Dagger on Kubernetes?

The minimum required components are:

* *Kubernetes cluster*, consisting of support nodes and runner nodes.

  * Runner nodes host CI runners and Dagger Engines.
  * Support nodes host support and management tools, such as certificate management, runner controller & other functions.
* *Certificates manager*, required by Runner controller for admission control.
* *Runner controller*, responsible for managing CI runners in response to CI job requests.

  * CI runners are the workhorses of a CI/CD system. They execute the jobs that are defined in the CI/CD pipeline.
* *Dagger Engine* on each runner node, running alongside one or more CI runners.

  * Responsible for running Dagger pipelines and caching intermediate and final build artifacts.

This architecture can be further optimized:

* A node auto-scaler, responsible for automatically adjusting the size of node groups based on the current workload.
* A cloud caching service, which caches data from all Dagger Engines in the cluster and uses it to "warm the cache" of new auto-provisioned Dagger Engines.

Here's an example of what this architecture might look like:

![](/img/pattern-cache-a289e38cd2f5e2e7154bf36a40fa68be.png)

### Next steps

If you're interested in implementing this architecture in your own Kubernetes cluster, we recommend the following resources:

* Our guide on [running Dagger on Kubernetes](https://docs.dagger.io/194031/kubernetes) describes a specific implementation of the above pattern using GitHub Actions, Amazon Elastic Kubernetes Service (EKS),  Karpenter and Dagger Cloud. It has step-by-step instructions to get you started.
* We also have guides on using Dagger with [Argo Workflows](https://docs.dagger.io/324301/argo-workflows) and with [Tekton](https://docs.dagger.io/213240/tekton).

If you get stuck and need help, contact us via the [\#kubernetes channel in our Discord](https://discord.com/channels/707636530424053791/1122942037096927353). Or, if you have feedback for us, or an improvement to suggest, join us on [Discord](https://discord.com/invite/dagger-io) or create a [GitHub issue](https://github.com/dagger/dagger/issues/new/choose).

Lastly, the Dagger Team will be at KubeCon NA 2023. We’ve been hard at work preparing a bunch of new stuff, so come by our booth (N37) to see some demos of our newest tech, or just come tell us about your Dagger experience and what you’d like to see from us going forward. If you want some dedicated time with us, for example, to discuss how Dagger can help you speed up your pipelines, [reach](mailto:sales@dagger.io) out to us and we’ll schedule some in-person time with you. We can’t wait to see you!

In this post, we discuss the challenges in application delivery on Kubernetes, how Dagger can solve these issues, and share the best ways to run Dagger on Kubernetes. 

Running Dagger on Kubernetes

We're always interested in hearing how our community is using Dagger – their use cases, their challenges, and their experiences with deploying Dagger in different environments. Our Discord is a great place to find these stories, and to benefit from the knowledge and experience of the Dagger community.

In this blog post, we’ll share the story of Daggernauts Augustin Lafanechère (aka @alafa on [Discord](https://discord.com/invite/dagger-io), [alafanechere on GitHub](https://github.com/alafanechere)) and Conor Barber (aka @conorba on [Discord](https://discord.com/invite/dagger-io), [cpdeethree on Github](https://github.com/cpdeethree)). Both work at [Airbyte](https://airbyte.com/), an open-source ELT platform managing hundreds of Docker containers. Augustin is part of the Connector Operations team responsible for managing CI for Airbyte connectors, while Conor works in the Platform Infrastructure team. Over the last few months, Augustin and Conor have transformed Airbyte's maze of GitHub Actions, YAML, and shell scripts into a streamlined CI/CD process powered by Dagger and significantly enhanced Airbyte's developer experience.

> "Dagger is the first thing we've seen that looks at the way CI/CD works and actually tries to turn some of the ideas on their head, which I think is pretty exciting. There's potential to dramatically improve the way our entire software-development supply chain works.”

### Legacy CI: A Maze of YAML and Shell Scripts

Airbyte has an extensive ecosystem of 350+ containerized connectors, all of which need to be tested regularly. On average, Airbyte publishes 17 connectors per day and tests 100+ connectors per night, both as part of regularly scheduled testing cycles and in response to incoming pull requests.  Airbyte also regularly tests connectors even if the code hasn’t recently changed - they ensure that upstream APIs haven’t changed and still work as expected.

Every new connector needs a new test suite; as a result, build and test requirements scale linearly with the number of connectors. Being able to consistently and reliably build, test, and publish connectors while also ensuring visibility to both platform and development teams is, therefore, a considerable CI/CD challenge.

Before Dagger, Airbyte's CI consisted of GitHub Actions YAML, shell scripts and Gradle scripts. Although the system worked, the overall team satisfaction was low:

* The GitHub Actions tooling was a maze of YAML and shell scripts that were slow to execute and difficult to maintain.
* The Gradle tooling was powerful, but complex to understand and use. It worked well for Java builds, but less so for Docker- and Python-related tasks (of which there were many).
* If a CI pipeline failed, developers had no way to simply replicate the failure locally to identify the error and had to spend time and effort searching the previously-mentioned maze for the source of the error.
* The CI scripts had been written by different team members at different times and therefore had no clear owner.
* The outsized nature of Airbyte's test requirements meant more, and more complex, CI pipelines and consequently high server costs.

> "Not being able to debug locally when there are problems is a pretty major pain point. We were looking for a better way."

### **“Daggerized” CI: Auto-Scaling Runners with Dagger Cloud Distributed Caching**

The Airbyte team investigated various other orchestration tools to solve these problems. In late 2022, Dagger's release of a Python SDK made it the most appealing choice, as Airbyte was already using Python extensively. The team started transitioning to Dagger in February 2023 and ran their first Dagger pipeline in production in May 2023.

The new system employs remote Dagger Engines on a Kubernetes cluster leveraging Dagger Cloud to utilize platform-neutral caching strategies, optimize start-up times, and scale flexibly according to load. Here’s how it works:

* Airbyte uses a slim GitHub Action which responds to repository events (pull requests, pushes, merges, ...) by triggering a [custom command-line wrapper (CLI) around Dagger](https://github.com/airbytehq/airbyte/tree/master/airbyte-ci/connectors/pipelines). This CLI is used for both remote CI and local development - Airbyte finally had a way to replicate the CI experience locally, and it was fast!
* Dagger, via this CLI wrapper, takes care of run orchestration and pipeline invocation.
* Pipelines are executed by a fleet of Dagger Engines running on Kubernetes. 
* Dagger Cloud provides a distributed caching service that all runners have access to. Each Dagger Engine communicates with Dagger Cloud to read from, and write to, the shared cache. 
* Karpenter, a Kubernetes-native node auto-scaler, dynamically adds or removes runner nodes depending on workload requirements.  Through this process, Airbyte also gained cost controls they lacked before.

> "We decided on Dagger because of the expressiveness of being able to do things completely in code. Being able to write things in a more simple, imperative style makes it a lot easier to understand and test."

### **Benefits: CI Pipelines as Code, Unified Local and Remote CI**

Some of the benefits of the new “Daggerized” system are:

**Faster local debugging:** Airbyte's development teams now use the same CLI wrapper as the CI engine. This ensures parity and portability between local and remote CI runs. As a result, developers are able to test code locally, identify errors early, and have fewer surprises when pushing to remote repositories.

> “Before Dagger it was 'Push and Pray.' Now, we have the same tool running locally and in CI, and that just streamlines everything. Now our mantra is test, test, test, test.”

**Better maintainability:** By writing pipelines in Python instead of YAML, CI code is easier to understand, maintain, and optimize. Native-language support makes it easy to create complex conditional pipelines (for example, running a pipeline based on the result of another pipeline). Working entirely in a programming language also makes it easier to follow best practices for testing and packaging CI pipelines as code.

> “We have made a valuable shift. Our CI pipelines are now expressed as code, so refactoring and optimization is much easier. Much of our speed gains were achieved because we can now spot poor, unoptimized CI logic in a simplified way.”

**Quicker pipeline execution:** Dagger's built-in caching support and integration with Dagger Cloud alleviates the problem of cold starts and enables pipelines to run quicker. Dagger's "caching by default" approach is also easier to work with compared to Gradle, which requires caching to be explicitly configured.

### **Dagger Cloud: 75% Lower Costs, Plus a 2-5x Performance Boost**

Airbyte worked closely with Dagger to integrate Dagger Cloud into its CI architecture, and the effort paid off in terms of speed, scalability, and cost.

**Speed**

With Dagger Cloud, CI pipelines are now 2-5 times faster on average. This is because Dagger Cloud intelligently stores the cached data and operations from previous pipeline runs in the cloud and sends this data to new Dagger Engines whenever they are provisioned. As a result, new Dagger Engines start up with a "warm cache" and have all the caching benefits of previous runs.

> "In the worst-case scenario, it's about twice as fast as the legacy way and in the best-case scenario, it's often four to five times as fast. The shared remote cache is a core component of why we're able to see a lot of these gains, because it's the foundation upon which we built this distributed architecture."

**Scalability and Cost**

Running Dagger inside a single Kubernetes cluster with auto-scaling runners has improved overall efficiency and drastically reduced CI costs for Airbyte. Airbyte is now able to intelligently scale up and down depending on the workload in CI. Runner nodes are provisioned on-demand and automatically de-provisioned when they are not needed. As a result, server costs are significantly lower than before.

> "We examined our spend and there were times in the past six to eight months that it was at least four times what it is now. Switching to a single-runner architecture has both reduced our cost and made our usage of server resources more efficient."

### **Future Plans**

Going forward, Augustin and Conor plan to continue building out Airbyte's CI infrastructure with Dagger. They are working to get other internal teams to adopt Dagger and hope to see those teams contributing more tests to the current pipelines as well as adding their own pipelines in the future. With Dagger Cloud, they are looking to increase the observability of their pipeline internals and gain greater control over caching and parallelization configuration.

> "The level of trust between the Airbyte team and the Dagger team is high. It's been a very fruitful and helpful collaboration with the Dagger team, who are willing to step in and make sure that we get things resolved."

Do you have a Dagger story you’d like us to feature? Tell us all about it in [Discord](https://discord.gg/dagger-io)!

Airbyte, an open-source ELT platform managing hundreds of Docker containers, shares how they use Dagger. Augustin is part of the Connector Operations team responsible for managing CI for Airbyte connectors, while Conor works in the Platform Infrastructure team. Over the last few months, Augustin and Conor have transformed Airbyte's maze of GitHub Actions, YAML, and shell scripts into a streamlined CI/CD process powered by Dagger and significantly enhanced Airbyte's developer experience.

Dagger Cloud in Action: Going 100% Faster, Spending 75% Less at Airbyte

Service containers have been available since [Dagger 0.4](https://dagger.io/blog/dagger-engine-0-4-0), making it easy to run services - like databases or web applications - as ephemeral containers directly inside Dagger. This first implementation of service containers  only supported Container-to-Container (C2C) networking and didn't allow fine-grained control over the service lifecycle.

Today, we are pleased to announce Dagger 0.9, featuring a significantly improved services API that expands Dagger's built-in capabilities and brings us closer to the promise of Application Delivery as Code. This new API adds support for Container-to-Host (C2H) networking, Host-to-Container (H2C) networking, and explicit service lifecycle management.

NOTE: Dagger 0.9 includes a breaking change for binding service containers. The `Container.withServiceBinding` API now takes a `Service` instead of a `Container`, so you must call `Container.asService` on its argument. See the section on [binding service containers](https://docs.dagger.io/757394/use-services#bind-service-containers) in our documentation for examples.

### **Enabling new use cases**

With the new services API, it is now possible to expose services running inside a Dagger pipeline to the host and allow Dagger pipelines to access host network resources. 

We expect this API to unlock a vast number of new use cases, including but not limited to:

* Spinning up micro-services or entire development toolchains locally in Dagger, similar to  Docker Compose
* Using AI language models/services on the host from clients running in Dagger
* Running a web application in Dagger and connecting to it from the host
* Running an ephemeral database in a pipeline test suite and pointing your local tests at it
* Accessing resources on internal company networks (such as internal Git repositories) from a Dagger pipeline
* Connecting to a remote service, which is available via a VPN on the host only
* Performing DNS resolution through the host
* Connecting to Docker Desktop on the host from a Dagger pipeline
* Querying a database on the host from a Dagger pipeline
* Running integration tests inside a Dagger pipeline that rely on an API server on the host
* Running services within Dagger (for example, Docker) for a controlled duration of time or even after a pipeline finishes executing

Let's take a closer look at the new features. The following code samples are in Go, but you can find [code samples for other languages in our documentation](https://docs.dagger.io/757394/use-services).

### **Exposing service containers to the host - a.k.a. Host-to-Container (H2C) networking**

You can now expose service containers directly to the host so clients on the host can communicate with services running in Dagger. One common use case is application testing, where you need to be able to spin up temporary databases to run tests against.

Here's an example of a host sending requests to a web application running as a container.

```
package main

import (
	"context"
	"fmt"
	"io"
	"net/http"
	"os"

	"dagger.io/dagger"
)

func main() {
	ctx := context.Background()

	// create Dagger client
	client, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stderr))
	if err != nil {
		panic(err)
	}
	defer client.Close()

	// create web service container with exposed port 8080
	httpSrv := client.Container().
		From("python").
		WithDirectory("/srv", client.Directory().WithNewFile("index.html", "Hello, world!")).
		WithWorkdir("/srv").
		WithExec([]string{"python", "-m", "http.server", "8080"}).
		WithExposedPort(8080).
		AsService()

	// expose web service to host
	tunnel, err := client.Host().Tunnel(httpSrv).Start(ctx)
	if err != nil {
		panic(err)
	}
	defer tunnel.Stop(ctx)

	// get web service address
	srvAddr, err := tunnel.Endpoint(ctx)
	if err != nil {
		panic(err)
	}

	// access web service from host
	res, err := http.Get("http://" + srvAddr)
	if err != nil {
		panic(err)
	}
	defer res.Body.Close()

	// print response
	body, err := io.ReadAll(res.Body)
	if err != nil {
		panic(err)
	}
	fmt.Println(string(body))
}
```

### **Exposing host services to client containers - a.k.a. Container-to-Host (C2H) networking**

You can also bind containers to host services so that clients in Dagger pipelines can communicate with services running on the host. Of course, this implies that a service is already listening on a port on the host, out-of-band of Dagger.

Here's an example of a container in a Dagger pipeline querying a MariaDB database service on the host:

```
package main

import (
	"context"
	"fmt"
	"os"

	"dagger.io/dagger"
)

func main() {
	ctx := context.Background()

	// create Dagger client
	client, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stderr))
	if err != nil {
		panic(err)
	}
	defer client.Close()

	// expose host service on port 3306
	hostSrv := client.Host().Service([]dagger.PortForward{
		{Frontend: 3306, Backend: 3306},
	})

	// create MariaDB container
	// with host service binding
	// execute SQL query on host service
	out, err := client.Container().
		From("mariadb:10.11.2").
		WithServiceBinding("db", hostSrv).
		WithExec([]string{"/bin/sh", "-c", "/usr/bin/mysql --user=root --password=secret --host=db -e 'SELECT * FROM mysql.user'"}).
		Stdout(ctx)
	if err != nil {
		panic(err)
	}
	fmt.Println(out)
}
```

### **Controlling the service lifecycle**

Services are designed to be expressed as a Directed Acyclic Graph (DAG) with explicit bindings allowing services to be started lazily. But sometimes, you may need bespoke service lifecycle management. Starting with Dagger 0.9, you can explicitly start and stop services as needed.

Here's an example which demonstrates starting a Docker daemon for use in a test suite:

```
package main_test

import (
	"context"
	"testing"

	"dagger.io/dagger"
	"github.com/stretchr/testify/require"
)

func TestFoo(t *testing.T) {
	ctx := context.Background()

	c, err := dagger.Connect(ctx)
	require.NoError(t, err)

	dockerd, err := c.Container().From("docker:dind").AsService().Start(ctx)
	require.NoError(t, err)

	// dockerd is now running, and will stay running
	// so you don't have to worry about it restarting after a 10 second gap

	// then in all of your tests, continue to use an explicit binding:
	_, err = c.Container().From("golang").
		WithServiceBinding("docker", dockerd).
		WithEnvVariable("DOCKER_HOST", "tcp://docker:2375").
		WithExec([]string{"go", "test", "./..."}).
		Sync(ctx)
	require.NoError(t, err)

	// or, if you prefer
	// trust `Endpoint()` to construct the address
	//
	// note that this has the exact same non-cache-busting semantics as WithServiceBinding,
	// since hostnames are stable and content-addressed
	//
	// this could be part of the global test suite setup.
	dockerHost, err := dockerd.Endpoint(ctx, dagger.ServiceEndpointOpts{
		Scheme: "tcp",
	})
	require.NoError(t, err)

	_, err = c.Container().From("golang").
		WithEnvVariable("DOCKER_HOST", dockerHost).
		WithExec([]string{"go", "test", "./..."}).
		Sync(ctx)
	require.NoError(t, err)

	// Service.Stop() is available to explicitly stop the service if needed
}
```

See more examples in our [guide on using services in Dagger](https://docs.dagger.io/757394/use-services).

### **SDK updates**

We've also released new versions of our SDKs with support for Dagger 0.9, plus various SDK-specific bug fixes and improvements.

For a complete list of improvements, refer to the changelog for each SDK:

* Go SDK 0.9.0 [changelog](https://github.com/dagger/dagger/blob/sdk/go/v0.9.0/sdk/go/.changes/v0.9.0.md) and [documentation](https://pkg.go.dev/dagger.io/dagger@v0.9.0)
* Node.js SDK 0.9.0 [changelog](https://github.com/dagger/dagger/blob/sdk/nodejs/v0.9.0/sdk/nodejs/.changes/v0.9.0.md) and [documentation](https://docs.dagger.io/current/sdk/nodejs/reference/modules/api_client_gen/)
* Python SDK 0.9.0 [changelog](https://github.com/dagger/dagger/blob/sdk/python/v0.9.0/sdk/python/.changes/v0.9.0.md) and [documentation](https://dagger-io.readthedocs.io/en/sdk-python-v0.9.0/)

We are excited to hear about the use cases that Dagger’s new Host-to-Container (H2C), Container-to-Host (C2H) and explicit service lifecycle capabilities enable for you. Let us know via Discord, and perhaps come share with the rest of the Dagger Community in our [regular fortnightly calls](https://dagger.io/events). If you have feedback for us, or an improvement to suggest, join us on [Discord](https://discord.com/invite/dagger-io) or create a [GitHub issue](https://github.com/dagger/dagger/issues/new/choose).

Looking forward to the automation that you will build with Dagger!

We are pleased to announce Dagger 0.9, featuring a significantly improved services API that expands Dagger's built-in capabilities and brings us closer to the promise of Application Delivery as Code. This new API adds support for Container-to-Host (C2H) networking, Host-to-Container (H2C) networking, and explicit service lifecycle management.

Dagger 0.9: Host-to-container, container-to-host, and other networking improvements

We’re always interested in hearing how our community is using Dagger – their use cases, their challenges, and their experiences with deploying Dagger in different environments. Our Discord is a great place to find these stories, and to benefit from the knowledge and experience of the Dagger community.

In this blog post, we’ll share the story of Daggernaut Kasper Hermansen (aka @kjuulh on Discord). Kasper is a Platform Engineer at [Lunar](https://www.lunar.app/), a digital bank based in Denmark. He is also the author and maintainer of the [Dagger Rust SDK](https://github.com/dagger/dagger/tree/main/sdk/rust). By switching to Dagger, Kasper and his team were able to apply engineering best practices to Lunar's platform tooling, improve its reliability, and deliver a better experience for Lunar's internal software development teams.

> "The most important part about Dagger is that it enables engineering principles across our CI/CD. With Dagger, I can actually trust our roll-out of code." - Kasper Hermansen, Platform Engineer, Lunar

### The problem: An unreliable software delivery platform built on shell scripts and templates

Kasper is part of a team of 3 engineers in the DevEx/Tooling squad at Lunar. This small team serves more than 120 developers across the company, providing and improving general purpose developer tooling across the company. The goal of the team is to improve the developer experience by delivering a platform and related tooling for more efficient, consistent, and reliable software engineering.

Before Dagger, Lunar primarily relied on shell scripts and a single Dockerfile with templating. As requirements increased, the Dockerfile ballooned in complexity, and became increasingly difficult to maintain and test. The behavior of the shell scripts was inconsistent, resulting in unexpected results when running in environments that didn't have the expected flags or worked in subtly different ways. The overall reliability of the build system was low, resulting in loss of developer productivity.

Kasper had previously evaluated Dagger in early 2022 in its CUE-focused avatar, but could not find enough compatibility points at the time. In late 2022, Dagger's evolution to a multi-language model, and especially the introduction of the Go SDK, made it a much better fit for Lunar's needs. The open source aspect of Dagger was also important, as it allowed for easy vetting of the source code and the company behind Dagger. After evaluation, the team decided to move forward and trial Dagger as a more sophisticated and reliable alternative to shell scripting.

> "We wanted to definitely get out of shell scripts and Docker, because we have to deliver software for a lot of people. It's fine to have a single Dockerfile, but now you're templating the Dockerfile, now you have if statements in that Dockerfile, and now you have 450 lines of Dockerfile to build something that should have been quite simple. So that's why we started the process of moving to Dagger."

### The solution: Transitioning the platform to Dagger, one tool at a time

Kasper and his team are currently replacing small parts of the Lunar platform with Dagger:

* Lunar uses shuttle, an open source tool, to expose various services to the development team in their local environments. Some of these services are now implemented in Dagger. 

  * For example, developers can use a command like "shuttle run build" on their laptops to create and test a build. This command triggers a Dagger CI pipeline (written using Dagger's Go SDK) that performs all the required tasks locally. This Dagger pipeline also integrates with external services like Snyk and Semgrep for additional code security and analysis functions. 
  * When it's time to create a production image, a similar command and pipeline runs in Lunar's CI system, performing the same tasks and eventually deploying to a self-managed Kubernetes cluster using GitOps. This results in greater consistency between local and remote builds.
* As part of their transition from batch processing to real-time streaming, the platform team has developed an internal schema generation tool. This tool requires them to load various files to generate output artifacts. With Dagger, they can do this easily using native Go programming constructs such as loops and filesystem read/write functions.

> "I normally evaluate a bunch of tools each month, I test a lot of things all the time. So I'm very quick to adopt something but also discard it again because it doesn't live up to expectations. But for the most part, I've actually been super impressed with the API of Dagger. It just makes sense to me."

### The results: Happier developers, more efficient software delivery

Even though the transition to Dagger is still in progress, the platform and development teams are already seeing benefits:

* Reliability: The overall reliability of the platform has improved after replacing unstable shell scripts and complex templating with workflows encoded in Go. Developers spend less time debugging unexpected environment-related failures, and this has led to an improvement in productivity.
* Consistency: The CI pipeline works the same way for local and remote builds. As a result, errors in the build process are caught early and quickly and there are fewer surprises for developers when pushing final code.
* Collaboration: Lunar primarily uses Go for development. Dagger's native Go support makes it easy for developers across the company to inspect platform tools and collaborate with the platform engineering team to improve them. Working entirely in Go also makes it easier for the platform team to follow best practices for testing, packaging and delivering software internally.

> "With Dagger, we can test our tooling, we can package it, we can even test it on a test repository if we want to. We can get all this done before it goes out of the door, which is much more difficult with a traditional platform."

### The future: Ephemeral, on-demand local development environments and extensions

Although the overall reliability of Lunar's platform has increased after switching to Dagger, the team still sees a fair amount of build failures due to external factors. Kasper's next priority is to add internal telemetry to the Dagger pipeline and start reviewing and analyzing metrics, to gain better visibility into these failures and eventually eliminate them.

Apart from this, Kasper is eagerly awaiting support for extensions in Dagger and is continuing to experiment with ephemeral development environments. His goal with this project is to deliver a set of centrally-managed binaries, which internally use Dagger to create declarative, isolated and reproducible environments on developer laptops.

> "We saw Dagger as a CI replacement. But once I began to know it a little bit more and I began to use it a little bit more, I actually saw that you could do so much more with Dagger. There are just so many other uses of Dagger that are super interesting as well."

Do you have a Dagger story you’d like us to feature? Tell us all about it in [Discord](https://discord.gg/dagger-io)!

Kasper Hermansen, a Platform Engineer at Lunar, shares how his team improved software delivery and reliability by transitioning from unreliable shell scripts and complex templating to Dagger. They have found that Dagger's Go-native support and open-source nature have led to greater consistency, collaboration, and overall productivity, making it a promising solution for their platform needs.

Dagger in Action: Enabling Best-Practice Platform Engineering at Lunar

Today we’re thrilled to announce Early Access to our first commercial product: [Dagger Cloud](https://dagger.cloud/signup). 

### What is Dagger Cloud?

Dagger Cloud is a centralized control plane that complements the decentralized Dagger Engine. The result is a complete platform that you can confidently put in production.

More specifically, every team rolling out Dagger has asked us the same three questions:

1. Where can I see my pipeline?
2. Where can I see all my organization’s pipelines?
3. How do I get great caching in CI?

Dagger Cloud answers those three questions with three features: Pipeline Visualization; Operational Insights; and Distributed Caching.

#### Bring Your Own Compute

Before talking about all the things Dagger Cloud does, let’s start with what it doesn’t do. Dagger Cloud is NOT a place to host a bunch of Dagger Engines. Our goal isn’t to replace your CI, but to make it better. Dagger Cloud is a “bring your own compute” service; the Dagger Engine can run on a wide variety of machines, including most development and CI platforms. And if the Dagger Engine can run on it, then Dagger Cloud supports it.

![](/img/screenshot-2023-09-26-at-12.57.29-pm.png)

#### Pipeline Visibility

In a nutshell, we’re giving you the ability to visualize all aspects of a pipeline. One thing we see over and over again: a developer pushes code, the CI pipeline runs, and then…the pipeline fails and blocks the change from being merged. 

“It must be the pipeline! Let me just rerun it,” you say. But it fails again – is it breaking because of the changes you made to the code, or is the pipeline being flaky? 

Dagger Cloud helps you get to the root cause quickly so you can either fix your pipeline or fix your code and get your changes passing again in CI. We make it easy to drill down to detailed logs, understand how long operations took to run, and whether operations were cached. 

#### Operational Insights

We often see a Platform or IT ops team managing the CI/CD infrastructure and pipelines for app teams. Dagger Cloud gives those teams holistic visibility into the pipelines being used across the organization, enabling them to standardize and streamline processes, catch errors, and reduce duplication of effort. This includes observability of both pre-push and post-push pipelines. This is an area with huge potential and we expect to add a lot more trends and insights over time and the ability to optimize processes and pipelines. 

#### Distributed Caching

Wait, doesn’t the Dagger Engine already have caching? What’s Distributed Caching? One of Dagger’s superpowers is that it caches everything. On a single machine (like a laptop or long-running server), caching just works, because the same Dagger Engine writing to the cache is also reading from it. But in a multi-machine configuration (like an elastic CI cluster), things get more complicated because all machines are continuously producing and consuming large amounts of cache data. How do we get the right cache data to the right machine at the right time, without wasting compute, networking, or storage resources? This is a complex problem that requires a distributed caching service, to orchestrate the movement of data between all machines in the cluster, and a centralized storage service. Because Dagger Cloud receives telemetry from all Dagger Engines, it can model the state of the cluster and make optimal caching decisions. The more telemetry data it receives, the smarter it becomes.

![](/img/distributed-cache.png)

### Getting started with Dagger Cloud

Dagger Cloud needed to be easy to evaluate and use. It takes only one token to set up, and you can start experiencing the benefits of Dagger Cloud within a few minutes after configuring your CI. Dagger Cloud supports a huge number of CIs. Basically, if you can run containers, then you can run Dagger Engines and you can use Dagger Cloud. Users report successfully leveraging Dagger with GitLab, CircleCI, GitHub Actions, Jenkins, Tekton, and many more. Dagger Cloud is built to support you wherever your data needs to be, featuring a global data storage service spanning 26 regions across 3 cloud providers: AWS, Google Cloud Platform, and Cloudflare R2. Dagger Cloud automatically selects the region closest to your compute.

#### How do I try it? How much does it cost?

As part of Early Access of Dagger Cloud, we are offering the Team plan for those collaborating with others and who will benefit most from distributed caching, shared pipeline visibility, and operational insights (or for anyone wanting to experience the full breadth of Dagger Cloud’s capabilities). We’ve tried to keep our pricing as simple as possible - after your free 2-week trial, you’ll pay just 5 cents per minute to use Dagger Cloud (the clock only runs while Dagger is running). 

See more here: <https://dagger.io/pricing> 

We’re thrilled to launch these capabilities which our community has so highly requested. As you know, we’re a community-first organization keen to hear your thoughts and feedback. Please feel free to jump on [Discord](https://discord.com/invite/dagger-io) give us your feedback on how well we hit the mark and what you’d like to see from us next.

Today we’re thrilled to announce Early Access to our first commercial product: Dagger Cloud. 

Dagger Cloud Early Access

We’re always interested in hearing how our community is using Dagger – their use cases, their challenges, and their experiences with deploying Dagger in different environments. Our [Discord](https://discord.gg/dagger-io) is a great place to find these stories, and to benefit from the knowledge and experience of the Dagger community.

In this blog post, we’ll share the story of Daggernaut Sean Clayton (aka @clayton9821 on Discord). Sean recently joined a new company, and was surprised to learn he would be the SME for all its CI pipelines. He had never used GitLab's CI/CD feature before, but took initiative to make it better. By switching to Dagger and away from a YAML-based CI workflow, Sean was able to give the development team faster builds and increased reliability, particularly for Maven builds.

> "How many times have you pushed your code to the pipeline that works locally and then it fails for some reason in the remote pipeline and you have no idea why? Dagger solves that. By letting you see your pipeline as code, you can do everything that you can in production locally, and then push to the pipeline with so much more confidence." - Sean Clayton

### The problem: Team frustration with unreliable CI

Before Dagger, the existing CI pipeline consisted of 1000+ lines of YAML code, with both actions and templates mixed together. It was so complex that no one in the team understood it completely. Due to the high degree of complexity, the pipeline code was difficult to maintain and update, and CI failures could not be debugged and resolved quickly. These issues were increasing developer friction and frustration, and affecting the team's confidence in the entire CI workflow.

Sean was keen to improve the experience for the development team and started investigating alternatives. During the process, he attended a talk by the Dagger team at KubeHuddle in Toronto and immediately realized that Dagger provided the modern and maintainable approach he was searching for. He started by reading the Dagger documentation, built a proof-of-concept and then began the process of migrating the existing CI pipeline to Dagger.

> "When the pipeline failed, people would come to the SRE team and say, it must be the pipeline that's broken. And that was really what drove me to want to improve this, rather than just keep applying band-aids over and over".

### The solution: Faster feedback loops and more consistent CI pipelines

With Dagger, Sean has been able to both simplify the CI pipeline and make it more consistent. The Dagger pipeline performs the same steps as the previous YAML pipeline, but using Dagger's Python SDK (for example, replacing a Packer template with Python SDK methods). It operates in a blended environment: QA on Kubernetes and production on bare metal.

1. The pipeline begins by building and tagging a Docker image of the application.
2. It uses Terragrunt to access the Kubernetes cluster and configures traffic mirroring rules with Istio.
3. It builds and tags a build RPM and deploys the build to the Kubernetes QA environment.
4. It builds and tags a production RPM.
5. It deploys the production RPM to the production environment on bare metal.

> "Local testing was the big win because the biggest thing we heard is, the build works locally and it's not working through the pipeline. Having the developers be able to push to the new pre-QA environment and see how the pipeline works was huge."

Some of the key benefits of “Daggerizing” the pipeline are:

* Improved local loop: The same Dagger pipeline works consistently locally and remotely. As a result, developers are able to build and test locally, exactly as they would through GitLab. Instead of pushing code to GitLab and waiting to see if builds succeed, they now receive almost instantaneous local feedback on their builds.
* Quicker builds: Dagger's support for cache volumes produces significantly faster builds. Previously, a high percentage of build time went in downloading Maven packages. By using cache volumes to store Maven downloads, Dagger has reduced the time required for each Maven build, which in turn reduces the time taken for a new feature to go from development to production.
* Higher efficiency and confidence levels: The CI pipeline is now encoded in Python; there is no longer any lengthy and complex YAML configuration to deal with. This has made the pipeline easier to understand, debug, and extend, thus enabling the SRE team to work more efficiently. Since developers can also inspect and understand the pipeline code directly, their confidence levels in the CI process are also higher.

> "It's definitely faster, and the developer that I'm working with on this is extremely pleased. Giving devs the ability to replicate the pipeline locally increases development speed dramatically. We both know that as soon as we introduce this, it's going to be adopted very quickly."

### The future: Building platform-independent images and front-end testing with Dagger

Sean is working on moving more internal customers over to the new Dagger pipeline, and has various ideas for future improvements:

* Sean is currently working with the development team to move from an RPM build pipeline to a cloud-native Java application built with Dagger. Sean is able to do this much faster because of the local Dagger pipeline.
* The Dagger Engine currently runs on a Kubernetes cluster, alongside a self-hosted CI runner. When new Dagger Engine pods spin up, they start with an empty cache. As a result, initial pipeline runs take longer. Sean is planning to create a tool to prime the cache quickly, so that developers don't need to wait for the cache to fill.
* For front-end testing requirements, Sean is investigating how to have Python pipeline code integrate with JavaScript-based tests, either by directly calling the Dagger API using HTTP or by translating some of the Python pipeline code to JavaScript.

> "The possibility for innovation with Dagger is so huge. Our pipeline is so much shorter, much faster and can be run on any operating system. This was a win directly because we were using Dagger."

Do you have a Dagger story you’d like us to feature? Tell us all about it in [Discord](https://discord.gg/dagger-io)!

Sean Clayton shares his journey of revitalizing a complex and frustrating CI pipeline by adopting Dagger. With Dagger, Sean simplified the pipeline, achieved consistent builds, and empowered developers with faster feedback, ultimately boosting their confidence in the CI process.

Dagger in Action: Restoring Developer Confidence in CI

We’re always interested in hearing how our community is using Dagger – their use cases, their challenges, and their experiences with deploying Dagger in different environments. Our Discord is a great place to find these stories, and to benefit from the knowledge and experience of the Dagger community.

In this blog post, we’ll share the story of Daggernaut Tony Worm (aka @verdverm on Discord and Twitter). Tony is the developer of [Hof](https://github.com/hofstadter-io/hof), an open source code generation framework. Tony has been able to significantly improve the consistency and accuracy of Hof's test suite by using containerized test environments orchestrated with Dagger.

> "With Dagger, I can define what I want my desired results to be, and Dagger figures out what needs to happen to get those results." - Tony Worm, Hof

### The problem: Test failures in mock environments

Hof is an open source code generation tool. It supports shared data models and produces scaffolding code and boilerplate for any technology stack, letting users be in full control. Hof builds on a new language [CUE](https://cuelang.org), using it to power many of its features. As part of dependency management, hof writes various cache and configuration files to the user's home environment on the host. It also runs various containers on the host for automatic code formatting during the code generation process or as task environments in the DAG workflow engine.

To test these features, Tony initially relied on the Go `[testscript](https://github.com/rogpeppe/go-internal/tree/master/cmd/testscript)` package, which is intended for CLI and filesystem-based tests. With this package, Tony could "mock" a user's home environment and run the Hof test suite in this mock environment without polluting the live filesystem on the host. However, this approach was imperfect: path calculations were complex due to the testing peculiarities of both mock and original environments co-existing on the same host.

> " When we're testing and things look different enough from the real environment, it causes issues with our path calculations. We had a point where we just couldn't get our tests to pass one way or the other. And so, we were in the market for a new solution."

### The solution: Containerized, clean test environments and matrix testing with Dagger

As a member of the CUE community and maintainer of [Cuetorials](https://cuetorials.com/), Tony was already familiar with Dagger from its previous CUE-based implementation. He realized that he could create a more realistic testing environment using containers orchestrated by Dagger. Dagger being open-source also aligned closely with the Hof project's own values.

With Dagger, Tony has simplified and improved Hof's test suite by running it in a containerized home environment. Hof's Dagger pipeline (created with Dagger's Go SDK) begins by producing a base container image containing the build to be tested and the required environment. It then loops over the test suite, executing each test in a separate container spawned from the base image.

Key benefits are:

* The Dagger pipeline runs each test in a "real" home environment. As a result, there are no additional considerations or limitations related to mocked-up environments.
* The Dagger pipeline runs each test in a clean, isolated container derived from the base container image. This eliminates stale states or conflicts caused by previous tests.
* Native Go support has allowed the team to adopt a matrix test pattern and execute the test suite against different versions of Go to identify and eliminate version-related incompatibilities.
* Go support has also enabled the team to continue using \`testscript\` without needing to rewrite any tests, as well as continuously improve the pipeline by adding new tests and Go best practices.

> "In the end, what we wanted was a better testing environment, and that meant doing things in a container. After switching to Dagger, we were able to remove some really nasty code for dealing with our test environment issues."

### **The future: Broader testing and end-to-end release packaging with Dagger**

Tony is excited by the potential of Dagger and plans to use it in multiple ways:

* Hof already uses Docker to perform various core UX functions, including various Git and filesystem operations. Tony is exploring the possibility of shipping a Dagger pipeline with Hof to perform these core functions. As this pipeline would be fully programmable, he expects it would be significantly easier to maintain and extend.
* Currently, GoReleaser takes care of packaging and distributing the Hof build produced by the Dagger pipeline. In the future, Tony hopes to complement GoReleaser with Dagger and create an end-to-end release process integrating Dagger.
* Hof runs containers in the background and needs to support multiple container runtimes. The current Dagger pipeline is able to test Hof using Docker-in-Dagger. Tony is looking forward to Dagger adding support for container-to-container sockets in future, which will enable similar testing for Podman and Nerdctl.
* In addition to using Dagger's Go SDK to test Hof, Tony also uses Dagger's Node.js SDK to build Hof's Next.js-based documentation website. In the future, he hopes to extend this pipeline and integrate Dagger with GitHub Actions to automatically build, test, and publish Hof's documentation on every successful merge.

> "Dagger was a natural choice for us. Dagger is adding value by building on top of the language ecosystem. I can now create modules of Dagger snippets that I can use in other places, and that's a really powerful concept."

Do you have a Dagger story you’d like us to feature? Tell us all about it in [Discord](https://discord.gg/dagger-io)!

Tony shares how he has been able to significantly improve the consistency and accuracy of Hof's test suite by using containerized test environments orchestrated with Dagger.

Dagger Cloud in Action: Going 100% Faster, Spending 75% Less at Airbyte

Running Dagger on Kubernetes

Dagger Cloud in Action: Going 100% Faster, Spending 75% Less at Airbyte

Dagger 0.9: Host-to-container, container-to-host, and other networking improvements

Dagger in Action: Enabling Best-Practice Platform Engineering at Lunar

Dagger Cloud Early Access

Dagger in Action: Restoring Developer Confidence in CI

Dagger in Action: Better CI Testing for the Hof Project