Dagger 0.9: Host-to-container, container-to-host, and other networking improvements

Service containers have been available since Dagger 0.4, making it easy to run services - like databases or web applications - as ephemeral containers directly inside Dagger. This first implementation of service containers  only supported Container-to-Container (C2C) networking and didn't allow fine-grained control over the service lifecycle.

Today, we are pleased to announce Dagger 0.9, featuring a significantly improved services API that expands Dagger's built-in capabilities and brings us closer to the promise of Application Delivery as Code. This new API adds support for Container-to-Host (C2H) networking, Host-to-Container (H2C) networking, and explicit service lifecycle management.

NOTE: Dagger 0.9 includes a breaking change for binding service containers. The Container.withServiceBinding API now takes a Service instead of a Container, so you must call Container.asService on its argument. See the section on binding service containers in our documentation for examples.

Enabling new use cases

With the new services API, it is now possible to expose services running inside a Dagger pipeline to the host and allow Dagger pipelines to access host network resources. 

We expect this API to unlock a vast number of new use cases, including but not limited to:

• Spinning up micro-services or entire development toolchains locally in Dagger, similar to  Docker Compose
• Using AI language models/services on the host from clients running in Dagger
• Running a web application in Dagger and connecting to it from the host
• Running an ephemeral database in a pipeline test suite and pointing your local tests at it
• Accessing resources on internal company networks (such as internal Git repositories) from a Dagger pipeline
• Connecting to a remote service, which is available via a VPN on the host only
• Performing DNS resolution through the host
• Connecting to Docker Desktop on the host from a Dagger pipeline
• Querying a database on the host from a Dagger pipeline
• Running integration tests inside a Dagger pipeline that rely on an API server on the host
• Running services within Dagger (for example, Docker) for a controlled duration of time or even after a pipeline finishes executing

Let's take a closer look at the new features. The following code samples are in Go, but you can find code samples for other languages in our documentation.

Exposing service containers to the host - a.k.a. Host-to-Container (H2C) networking

You can now expose service containers directly to the host so clients on the host can communicate with services running in Dagger. One common use case is application testing, where you need to be able to spin up temporary databases to run tests against.

Here's an example of a host sending requests to a web application running as a container.

package main

import (
	"context"
	"fmt"
	"io"
	"net/http"
	"os"

	"dagger.io/dagger"
)

func main() {
	ctx := context.Background()

	// create Dagger client
	client, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stderr))
	if err != nil {
		panic(err)
	}
	defer client.Close()

	// create web service container with exposed port 8080
	httpSrv := client.Container().
		From("python").
		WithDirectory("/srv", client.Directory().WithNewFile("index.html", "Hello, world!")).
		WithWorkdir("/srv").
		WithExec([]string{"python", "-m", "http.server", "8080"}).
		WithExposedPort(8080).
		AsService()

	// expose web service to host
	tunnel, err := client.Host().Tunnel(httpSrv).Start(ctx)
	if err != nil {
		panic(err)
	}
	defer tunnel.Stop(ctx)

	// get web service address
	srvAddr, err := tunnel.Endpoint(ctx)
	if err != nil {
		panic(err)
	}

	// access web service from host
	res, err := http.Get("http://" + srvAddr)
	if err != nil {
		panic(err)
	}
	defer res.Body.Close()

	// print response
	body, err := io.ReadAll(res.Body)
	if err != nil {
		panic(err)
	}
	fmt.Println(string(body))
}

Exposing host services to client containers - a.k.a. Container-to-Host (C2H) networking

You can also bind containers to host services so that clients in Dagger pipelines can communicate with services running on the host. Of course, this implies that a service is already listening on a port on the host, out-of-band of Dagger.

Here's an example of a container in a Dagger pipeline querying a MariaDB database service on the host:

package main

import (
	"context"
	"fmt"
	"os"

	"dagger.io/dagger"
)

func main() {
	ctx := context.Background()

	// create Dagger client
	client, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stderr))
	if err != nil {
		panic(err)
	}
	defer client.Close()

	// expose host service on port 3306
	hostSrv := client.Host().Service([]dagger.PortForward{
		{Frontend: 3306, Backend: 3306},
	})

	// create MariaDB container
	// with host service binding
	// execute SQL query on host service
	out, err := client.Container().
		From("mariadb:10.11.2").
		WithServiceBinding("db", hostSrv).
		WithExec([]string{"/bin/sh", "-c", "/usr/bin/mysql --user=root --password=secret --host=db -e 'SELECT * FROM mysql.user'"}).
		Stdout(ctx)
	if err != nil {
		panic(err)
	}
	fmt.Println(out)
}

Controlling the service lifecycle

Services are designed to be expressed as a Directed Acyclic Graph (DAG) with explicit bindings allowing services to be started lazily. But sometimes, you may need bespoke service lifecycle management. Starting with Dagger 0.9, you can explicitly start and stop services as needed.

Here's an example which demonstrates starting a Docker daemon for use in a test suite:

package main_test

import (
	"context"
	"testing"

	"dagger.io/dagger"
	"github.com/stretchr/testify/require"
)

func TestFoo(t *testing.T) {
	ctx := context.Background()

	c, err := dagger.Connect(ctx)
	require.NoError(t, err)

	dockerd, err := c.Container().From("docker:dind").AsService().Start(ctx)
	require.NoError(t, err)

	// dockerd is now running, and will stay running
	// so you don't have to worry about it restarting after a 10 second gap

	// then in all of your tests, continue to use an explicit binding:
	_, err = c.Container().From("golang").
		WithServiceBinding("docker", dockerd).
		WithEnvVariable("DOCKER_HOST", "tcp://docker:2375").
		WithExec([]string{"go", "test", "./..."}).
		Sync(ctx)
	require.NoError(t, err)

	// or, if you prefer
	// trust `Endpoint()` to construct the address
	//
	// note that this has the exact same non-cache-busting semantics as WithServiceBinding,
	// since hostnames are stable and content-addressed
	//
	// this could be part of the global test suite setup.
	dockerHost, err := dockerd.Endpoint(ctx, dagger.ServiceEndpointOpts{
		Scheme: "tcp",
	})
	require.NoError(t, err)

	_, err = c.Container().From("golang").
		WithEnvVariable("DOCKER_HOST", dockerHost).
		WithExec([]string{"go", "test", "./..."}).
		Sync(ctx)
	require.NoError(t, err)

	// Service.Stop() is available to explicitly stop the service if needed
}

See more examples in our guide on using services in Dagger.

SDK updates

We've also released new versions of our SDKs with support for Dagger 0.9, plus various SDK-specific bug fixes and improvements.

For a complete list of improvements, refer to the changelog for each SDK:

• Go SDK 0.9.0 changelog and documentation
• Node.js SDK 0.9.0 changelog and documentation
• Python SDK 0.9.0 changelog and documentation

We are excited to hear about the use cases that Dagger’s new Host-to-Container (H2C), Container-to-Host (C2H) and explicit service lifecycle capabilities enable for you. Let us know via Discord, and perhaps come share with the rest of the Dagger Community in our regular fortnightly calls. If you have feedback for us, or an improvement to suggest, join us on Discord or create a GitHub issue.

Looking forward to the automation that you will build with Dagger!