Dagger 0.9: Host-to-container, container-to-host, and other networking improvements
October 20, 2023
Oct 20, 2023
Product
Service containers have been available since Dagger 0.4, making it easy to run services - like databases or web applications - as ephemeral containers directly inside Dagger. This first implementation of service containers only supported Container-to-Container (C2C) networking and didn't allow fine-grained control over the service lifecycle.
Today, we are pleased to announce Dagger 0.9, featuring a significantly improved services API that expands Dagger's built-in capabilities and brings us closer to the promise of Application Delivery as Code. This new API adds support for Container-to-Host (C2H) networking, Host-to-Container (H2C) networking, and explicit service lifecycle management.
NOTE: Dagger 0.9 includes a breaking change for binding service containers. The Container.withServiceBinding
API now takes a Service
instead of a Container
, so you must call Container.asService
on its argument. See the section on binding service containers in our documentation for examples.
Enabling new use cases
With the new services API, it is now possible to expose services running inside a Dagger pipeline to the host and allow Dagger pipelines to access host network resources.
We expect this API to unlock a vast number of new use cases, including but not limited to:
Spinning up micro-services or entire development toolchains locally in Dagger, similar to Docker Compose
Using AI language models/services on the host from clients running in Dagger
Running a web application in Dagger and connecting to it from the host
Running an ephemeral database in a pipeline test suite and pointing your local tests at it
Accessing resources on internal company networks (such as internal Git repositories) from a Dagger pipeline
Connecting to a remote service, which is available via a VPN on the host only
Performing DNS resolution through the host
Connecting to Docker Desktop on the host from a Dagger pipeline
Querying a database on the host from a Dagger pipeline
Running integration tests inside a Dagger pipeline that rely on an API server on the host
Running services within Dagger (for example, Docker) for a controlled duration of time or even after a pipeline finishes executing
Let's take a closer look at the new features. The following code samples are in Go, but you can find code samples for other languages in our documentation.
Exposing service containers to the host - a.k.a. Host-to-Container (H2C) networking
You can now expose service containers directly to the host so clients on the host can communicate with services running in Dagger. One common use case is application testing, where you need to be able to spin up temporary databases to run tests against.
Here's an example of a host sending requests to a web application running as a container.
package main
import (
"context"
"fmt"
"io"
"net/http"
"os"
"dagger.io/dagger"
)
func main() {
ctx := context.Background()
// create Dagger client
client, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stderr))
if err != nil {
panic(err)
}
defer client.Close()
// create web service container with exposed port 8080
httpSrv := client.Container().
From("python").
WithDirectory("/srv", client.Directory().WithNewFile("index.html", "Hello, world!")).
WithWorkdir("/srv").
WithExec([]string{"python", "-m", "http.server", "8080"}).
WithExposedPort(8080).
AsService()
// expose web service to host
tunnel, err := client.Host().Tunnel(httpSrv).Start(ctx)
if err != nil {
panic(err)
}
defer tunnel.Stop(ctx)
// get web service address
srvAddr, err := tunnel.Endpoint(ctx)
if err != nil {
panic(err)
}
// access web service from host
res, err := http.Get("http://" + srvAddr)
if err != nil {
panic(err)
}
defer res.Body.Close()
// print response
body, err := io.ReadAll(res.Body)
if err != nil {
panic(err)
}
fmt.Println(string(body))
}
Exposing host services to client containers - a.k.a. Container-to-Host (C2H) networking
You can also bind containers to host services so that clients in Dagger pipelines can communicate with services running on the host. Of course, this implies that a service is already listening on a port on the host, out-of-band of Dagger.
Here's an example of a container in a Dagger pipeline querying a MariaDB database service on the host:
package main
import (
"context"
"fmt"
"os"
"dagger.io/dagger"
)
func main() {
ctx := context.Background()
// create Dagger client
client, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stderr))
if err != nil {
panic(err)
}
defer client.Close()
// expose host service on port 3306
hostSrv := client.Host().Service([]dagger.PortForward{
{Frontend: 3306, Backend: 3306},
})
// create MariaDB container
// with host service binding
// execute SQL query on host service
out, err := client.Container().
From("mariadb:10.11.2").
WithServiceBinding("db", hostSrv).
WithExec([]string{"/bin/sh", "-c", "/usr/bin/mysql --user=root --password=secret --host=db -e 'SELECT * FROM mysql.user'"}).
Stdout(ctx)
if err != nil {
panic(err)
}
fmt.Println(out)
}
Controlling the service lifecycle
Services are designed to be expressed as a Directed Acyclic Graph (DAG) with explicit bindings allowing services to be started lazily. But sometimes, you may need bespoke service lifecycle management. Starting with Dagger 0.9, you can explicitly start and stop services as needed.
Here's an example which demonstrates starting a Docker daemon for use in a test suite:
package main_test
import (
"context"
"testing"
"dagger.io/dagger"
"github.com/stretchr/testify/require"
)
func TestFoo(t *testing.T) {
ctx := context.Background()
c, err := dagger.Connect(ctx)
require.NoError(t, err)
dockerd, err := c.Container().From("docker:dind").AsService().Start(ctx)
require.NoError(t, err)
// dockerd is now running, and will stay running
// so you don't have to worry about it restarting after a 10 second gap
// then in all of your tests, continue to use an explicit binding:
_, err = c.Container().From("golang").
WithServiceBinding("docker", dockerd).
WithEnvVariable("DOCKER_HOST", "tcp://docker:2375").
WithExec([]string{"go", "test", "./..."}).
Sync(ctx)
require.NoError(t, err)
// or, if you prefer
// trust `Endpoint()` to construct the address
//
// note that this has the exact same non-cache-busting semantics as WithServiceBinding,
// since hostnames are stable and content-addressed
//
// this could be part of the global test suite setup.
dockerHost, err := dockerd.Endpoint(ctx, dagger.ServiceEndpointOpts{
Scheme: "tcp",
})
require.NoError(t, err)
_, err = c.Container().From("golang").
WithEnvVariable("DOCKER_HOST", dockerHost).
WithExec([]string{"go", "test", "./..."}).
Sync(ctx)
require.NoError(t, err)
// Service.Stop() is available to explicitly stop the service if needed
}
See more examples in our guide on using services in Dagger.
SDK updates
We've also released new versions of our SDKs with support for Dagger 0.9, plus various SDK-specific bug fixes and improvements.
For a complete list of improvements, refer to the changelog for each SDK:
Go SDK 0.9.0 changelog and documentation
Node.js SDK 0.9.0 changelog and documentation
Python SDK 0.9.0 changelog and documentation
We are excited to hear about the use cases that Dagger’s new Host-to-Container (H2C), Container-to-Host (C2H) and explicit service lifecycle capabilities enable for you. Let us know via Discord, and perhaps come share with the rest of the Dagger Community in our reg ular fortnightly call. If you have feedback for us, or an improvement to suggest, join us on Discord or create a GitHub issue.
Looking forward to the automation that you will build with Dagger!
Service containers have been available since Dagger 0.4, making it easy to run services - like databases or web applications - as ephemeral containers directly inside Dagger. This first implementation of service containers only supported Container-to-Container (C2C) networking and didn't allow fine-grained control over the service lifecycle.
Today, we are pleased to announce Dagger 0.9, featuring a significantly improved services API that expands Dagger's built-in capabilities and brings us closer to the promise of Application Delivery as Code. This new API adds support for Container-to-Host (C2H) networking, Host-to-Container (H2C) networking, and explicit service lifecycle management.
NOTE: Dagger 0.9 includes a breaking change for binding service containers. The Container.withServiceBinding
API now takes a Service
instead of a Container
, so you must call Container.asService
on its argument. See the section on binding service containers in our documentation for examples.
Enabling new use cases
With the new services API, it is now possible to expose services running inside a Dagger pipeline to the host and allow Dagger pipelines to access host network resources.
We expect this API to unlock a vast number of new use cases, including but not limited to:
Spinning up micro-services or entire development toolchains locally in Dagger, similar to Docker Compose
Using AI language models/services on the host from clients running in Dagger
Running a web application in Dagger and connecting to it from the host
Running an ephemeral database in a pipeline test suite and pointing your local tests at it
Accessing resources on internal company networks (such as internal Git repositories) from a Dagger pipeline
Connecting to a remote service, which is available via a VPN on the host only
Performing DNS resolution through the host
Connecting to Docker Desktop on the host from a Dagger pipeline
Querying a database on the host from a Dagger pipeline
Running integration tests inside a Dagger pipeline that rely on an API server on the host
Running services within Dagger (for example, Docker) for a controlled duration of time or even after a pipeline finishes executing
Let's take a closer look at the new features. The following code samples are in Go, but you can find code samples for other languages in our documentation.
Exposing service containers to the host - a.k.a. Host-to-Container (H2C) networking
You can now expose service containers directly to the host so clients on the host can communicate with services running in Dagger. One common use case is application testing, where you need to be able to spin up temporary databases to run tests against.
Here's an example of a host sending requests to a web application running as a container.
package main
import (
"context"
"fmt"
"io"
"net/http"
"os"
"dagger.io/dagger"
)
func main() {
ctx := context.Background()
// create Dagger client
client, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stderr))
if err != nil {
panic(err)
}
defer client.Close()
// create web service container with exposed port 8080
httpSrv := client.Container().
From("python").
WithDirectory("/srv", client.Directory().WithNewFile("index.html", "Hello, world!")).
WithWorkdir("/srv").
WithExec([]string{"python", "-m", "http.server", "8080"}).
WithExposedPort(8080).
AsService()
// expose web service to host
tunnel, err := client.Host().Tunnel(httpSrv).Start(ctx)
if err != nil {
panic(err)
}
defer tunnel.Stop(ctx)
// get web service address
srvAddr, err := tunnel.Endpoint(ctx)
if err != nil {
panic(err)
}
// access web service from host
res, err := http.Get("http://" + srvAddr)
if err != nil {
panic(err)
}
defer res.Body.Close()
// print response
body, err := io.ReadAll(res.Body)
if err != nil {
panic(err)
}
fmt.Println(string(body))
}
Exposing host services to client containers - a.k.a. Container-to-Host (C2H) networking
You can also bind containers to host services so that clients in Dagger pipelines can communicate with services running on the host. Of course, this implies that a service is already listening on a port on the host, out-of-band of Dagger.
Here's an example of a container in a Dagger pipeline querying a MariaDB database service on the host:
package main
import (
"context"
"fmt"
"os"
"dagger.io/dagger"
)
func main() {
ctx := context.Background()
// create Dagger client
client, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stderr))
if err != nil {
panic(err)
}
defer client.Close()
// expose host service on port 3306
hostSrv := client.Host().Service([]dagger.PortForward{
{Frontend: 3306, Backend: 3306},
})
// create MariaDB container
// with host service binding
// execute SQL query on host service
out, err := client.Container().
From("mariadb:10.11.2").
WithServiceBinding("db", hostSrv).
WithExec([]string{"/bin/sh", "-c", "/usr/bin/mysql --user=root --password=secret --host=db -e 'SELECT * FROM mysql.user'"}).
Stdout(ctx)
if err != nil {
panic(err)
}
fmt.Println(out)
}
Controlling the service lifecycle
Services are designed to be expressed as a Directed Acyclic Graph (DAG) with explicit bindings allowing services to be started lazily. But sometimes, you may need bespoke service lifecycle management. Starting with Dagger 0.9, you can explicitly start and stop services as needed.
Here's an example which demonstrates starting a Docker daemon for use in a test suite:
package main_test
import (
"context"
"testing"
"dagger.io/dagger"
"github.com/stretchr/testify/require"
)
func TestFoo(t *testing.T) {
ctx := context.Background()
c, err := dagger.Connect(ctx)
require.NoError(t, err)
dockerd, err := c.Container().From("docker:dind").AsService().Start(ctx)
require.NoError(t, err)
// dockerd is now running, and will stay running
// so you don't have to worry about it restarting after a 10 second gap
// then in all of your tests, continue to use an explicit binding:
_, err = c.Container().From("golang").
WithServiceBinding("docker", dockerd).
WithEnvVariable("DOCKER_HOST", "tcp://docker:2375").
WithExec([]string{"go", "test", "./..."}).
Sync(ctx)
require.NoError(t, err)
// or, if you prefer
// trust `Endpoint()` to construct the address
//
// note that this has the exact same non-cache-busting semantics as WithServiceBinding,
// since hostnames are stable and content-addressed
//
// this could be part of the global test suite setup.
dockerHost, err := dockerd.Endpoint(ctx, dagger.ServiceEndpointOpts{
Scheme: "tcp",
})
require.NoError(t, err)
_, err = c.Container().From("golang").
WithEnvVariable("DOCKER_HOST", dockerHost).
WithExec([]string{"go", "test", "./..."}).
Sync(ctx)
require.NoError(t, err)
// Service.Stop() is available to explicitly stop the service if needed
}
See more examples in our guide on using services in Dagger.
SDK updates
We've also released new versions of our SDKs with support for Dagger 0.9, plus various SDK-specific bug fixes and improvements.
For a complete list of improvements, refer to the changelog for each SDK:
Go SDK 0.9.0 changelog and documentation
Node.js SDK 0.9.0 changelog and documentation
Python SDK 0.9.0 changelog and documentation
We are excited to hear about the use cases that Dagger’s new Host-to-Container (H2C), Container-to-Host (C2H) and explicit service lifecycle capabilities enable for you. Let us know via Discord, and perhaps come share with the rest of the Dagger Community in our reg ular fortnightly call. If you have feedback for us, or an improvement to suggest, join us on Discord or create a GitHub issue.
Looking forward to the automation that you will build with Dagger!
Service containers have been available since Dagger 0.4, making it easy to run services - like databases or web applications - as ephemeral containers directly inside Dagger. This first implementation of service containers only supported Container-to-Container (C2C) networking and didn't allow fine-grained control over the service lifecycle.
Today, we are pleased to announce Dagger 0.9, featuring a significantly improved services API that expands Dagger's built-in capabilities and brings us closer to the promise of Application Delivery as Code. This new API adds support for Container-to-Host (C2H) networking, Host-to-Container (H2C) networking, and explicit service lifecycle management.
NOTE: Dagger 0.9 includes a breaking change for binding service containers. The Container.withServiceBinding
API now takes a Service
instead of a Container
, so you must call Container.asService
on its argument. See the section on binding service containers in our documentation for examples.
Enabling new use cases
With the new services API, it is now possible to expose services running inside a Dagger pipeline to the host and allow Dagger pipelines to access host network resources.
We expect this API to unlock a vast number of new use cases, including but not limited to:
Spinning up micro-services or entire development toolchains locally in Dagger, similar to Docker Compose
Using AI language models/services on the host from clients running in Dagger
Running a web application in Dagger and connecting to it from the host
Running an ephemeral database in a pipeline test suite and pointing your local tests at it
Accessing resources on internal company networks (such as internal Git repositories) from a Dagger pipeline
Connecting to a remote service, which is available via a VPN on the host only
Performing DNS resolution through the host
Connecting to Docker Desktop on the host from a Dagger pipeline
Querying a database on the host from a Dagger pipeline
Running integration tests inside a Dagger pipeline that rely on an API server on the host
Running services within Dagger (for example, Docker) for a controlled duration of time or even after a pipeline finishes executing
Let's take a closer look at the new features. The following code samples are in Go, but you can find code samples for other languages in our documentation.
Exposing service containers to the host - a.k.a. Host-to-Container (H2C) networking
You can now expose service containers directly to the host so clients on the host can communicate with services running in Dagger. One common use case is application testing, where you need to be able to spin up temporary databases to run tests against.
Here's an example of a host sending requests to a web application running as a container.
package main
import (
"context"
"fmt"
"io"
"net/http"
"os"
"dagger.io/dagger"
)
func main() {
ctx := context.Background()
// create Dagger client
client, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stderr))
if err != nil {
panic(err)
}
defer client.Close()
// create web service container with exposed port 8080
httpSrv := client.Container().
From("python").
WithDirectory("/srv", client.Directory().WithNewFile("index.html", "Hello, world!")).
WithWorkdir("/srv").
WithExec([]string{"python", "-m", "http.server", "8080"}).
WithExposedPort(8080).
AsService()
// expose web service to host
tunnel, err := client.Host().Tunnel(httpSrv).Start(ctx)
if err != nil {
panic(err)
}
defer tunnel.Stop(ctx)
// get web service address
srvAddr, err := tunnel.Endpoint(ctx)
if err != nil {
panic(err)
}
// access web service from host
res, err := http.Get("http://" + srvAddr)
if err != nil {
panic(err)
}
defer res.Body.Close()
// print response
body, err := io.ReadAll(res.Body)
if err != nil {
panic(err)
}
fmt.Println(string(body))
}
Exposing host services to client containers - a.k.a. Container-to-Host (C2H) networking
You can also bind containers to host services so that clients in Dagger pipelines can communicate with services running on the host. Of course, this implies that a service is already listening on a port on the host, out-of-band of Dagger.
Here's an example of a container in a Dagger pipeline querying a MariaDB database service on the host:
package main
import (
"context"
"fmt"
"os"
"dagger.io/dagger"
)
func main() {
ctx := context.Background()
// create Dagger client
client, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stderr))
if err != nil {
panic(err)
}
defer client.Close()
// expose host service on port 3306
hostSrv := client.Host().Service([]dagger.PortForward{
{Frontend: 3306, Backend: 3306},
})
// create MariaDB container
// with host service binding
// execute SQL query on host service
out, err := client.Container().
From("mariadb:10.11.2").
WithServiceBinding("db", hostSrv).
WithExec([]string{"/bin/sh", "-c", "/usr/bin/mysql --user=root --password=secret --host=db -e 'SELECT * FROM mysql.user'"}).
Stdout(ctx)
if err != nil {
panic(err)
}
fmt.Println(out)
}
Controlling the service lifecycle
Services are designed to be expressed as a Directed Acyclic Graph (DAG) with explicit bindings allowing services to be started lazily. But sometimes, you may need bespoke service lifecycle management. Starting with Dagger 0.9, you can explicitly start and stop services as needed.
Here's an example which demonstrates starting a Docker daemon for use in a test suite:
package main_test
import (
"context"
"testing"
"dagger.io/dagger"
"github.com/stretchr/testify/require"
)
func TestFoo(t *testing.T) {
ctx := context.Background()
c, err := dagger.Connect(ctx)
require.NoError(t, err)
dockerd, err := c.Container().From("docker:dind").AsService().Start(ctx)
require.NoError(t, err)
// dockerd is now running, and will stay running
// so you don't have to worry about it restarting after a 10 second gap
// then in all of your tests, continue to use an explicit binding:
_, err = c.Container().From("golang").
WithServiceBinding("docker", dockerd).
WithEnvVariable("DOCKER_HOST", "tcp://docker:2375").
WithExec([]string{"go", "test", "./..."}).
Sync(ctx)
require.NoError(t, err)
// or, if you prefer
// trust `Endpoint()` to construct the address
//
// note that this has the exact same non-cache-busting semantics as WithServiceBinding,
// since hostnames are stable and content-addressed
//
// this could be part of the global test suite setup.
dockerHost, err := dockerd.Endpoint(ctx, dagger.ServiceEndpointOpts{
Scheme: "tcp",
})
require.NoError(t, err)
_, err = c.Container().From("golang").
WithEnvVariable("DOCKER_HOST", dockerHost).
WithExec([]string{"go", "test", "./..."}).
Sync(ctx)
require.NoError(t, err)
// Service.Stop() is available to explicitly stop the service if needed
}
See more examples in our guide on using services in Dagger.
SDK updates
We've also released new versions of our SDKs with support for Dagger 0.9, plus various SDK-specific bug fixes and improvements.
For a complete list of improvements, refer to the changelog for each SDK:
Go SDK 0.9.0 changelog and documentation
Node.js SDK 0.9.0 changelog and documentation
Python SDK 0.9.0 changelog and documentation
We are excited to hear about the use cases that Dagger’s new Host-to-Container (H2C), Container-to-Host (C2H) and explicit service lifecycle capabilities enable for you. Let us know via Discord, and perhaps come share with the rest of the Dagger Community in our reg ular fortnightly call. If you have feedback for us, or an improvement to suggest, join us on Discord or create a GitHub issue.
Looking forward to the automation that you will build with Dagger!