CI that keeps up.
← Changelog
Coming soon

Distributed cache v2

We are rebuilding how Dagger Cloud engines share their cache. Today they copy full disks, which is slow and causes cache misses. Version 2 moves single results. The scheduler decides which results each engine needs, and sends each pipeline to an engine that already has them. More cache hits, with no configuration.

Dagger Cloud runs your pipelines on a cluster of open-source Dagger Engines, which we host and manage. Each engine caches the result of each operation, for example a built container or a compiled binary, so that it does not run the operation again.

In a cluster, engines must share their caches. Today, Dagger Cloud does this with disk copies. Each engine starts from a copy of a shared cache disk. When the engine stops, its full disk becomes the new shared copy. This has three problems:

  1. Copies are best effort. When two engines run at the same time, only the cache of the last engine to stop is kept. New copies also get to other machines slowly, so an engine can start with an old copy, or with no copy.
  2. Transfers are large. An engine gets a full disk, even when it needs only one result.
  3. All results are shared, even when not useful. Some results are never used again, but they are uploaded and stored. Others are useful only to some engines, but all engines get them.

To fix these problems, we are building version 2 of the distributed cache. It shares each result separately. And it is integrated with the Dagger Cloud scheduler, the system that decides which engine runs each pipeline. With this integration, the scheduler can:

  • send a pipeline to an engine that already has the results it needs, so that no transfer is necessary
  • tell engines to upload only the results that are likely to be used again
  • tell each engine which results to download before a pipeline starts

An engine first downloads a small description of each result, and downloads the files only when an operation needs them, so pipelines start faster. If a download fails, the engine runs the operation again, so the pipeline does not fail.