Daggerizing Production Pipelines - How OpenMeter Overcame Push and Pray

We’re always interested in hearing how our community is using Dagger – their use cases, their challenges, and their experiences with deploying Dagger in different environments. Our Discord is a great place to find these stories, and to benefit from the knowledge and experience of the Dagger community.

In this blog post, we’ll share the story of Daggernaut Márk Sági-Kazár (aka @sagikazarmark on Discord). Mark is the Head of Open Source at OpenMeter, which develops an open source product to help engineers monetize AI products and standardize usage metering. When Mark joined OpenMeter, he was already an active Dagger user and contributor to the Dagger project. He was determined not to rebuild the typical CI mess he'd seen elsewhere and so, he decided to build OpenMeter’s delivery pipelines with Dagger. Today, OpenMeter is using Dagger Functions and Dagger Cloud in multiple production pipelines, and Mark is extremely happy with the results.

"Dagger is going to be a market disruptor because it completely changes how software development works. It's just completely different from all of the other approaches out there, and obviously superior in a lot of ways." - Márk Sági-Kazár, Head of Open Source, OpenMeter

Exploring Solutions to Improve Developer Effectiveness

Mark is a seasoned platform engineer, and has a deep understanding of the struggles and challenges with popular CI/CD tools. Some of the problems he has experienced in the past are:

• The “push and pray” problem: Mark said that to him, this is not a marketing slogan, but a very real problem that he sees developers experiencing every day. He has seen developers waste hours re-running their pipelines in CI, unable to figure out if the bug is in their code, their tests or the pipeline itself.

• Inconsistent environments: Very often, tests succeed locally but fail in CI. This "works on my machine" problem is familiar to almost all developers and platform engineers, and the reason is that development and CI environments often differ drastically.

• Onboarding friction: The previous problem also directly affects developer productivity. Manually configuring developer environments with all the required dependencies and tools can take days, increasing the time for developers to become productive and start contributing useful code.

• Lack of reusability: Creating a bespoke pipeline for every application is time-consuming. Maintaining those pipelines as tools and needs evolve over time is an additional challenge.

When Mark joined OpenMeter, he was keen to implement a platform that solved all these problems. He wanted a toolchain that eliminated push and pray for developers, made it easy to create standardized environments, and was compatible with DRY (Don't Repeat Yourself) principles so that he could quickly leverage existing pipelines to create new ones. 

This focus on improving the overall experience, for both OpenMeter's developers and platform engineers, first led him to Dagger in its Europa release, and he has been a fan and early adopter of Dagger since then.

Mark was attracted to Dagger initially because of its ability to create completely self-contained project environments with all dependencies installed, regardless of operating system. Built-in caching, native-language SDKs, and fast local runs were also key factors in his selection of Dagger for OpenMeter's delivery pipelines.

"I know there are lots of struggles with tools and environments all the time. I've always been trying to minimize that struggle. When I discovered Dagger, I realized it was a huge step forward. Being able to write the pipeline in the language that you and your developers know - that’s a very important and very underappreciated feature." 

Using Dagger in Production: Testing, Releasing, and Deploying 

To implement OpenMeter's Daggerized pipelines, Mark followed a "bottom up" approach. 

1. He first created a number of lower-order Dagger Functions, to handle individual tasks like linting, code formatting, compiling, running tests, and so on. 

2. He then created higher-order Dagger Functions that sequenced the lower-order ones to "run all linters", "run all tests", "build for all platforms", and so on. 

3. Finally, he connected the higher-order Dagger Functions together to create a complete pipeline, also expressed as a Dagger Function.

The advantage of this approach is that developers can now run an entire pipeline with a single Dagger Function call instead of running each step individually. At the same time, they have the option to call the lower-order Dagger Functions individually to hunt down specific bugs or localized problems. To make debugging easier, Mark and his colleagues also use the Traces feature in Dagger Cloud to visualize Dagger Function calls and zoom into errors.

"When it comes to the actual software pipelines, everything is implemented in Dagger. So before I commit and push code, I can run everything locally. The caching is also great; I’ve experienced a lot of improvements after converting some projects to Dagger.” 

In addition to Daggerizing OpenMeter’s CI pipeline, Mark is using Dagger for:

• Release management: Mark has implemented OpenMeter's release pipelines entirely in Dagger, for both its open source and private repositories. This Dagger pipeline is responsible for building containers, Helm charts and client SDKs, and then pushing these artifacts to various registries and application package repositories like PyPi and NPM.

• Continuous delivery: Mark has recently updated OpenMeter's continuous delivery pipeline to use a combination of Dagger and Flux. Previously, OpenMeter used Flux to update its GitOps resources based on the latest versions in the container registry. Since releases are now managed through Dagger, Mark decided to Daggerize part of this pipeline and have Dagger update the GitOps resources on every release (Flux continues to manage the actual deployment process).

• Snapshotting: OpenMeter uses Dagger to snapshot its ongoing development, by automatically publishing container images for every commit made to its `main` branch.

Benefits: Maximum Satisfaction for both Developers and Platform Engineers

Mark is extremely happy with Dagger, for a number of reasons:

• No "push and pray": With Dagger, developers are able to run tests and create builds locally in exactly the same way as the real CI environment. Failures, if any, are identified and fixed locally. Developers know that if the pipeline passes locally, it will also pass in CI, and this consistency eliminates both project risk and developer frustration.

• Zero onboarding friction: When new developers join OpenMeter, the onboarding is almost an afterthought. The only thing they need is Dagger (and Docker) installed on their development machines, and everything else is installed or run automatically. Every developer operates in the same standardized environment.

"Developers at OpenMeter are able to submit PRs on day one. Dagger plays a huge role in that." 

• Fast feedback loop: Streamlined pipelines, caching, and the ability to run pipelines locally mean that time waiting is minimized.  For example, if a developer changes a dependency, he or she can run the pipeline locally to confirm it all still works, instead of having to wait for a remote CI runner to pick up the change. This is a huge benefit to the business, because the shorter feedback loop allows the team to develop and push code much faster.

• Dagger modules are reusable and extensible, which means that Mark is able to create new Daggerized pipelines simply by chaining his existing battle-tested and proven Dagger Functions together in new combinations. This has improved his productivity significantly, and also allowed him to gain greater insights into how to optimize his Dagger Functions for different usage scenarios.

"I spend a fair bit of time on building Dagger modules or trying to find the right API and the right experience. A lot of people told me that they were going to build a certain module, but then found on the Daggerverse that I'd already built the module they needed." 

Leveraging Dagger Cloud to Analyze and Improve Performance

Although Mark and the OpenMeter team are happy with their Daggerized pipelines, Mark is keen to improve them even further. For example, he plans to enhance the snapshotting workflow to also publish Helm charts and some of the SDK packages as build artifacts.

Mark is also looking to Dagger Cloud to provide better observability and metrics. On his wishlist are metrics to better understand changes in build performance over time, improved visibility into cache hits and misses, and UX improvements to find the source of pipeline failures quicker.

"Dagger has never stopped amazing me. The first release of the language SDKs was revolutionary, and then there was the Zenith release with Dagger Functions. I really believe that Dagger is on the right path to revolutionize the whole developer environment and CI/CD space." 

Do you have a Dagger story you’d like us to feature? Tell us all about it in Discord.

Mark’s Top Tips for Aspiring Daggernauts

Mark is an active contributor to the Dagger community, and often provides feedback to the Dagger team on new features. He enjoys building and contributing Dagger modules, all of which are publicly available in the Daggerverse. 

So, we asked Mark for his top suggestions for platform engineers and DevOps professionals just getting started with Dagger. Here's what he said:

1. If you're like me and you like learning by example, start with the Dagger quickstart. It shows you how to build a simple Dagger module and along the way, it introduces you to all the key components you need to know, including the Dagger CLI, the Dagger API, Dagger Functions, the Daggerverse, ...

2. Take a look at my modules in the Daggerverse to understand how Dagger works. A good one to start with if you have a Go project, is my Go builder module. It's built very similarly to how you would use the Go tool itself. So, for example, if you want to build a binary, I have a build function in my Go module and you can “dagger call” it to build your Go project.

3. Pick an existing project with a Dockerfile and convert it to Dagger. The API of Dagger is fairly similar to that of a Dockerfile. Producing the same container image with Dagger will give you an easy bridge to start understanding how Dagger works. Then, try to do something more complex: once you have your image, try using Dagger to push it to a registry or deploy it somewhere. The Dagger Cookbook and the Daggerverse have lots of example code you can use for inspiration.

4. If you have questions, ping the Dagger team in the #help channel on Discord. They are super responsive and quick to help or provide advice.