“Replacing our makefiles with Dagger has greatly simplified our process for deployments since it’s much faster (1-3 hours to 3 minutes)...The power, flexibility and capability behind Dagger is something that gets me excited.”
We’re always interested in hearing how our community is using Dagger – their use cases, their challenges, and their experiences with deploying Dagger in different environments. Our Discord is a great place to find these stories, and to benefit from the knowledge and experience of the Dagger community.
In this blog post, we’ll share the story of Dagger community member, Andy Gonzalez (aka @duchi on Discord). Andy, an SRE team leader, used Dagger to speed up his production AWS Lambda deployments from a few hours to just a few minutes…a 93% reduction in processing time!
Andy and his team deploy their application using a CircleCI pipeline. This pipeline is a 3000+-line YAML file that performs multiple tasks for their core application. Besides those core tasks, there is a need to deploy 15 different AWS Lambdas (serverless functions) to support the primary application across multiple environments and using different accounts. To begin with, this was a totally manual process, and incorporating it into the existing CI YAML was daunting. Nevertheless, something had to be done since the manual Lambda deployments took between 1 and 3 hours every time!
Each Lambda is unique – it performs a different task and has its own set of dependencies. Some examples include antivirus scanning of file uploads by customers, RDS snapshot backups in multiple regions, client side logging and more. These functions are updated and re-deployed once per week.
There were a few challenges with this approach:
“Before we started to use Dagger, we deployed our Lambdas one at a time between all of our environments. We have four accounts with each containing one or more ‘environments’, so this would take a long time. Here is what our old manual steps would look like: (1) Get credentials and specify environment; (2) Deploy or update Lambda using Terraform CLI; (3) Verify Lambdas are behaving correctly; (4) Move on to the next environment.”
Andy had heard about Dagger and, as an initial proof-of-concept, he decided to try and automate the AWS Lambda deployment sub-process using Dagger. With Dagger, he was able to consolidate all the tasks related to Lambda deployment in a separate Dagger pipeline and run them alongside matrix jobs in CircleCI.
Under the new approach, the AWS Lambda deployment process is handled in a separate Dagger pipeline outside the CircleCI pipeline. When triggered, this pipeline performs the following tasks:
Key benefits of this new approach are:
“Dagger has allowed our engineers to deploy at a higher velocity and focus on other items instead of the manual workflow of deploying Lambdas. Our new Dagger process looks like this: (1) Run Dagger pipeline in CircleCI; (2) Verify Lambdas are behaving correctly.”
Going forward, Andy plans to combine the separate Lambda deployment pipeline with the existing CircleCI pipeline and simultaneously begin migrating the CircleCI pipeline to Dagger. He expects this will make the deployment process easier to test and maintain, and less reliant on the CircleCI platform.
In the CircleCI matrix, each job currently uses OIDC to connect to AWS and pass credentials to Dagger via environment variables. Andy will be looking for a better way to pass secrets through to executions – for example, using Dagger to connect to AWS directly for secrets.
“Dagger is an extremely powerful tool which allows users to upshift from any CI/CD platform without hesitation. I believe that Dagger is the future for CI/CD since the tool allows users to be so decoupled by the actual platform itself.”
Do you have a Dagger story you’d like us to feature? Tell us all about it in Discord!