Powering the CMS Software Factory at Craft CMS

We’re always interested in hearing how our community is using Dagger – their use cases, their challenges, and their experiences with deploying Dagger in different environments. Our Discord is a great place to find these stories, and to benefit from the knowledge and experience of the Dagger community.

In this blog post, we’ll share the story of Daggernaut Jason McCallister (aka @jasonmccallister on Discord). Jason is the Lead Cloud Platform Engineer and Architect of Craft Cloud, a cloud-based, serverless hosting platform for Craft CMS projects. Craft Cloud operates a fast, flexible build system that uses Dagger to build and deploy customer websites to the cloud.

"Dagger powers our build system. It allows us to programmatically construct a highly scalable and efficient pipeline for orchestrating complex builds, eliminating the need for manual Dockerfile manipulation." - Jason McCallister, Craft CMS

The Need for Programmable Builds

Craft CMS is an open source PHP CMS with an intuitive, user-friendly dashboard for content creation and management. Craft Cloud is a managed hosting platform for Craft CMS projects. First introduced in 2019, it has evolved over time from a headless CMS system to a complex cloud platform that integrates Cloudflare, AWS Lambda, S3, and Cloudflare Workers for edge transformations and HTTP routing. 

Users interact with Craft Cloud through a browser-based Web console called Craft Console, which they can use to connect with their project repository and define their Craft CMS environments, domains, variables, and deployment strategy. Once done, Craft Cloud takes care of building the project on every commit and deploying it at a public URL using AWS Lambda.

Each Craft Cloud project is built as a container image containing everything it needs to run: environment configuration, application codebase, database credentials, web server, queue worker, and so on. In essence, Craft Cloud is a software factory, continuously churning out container images of user projects…and for such a software factory to work, a reliable build system is paramount.

Prior to Dagger, the build system consisted of various Go programs that would manually manipulate Dockerfiles to match user-specified configuration and build images accordingly. This was tedious, error-prone, and hard to manage. With the release of Dagger's Go SDK in 2022, things started looking up. By making it possible to create container images programmatically, Dagger's Go SDK gave Jason the tools he needed to create a powerful, reliable container build system for Craft Cloud. Work on the project started in Jan 2023, and Craft Cloud's Dagger-powered build system entered production shortly after.

"We try to use as many open source tools as we can, so the fact that Dagger was open source was pretty key for us. That apart, it was getting traction, had good documentation, the Go SDK was pretty easy to get started with without even looking at docs...it all added up to a strong case for trying it out."

Fast, Flexible Builds and Backups, Powered by Dagger

Today, Craft Cloud’s Dagger-powered build system runs continuously, all day, every day. It handles between 25-50 builds per day and, on average, it is able to build Craft CMS sites in about 1 minute. Node.js builds usually complete in 3 additional minutes depending on the complexity.

Here's how it works:

1. A change in a user's project Git (Craft Cloud supports Github, GitLab, and BitBucket) repository (or a manual deployment request) triggers a build in the Craft Cloud API.

2. The API checks for an environment (dev, staging, production) matching the branch on which the change occurred.

3. If an environment exists, the API retrieves the necessary secrets and runs a Go program to clone the project at the correct branch and commit.

4. The API requests a Dagger build. Builds are performed on-demand in AWS ECS that communicates with an internal Dagger Engine running on AWS EC2 with an auto-scaling group.

5. The Dagger pipeline pulls a base image (from Bref PHP) matching the PHP version configured in the project.

6. Sets up the project environment, including downloading Composer or Node.js dependencies, configuring a MySQL or PostgreSQL database, transforming images, etc.

7. The API publishes a new version of the AWS Lambda function using the new container image.

8. Once published, the API notifies the user and updates the status of the deployment in the Craft Cloud console.

The primary benefit to Craft Cloud from Dagger is its programmability. Craft Cloud users define their Craft CMS environment using a YAML configuration file. At build time, the Craft Cloud build system must parse this file and apply it to the build environment. Dagger makes it easy to programmatically control the build environment and build process based on the configuration specified by a user. In addition, native Go support means that common concerns like error handling, unit testing, and logging can be quickly satisfied in line with language best practices.

Dagger is at the core of Craft Cloud's build system, but that's not the only place where it performs a critical function. Dagger also powers Craft Cloud's backup system, handling both manual and automated nightly backups of production databases. In this case, a Go application inspects project configuration, identifies the database type (Craft Cloud supports both MySQL and PostgreSQL) and version in use by talking directly to the database. It then uses Dagger to download the corresponding image based on the actual version of the database, performs a version-compatible backup, uploads the backup file to AWS S3, and provides an API for users to download the backup.

" We use Go, so it just made sense for us to be able to continue use it for this project. And then, finding out how flexible the Go SDK was in terms of using an image, setting the entrypoint and environment variables, using secrets, was great."

Future Improvements: Smoke Testing, Matrix Testing, and Dagger Functions

Going forward, Jason sees multiple opportunities to embed Dagger even more deeply into Craft Cloud's software factory.

• Smoke-testing website deployments: Jason is interested in making Craft Cloud website deployments completely error-free, by spinning up project environments and "smoke testing" them before deployment. He expects to use Dagger's built-in services support for this, and in the process also make debugging easier by improving the logs that users see during the deployment process.

• Matrix-testing plugins: CraftCMS can be extended with plugins, and Craft operates a plugin store which allows developers to create and sell Craft CMS plugins. Jason would like to use Dagger in a test matrix, to automate testing of plugins against different versions of CraftCMS and identify compatibility issues upfront.

• Caching: Dagger is able to cache both operations (such as copying files or directories to a container, running tests, compiling code, etc.) and volumes (such as data caches or package manager caches). The Craft Cloud build system doesn't currently use caching to its fullest extent; Jason hopes to explore this further to reduce the build times for React/Vue-based websites.

• Migrating to Dagger Functions: The current Dagger implementation uses a custom Go tool built with the Dagger SDK, and so does not benefit from the portability and reusability of Dagger Functions. Jason hopes to convert the Go codebase into one or more Dagger Functions soon.

"Dagger is at the heart of everything we do. We use it to serve our customers with on-demand builds and database backups, and it's deeply embedded in our process. We have complete trust in it, it just works, and the feedback we've been getting from our users is really good!"

Jason's Power Tip for Aspiring Daggernauts

We asked Jason what he thought was the best way for platform engineers and DevOps professionals to get started with Dagger. Here's what he said:

“My suggestion would be to take something you already have working and just try to replace it with Dagger. Start with your repository, look at your GitHub action, your Makefile, or whatever you have set up, and try converting some of it to use Dagger. If you're building a container image, that's easy to do, but what if it's something else, like maybe an S3 sync? Instead of using a GitHub Action for that, try to get it working with Dagger. And once you have that first piece working, move on to the next one, and so on.”