Better CI Testing for the Hof Project
August 22, 2023
Aug 22, 2023
Case Study
We’re always interested in hearing how our community is using Dagger – their use cases, their challenges, and their experiences with deploying Dagger in different environments. Our Discord is a great place to find these stories, and to benefit from the knowledge and experience of the Dagger community.
In this blog post, we’ll share the story of Daggernaut Tony Worm (aka @verdverm on Discord and Twitter). Tony is the developer of Hof, an open source code generation framework. Tony has been able to significantly improve the consistency and accuracy of Hof's test suite by using containerized test environments orchestrated with Dagger.
"With Dagger, I can define what I want my desired results to be, and Dagger figures out what needs to happen to get those results."
- Tony Worm, Hof
The problem: Test failures in mock environments
Hof is an open source code generation tool. It supports shared data models and produces scaffolding code and boilerplate for any technology stack, letting users be in full control. Hof builds on a new language CUE, using it to power many of its features. As part of dependency management, hof writes various cache and configuration files to the user's home environment on the host. It also runs various containers on the host for automatic code formatting during the code generation process or as task environments in the DAG workflow engine.
To test these features, Tony initially relied on the Go [testscript](https://github.com/rogpeppe/go-internal/tree/master/cmd/testscript) package, which is intended for CLI and filesystem-based tests. With this package, Tony could "mock" a user's home environment and run the Hof test suite in this mock environment without polluting the live filesystem on the host. However, this approach was imperfect: path calculations were complex due to the testing peculiarities of both mock and original environments co-existing on the same host.
"When we're testing and things look different enough from the real environment, it causes issues with our path calculations. We had a point where we just couldn't get our tests to pass one way or the other. And so, we were in the market for a new solution."
The solution: Containerized, clean test environments and matrix testing with Dagger
As a member of the CUE community and maintainer of Cuetorials, Tony was already familiar with Dagger from its previous CUE-based implementation. He realized that he could create a more realistic testing environment using containers orchestrated by Dagger. Dagger being open-source also aligned closely with the Hof project's own values.
With Dagger, Tony has simplified and improved Hof's test suite by running it in a containerized home environment. Hof's Dagger pipeline (created with Dagger's Go SDK) begins by producing a base container image containing the build to be tested and the required environment. It then loops over the test suite, executing each test in a separate container spawned from the base image.
Key benefits are:
The Dagger pipeline runs each test in a "real" home environment. As a result, there are no additional considerations or limitations related to mocked-up environments.
The Dagger pipeline runs each test in a clean, isolated container derived from the base container image. This eliminates stale states or conflicts caused by previous tests.
Native Go support has allowed the team to adopt a matrix test pattern and execute the test suite against different versions of Go to identify and eliminate version-related incompatibilities.
Go support has also enabled the team to continue using `testscript` without needing to rewrite any tests, as well as continuously improve the pipeline by adding new tests and Go best practices.
"In the end, what we wanted was a better testing environment, and that meant doing things in a container. After switching to Dagger, we were able to remove some really nasty code for dealing with our test environment issues."
The future: Broader testing and end-to-end release packaging with Dagger
Tony is excited by the potential of Dagger and plans to use it in multiple ways:
Hof already uses Docker to perform various core UX functions, including various Git and filesystem operations. Tony is exploring the possibility of shipping a Dagger pipeline with Hof to perform these core functions. As this pipeline would be fully programmable, he expects it would be significantly easier to maintain and extend.
Currently, GoReleaser takes care of packaging and distributing the Hof build produced by the Dagger pipeline. In the future, Tony hopes to complement GoReleaser with Dagger and create an end-to-end release process integrating Dagger.
Hof runs containers in the background and needs to support multiple container runtimes. The current Dagger pipeline is able to test Hof using Docker-in-Dagger. Tony is looking forward to Dagger adding support for container-to-container sockets in future, which will enable similar testing for Podman and Nerdctl.
In addition to using Dagger's Go SDK to test Hof, Tony also uses Dagger's Node.js SDK to build Hof's Next.js-based documentation website. In the future, he hopes to extend this pipeline and integrate Dagger with GitHub Actions to automatically build, test, and publish Hof's documentation on every successful merge.
"Dagger was a natural choice for us. Dagger is adding value by building on top of the language ecosystem. I can now create modules of Dagger snippets that I can use in other places, and that's a really powerful concept."
Do you have a Dagger story you’d like us to feature? Tell us all about it in Discord!
We’re always interested in hearing how our community is using Dagger – their use cases, their challenges, and their experiences with deploying Dagger in different environments. Our Discord is a great place to find these stories, and to benefit from the knowledge and experience of the Dagger community.
In this blog post, we’ll share the story of Daggernaut Tony Worm (aka @verdverm on Discord and Twitter). Tony is the developer of Hof, an open source code generation framework. Tony has been able to significantly improve the consistency and accuracy of Hof's test suite by using containerized test environments orchestrated with Dagger.
"With Dagger, I can define what I want my desired results to be, and Dagger figures out what needs to happen to get those results."
- Tony Worm, Hof
The problem: Test failures in mock environments
Hof is an open source code generation tool. It supports shared data models and produces scaffolding code and boilerplate for any technology stack, letting users be in full control. Hof builds on a new language CUE, using it to power many of its features. As part of dependency management, hof writes various cache and configuration files to the user's home environment on the host. It also runs various containers on the host for automatic code formatting during the code generation process or as task environments in the DAG workflow engine.
To test these features, Tony initially relied on the Go [testscript](https://github.com/rogpeppe/go-internal/tree/master/cmd/testscript) package, which is intended for CLI and filesystem-based tests. With this package, Tony could "mock" a user's home environment and run the Hof test suite in this mock environment without polluting the live filesystem on the host. However, this approach was imperfect: path calculations were complex due to the testing peculiarities of both mock and original environments co-existing on the same host.
"When we're testing and things look different enough from the real environment, it causes issues with our path calculations. We had a point where we just couldn't get our tests to pass one way or the other. And so, we were in the market for a new solution."
The solution: Containerized, clean test environments and matrix testing with Dagger
As a member of the CUE community and maintainer of Cuetorials, Tony was already familiar with Dagger from its previous CUE-based implementation. He realized that he could create a more realistic testing environment using containers orchestrated by Dagger. Dagger being open-source also aligned closely with the Hof project's own values.
With Dagger, Tony has simplified and improved Hof's test suite by running it in a containerized home environment. Hof's Dagger pipeline (created with Dagger's Go SDK) begins by producing a base container image containing the build to be tested and the required environment. It then loops over the test suite, executing each test in a separate container spawned from the base image.
Key benefits are:
The Dagger pipeline runs each test in a "real" home environment. As a result, there are no additional considerations or limitations related to mocked-up environments.
The Dagger pipeline runs each test in a clean, isolated container derived from the base container image. This eliminates stale states or conflicts caused by previous tests.
Native Go support has allowed the team to adopt a matrix test pattern and execute the test suite against different versions of Go to identify and eliminate version-related incompatibilities.
Go support has also enabled the team to continue using `testscript` without needing to rewrite any tests, as well as continuously improve the pipeline by adding new tests and Go best practices.
"In the end, what we wanted was a better testing environment, and that meant doing things in a container. After switching to Dagger, we were able to remove some really nasty code for dealing with our test environment issues."
The future: Broader testing and end-to-end release packaging with Dagger
Tony is excited by the potential of Dagger and plans to use it in multiple ways:
Hof already uses Docker to perform various core UX functions, including various Git and filesystem operations. Tony is exploring the possibility of shipping a Dagger pipeline with Hof to perform these core functions. As this pipeline would be fully programmable, he expects it would be significantly easier to maintain and extend.
Currently, GoReleaser takes care of packaging and distributing the Hof build produced by the Dagger pipeline. In the future, Tony hopes to complement GoReleaser with Dagger and create an end-to-end release process integrating Dagger.
Hof runs containers in the background and needs to support multiple container runtimes. The current Dagger pipeline is able to test Hof using Docker-in-Dagger. Tony is looking forward to Dagger adding support for container-to-container sockets in future, which will enable similar testing for Podman and Nerdctl.
In addition to using Dagger's Go SDK to test Hof, Tony also uses Dagger's Node.js SDK to build Hof's Next.js-based documentation website. In the future, he hopes to extend this pipeline and integrate Dagger with GitHub Actions to automatically build, test, and publish Hof's documentation on every successful merge.
"Dagger was a natural choice for us. Dagger is adding value by building on top of the language ecosystem. I can now create modules of Dagger snippets that I can use in other places, and that's a really powerful concept."
Do you have a Dagger story you’d like us to feature? Tell us all about it in Discord!
We’re always interested in hearing how our community is using Dagger – their use cases, their challenges, and their experiences with deploying Dagger in different environments. Our Discord is a great place to find these stories, and to benefit from the knowledge and experience of the Dagger community.
In this blog post, we’ll share the story of Daggernaut Tony Worm (aka @verdverm on Discord and Twitter). Tony is the developer of Hof, an open source code generation framework. Tony has been able to significantly improve the consistency and accuracy of Hof's test suite by using containerized test environments orchestrated with Dagger.
"With Dagger, I can define what I want my desired results to be, and Dagger figures out what needs to happen to get those results."
- Tony Worm, Hof
The problem: Test failures in mock environments
Hof is an open source code generation tool. It supports shared data models and produces scaffolding code and boilerplate for any technology stack, letting users be in full control. Hof builds on a new language CUE, using it to power many of its features. As part of dependency management, hof writes various cache and configuration files to the user's home environment on the host. It also runs various containers on the host for automatic code formatting during the code generation process or as task environments in the DAG workflow engine.
To test these features, Tony initially relied on the Go [testscript](https://github.com/rogpeppe/go-internal/tree/master/cmd/testscript) package, which is intended for CLI and filesystem-based tests. With this package, Tony could "mock" a user's home environment and run the Hof test suite in this mock environment without polluting the live filesystem on the host. However, this approach was imperfect: path calculations were complex due to the testing peculiarities of both mock and original environments co-existing on the same host.
"When we're testing and things look different enough from the real environment, it causes issues with our path calculations. We had a point where we just couldn't get our tests to pass one way or the other. And so, we were in the market for a new solution."
The solution: Containerized, clean test environments and matrix testing with Dagger
As a member of the CUE community and maintainer of Cuetorials, Tony was already familiar with Dagger from its previous CUE-based implementation. He realized that he could create a more realistic testing environment using containers orchestrated by Dagger. Dagger being open-source also aligned closely with the Hof project's own values.
With Dagger, Tony has simplified and improved Hof's test suite by running it in a containerized home environment. Hof's Dagger pipeline (created with Dagger's Go SDK) begins by producing a base container image containing the build to be tested and the required environment. It then loops over the test suite, executing each test in a separate container spawned from the base image.
Key benefits are:
The Dagger pipeline runs each test in a "real" home environment. As a result, there are no additional considerations or limitations related to mocked-up environments.
The Dagger pipeline runs each test in a clean, isolated container derived from the base container image. This eliminates stale states or conflicts caused by previous tests.
Native Go support has allowed the team to adopt a matrix test pattern and execute the test suite against different versions of Go to identify and eliminate version-related incompatibilities.
Go support has also enabled the team to continue using `testscript` without needing to rewrite any tests, as well as continuously improve the pipeline by adding new tests and Go best practices.
"In the end, what we wanted was a better testing environment, and that meant doing things in a container. After switching to Dagger, we were able to remove some really nasty code for dealing with our test environment issues."
The future: Broader testing and end-to-end release packaging with Dagger
Tony is excited by the potential of Dagger and plans to use it in multiple ways:
Hof already uses Docker to perform various core UX functions, including various Git and filesystem operations. Tony is exploring the possibility of shipping a Dagger pipeline with Hof to perform these core functions. As this pipeline would be fully programmable, he expects it would be significantly easier to maintain and extend.
Currently, GoReleaser takes care of packaging and distributing the Hof build produced by the Dagger pipeline. In the future, Tony hopes to complement GoReleaser with Dagger and create an end-to-end release process integrating Dagger.
Hof runs containers in the background and needs to support multiple container runtimes. The current Dagger pipeline is able to test Hof using Docker-in-Dagger. Tony is looking forward to Dagger adding support for container-to-container sockets in future, which will enable similar testing for Podman and Nerdctl.
In addition to using Dagger's Go SDK to test Hof, Tony also uses Dagger's Node.js SDK to build Hof's Next.js-based documentation website. In the future, he hopes to extend this pipeline and integrate Dagger with GitHub Actions to automatically build, test, and publish Hof's documentation on every successful merge.
"Dagger was a natural choice for us. Dagger is adding value by building on top of the language ecosystem. I can now create modules of Dagger snippets that I can use in other places, and that's a really powerful concept."
Do you have a Dagger story you’d like us to feature? Tell us all about it in Discord!
