CI that keeps up
how-toby Kyle Penfound

How do I run Playwright tests in CI and locally in the same container, with cached browsers?

A reliable Playwright CI setup does four things. It runs the tests in Microsoft’s Playwright Docker image, mcr.microsoft.com/playwright, at the tag for your exact Playwright version. It does not cache the browsers, because the image already contains them. It updates screenshot baselines in that same image, and it uses one CPU architecture on every machine. You can meet each requirement by hand in each CI config and local script. Dagger’s official Playwright module meets all four in one place, and runs the same way on a laptop and in CI.

What a reliable Playwright CI setup needs

Playwright’s own docs give four requirements for test runs that behave the same in CI and on a laptop. This guide uses them to judge each approach, Dagger included:

  1. Run the tests in the Playwright image for your exact version. The Playwright Docker image “includes the Playwright browsers and browser system dependencies”. It does not include the Playwright package, so the job still installs your dependencies. Pin the image tag to the exact version of @playwright/test, such as v1.64.1-noble for Ubuntu 24.04. If the two versions differ, Playwright “will be unable to locate browser executables”, says its Docker page.
  2. Do not cache the browsers. Playwright’s CI guide says that a restore of the browser cache takes about as long as a fresh download of the browsers. On Linux, a job must still install the operating system dependencies, and Playwright says these are not cacheable. The cache would hold the browser directory, ~/.cache/ms-playwright on Linux. In the Playwright image, the browsers are part of the image, so there is no browser directory to cache.
  3. Update screenshot baselines in the same image. Playwright’s visual comparisons page warns that “browser rendering can vary based on the host OS, version, settings, hardware, power source (battery vs. power adapter), headless mode, and other factors.” It tells you to run the tests in the same environment that generated the baselines. Playwright also puts the browser and the platform in the name of each baseline file, such as home-chromium-linux.png. A baseline from a Mac is home-chromium-darwin.png, so a Linux CI run does not find it.
  4. Use one CPU architecture. The same image gives the same fonts, libraries, and browser builds on every machine. It does not give the same CPU architecture. The image has arm64 and amd64 variants, and each machine pulls the variant for its own architecture. In a Playwright issue about this, a maintainer wrote that “it is expected that arm docker image vs intel docker image produce different screenshots.”

Meet them in your CI config

A container job meets the first two requirements. Playwright’s CI guide shows the image on several CI systems. GitHub Actions uses jobs.<job_id>.container, GitLab CI uses a job image, Jenkins uses a Docker agent, and CircleCI uses a docker entry in the job. The job runs npm ci and then npx playwright test in the image. It downloads no browsers, so it has nothing to cache.

The CI guide also gives a second way to prepare a CI agent: install the browsers with the Playwright CLI in each job. After npm ci, one command downloads the browsers and installs the operating system packages that they need:

npx playwright install --with-deps chromium

This approach does not cache the browsers, but it does not use the Playwright image, and each job pays for the download again. In this guide’s project, the command took 30 seconds in a node:22-bookworm container on an arm64 laptop. It put 680 MB in ~/.cache/ms-playwright: 401 MB for Chromium, 276 MB for the headless shell, and 4 MB for ffmpeg.

To meet the third requirement, update the baselines in the image, not on the host. From the project root, run --update-snapshots in the image with docker run:

docker run --rm --init --ipc=host --platform linux/amd64 -v "$PWD":/work -v /work/node_modules -w /work mcr.microsoft.com/playwright:v1.64.1-noble sh -c "npm ci && npx playwright test --update-snapshots"

Playwright’s Docker page recommends the --init and --ipc=host flags. Without --ipc=host, Chromium can run out of memory and crash. The second -v flag gives node_modules its own volume, so the Linux packages do not replace the ones in your checkout. In this guide’s project, the command wrote tests/home.spec.ts-snapshots/home-chromium-linux.png. Review the file and commit it.

To meet the fourth requirement, choose one platform and pass it with --platform everywhere the image runs. The command above uses linux/amd64. If the team uses both arm64 and amd64 machines, run every machine on one platform, or update baselines from one platform only.

Where the hand-built setup strains

The hand-built setup puts the same container settings in several places, and each place must change at the same time:

  • The image tag is in the CI config, in local docker run scripts, and often in a devcontainer. Each copy must change in the commit that upgrades @playwright/test. The --platform value must match in each copy too.
  • A developer who skips the docker run script updates baselines on the host, not in the image.
  • A move to another CI system means a rewrite of the container job in the syntax of that system.

Meet them with the Dagger Playwright module

The Dagger Playwright module meets the same four requirements, with the image, the install, and the test command defined in one place. Dagger is a CI orchestration engine. You define the pipeline once, in code, and it runs the same way on a laptop and in CI. The open-source Dagger Engine runs the pipelines on your machine or on a server. A module is a reusable package of functions, checks, and services that a project installs.

From the project root, install the official Playwright module:

dagger module install dagger.io/js/playwright

The command adds the module to dagger.toml, the workspace configuration, and creates that file if the project has none. It also pins the Git ref of the module in dagger.lock. Commit both files. The Quickstart covers the rest of the Dagger setup for a project.

A check is a function that validates the project, such as a test run. List the checks:

dagger check -l
MODULE      PLAYWRIGHT-PROJECT  PLAYWRIGHT-TEST-FILE  CHECK  DESCRIPTION
playwright  *                   *                     test   Run this test file.
# Use --all to expand collections and list each item.

Each directory with a playwright.config is a project, and the check can run each test file on its own. A * stands for every item. Run the tests:

dagger check

The module installs your dependencies in the Playwright image and runs the project’s own playwright test. If the config declares a webServer, Playwright starts the app inside the container, as it does on your machine. The next four sections show how the module meets each requirement.

Run the tests in the Playwright image for your exact version

The module runs your tests in the same mcr.microsoft.com/playwright image as a hand-built container job. It reads your @playwright/test version from package-lock.json, pnpm-lock.yaml, yarn.lock, or bun.lock, and picks the image tag for that version. Without a lockfile, it reads package.json, and it reads a range such as ^1.58.2 as its lower bound. Pin that version exactly, or commit a lockfile. Show the image that the module chose:

dagger api call playwright project --path=. image-address
mcr.microsoft.com/playwright:v1.64.1-noble

When you upgrade @playwright/test, the lockfile changes, and the module picks the new tag. No other file has a copy of the tag.

Do not cache the browsers

With the module, there is no browser directory to cache. The browsers come with the Playwright image, and the Dagger Engine caches the pull of that image. Dagger caches each step by its inputs and runs a step again only when an input changes. A later run does not download the browsers again. The module also mounts a cache volume, a directory whose contents persist across runs, for the download cache of the package manager.

The step that installs dependencies reads only the package manifests, the lockfiles, and the package manager config. A change to a test or source file does not start a new install.

In this guide’s project, on an arm64 laptop, the first dagger check took 46.6 seconds. That run pulled the Playwright image and ran npm install. A rerun with no changes took 2.7 seconds, with every step from the cache:

== CHECKS ==  ✔ 1 passed
✔ dag://?check=test&playwright-project=. 0.3s OK

After an edit to the test file, dagger check took 4.8 seconds. Dagger ran the tests again and used the cached npm install. These numbers come from one run on one machine, so treat them as an example.

Update screenshot baselines in the same image

The module updates baselines in its test container, which is the Playwright image with your dependencies installed. In a project without baselines, the first dagger check fails as Playwright does anywhere else:

== CHECKS ==  ✘ 1 failed
✘ dag://?check=test&playwright-project=. 1.5s ERROR
  == TESTS ==
    ✘ chromium FAIL
    ✘ chromium::home.spec.ts FAIL

    ✘ chromium › home.spec.ts › home page FAIL
          Error: A snapshot doesn't exist at /app/tests/home.spec.ts-snapshots/home-chromium-linux.png, writing actual.
              at /app/tests/home.spec.ts:5:3
          
            3 | test('home page', async ({ page }) => {
            4 |   await page.goto('/');
          > 5 |   await expect(page).toHaveScreenshot('home.png');
              |   ^
            6 | });
            7 |

    ✘ 1 failed

== RUN LOCALLY ==
dagger check "dag://?check=test&playwright-project=."

Run --update-snapshots in the test container, then export the test directory to your checkout:

dagger api call playwright project --path=. base with-exec --args=npx,playwright,test,--update-snapshots directory --path=tests export --path=tests

The command writes tests/home.spec.ts-snapshots/home-chromium-linux.png from inside the image, as the docker run command does. Review the file and commit it. The next dagger check compares against it:

== CHECKS ==  ✔ 1 passed
✔ dag://?check=test&playwright-project=. 1.2s OK
  == TESTS ==
    ✔ chromium  1 passed

    ✔ 1 passed

If your tests are in a directory other than tests, change both paths in the command.

Use one CPU architecture

To use one CPU architecture, give the module a base container for one platform. By default, a container uses the native platform of the engine. An arm64 laptop and an amd64 CI engine then pull different variants of the image. Cloud Engines are the managed Dagger Engines that run Cloud Checks. They run on amd64 and on arm64 (arm64 Cloud Engines). Pick one platform for the whole team and pin it. The example below pins amd64.

The baseCtr setting of the module takes a container from a function in another module. This is module wiring. A setting whose value is a dag://<module>/<function> address gets the value that the function returns.

Write the function in Dang, the native language of Dagger for modules. Install the Dang SDK, then create the dev module of the project:

dagger module install dagger.io/sdk/dang
dagger module init dang

dagger module init creates the module at .dagger/modules/<project>-dev, records it in dagger.toml, and shows the changes for you to apply. Replace the contents of .dagger/modules/<project>-dev/main.dang with this function. WebDev stands in for the type name that dagger module init generated, so keep your own:

type WebDev {
  """
  The Playwright image for browser tests, pinned to linux/amd64 so
  screenshots match on every machine.
  """
  pub playwrightBase: Container! {
    container(platform: "linux/amd64").from("mcr.microsoft.com/playwright:v1.64.1-noble")
  }
}

Wire the function into the Playwright module:

dagger module settings playwright baseCtr playwright-base

The command stores the full address in dagger.toml:

[modules.playwright.settings]
baseCtr = "dag://<project>-dev/playwright-base"

Checks and baseline updates now run in the amd64 image on every machine. On an arm64 machine, the amd64 image runs under emulation. In this guide’s project on an arm64 laptop, the test container reported x86_64. The first check in it pulled the amd64 image and took 43.4 seconds. The Dang SDK also adds its own check, dang-sdk/generate/stale, which dagger check now runs.

With baseCtr set, the module no longer takes the image tag from your lockfile. Change the tag in main.dang in the commit that upgrades @playwright/test. If every machine on the team has the same architecture, skip this section.

Run the same checks with Cloud Checks

In CI, run the same check with Dagger Cloud Checks. Cloud Checks run your checks on Dagger’s Cloud Engines after each push and pull request, with no CI workflow file. Turn them on from the project root:

dagger cloud checks on

The command connects your GitHub repository to Dagger Cloud. The Git origin remote of the project must point to that repository. In an interactive terminal, the command offers to sign you in and connect GitHub. Set up Cloud Checks covers each step.

Cloud Engines are managed Dagger Engines with scale-out and a distributed cache. Dagger Cloud bills the CPU, memory, and data that your pipelines use, as the pricing page shows. dagger.toml and the module define the image, the install, and the test command once. Your laptop and Cloud Checks run that same definition, so there is no second container job to keep in step.

Bottom line: Run Playwright in the mcr.microsoft.com/playwright image whose tag matches your @playwright/test version, and do not cache the browsers. Update screenshot baselines in that image, on one CPU architecture. Dagger’s Playwright module meets these requirements in one place and picks the image from your lockfile. It runs the same check on your machine and with Cloud Checks.