visual-testing.md

Visual regression testing (VRT)

Pixel-diff every component variant against a committed baseline so a visual change (often a tokens.css / global.css cascade, or CMS content the site has never rendered) is surfaced on the PR. Decision and rationale: ADR-0007.

$0, EU-resident, no SaaS: Playwright screenshots a dev-only gallery built from the real Astro pipeline; baselines are PNGs in this repo. Nothing ships to visitors (the gallery is absent from prod builds). Ported from the Noa-IQ marketing-website VRT.

The model in one paragraph

Each component declares its variants in a co-located *.vrt.ts. A VRT=1 build emits a page per variant under /vrt/ (into dist-vrt/, never dist/). Playwright renders each at three widths and diffs against apps/web/tests/visual/__screenshots__/. On a PR this runs as an advisory job (visual.yml): a diff never blocks the merge; instead it posts a sticky comment linking a click-to-view report and, when enabled, auto-pushes the refreshed baselines onto the PR branch. The reviewer sees the change and approves (or fixes) it. The job runs inside the pinned Playwright image, so rendering is identical to how the baselines were generated.

Daily flow

1. You open a PR. If it touches UI files, the advisory Visual review job renders every variant and diffs.
2. No visual change -> the job is silent (no comment), status is green. Orphan-only baseline cleanup still posts (there is something to adopt).
3. A visual change -> a sticky PR comment appears with an added · changed · same · failed tally (plus removed when orphans were pruned). Review lists put changed first, then failed, then compact added. The hosted report (published whenever there is something to review) uses the same buckets as filter chips; Playwright's own Passed/Failed chips are hidden because they treat missing baselines as Failed. The report list is grouped by component (from each *.vrt.ts), not one giant gallery.spec.ts shell. The check stays advisory (never red).
   • With AUTO mode on (a VRT_PAT secret), the job also commits the refreshed baselines to your PR branch (chore(vrt): refresh visual baselines), so the before/after also shows as GitHub's native image diff in "Files changed", and merging adopts them. No manual step. The commit is authored as you (squash merges stay yours); github-actions[bot] is only the committer so the job can skip its own push.
   • Without AUTO, the comment shows the diff and you accept it with the manual step below.
4. You review the diff. Intended -> merge. A regression -> fix the code and push again; the comment updates in place and flips to the all-clear when resolved.
5. On merge to main, visual-propagate.yml updates every open PR's branch with main, so their baselines re-render against the latest (no cross-PR clobbering) with no manual "update branch" click. You do not need branch-protection "require up to date": baselines are committed PNGs that git 3-way-merges (same-component changes hit a binary conflict that forces a re-render), and Checks re-runs on push: main, so correctness holds without the per-PR update chore.

Commands (local)

Run from the repo root.

pnpm --filter web run vrt          # build the gallery + diff vs baselines (needs the pinned env; see below)
pnpm --filter web run vrt:docker   # the same, but inside the pinned Playwright image (accurate anywhere)
pnpm --filter web run vrt:report   # open the last HTML report (expected/actual/diff slider)
pnpm --filter web run vrt:docker:update   # regenerate baselines in the pinned image (accept locally)

Local rendering (fonts/anti-aliasing) differs from CI, so a bare pnpm --filter web run vrt will diff on text unless you are on the exact pinned environment. Use vrt:docker for an accurate local compare, or just rely on the PR's advisory report.

Accepting an intended change (regenerating baselines)

Baselines must be generated in the same environment they are compared in (the pinned mcr.microsoft.com/playwright:v<version>-noble image, which the CI job also runs inside). Three ways, in order of preference:

1. AUTO mode (automatic): if VRT_PAT is set, the advisory job commits the refreshed baselines to your PR branch for you. Nothing to do but review and merge.
2. The Update visual baselines workflow (manual, in CI): from the Actions tab, dispatch it against the PR's branch. It regenerates in the pinned image on the runner and pushes the PNGs back. Use it when AUTO is off or to force a full refresh.
3. pnpm --filter web run vrt:docker:update (local): regenerate in the pinned image on your machine and commit the PNGs. On macOS, the container writes Linux dependency binaries into the repo's node_modules; run pnpm install once afterward to restore your host binaries. On Linux nothing is needed.

Never commit baselines generated by a bare vrt:update on your own OS; they will differ from CI's rendering.

Adding or removing coverage

• Add a variant: write it in the co-located *.vrt.ts, open a PR. CI treats the missing PNGs as added; AUTO commits them. Locally: pnpm --filter web run vrt:docker:update.
• Remove a variant or *.vrt.ts: delete the declaration. CI lists the leftover PNGs as orphans; AUTO (and vrt:update, which prunes with --trust-gallery after a fresh build) deletes them. A standalone node scripts/check/vrt-report.mjs --prune-only stays cautious and refuses if dist-vrt looks incomplete, so a stale gallery cannot wipe the corpus. Playwright itself never removes baselines.

Adding coverage: write a *.vrt.ts

Co-locate a <Component>.vrt.ts next to the component. It default-exports the component plus named variants; each variant is the component's props. The gallery collects every *.vrt.ts automatically (src/vrt/collect.ts).

// src/components/home/StatsBand.vrt.ts
import StatsBand from "./StatsBand.astro";

export default {
  component: StatsBand,
  variants: {
    default: {
      heading: "Q-Summit in numbers",
      intro: "The event at a glance.",
      stats: [
        { value: "3000", label: "Attendees" },
        { value: "120", label: "Speakers" },
      ],
    },
    "long-labels": {
      heading: "Impact",
      intro: "Long labels can overflow a fixed grid cell.",
      stats: [
        {
          value: "3000",
          label: "Ambitious students, founders and investors in one hall",
        },
      ],
    },
  },
};

Conventions

• Cover the content edges, not just the happy path. The point is to guard layout against what a CMS produces: a default, then the stressful cases the component can hit (long-* copy that must wrap, empty lists, many-items, and German if the string differs by locale). Two to five variants is typical.
• Realistic English copy, never lorem ipsum. Lorem trips the spell checker and does not wrap like real content. Write plausible Q-Summit copy.
• Slotted components: a string children key fills the default slot, e.g. a Button label: { href: "#", children: "Register now" }.
• Dynamic or non-deterministic regions (a live counter, a date, a rotating image) get data-vrt-mask on the element in the component; the spec masks them automatically. A build-time date (the footer year) is masked the same way.
• Interactive/JS-toggled states are not captured. The gallery screenshots the static SSR render, so a disclosure such as the mobile nav open menu is shot closed. A client enhancement that settles to a non-deterministic resting state (the speaker carousel) is skipped in VRT via the data-vrt flag on the gallery <html>, leaving the deterministic SSR layout. One consequence: because the carousel's prev/next arrows and dots only appear once its JS runs, they are absent from the speaker-grid baselines (that control chrome is not visually covered; the card layout still is).
• Images need no special handling. Every <img> (plain or astro:assets <picture>) is swapped for a deterministic grey image-placeholder glyph of the same intrinsic size before the screenshot, so pass whatever image props are realistic. This keeps real photos (gitignored, absent on a fresh checkout) out of committed baselines and makes diffs flat; the element and its CSS (border-radius, object-fit, box) are untouched, so image-layout regressions are still caught. A background <video> (the hero) is paused and hidden so the section backdrop and content over it stay visible.
• Props must type-check. pnpm --filter web check type-checks every *.vrt.ts (it imports the real component), so a wrong prop shape fails the build, not just the screenshot.

The variant id and baseline filename derive from the path and variant name: components/home/StatsBand.vrt.ts + long-labels -> home-stats-band--long-labels -> home-stats-band--long-labels-1280.png.

What runs in CI

The visual diff is advisory, so it is not part of the blocking pnpm run check; it runs in its own PR workflow.

• visual.yml (on pull_request): a changes job skips it on docs-only PRs and on the bot's own baseline commit (so AUTO does not loop); the visual job runs inside the pinned image, diffs, publishes the report (PUBLISH), upserts the sticky comment, uploads the report artifact, and auto-pushes baselines (AUTO). continue-on-error keeps it non-blocking.
  • The PR is rendered on top of current main and always diffed against main's baselines, never the PNGs committed on the branch. The job checks out the PR head, git merges main in, then wipes and restores the baselines from main (so PR-only PNGs cannot compare against themselves) and renders. It does not rely on an in-tree merge driver (a PR could edit .gitattributes to neutralize it); instead, after the merge it inspects the actual conflicted paths: a conflict only in the baseline images is accepted (they are reset to main anyway), any conflict outside is a real code conflict. This matters for two reasons: (1) a contributor who updates the look and commits the matching baseline cannot hide the change from review (diffing against their own PNG would render==baseline, showing "no changes"); and (2) a component only main changed does not false-diff, because the render sits on top of main. A real code conflict (outside the baselines) fails the merge, and the job then skips with a "update your branch" comment rather than diff stale code. On the first-ever seed (no baselines on main yet) it treats all snapshots as new. In AUTO the correct baselines are re-derived (container render) and committed to the branch as a single baseline-only commit anchored to the PR head; it commits only the baselines it refreshed or deleted, so a baseline the PR deleted is not resurrected, and a removed *.vrt.ts variant's orphan PNGs are pruned. Orphan prune also runs at the end of vrt:update / --prune-only, because Playwright's --update-snapshots never deletes. They land on main at merge. Dependencies are installed after the in-job merge, so a branch behind main renders with the merged tree's dependencies, not its own stale ones. The only way a baseline becomes truth is a container render (AUTO, the Update visual baselines workflow, or vrt:docker:update).
  • If the review cannot run (the Playwright JSON report is missing or unparseable, e.g. the gallery build or browser failed), the job does not fall silent-green: vrt-report.mjs marks the run unavailable and the sticky comment says "report unavailable" with a link to the job log, so an infrastructure failure is never mistaken for a clean pass.
• visual-propagate.yml (on push to main): updates open PR branches with main so their baselines stay current.
• visual-cleanup.yml (on PR close): deletes that PR's hosted report deployment.
• update-visual-baselines.yml (workflow_dispatch): the manual regeneration fallback.

Modes light up as secrets are present and degrade gracefully:

Mode	Needs	Effect
COMPARE	nothing	sticky comment listing changed shots + a downloadable report artifact
PUBLISH	CLOUDFLARE_API_TOKEN, CLOUDFLARE_ACCOUNT_ID	also hosts the click-to-view report on Cloudflare Pages, linked from the comment
AUTO	VRT_PAT (contents:write, fine-grained)	also commits refreshed baselines to the PR branch

One-time setup (for full seamless mode)

1. Cloudflare Pages project named q-web-vrt-reports (Direct Upload). Add repo secrets CLOUDFLARE_API_TOKEN (Pages:Edit) and CLOUDFLARE_ACCOUNT_ID. The hosted report is public, which is fine here: every screenshot renders template fixtures with images neutralized to placeholder glyphs, so a report never contains production content. Add a Cloudflare Access policy only if you want to restrict viewing anyway (optional).
2. VRT_PAT repo secret: a fine-grained token with contents: write on this repo (the default GITHUB_TOKEN cannot re-trigger CI after the bot push). Enables AUTO baseline commits and lets visual-propagate re-run updated PRs' CI.
3. Branch protection on main: require the Checks status. "Require branches up to date" is optional and off by default (see go-live.md); visual-propagate plus git's merge of committed PNGs keeps baselines correct without it.

Without any of these it still works in COMPARE mode. The Playwright image ships its own Node (currently 24.x, above the repo's >=22.19). Bumping Playwright means bumping, in lockstep: @playwright/test, the v<x>-noble tag in the vrt:docker* scripts, the container: tags in visual.yml and update-visual-baselines.yml, then regenerating baselines.

Files

apps/web/
  src/
    components/**/<Name>.vrt.ts     variant declarations (co-located, one per component)
    vrt/collect.ts                 globs *.vrt.ts -> flat entry list
    layouts/VrtGallery.astro       minimal layout: global.css + Montserrat, no nav/footer
    pages/vrt/[slug].astro         one page per variant (getStaticPaths gated on VRT=1)
  tests/visual/
    gallery.spec.ts                screenshots each entry x width (enumerated from dist-vrt/vrt/)
    __screenshots__/               committed baselines (the tracked source of truth)
    tsconfig.json                  isolated tsconfig for the specs
  playwright.config.ts             serves dist-vrt via astro preview; thresholds
scripts/check/
  vrt-report.mjs                   Playwright JSON -> sticky comment (+ AUTO baseline staging)
  vrt-cleanup.mjs                  sweep closed PRs' Cloudflare Pages reports
.github/workflows/
  visual.yml                       advisory PR job: merge main, diff vs main, report, comment, AUTO push
  visual-propagate.yml             on merge, refresh open PR branches with main
  visual-cleanup.yml               on PR close, delete the hosted report
  update-visual-baselines.yml      workflow_dispatch: manual baseline regeneration