Skip to content

test(e2e): run the Node suite against an astro build --devOutput build - #6

Draft
ostenbom wants to merge 3 commits into
mainfrom
e2e/dev-output-build
Draft

ostenbom wants to merge 3 commits into
mainfrom
e2e/dev-output-build

Conversation

@ostenbom

@ostenbom ostenbom commented Oct 27, 2026 •

Copy link
Copy Markdown

What does this PR do?

Runs the Node Playwright suite against a built site instead of astro dev, using Astro's astro build --devOutput. It is one of four PRs comparing ways to speed up the E2E sweep (see Related PRs). This one keeps EmDash's behaviour under test as close to astro dev as possible and changes one line of core.

EMDASH_E2E_BUILD=production switches e2e/global-setup.ts from astro dev to astro build --devOutput plus node dist/server/entry.mjs. A dev-output build keeps import.meta.env.DEV true, so the dev-bypass login, dev-reset and captured-email endpoints the suite depends on work with no test switch. Runs without the variable still use astro dev. CI sets it for the Node shards only; the Cloudflare step and the 8-shard layout are unchanged.

Why each part is needed

Change Why
global-setup.ts: build with --devOutput, serve dist/server/entry.mjs, skip the page and admin warm-ups The warm-ups exist only because astro dev compiles on first request and reloads after optimizing dependencies. A build has neither. The SQLite path is baked into the build, so the data folder is fixed ($TMPDIR/emdash-pw-production) and cleared each run.
integration/index.ts: build-time registry check accepts localhost when NODE_ENV=development The suite's mock marketplace is http://127.0.0.1:4445. The build-time check allowed localhost only for astro dev/astro sync, so astro build --devOutput failed with "must use HTTPS outside development", even though the built site's runtime check (import.meta.env.DEV) accepts it. Astro sets NODE_ENV=development for --devOutput before loading the config. Production builds still reject it (covered by a new unit test).
EMDASH_SITE_URL set for the build and server image.remotePatterns for EmDash media is derived from the site URL at build time.
e2e/fixture: i18n.routing: "manual" plus a pass-through src/middleware.ts Built sites add Astro locale-fallback routes that take priority over the fixture's [locale] pages, so /fr/posts/<slug> showed the English post.
e2e/fixture: sharp dependency The built server's image endpoint fails with MissingSharp; astro dev found Astro's optional copy.
outer-middleware.ts installs the registry fixture at startup astro.config.mjs installs it, and a built server never runs the config.
editor-image-panel.spec.ts uses an https:// image URL The admin CSP allows only https: images. Playwright intercepts the URL either way.
.gitignore: e2e/fixture*/.emdash/migrations.json Builds write a migrations journal into the fixture.

Three tests that fail only when the whole suite runs against one server are fixed in #10. The CI shards don't need that fix; a full local run does.

How much it helps

Bottom line: after syncing with upstream, a real production build cuts the Node E2E time by about 60% with the 8-shard layout unchanged (#7: slowest shard 7.1 → 2.8 min). A dev-output build saves about a quarter (#6: 7.1 → 5.5 min) and fails on Endform at concurrency 64 on every attempt (#8). The production build on Endform (#9) runs the whole suite in one job in under 6 minutes, install and build included.

Local, native Playwright, 1 worker, full Node suite (378 tests), same Mac, run back to back:

Server under test PR Wall time vs astro dev Result
astro dev main 12.9 min baseline 344 passed, 34 skipped
astro build --devOutput #6 10.3 min −20% 344 passed, 34 skipped
astro build + dangerouslyEnableTestHooks #7 6.0 min −53% 343 passed, 1 failed*
astro build + runtime EMDASH_INSECURE_TEST_BYPASS (prototype) #4 6.1 min −53% 343 passed, 1 failed*

* The same flaky test both times: media-library.spec.ts "moves one local media item into a visible folder by dragging", which fails about 1 run in 3 when the root folder is crowded. It isn't specific to production builds.

CI, GitHub Actions ubuntu-latest, after syncing with upstream main (c1efce5cc):

Method PR Layout Node E2E wall time Node E2E runner time Result
astro dev upstream main 8 shards 7.1 min (slowest shard, mean of the 2 green runs since emdash-cms#4317 landed) 35.4 shard-min green
astro build --devOutput #6 8 shards 5.5 min (−23%) 24.1 shard-min (−32%) all 8 shards green
astro build + dangerouslyEnableTestHooks #7 8 shards 2.8 min (−61%) 14.6 shard-min (−59%) all 8 shards green
astro build --devOutput on Endform #8 1 job, concurrency 64 6m38s for the whole job (parallel run 3.1 min) 145 Endform runner-min, plus 1 GitHub job red on every attempt (12, 9 and 16 failed), mostly 5 s assertion timeouts in different tests each time**
astro build + dangerouslyEnableTestHooks on Endform #9 1 job, concurrency 64 5m44s for the whole job (parallel run 2.2 min, exclusive 1.4 min) 91 Endform runner-min, plus 1 GitHub job 344 passed, 2 passed on retry***
Prototype: astro build + runtime switch on Endform #4 1 job, concurrency 64 4 min 50 s for the whole job (about 1 min 45 s of it install and build) 1 job plus Endform runner time 378 passed, 0 retries

For the shard rows, wall time is the slowest shard's "Run Node E2E" step and runner time is that step summed over all 8 shards. Install, build and the Cloudflare step are left out because they don't change between methods. emdash-cms#4317 now builds the fixtures before that step, which is why every shard number is lower than before the sync. Before the sync, against the older baseline (8.0 min, 42.6 shard-min), the shard results were the same picture: #6 −26% and #7 −51%, and on Endform #9 took 4m55s with no retries. For the Endform rows, wall time is the whole job, install and build included.

*** visual-editing.spec.ts "flushes unsaved edits when navigating away", the known Endform keepalive issue, and media-library.spec.ts "matches the compact responsive folder layout".

** The same dev-output build passes natively (344 of 344), and its client bundles are within 3% of the production build's (4.9 MB vs 4.8 MB of gzipped JS). The extra time is most likely server-side rendering in development mode, which the one shared server can't keep up with at 64 tests at once. That is the same gap as in the native timings (10.3 vs 6.0 min). Not profiled further.

Runs after the sync: #6 CI, #7 CI, #8 Endform E2E, #9 Endform E2E. Before the sync: #6, #7, #8, #9. One or two runs each, so treat small differences as noise.

Other checks: Smoke Isolated Install passes after the sync, since upstream now installs the newest Node 22 for it. Browser Tests (the admin Vitest browser suite, which none of these PRs touch) still fails intermittently on these runners. It fails with "Failed to fetch dynamically imported module" / "Cannot connect to the iframe" on a different file each attempt, and once locally with a timing assertion (PortableTextEditor.test.tsx, toolbar fade) that passes 3 of 3 times on its own. After the sync it passed first time on #6 and #10, and on the third attempt on #7.

The dev-output build keeps less than half of the true production build's gain. A guess, not yet profiled: development React and unminified bundles, plus dev-only code paths in Astro and EmDash, stay on in a dev-output build.

Related PRs

PR Server under test Where it runs Core change
#6 (this PR) astro build --devOutput 8 GitHub shards (layout unchanged) One line: the build-time registry check accepts localhost in a dev-output build
#7 astro build 8 GitHub shards (layout unchanged) New build-time dangerouslyEnableTestHooks option
#8 (stacked on #6) astro build --devOutput Endform, one job None beyond #6
#9 (stacked on #7) astro build Endform, one job None beyond #7
#10 any Wherever the whole suite shares one server: a full local run, or Endform None; three test fixes, included in #8 and #9
#4 astro build with a runtime env switch Endform Prototype that the others replace

Type of change

  • Bug fix
  • Feature (requires an accepted feature plan or RFC)
  • Refactor (no behavior change)
  • Translation
  • Documentation
  • Performance improvement
  • Tests
  • Chore (dependencies, CI, tooling)

Your connection to this change

  • I use EmDash and ran into the problem this PR addresses
  • I maintain a tool, dependency, or service that EmDash uses, and this change concerns that integration
  • A Maintainer approved this specific change before I opened the PR

Checklist

  • I have read CONTRIBUTING.md
  • pnpm typecheck passes
  • pnpm lint passes
  • pnpm test passes (or targeted tests for my change): full Node E2E suite against the dev-output build (344 passed, 34 skipped), plus the new registry-localhost-build unit test, which fails without the fix
  • pnpm format has been run
  • I have added/updated tests for my changes (if applicable)
  • User-visible strings in the admin UI are wrapped for translation (if applicable): not applicable, no admin UI change
  • I have added and reviewed the user-facing changeset (if this PR changes a published package)
  • New features link to their merged design PR: not applicable
  • I have included screenshots below if this PR changes the UI: not applicable

AI assistance disclosure

  • This PR includes AI-generated code or text — model/tool: Claude Opus 5.5 (Claude Code)

Screenshots / test output

Not applicable. Timings are in the table above.

🤖 Generated with Claude Code

ostenbom and others added 3 commits October 27, 2026 13:45
EMDASH_E2E_BUILD=production now builds the Node fixture with
`astro build --devOutput` and serves dist/server/entry.mjs instead of
running `astro dev`. The build keeps `import.meta.env.DEV` true, so the
dev-bypass, dev-reset and captured-email endpoints the suite depends on
work without any test switch in EmDash.

Also fixes what a built fixture exposes: manual i18n routing so the
fixture's [locale] pages win over Astro's fallback routes, sharp for
the image endpoint, the registry fixture installed at server start, an
https image URL for the CSP, and an ignore rule for the migrations
journal that builds write into the fixture.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The build-time registry check only allowed localhost for `astro dev`
and `astro sync`, while the runtime check follows import.meta.env.DEV.
A --devOutput build therefore failed at build time although the built
site would accept the URL. Astro sets NODE_ENV=development for
--devOutput before loading the config, so the build check uses that.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant