Skip to content

feat(core): add dangerouslyEnableTestHooks and run Node E2E against a production build - #7

Draft
ostenbom wants to merge 3 commits into
mainfrom
e2e/production-build-flag
Draft

ostenbom wants to merge 3 commits into
mainfrom
e2e/production-build-flag

Conversation

@ostenbom

@ostenbom ostenbom commented Oct 27, 2026 •

Copy link
Copy Markdown

What does this PR do?

Runs the Node Playwright suite against a real production build (astro build) instead of astro dev. It adds a build-time dangerouslyEnableTestHooks option to the emdash() integration, so the suite can still sign in and reset the site. It is one of four PRs comparing ways to speed up the E2E sweep (see Related PRs). This one tests production behaviour and gets the largest speedup.

EMDASH_E2E_BUILD=production switches e2e/global-setup.ts from astro dev to astro build plus node dist/server/entry.mjs. Runs without the variable still use astro dev. CI sets it for the Node shards only; the Cloudflare step and the 8-shard layout are unchanged.

Why each part is needed

Change Why
dangerouslyEnableTestHooks integration option, read through a new isDevOnlyEnabled() gate The suite signs in through /_emdash/api/setup/dev-bypass and /_emdash/api/auth/dev-bypass, resets the site through dev-reset, reads invites through /_emdash/api/dev/emails, and installs from a localhost mock marketplace. All of these were gated on import.meta.env.DEV, which is false in a production build. The gate now also reads a build-time constant (__EMDASH_TEST_HOOKS__, a Vite define) that the option sets.
Replaces import.meta.env.DEV at those call sites (dev-bypass x2, dev-reset, dev emails, console email provider, registry localhost checks in registry/config.ts, registry/artifact-fetch.ts, api/handlers/registry.ts and the registry artifact route) These are the only dev-only paths the suite needs. Other import.meta.env.DEV checks are untouched.
Build-time registry check accepts localhost when the option is set, and astro build warns while it is on The integration validates registry in Node at build time, before Vite runs, so it reads the option directly.
e2e/fixture/astro.config.mjs sets dangerouslyEnableTestHooks: true The only place it is turned on.
global-setup.ts: build, serve dist/server/entry.mjs, skip the warm-ups The warm-ups exist only because astro dev compiles on first request and reloads after optimizing dependencies. The SQLite path is baked into the build, so the data folder is fixed ($TMPDIR/emdash-pw-production) and cleared each run.
EMDASH_SITE_URL set for the build and server Production setup won't infer the site URL from the request, and image.remotePatterns is derived from it at build time.
e2e/fixture: i18n.routing: "manual" plus a pass-through src/middleware.ts Production builds add Astro locale-fallback routes that take priority over the fixture's [locale] pages, so /fr/posts/<slug> showed the English post.
e2e/fixture: sharp dependency The built server's image endpoint fails with MissingSharp; astro dev found Astro's optional copy.
outer-middleware.ts installs the registry fixture at startup astro.config.mjs installs it, and a built server never runs the config.
editor-image-panel.spec.ts uses an https:// image URL The admin CSP allows only https: images. Playwright intercepts the URL either way.
.gitignore: e2e/fixture*/.emdash/migrations.json Builds write a migrations journal into the fixture.

Three tests that fail only when the whole suite runs against one server are fixed in #10. The CI shards don't need that fix; a full local run does.

Safety

The option is fixed at build time. In a build without it, the fixture's compiled isDevOnlyEnabled() is literally return false;. Its handlers are still in the bundle but can't be reached, and no environment variable or request can turn them on. That is stricter than the earlier prototype on #4, which used a runtime EMDASH_INSECURE_TEST_BYPASS environment variable that any production deployment could have set. The dev routes are still injected in every build, as on main, and return 403. Skipping their injection unless the option is set would be a separate change.

How much it helps

Bottom line: after syncing with upstream, a real production build cuts the Node E2E time by about 60% with the 8-shard layout unchanged (#7: slowest shard 7.1 → 2.8 min). A dev-output build saves about a quarter (#6: 7.1 → 5.5 min) and fails on Endform at concurrency 64 on every attempt (#8). The production build on Endform (#9) runs the whole suite in one job in under 6 minutes, install and build included.

Local, native Playwright, 1 worker, full Node suite (378 tests), same Mac, run back to back:

Server under test PR Wall time vs astro dev Result
astro dev main 12.9 min baseline 344 passed, 34 skipped
astro build --devOutput #6 10.3 min −20% 344 passed, 34 skipped
astro build + dangerouslyEnableTestHooks #7 6.0 min −53% 343 passed, 1 failed*
astro build + runtime EMDASH_INSECURE_TEST_BYPASS (prototype) #4 6.1 min −53% 343 passed, 1 failed*

* The same flaky test both times: media-library.spec.ts "moves one local media item into a visible folder by dragging", which fails about 1 run in 3 when the root folder is crowded. It isn't specific to production builds.

CI, GitHub Actions ubuntu-latest, after syncing with upstream main (c1efce5cc):

Method PR Layout Node E2E wall time Node E2E runner time Result
astro dev upstream main 8 shards 7.1 min (slowest shard, mean of the 2 green runs since emdash-cms#4317 landed) 35.4 shard-min green
astro build --devOutput #6 8 shards 5.5 min (−23%) 24.1 shard-min (−32%) all 8 shards green
astro build + dangerouslyEnableTestHooks #7 8 shards 2.8 min (−61%) 14.6 shard-min (−59%) all 8 shards green
astro build --devOutput on Endform #8 1 job, concurrency 64 6m38s for the whole job (parallel run 3.1 min) 145 Endform runner-min, plus 1 GitHub job red on every attempt (12, 9 and 16 failed), mostly 5 s assertion timeouts in different tests each time**
astro build + dangerouslyEnableTestHooks on Endform #9 1 job, concurrency 64 5m44s for the whole job (parallel run 2.2 min, exclusive 1.4 min) 91 Endform runner-min, plus 1 GitHub job 344 passed, 2 passed on retry***
Prototype: astro build + runtime switch on Endform #4 1 job, concurrency 64 4 min 50 s for the whole job (about 1 min 45 s of it install and build) 1 job plus Endform runner time 378 passed, 0 retries

For the shard rows, wall time is the slowest shard's "Run Node E2E" step and runner time is that step summed over all 8 shards. Install, build and the Cloudflare step are left out because they don't change between methods. emdash-cms#4317 now builds the fixtures before that step, which is why every shard number is lower than before the sync. Before the sync, against the older baseline (8.0 min, 42.6 shard-min), the shard results were the same picture: #6 −26% and #7 −51%, and on Endform #9 took 4m55s with no retries. For the Endform rows, wall time is the whole job, install and build included.

*** visual-editing.spec.ts "flushes unsaved edits when navigating away", the known Endform keepalive issue, and media-library.spec.ts "matches the compact responsive folder layout".

** The same dev-output build passes natively (344 of 344), and its client bundles are within 3% of the production build's (4.9 MB vs 4.8 MB of gzipped JS). The extra time is most likely server-side rendering in development mode, which the one shared server can't keep up with at 64 tests at once. That is the same gap as in the native timings (10.3 vs 6.0 min). Not profiled further.

Runs after the sync: #6 CI, #7 CI, #8 Endform E2E, #9 Endform E2E. Before the sync: #6, #7, #8, #9. One or two runs each, so treat small differences as noise.

Other checks: Smoke Isolated Install passes after the sync, since upstream now installs the newest Node 22 for it. Browser Tests (the admin Vitest browser suite, which none of these PRs touch) still fails intermittently on these runners. It fails with "Failed to fetch dynamically imported module" / "Cannot connect to the iframe" on a different file each attempt, and once locally with a timing assertion (PortableTextEditor.test.tsx, toolbar fade) that passes 3 of 3 times on its own. After the sync it passed first time on #6 and #10, and on the third attempt on #7.

Related PRs

PR Server under test Where it runs Core change
#6 astro build --devOutput 8 GitHub shards (layout unchanged) One line: the build-time registry check accepts localhost in a dev-output build
#7 (this PR) astro build 8 GitHub shards (layout unchanged) New build-time dangerouslyEnableTestHooks option
#8 (stacked on #6) astro build --devOutput Endform, one job None beyond #6
#9 (stacked on #7) astro build Endform, one job None beyond #7
#10 any Wherever the whole suite shares one server: a full local run, or Endform None; three test fixes, included in #8 and #9
#4 astro build with a runtime env switch Endform Prototype that the others replace

Type of change

  • Bug fix
  • Feature (requires an accepted feature plan or RFC)
  • Refactor (no behavior change)
  • Translation
  • Documentation
  • Performance improvement
  • Tests
  • Chore (dependencies, CI, tooling)

Your connection to this change

  • I use EmDash and ran into the problem this PR addresses
  • I maintain a tool, dependency, or service that EmDash uses, and this change concerns that integration
  • A Maintainer approved this specific change before I opened the PR

Checklist

  • I have read CONTRIBUTING.md
  • pnpm typecheck passes
  • pnpm lint passes
  • pnpm test passes (or targeted tests for my change): core unit tests (6100 passed), full Node E2E suite against the production build (see table); core integration tests not run locally
  • pnpm format has been run
  • I have added/updated tests for my changes (if applicable)
  • User-visible strings in the admin UI are wrapped for translation (if applicable): not applicable, no admin UI change
  • I have added and reviewed the user-facing changeset (if this PR changes a published package)
  • New features link to their merged design PR: none yet; this is a prototype for comparison, and the option would need a design PR before going upstream
  • I have included screenshots below if this PR changes the UI: not applicable

Not done: the option isn't in the configuration reference docs yet.

AI assistance disclosure

  • This PR includes AI-generated code or text — model/tool: Claude Opus 5.5 (Claude Code)

Screenshots / test output

Not applicable. Timings are in the table above.

🤖 Generated with Claude Code

@github-actions

Copy link
Copy Markdown

PR template validation failed

Please fix the following issues by editing your PR description:

  • Feature PRs require a link to their merged design PR. Follow the feature-plan or RFC process in proposals/README.md first.

See CONTRIBUTING.md for the full contribution policy.

ostenbom and others added 3 commits October 27, 2026 13:45
The dev-bypass, dev-reset and captured-email endpoints, the console
email provider and localhost registry URLs are gated on
import.meta.env.DEV, so an end-to-end suite that signs in through them
can only run against `astro dev`.

The new integration option sets a build-time constant that the shared
isDevOnlyEnabled() gate reads alongside import.meta.env.DEV. A build
without the option gets `false`, so nothing at runtime can switch the
endpoints on, and the build-time registry check accepts localhost only
when the option is set. `astro build` warns while it is on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
EMDASH_E2E_BUILD=production now runs `astro build` on the Node fixture
and serves dist/server/entry.mjs instead of running `astro dev`. The
fixture sets dangerouslyEnableTestHooks so the suite can still sign in
and reset the site, and EMDASH_SITE_URL because production setup and
image.remotePatterns need it.

Also fixes what a built fixture exposes: manual i18n routing so the
fixture's [locale] pages win over Astro's fallback routes, sharp for
the image endpoint, the registry fixture installed at server start, an
https image URL for the CSP, and an ignore rule for the migrations
journal that builds write into the fixture.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant