Skip to content

test(e2e): Node suite on Endform against the dev-output build - #8

Draft
ostenbom wants to merge 3 commits into
e2e/dev-output-buildfrom
e2e/endform-dev-output
Draft

ostenbom wants to merge 3 commits into
e2e/dev-output-buildfrom
e2e/endform-dev-output

Conversation

@ostenbom

@ostenbom ostenbom commented Oct 27, 2026 •

Copy link
Copy Markdown

What does this PR do?

Runs the Node Playwright suite on Endform against the dev-output build (astro build --devOutput) from #6, which this PR is stacked on. It is one of four PRs comparing ways to speed up the E2E sweep (see Related PRs). Instead of 8 GitHub shards, one CI job builds and serves the site, and Endform runs every test on its own remote runner (concurrency 64, organization jOXdcdjD), proxying the runners' localhost traffic back to that job.

The Endform layer is the same in this PR and its counterpart, so the two differ only in the build under test. It is squashed from the prototype on #4, which has the commit-by-commit history and the triage notes.

Why each part is needed

Change Why
playwright.config.ts switches on ENDFORM=true Endform sets it. Under Endform tests run fullyParallel, globalSetup is replaced by a webServer, and specs that reset the whole site (setup-wizard, passkey-full-setup-virtual-auth) move to a separate exclusive project. Native runs are unchanged.
e2e/endform-server.ts Runs the normal global setup with EMDASH_E2E_BUILD=production on the CLI machine and serves the setup state (token, seeded IDs) on 127.0.0.1:4446. On a runner the temp file that global setup writes doesn't exist.
Auto fixture in e2e/fixtures/index.ts, refresh-server-pat.ts Each runner fetches the setup state before its test and writes it to its own temp file, so specs read it as before. The setup-wizard tests post a new token back after resetting the site.
endform.config.ts proxyNetworkHosts: ["<loopback>"] carries the runners' requests for localhost:4444/127.0.0.1:4446 back to the CI job. Concurrency limits: 64 overall, 1 for the exclusive project, and 1 each for tests tagged @comments and @search. additionalFiles: ["e2e/fixtures/assets/*"] sends the test image; dir/** sends nothing on Endform 0.81.5.
e2e/fixtures/compression-proxy.ts (built server moves to 127.0.0.1:4447) Every runner starts with an empty cache and downloads the admin bundle through the tunnel, and Astro's Node server doesn't compress it (lib.js 5.7 MB, 1.26 MB gzipped). The proxy gzips each /_astro/ asset once and serves it from memory; compressing per request saturated the CI machine at concurrency 64.
About 25 tests across 20 specs (content-*, redirects, media-library, allowed-domains, i18n, visual-editing, plugin-editor-extensions …) and the createPost fixture They assumed they ran alone against one database: fixed names and slugs, edits to shared seeded posts, "delete everything of this kind", empty-state assertions. Each now owns its data; empty states are stubbed with page.route.
.github/workflows/endform-e2e.yml One ubuntu-latest job: install, build the fixture's dependencies, run the chromium project and then the exclusive project on Endform. Authenticates with OIDC through the Endform GitHub App, so no secret is stored.
Includes the commit from #10 Endform runs every test against one shared server, so it hits the three failures that #10 fixes.
Root package.json: endform 0.81.5, tsx 4.21.0 The CLI, and the runner for endform-server.ts.

Known Endform issues, both handed off to Endform: a keepalive fetch sent during pagehide is sometimes lost through the proxy (visual-editing "flushes unsaved edits when navigating away", absorbed by CI's one retry), and the additionalFiles glob above.

How much it helps

Bottom line: after syncing with upstream, a real production build cuts the Node E2E time by about 60% with the 8-shard layout unchanged (#7: slowest shard 7.1 → 2.8 min). A dev-output build saves about a quarter (#6: 7.1 → 5.5 min) and fails on Endform at concurrency 64 on every attempt (#8). The production build on Endform (#9) runs the whole suite in one job in under 6 minutes, install and build included.

Local, native Playwright, 1 worker, full Node suite (378 tests), same Mac, run back to back:

Server under test PR Wall time vs astro dev Result
astro dev main 12.9 min baseline 344 passed, 34 skipped
astro build --devOutput #6 10.3 min −20% 344 passed, 34 skipped
astro build + dangerouslyEnableTestHooks #7 6.0 min −53% 343 passed, 1 failed*
astro build + runtime EMDASH_INSECURE_TEST_BYPASS (prototype) #4 6.1 min −53% 343 passed, 1 failed*

* The same flaky test both times: media-library.spec.ts "moves one local media item into a visible folder by dragging", which fails about 1 run in 3 when the root folder is crowded. It isn't specific to production builds.

CI, GitHub Actions ubuntu-latest, after syncing with upstream main (c1efce5cc):

Method PR Layout Node E2E wall time Node E2E runner time Result
astro dev upstream main 8 shards 7.1 min (slowest shard, mean of the 2 green runs since emdash-cms#4317 landed) 35.4 shard-min green
astro build --devOutput #6 8 shards 5.5 min (−23%) 24.1 shard-min (−32%) all 8 shards green
astro build + dangerouslyEnableTestHooks #7 8 shards 2.8 min (−61%) 14.6 shard-min (−59%) all 8 shards green
astro build --devOutput on Endform #8 1 job, concurrency 64 6m38s for the whole job (parallel run 3.1 min) 145 Endform runner-min, plus 1 GitHub job red on every attempt (12, 9 and 16 failed), mostly 5 s assertion timeouts in different tests each time**
astro build + dangerouslyEnableTestHooks on Endform #9 1 job, concurrency 64 5m44s for the whole job (parallel run 2.2 min, exclusive 1.4 min) 91 Endform runner-min, plus 1 GitHub job 344 passed, 2 passed on retry***
Prototype: astro build + runtime switch on Endform #4 1 job, concurrency 64 4 min 50 s for the whole job (about 1 min 45 s of it install and build) 1 job plus Endform runner time 378 passed, 0 retries

For the shard rows, wall time is the slowest shard's "Run Node E2E" step and runner time is that step summed over all 8 shards. Install, build and the Cloudflare step are left out because they don't change between methods. emdash-cms#4317 now builds the fixtures before that step, which is why every shard number is lower than before the sync. Before the sync, against the older baseline (8.0 min, 42.6 shard-min), the shard results were the same picture: #6 −26% and #7 −51%, and on Endform #9 took 4m55s with no retries. For the Endform rows, wall time is the whole job, install and build included.

*** visual-editing.spec.ts "flushes unsaved edits when navigating away", the known Endform keepalive issue, and media-library.spec.ts "matches the compact responsive folder layout".

** The same dev-output build passes natively (344 of 344), and its client bundles are within 3% of the production build's (4.9 MB vs 4.8 MB of gzipped JS). The extra time is most likely server-side rendering in development mode, which the one shared server can't keep up with at 64 tests at once. That is the same gap as in the native timings (10.3 vs 6.0 min). Not profiled further.

Runs after the sync: #6 CI, #7 CI, #8 Endform E2E, #9 Endform E2E. Before the sync: #6, #7, #8, #9. One or two runs each, so treat small differences as noise.

Other checks: Smoke Isolated Install passes after the sync, since upstream now installs the newest Node 22 for it. Browser Tests (the admin Vitest browser suite, which none of these PRs touch) still fails intermittently on these runners. It fails with "Failed to fetch dynamically imported module" / "Cannot connect to the iframe" on a different file each attempt, and once locally with a timing assertion (PortableTextEditor.test.tsx, toolbar fade) that passes 3 of 3 times on its own. After the sync it passed first time on #6 and #10, and on the third attempt on #7.

Related PRs

PR Server under test Where it runs Core change
#6 astro build --devOutput 8 GitHub shards (layout unchanged) One line: the build-time registry check accepts localhost in a dev-output build
#7 astro build 8 GitHub shards (layout unchanged) New build-time dangerouslyEnableTestHooks option
#8 (this PR) (stacked on #6) astro build --devOutput Endform, one job None beyond #6
#9 (stacked on #7) astro build Endform, one job None beyond #7
#10 any Wherever the whole suite shares one server: a full local run, or Endform None; three test fixes, included in #8 and #9
#4 astro build with a runtime env switch Endform Prototype that the others replace

Type of change

  • Bug fix
  • Feature (requires an accepted feature plan or RFC)
  • Refactor (no behavior change)
  • Translation
  • Documentation
  • Performance improvement
  • Tests
  • Chore (dependencies, CI, tooling)

Your connection to this change

  • I use EmDash and ran into the problem this PR addresses
  • I maintain a tool, dependency, or service that EmDash uses, and this change concerns that integration
  • A Maintainer approved this specific change before I opened the PR

Checklist

  • I have read CONTRIBUTING.md
  • pnpm typecheck passes
  • pnpm lint passes
  • pnpm test passes (or targeted tests for my change): the Endform E2E workflow on this PR (see table)
  • pnpm format has been run
  • I have added/updated tests for my changes (if applicable)
  • User-visible strings in the admin UI are wrapped for translation (if applicable): not applicable, no admin UI change
  • I have added and reviewed the user-facing changeset (if this PR changes a published package): not applicable, test and CI changes only
  • New features link to their merged design PR: not applicable
  • I have included screenshots below if this PR changes the UI: not applicable

AI assistance disclosure

  • This PR includes AI-generated code or text — model/tool: Claude Opus 5.5 (Claude Code)

Screenshots / test output

Not applicable. Timings are in the table above.

🤖 Generated with Claude Code

ostenbom and others added 3 commits October 27, 2026 13:46
CI splits the Node suite over 8 shards, each with its own server, so
these only fail in a full run against a single server:

- The sandbox hook-event test listed the test plugin's events with
  limit 100; a full run records more. The plugin's events-list route
  now accepts `ids` for a direct lookup.
- The sandbox user-list check required exactly one user, but the
  passkey setup test adds a second.
- In a crowded root folder the dragged media row sat at the viewport's
  bottom edge, where dnd-kit auto-scrolls and the drop target moves.
  pointerDrag now scrolls the source to the centre and re-measures the
  target during the drag.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Under Endform (ENDFORM=true) Playwright starts e2e/endform-server.ts as
its webServer on the CLI machine. It runs the normal global setup with
EMDASH_E2E_BUILD=production and serves the setup state on
127.0.0.1:4446, which an auto fixture fetches on each runner. Endform
proxies loopback traffic from its runners back to the CLI machine, so
every test runs on its own runner against the one built server.

A gzip proxy in front of the built server compresses /_astro/ assets
once, since each runner starts with an empty cache and downloads the
admin bundle through the tunnel. About 25 tests that assumed they ran
alone against one database now create their own data. Specs that reset
the whole site run in a separate exclusive project, and tests that wipe
every comment or toggle search for a collection are limited by tag.

CI runs this as one job in .github/workflows/endform-e2e.yml.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The repository runs Endform for more than one organization, so the
GitHub App's OIDC sign-in can't pick this one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant