[Feature]: Run full-site publish off the server's main thread #650

Problem

A full-site publish (publishDraftSite, server/publish/publishSite.ts) runs entirely on the server's main thread, inside the request that triggered it:

• POST /admin/api/cms/publish (server/handlers/cms/publish.ts) and the MCP publish tool (server/ai/mcp/tools/publishTool.ts) both await publishDraftSite(...) and only answer when the whole pipeline is done.
• Phase 1 runs buildSiteRuntimeScripts once per page, one after another. The esbuild work itself happens in esbuild's service process, but the inputs and outputs are assembled on the main thread.
• Layer A then renders every page (renderPublishedSnapshot), runs buildPublishedSiteCssBundle and applyPublishedHtmlPipeline, bakes data rows and writes the slot. These are all CPU-bound and run on the same event loop that serves the editor, the collab relay, the MCP bridge stream and public traffic. The only breaks are the awaits between pages.

Numbers from a self-hosted 0.0.21 instance with 75 pages, on 2026-10-14. They compare site_snapshots.created_at (Phase 2 commit) with the publish audit_events.created_at (end of the request):

• Layer A alone takes 4.1–4.8 s for every publish. Phase 1 comes on top of that, and these timestamps can't show how long it takes.
• 12 full publishes ran that day, and 9 of them came from the MCP agent. The agent publishes after every round of edits, so the editor shares the event loop with a multi-second bake again and again while someone is working in it.

We don't have event-loop lag measurements yet, so how much this stalls interactive requests is an inference from the code, not a measured result. Both get worse as the page count grows.

Proposed solution

Smallest useful version, in steps:

1. Measure. Log the duration of each publish phase (Phase 1, Phase 2, Layer A, data-row bake), and sample event-loop lag while a publish runs (monitorEventLoopDelay from perf_hooks). That shows how big the problem is before anything else changes.
2. Yield between pages. In the Phase 1 and Layer A loops, yield to the event loop (await new Promise(setImmediate)) after each page. Editor, collab and MCP requests then get a turn between pages. It is cheap, but it doesn't reduce the total CPU time.
3. Move the build and bake into a worker. The server already runs Bun Workers for plugins (server/plugins/host/workerPool.ts) and image variants (server/handlers/cms/imageVariantWorkerHost.ts). The work involved:
   • Phase 1 and Layer A run in a worker with its own DB client and module registry.
   • The main thread keeps withPublishLock, the Phase 2 transaction, swapSlot and bumpPublishVersion, so the version-stamping invariant (ISS-038) and the "no await between swap and bump" rule stay where they are.
4. Optional: don't hold the request open. The publish endpoint and the MCP tool could start a publish job and return straight away. The existing /admin/api/cms/publish/status would report progress. This is a bigger change to the API and the agent tool contract, so it should be decided separately from step 3.

Alternatives considered

• Incremental publish (only rebuild pages whose inputs changed) would cut the work itself, but it is a much larger change. The per-version runtime-asset paths would also need to be content-addressed first.
• A separate publisher process/container isolates it best, but self-hosters then have to deploy another service. A worker gets most of the benefit without that.

Related: #647 (prunes runtime assets of superseded page versions; it is the storage side of the same frequent-publish pattern).

Area

Publishing