Repository navigation
fix(collaboration): survive strict-Event dispatch on Workers; log sync frame decode failures - #6047
Merged
Merged
Conversation
…rumentation Every prod EditDocument since 2026-08-26 fails with an opaque "initial sync failed: timeout (10000ms)". Add passive WebSocket diagnostics to the worker so the next timeout says how far the connection got: never opened, raw frame arrived but never decoded (e.g. Blob where Bebop expects ArrayBuffer), or decoded frames with no RemoteInitialSync. - new sync-diagnostics recorder wraps the platform WebSocket factory and observes open/message/close/error on the raw socket - decoded-frame kinds counted via the wrapper's public Message event - edit.sync_init span gets sync.ws.* attrs; the timeout error message carries a compact summary into the 502 body - createSyncSocket accepts an optional socket factory (additive) - WebsocketBuilder.withSerializer no longer drops a factory set before it Co-authored-by: teo <synoet@users.noreply.github.com>
…ial sync is lost Every prod edit has died on 'initial sync failed: timeout (10000ms)' since 2026-08-26 14:02 UTC. Sync-service accepts the worker's /connect and sends the snapshot (span records snapshot.bytes), but the frame never reaches the worker: a workerd hibernation regression (STOR-5552; cloudflare/workerd#7163, follow-up #7147) dropped sends caught in the connect-request handover to the hibernation manager. PeerRequestSnapshot -> RemoteSnapshot is served from the message handler on the manager-owned socket, which does not cross that boundary and demonstrably still works in prod. WorkerSyncSource (worker-only subclass) requests a snapshot when the pushed RemoteInitialSync stays silent for 2s and bootstraps from the reply; first success wins. Browsers keep the plain push-based flow — the shared source only widens initialSyncReceived to protected. Co-authored-by: teo <synoet@users.noreply.github.com>
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
Included review availability: Your plan provides up to 8 included reviews per hour; 6 remain after this review. 📝 WalkthroughSummary by CodeRabbit
WalkthroughThe WebSocket message handler now describes frame data without reading its bytes and redacts URL query strings before logging. Deserialization failures are logged with the frame description and redacted URL, then rethrown. Message event construction now falls back to a data-only Merge Risk: ⚪ Minimal · up to This localized update improves WebSocket resilience and makes decode failures visible without changing the established message contract; no actionable merge-blocking risk remains after normal checks and review. 🚥 Pre-merge checks | ✅ 3 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (3 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
… break delivery The diagnostics listeners attach to the native socket ahead of the transport wrapper's handlers. A listener exception there could suppress the wrapper's dispatch for that event (or error out the socket under workerd's fail-fast event dispatch), making the instrumentation cause the failure it measures. The bodies are all typeof/instanceof checks that should not throw; swallow exceptions anyway so observation stays observation. Co-authored-by: teo <synoet@users.noreply.github.com>
Dev deploy of the fallback still failed, and its diagnostics finally showed why: the initial sync ARRIVES (785-byte ArrayBuffer, matching the 771-byte snapshot the sync service recorded sending) but fails to deserialize, and the deserialize throw escapes the wrapper's message listener into the runtime's event dispatch, which errors out the socket without a close. The zombie socket never retries and silently swallowed the snapshot fallback request. Two changes: - Tolerant serializer (worker-only): deserialize failures report to diagnostics and yield an inert message instead of throwing, so the socket survives and the snapshot fallback still runs. createSyncSocket gains an optional serializer parameter (additive; browser flow unchanged). - Diagnostics v2: hex prefix + independent probe decode (with exception text) of the first frame, ordered event log with ms offsets, and decode-failure counters — the next dev run will say exactly what the bytes are. Co-authored-by: teo <synoet@users.noreply.github.com>
…eEvent copy The probe build on dev showed the frame is VALID (probe=ok:initial_sync, hex has the Bebop envelope + loro magic) yet decoded=0 with no deserialize attempt and an error event in the same ms as the frame: dispatch dies between the first-registered native listener and the wrapper's handler. - Wrapper: copying origin/lastEventId/source into the re-dispatched MessageEvent can throw on runtimes with stricter Event internals; fall back to a data-only event (consumers only read data). - Diagnostics: route every native-socket listener through a guard that contains exceptions and records a per-listener start/ok/threw trail plus the first error with stack line (sync.ws.first_listener_error) — the next run distinguishes 'listener threw' from 'runtime aborted dispatch'. Co-authored-by: teo <synoet@users.noreply.github.com>
cursor
Bot
changed the title
…cs + dispatch fix Scope the PR to instrumentation and the dispatch fix per review: remove WorkerSyncSource, the tolerant serializer, createSyncSocket's serializer param, the protected widening of initialSyncReceived, and the worker's neverthrow dependency. Deserialize throws are still contained and recorded by the diagnostics listener guards (listener_errors / first_listener_error), so a poison frame degrades to a fully-described timeout instead of a zombie socket. Co-authored-by: teo <synoet@users.noreply.github.com>
cursor
Bot
changed the title
… dispatch fix + decode-failure log Net change vs main is now a single file. The wrapper's message path: - logs a clear, token-free error when an incoming frame fails to deserialize (frame shape + socket path + exception), so an undecodable initial sync is distinguishable from one that never arrived — previously the throw died inside the runtime's native dispatch with no trace; - constructs the re-dispatched MessageEvent with a data-only fallback, since copying origin/lastEventId/source can throw on runtimes with stricter Event internals (Cloudflare Workers) and error out the socket. Co-authored-by: teo <synoet@users.noreply.github.com>
cursor
Bot
changed the title
…ilure log Co-authored-by: teo <synoet@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
Every production
EditDocument(MCP and in-app AI edit) has failed since 2026-08-26 14:02 UTC with an opaqueinitial sync failed: timeout (10000ms)— zero successes since, across all documents, in prod and dev alike, with no Macro deploy at the cutoff. Browsers were unaffected.Instrumented dev deploys during the investigation established the failure chain precisely:
/connectspan recordssnapshot.bytes).ok:initial_sync, valid Bebop envelope +lorosnapshot magic).The timing matches workerd
v1.20260825.1(in production at the cutoff), which shipped a large Event/EventTarget standards refactor (workerd#7053). The wrapper's message handler copiesorigin/lastEventId/sourceoff the native event into a re-dispatchedMessageEvent; under stricter Event internals that copy can throw, and a throw escaping a message listener into native dispatch errors out the socket.Change
One file:
packages/collaboration/src/websocket/core/websocket.ts.MessageEventwith a data-only fallback when copyingorigin/lastEventId/sourcethrows. Consumers only readdata.ArrayBuffer of N bytes), token-stripped socket path, and the exception. Previously that throw died inside the runtime's native dispatch with no trace, making an undecodable frame indistinguishable from a frame that never arrived.The instrumentation and snapshot-fallback commits from earlier revisions of this branch are fully reverted; diagnostics served their purpose during the investigation.
Testing
tsc --noEmitand biome clean in both packages.POST /editagainst the local stack → HTTP 200 in 7.3s. Behavior unchanged.websocket: failed to deserialize incoming message (ArrayBuffer of 3740 bytes) from ws://…/document/<id>/connect: BebopRuntimeError: Unrecognized discriminator while decoding FromRemote— decode failures are no longer silent.
Rollout
Deploy dev (
bun run deploy-devinservices/ai-editing-worker, or theDeploy AI Editing Workerworkflow dispatch from this branch) and run one edit. Dev currently fails 100%, so one successful edit confirms theMessageEventcopy was the killing throw. If it still times out,wrangler tail --env devnow distinguishes the remaining possibilities: a deserialize-failure log means bad bytes; silence with an instant socket error means the runtime is aborting multi-listener dispatch — evidence for a Cloudflare ticket referencing workerd#7053 / STOR-5552.