fix(sdk): recover backlog past exhausted failed messages

[bandzalkin]

Rebased onto main d7ab3429cbdb (2026-10-15)

Rebased onto current main (after #709, #804, #789/#775, #796 and #802) as one commit; head b9158829eb142311a7ba99761665135f9fc74679. Ported onto #789/#775's TurnOutcome and #804's module layout. Adds a fix for a stall the previous head had: snapshot recovery now yields to queued local turns only outside bootstrap and once no older deferred message is outstanding, and a deferral ends as soon as its row is settled locally (including under a STOP/PLAY/INTERRUPT fence or via a processed-ack retry); regressions in tests/runtime/resync/test_actionable_recovery.py. Also carries the deferred-local-event stopped-room probe, its docs/websocket-events.md note, and a bounded watchdog-cleanup test fixture.

CHANGELOG.md edits are dropped: Release Please generates it.

Verified at the new head (macOS, Python 3.13): uv sync --all-packages --locked --extra dev, all pre-commit hooks, and the full pytest -q: 8628 passed, 738 skipped. Verification notes further down refer to the previous head.

---

Summary

A retained skipped or retry-exhausted /next head must not starve later eligible deliveries. Refresh this fix onto current upstream d7ab3429cbdb5381b9ca572cf7eac210a8196052, preserving the current reconciliation model and the failed head's diagnostics rather than acknowledging it as processed.

Changes

• Collect one exhaustive, ordered, unfiltered actionable snapshot before any claim or delivery-status mutation. Preserve server ordering and deduplicate delivery IDs; finish cursor pagination before recovery starts.
• Recover later eligible deliveries behind a skipped or exhausted head without bypassing a quiet /next, overtaking retryable/in-flight work, or introducing a preliminary processing-status sweep.
• Preserve STOP/PLAY/INTERRUPT fences, accepted claims, terminal WebSocket handoff, deferred-head handling and explicit resync outcomes. Propagate listing failures rather than treating an incomplete listing as empty.
• Keep recovery in the existing lifecycle/execution owners; update regressions, operator documentation and the Unreleased changelog without changing released 5.1.0 entries.

Testing

Fresh current-main affected suite: 491 passed, 1 warning, exit 0. Scope: tests/runtime/resync, executioninterrupt, rooms, execution/working-state/shutdown, platform link and session sync/shutdown; includes all 67 actionable-recovery cases.
uv sync --all-packages --locked --extra dev and all five pre-commit hooks passed. The published head adds only the required changelog entry to tested code head 2c15fb08a18af243232a96fd5243fda57fca0aac.
Local checks are not remote CI or live-room/native provider receipts.

Commit range and merge order

Complete prerequisite range: d7ab3429cbdb5381b9ca572cf7eac210a8196052..b9158829eb142311a7ba99761665135f9fc74679.
This fix is independent of the ACP and OMP prerequisites and can merge independently. The separate resolver stacks retain their own prerequisite-first order. Related: #709.

[bandzalkin]

Architectural review → fix round

Scope: Reviewed #730 from merge base 8921c8ce8e5b78d4264a5e286880df87c74dc8a9 through correction commit 49d3dff9179bb92bfa08ecbe07b816ed7f75d447, including message lifecycle, execution, regressions, operator docs, and changelog. Outcome: recover later actionable work without acknowledging an exhausted failed head.

Constraints: Platform delivery state remains authoritative; local retry exhaustion must not erase failure evidence. MessageLifecycle owns REST pagination and ordering, ExecutionContext owns eligibility, claims, stop/play, and WebSocket handoff. An incomplete listing is not proof of a drained backlog. Existing cursor and legacy-page compatibility stays intact.

Decisions: Keep the existing recovery owners. Replace three status scans and synthesized timestamp sorting with one fully paginated, oldest-first unprocessed server stream and stable deduplication. This avoids status-transition omissions and invented ordering without adding another queue or retry policy. Stop before claiming each snapshot row and hand back to the WebSocket queue at its marker; preserve later reconnect synchronization.

Findings and proof: B1 marker overrun, B2 stop-during-listing, and B3 status/order loss were confirmed, corrected, and re-probed. A reconnect regression exposed during correction was also fixed. No in-scope blocker remains. Fresh local gate: 6,327 passed / 684 skipped; focused execution suite 151 passed; all pre-commit hooks passed; both lock checks passed; baseline E2E collection 526 items / 1 skipped; markdown snippets 23 passed. Standalone runtime/REST smoke preserved A → new-user → B, executed nothing while stopped, and retained status-transition and missing-timestamp order.

Architectural verdict: APPROVED for #730 independently. No new dependency, durable format, or policy owner. Approval assumes the existing REST listing contract. Combined integration with #729 remains HELD on the separately demonstrated ACP effect-commit finding; this is not remote CI or merge approval.

[AlexanderZ-Band]

Current-main follow-up

The underlying head-of-line failure remains on current main: /next continues to return the retained, locally exhausted failed head; startup stops at it, and idle reconciliation reports BLOCKED rather than listing later actionable rows. Later deliveries therefore remain unrecoverable through REST after a missed WebSocket delivery.

This PRs core contract is still needed: preserve the failed head and its diagnostics, but snapshot the complete ordered actionable stream before any status transition, then process eligible later rows without overtaking retryable or in-flight work.

The implementation cannot merge unchanged: GitHub now reports a conflict with main, whose reconciliation model has since acquired ResyncOutcome, deferred-head handling, and cursor-page APIs. The fix should be rebuilt on current main and retain this PRs regression coverage rather than resolving the old diff mechanically.

[bandzalkin]

Rebased onto current main (d7ab3429) at b9158829eb14; the description's new first section lists the conflict resolutions and the verification at this head.