Skip to content

fix: end /next drains that re-offer an already-skipped message - #753

Closed
bandzalkin wants to merge 1 commit into
mainfrom
fix/processed-ack-resync-livelock
Closed

bandzalkin wants to merge 1 commit into
mainfrom
fix/processed-ack-resync-livelock

Conversation

@bandzalkin

Copy link
Copy Markdown
Contributor

Problem

A room's /next catch-up loops could livelock, pinning the room's REST backlog behind one message forever (busy /next polling, no durable progress).

_retry_processed_ack (src/band/runtime/execution.py) deliberately converts an exhausted processed-ack retry budget into local completion: the turn ran, so re-running it could duplicate its side effects, and blocking forever on the failed ack would starve every newer message. But from that moment on:

  • the platform copy of the message is unchanged (typically still processing), so /next keeps returning it as the oldest not-processed message;
  • _process_backlog_message advances it from the local completed marker without touching the platform;
  • _synchronize_with_next and _resync_pending_messages read ADVANCED as server-side progress and loop until /next yields None.

Result: a drain that spins on one message it can never retire, and a REST-only backlog that never gets past it. This also widens from a single trigger to whole bursts for agents that merge waiting messages into one turn (band-agent's merged turns ack every merged message on the turn's tools; one lost ack wedges one fetch each).

Fix

Classify the local-knowledge skips (locally completed, permanently failed, own echo) as a new BacklogProcessResult.NO_PROGRESS, and have both drain loops end a drain that re-receives a message they already skipped in the same pass — the same way the existing stopped and permanently-failed guards end a drain without durable completion.

Retryable outcomes are untouched: a handler failure with attempt budget left, a pending processed-ack, or an unacquired claim keep today's in-drain retry semantics (new test test_startup_sync_retries_failed_message_in_same_drain pins this; the guard records only NO_PROGRESS, never those).

Scope and limits

  • Ends the local busy-poll livelock; startup/resync now complete instead of spinning.
  • Does not make the wedged message durable on the platform — nothing can, once the ack endpoint keeps refusing it — and does not change the /next contract. New WS-pushed messages process normally while a wedged message waits to be retired (platform stale-processing re-serve, local completion cache eviction, or restart).
  • No wire, persistence, or public API changes; BacklogProcessResult is internal.

Tests

  • tests/runtime/test_resync.py: two wedge tests (repeating /next + locally completed message → both drains return synced after two fetches; they fail pre-fix by spinning to a 10-fetch cap) and the retry-preservation test.
  • tests/runtime/test_execution.py: two local-skip assertions updated to the new NO_PROGRESS classification (behavioral assertions unchanged).
  • Gate: ruff check, ruff format --check, pyrefly check, pytest full unit suite — 6703 passed / 711 skipped; pre-commit run --all-files clean.
  • Review: adversarial round 1 confirmed a first-cut regression (guard recorded retryable ADVANCED results and deferred in-drain retries) → corrected by the NO_PROGRESS classification; adversarial round 2 traced all 13 result sites and every repeat class to bounded termination, zero findings; architectural pass approved the drain-loop ownership seam.

Found while reviewing band-agent's merged-turn work (ack path shared with ExecutionContext); no behavioral coupling to it.

@bandzalkin
bandzalkin requested a review from a team October 6, 2026 17:08
github-actions Bot pushed a commit to bandzalkin/band-sdk-python that referenced this pull request Oct 8, 2026
github-actions Bot pushed a commit to bandzalkin/band-sdk-python that referenced this pull request Oct 8, 2026
Squashed from band-ai#730 at fb7403a, resolved over band-ai#753: a locally skipped /next head now hands the drain to the actionable snapshot, so band-ai#753's re-offer guard is a backstop and its wedge tests assert one /next poll; the sync point still accepts a no-progress result but only once the message is terminal.

Upstream-PR: band-ai#730
bandzalkin added a commit to bandzalkin/band-sdk-python that referenced this pull request Oct 8, 2026
bandzalkin added a commit to bandzalkin/band-sdk-python that referenced this pull request Oct 8, 2026
Squashed from band-ai#730 at fb7403a, resolved over band-ai#753: a locally skipped /next head now hands the drain to the actionable snapshot, so band-ai#753's re-offer guard is a backstop and its wedge tests assert one /next poll; the sync point still accepts a no-progress result but only once the message is terminal.

Upstream-PR: band-ai#730
github-actions Bot pushed a commit to bandzalkin/band-sdk-python that referenced this pull request Oct 8, 2026
github-actions Bot pushed a commit to bandzalkin/band-sdk-python that referenced this pull request Oct 8, 2026
Squashed from band-ai#730 at fb7403a, resolved over band-ai#753: a locally skipped /next head now hands the drain to the actionable snapshot, so band-ai#753's re-offer guard is a backstop and its wedge tests assert one /next poll; the sync point still accepts a no-progress result but only once the message is terminal.

Upstream-PR: band-ai#730
github-actions Bot pushed a commit to bandzalkin/band-sdk-python that referenced this pull request Oct 9, 2026
github-actions Bot pushed a commit to bandzalkin/band-sdk-python that referenced this pull request Oct 9, 2026
Squashed from band-ai#730 at fb7403a, resolved over band-ai#753: a locally skipped /next head now hands the drain to the actionable snapshot, so band-ai#753's re-offer guard is a backstop and its wedge tests assert one /next poll; the sync point still accepts a no-progress result but only once the message is terminal.

Upstream-PR: band-ai#730
github-actions Bot pushed a commit to bandzalkin/band-sdk-python that referenced this pull request Oct 9, 2026
github-actions Bot pushed a commit to bandzalkin/band-sdk-python that referenced this pull request Oct 9, 2026
Squashed from band-ai#730 at fb7403a, resolved over band-ai#753: a locally skipped /next head now hands the drain to the actionable snapshot, so band-ai#753's re-offer guard is a backstop and its wedge tests assert one /next poll; the sync point still accepts a no-progress result but only once the message is terminal.

Upstream-PR: band-ai#730
github-actions Bot pushed a commit to bandzalkin/band-sdk-python that referenced this pull request Oct 9, 2026
github-actions Bot pushed a commit to bandzalkin/band-sdk-python that referenced this pull request Oct 9, 2026
Squashed from band-ai#730 at fb7403a, resolved over band-ai#753: a locally skipped /next head now hands the drain to the actionable snapshot, so band-ai#753's re-offer guard is a backstop and its wedge tests assert one /next poll; the sync point still accepts a no-progress result but only once the message is terminal.

Upstream-PR: band-ai#730
github-actions Bot pushed a commit to bandzalkin/band-sdk-python that referenced this pull request Oct 9, 2026
github-actions Bot pushed a commit to bandzalkin/band-sdk-python that referenced this pull request Oct 9, 2026
Squashed from band-ai#730 at fb7403a, resolved over band-ai#753: a locally skipped /next head now hands the drain to the actionable snapshot, so band-ai#753's re-offer guard is a backstop and its wedge tests assert one /next poll; the sync point still accepts a no-progress result but only once the message is terminal.

Upstream-PR: band-ai#730
bandzalkin added a commit to bandzalkin/band-sdk-python that referenced this pull request Oct 9, 2026
bandzalkin added a commit to bandzalkin/band-sdk-python that referenced this pull request Oct 9, 2026
Squashed from band-ai#730 at fb7403a, resolved over band-ai#753: a locally skipped /next head now hands the drain to the actionable snapshot, so band-ai#753's re-offer guard is a backstop and its wedge tests assert one /next poll; the sync point still accepts a no-progress result but only once the message is terminal.

Upstream-PR: band-ai#730
github-actions Bot pushed a commit to bandzalkin/band-sdk-python that referenced this pull request Oct 9, 2026
github-actions Bot pushed a commit to bandzalkin/band-sdk-python that referenced this pull request Oct 9, 2026
Squashed from band-ai#730 at fb7403a, resolved over band-ai#753: a locally skipped /next head now hands the drain to the actionable snapshot, so band-ai#753's re-offer guard is a backstop and its wedge tests assert one /next poll; the sync point still accepts a no-progress result but only once the message is terminal.

Upstream-PR: band-ai#730
github-actions Bot pushed a commit to bandzalkin/band-sdk-python that referenced this pull request Oct 10, 2026
github-actions Bot pushed a commit to bandzalkin/band-sdk-python that referenced this pull request Oct 10, 2026
Squashed from band-ai#730 at fb7403a, resolved over band-ai#753: a locally skipped /next head now hands the drain to the actionable snapshot, so band-ai#753's re-offer guard is a backstop and its wedge tests assert one /next poll; the sync point still accepts a no-progress result but only once the message is terminal.

Upstream-PR: band-ai#730
The /next catch-up loops polled until the endpoint yielded nothing. A
message skipped from local knowledge alone -- completed, permanently
failed, or the agent's own -- never changes its platform state, so once
the processed-ack retry budget exhausted and recorded local completion
without a durable mark, /next kept returning that same oldest message:
the drain spun on it and newer REST-only backlog starved behind it.

Classify those skips as BacklogProcessResult.NO_PROGRESS and end a
drain that re-receives a message it already skipped in the same pass,
the way the stop and permanently-failed guards already end a drain.
Retryable outcomes -- a handler failure with attempt budget left, a
pending processed ack, an unacquired claim -- keep their in-drain
retry semantics unchanged.
@AlexanderZ-Band
AlexanderZ-Band force-pushed the fix/processed-ack-resync-livelock branch from c02eb66 to 6f5df16 Compare October 10, 2026 06:24
bandzalkin added a commit to bandzalkin/band-sdk-python that referenced this pull request Oct 10, 2026
bandzalkin added a commit to bandzalkin/band-sdk-python that referenced this pull request Oct 10, 2026
Squashed from band-ai#730 at adebc7d; preserves band-ai#753 terminal sync handling and removes a duplicate recovery wiring assertion in favor of behavioral recovery coverage.

Upstream-PR: band-ai#730
github-actions Bot pushed a commit to bandzalkin/band-sdk-python that referenced this pull request Oct 10, 2026
github-actions Bot pushed a commit to bandzalkin/band-sdk-python that referenced this pull request Oct 10, 2026
Squashed from band-ai#730 at adebc7d; preserves band-ai#753 terminal sync handling and removes a duplicate recovery wiring assertion in favor of behavioral recovery coverage.

Upstream-PR: band-ai#730
bandzalkin added a commit to bandzalkin/band-sdk-python that referenced this pull request Oct 11, 2026
bandzalkin added a commit to bandzalkin/band-sdk-python that referenced this pull request Oct 11, 2026
Squashed from band-ai#730 at adebc7d; preserves band-ai#753 terminal sync handling and removes a duplicate recovery wiring assertion in favor of behavioral recovery coverage.

Upstream-PR: band-ai#730
github-actions Bot pushed a commit to bandzalkin/band-sdk-python that referenced this pull request Oct 11, 2026
github-actions Bot pushed a commit to bandzalkin/band-sdk-python that referenced this pull request Oct 11, 2026
Squashed from band-ai#730 at adebc7d; preserves band-ai#753 terminal sync handling and removes a duplicate recovery wiring assertion in favor of behavioral recovery coverage.

Upstream-PR: band-ai#730
github-actions Bot pushed a commit to bandzalkin/band-sdk-python that referenced this pull request Oct 11, 2026
github-actions Bot pushed a commit to bandzalkin/band-sdk-python that referenced this pull request Oct 11, 2026
Squashed from band-ai#730 at adebc7d; preserves band-ai#753 terminal sync handling and removes a duplicate recovery wiring assertion in favor of behavioral recovery coverage.

Upstream-PR: band-ai#730
github-actions Bot pushed a commit to bandzalkin/band-sdk-python that referenced this pull request Oct 11, 2026
github-actions Bot pushed a commit to bandzalkin/band-sdk-python that referenced this pull request Oct 11, 2026
Squashed from band-ai#730 at adebc7d; preserves band-ai#753 terminal sync handling and removes a duplicate recovery wiring assertion in favor of behavioral recovery coverage.

Upstream-PR: band-ai#730
@AlexanderZ-Band

Copy link
Copy Markdown
Collaborator

Closing as superseded by #761. Current main already detects a repeated locally nonrunnable /next head—including the locally completed state created when the processed-ack retry budget exhausts—and ends recovery after the repeat, preventing this drain from busy-polling.

This PR would duplicate that no-spin behavior. It also cannot unblock later REST-only work behind a server-retained head; that remaining recovery problem needs a separate, current-main design and is being considered in #730.

bandzalkin added a commit that referenced this pull request Oct 13, 2026
Refresh #730 onto current upstream. Collect complete unfiltered chronological pages before claims; retain failed heads, eligible delivery order, terminal WebSocket handoff and current recovery/admission ownership.

Source-Base: 55c70f2
Source-Head: adebc7d
Applied-Over: cd6bde7
Conflict-Notes: keep authoritative /next and stopped-room 204 gating, no filtered processing pre-sweep; reuse shared nonrunnable predicate instead of #753 NO_PROGRESS bookkeeping; fold 0537758 snapshot revision fence into explicit recovery outcomes; preserve accepted claim, pending input, acknowledgement, #776 STOP and TurnTask custody. CHANGELOG.md excluded.
Verification: parent assembled HTTP-boundary/custody proof 589 passed; #761 plus refreshed #730 covers #753 no-spin behavior. Source hunk ledger retained in u3-preparation/fork.
Upstream-PR: #730
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants