Skip to content

fix(harness): retry silent model stream interruptions - #2157

Merged
justinhelmer merged 1 commit into
mainfrom
plan/fix-the-remaining-half-o-9f6aca/u1
Sep 21, 2026
Merged

justinhelmer merged 1 commit into
mainfrom
plan/fix-the-remaining-half-o-9f6aca/u1

Conversation

@coreplane-switchboard

@coreplane-switchboard coreplane-switchboard Bot commented Sep 21, 2026

Copy link
Copy Markdown
Contributor

Retries OpenAI EOFs and open-stream aborts on the lease-backed ladder. Lease-sized pi timeouts and SSE keepalives prevent quiet reasoning turns from being mistaken for transport failures.

Why: #1932 showed a 29 s silent reasoning pause ending a coding child after #2136 could only salvage its work. Under record 0046, recovery must spend the existing lease instead of ending the child.

Where to look

  1. Transient wire classifier Recognizes OpenAI's missing-finish EOF and the anchored open-stream abort shape beside the existing Anthropic stream cut.
  2. Intentional versus transient aborts Books tool-cut aborts first, then holds ordinary open-stream aborts on the existing retry ladder. ⚠ Wrong ordering can reissue work deliberately cut by a budget.
  3. Silent-stream keepalive Races one retained provider read with unmetered SSE comments while preserving provider chunk order. ⚠ A stream race could duplicate, drop, or mis-meter provider output.
  4. Lease-backed pi timeout Writes the remaining lease into pi's HTTP idle/provider timeout instead of accepting a shorter library default.
  5. Launch settings contract Pins the lease timeout in compaction and plain settings, correcting the failed shard-1 assertion.
  6. Transport retry regression Proves the observed OpenAI EOF holds and retries the same turn so the child survives.
  7. Provider and harness contract Binds the silent-stream keepalive and metering behavior to exact unit proofs.

Feedback wanted: Please focus on whether SSE comments preserve each wire dialect and whether intentional aborts remain safely ahead of transient classification.

Risk: If wrong, quiet streams can still die, provider bytes can be mishandled, or deliberate cuts can duplicate turns. The blast radius is pi-backed model calls; revert this PR to restore the prior behavior.

Verified: 347 scoped tests and all local changed-set gates passed; 29/29 CI checks completed without failure at d6369de. Deployed-hop behavior remains human-gated.

Decisions (3)
  • Keep the silent stream alive, not merely the client. Raising pi's timeout alone cannot fix an intermediary idle cut: pi already allowed longer than the observed 29 s close. SSE comments keep the public hop active while wire parsers and metering ignore them.
  • Derive pi's stream timeout from the lease. A fixed timeout can end a healthy turn before record 0046's bound. The process receives the remaining lease at launch, while the harness retains its earlier loop, finale, and follow-up boundaries.
  • Classify intentional aborts before transient ones. The same abort wording can come from a budget cut or an ordinary stream loss. Existing cut and finale state distinguishes them, so deliberate cancellation is booked before the transient retry classifier.
Validation (9 criteria)
Criterion Proof
The failed shard assertion reproduces and is corrected Before the fix, npx vitest run src/core/harness/contract.test.ts failed 1/17 on missing httpIdleTimeoutMs; after the fix, 17/17 passed.
Affected harness, proxy, budget, and conformance behavior passes npx vitest run on 7 named affected test files — 7 files, 347 tests passed.
Changed TypeScript compiles NODE_OPTIONS=--max-old-space-size=6144 npx tsc --noEmit -p tsconfig.json — passed.
Changed files match formatting npx prettier --check on the 12 changed files — all matched.
Changed TypeScript passes lint npx eslint on changed TypeScript files — passed.
Public hygiene and duration centralization remain ratcheted npm run hygiene:check and npm run clock:check — passed.
Spec proofs resolve and every changed path remains covered npm run specs:check and npm run specs:coverage -- --changed origin/main...HEAD --test-guard --require — passed.
Pull request title is accepted npm run check:pr-title -- "fix(harness): retry silent model stream interruptions" — passed.
The original CI findings are closed at the pushed head At d6369de, ci / bot / test 1 of 4 and aggregate ci / bot passed; all 29 check runs completed with zero failures.
For agents

Rebased without conflict onto origin/main at 4bcc7f0, folded the stale contract expectation into the single implementation commit, and force-pushed d6369de. The original shard failure was src/core/harness/contract.test.ts: settings now correctly include the 45-minute lease-backed httpIdleTimeoutMs with and without compaction. Reproduction wording remains Stream ended without finish_reason and This operation was aborted. The heartbeat is an SSE comment and deliberately bypasses TTFT, token metering, and provider-byte accounting.
Requested by @justinhelmer in slack:C0BRRHKFLCB

🤖 Generated with Claude Code

@coreplane-switchboard coreplane-switchboard Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM: The transient retry and lease-backed stream-silence handling are coherent and covered by the required regression scenarios.

Note

Approved · head b82be7f · no findings

Full review

No findings; the retry classification and stream-silence handling satisfy the contract without contradicting the touched specs.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Auto-approved: coreplane-switchboard[bot] reviewed this PR and posted an LGTM verdict (see its review). This repository opted in through its REVIEW_BOT_LOGIN and REVIEW_BOT_ID variables.

@justinhelmer justinhelmer left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not a finding for this PR (revised 20:02Z, after a second read with the tracker session). The lease-derived httpIdleTimeoutMs is the principled bound for pi's single knob, and the red contract test only needs its expected settings file updated to the new field. Merge on that.

Residual, carried to #1932 as the remaining item (Wave B, not this PR): pi's one knob sets both the request bound and the between-byte idle bound to the remaining lease, so an upstream that hangs silently without closing is detected only at lease end — up to the coding lease of "thinking". The right hop for a short idle bound is the proxy (src/channels/modelProxy.ts), which sees OpenRouter's SSE keepalive comments (: OPENROUTER PROCESSING) while a model reasons: it can hold a minutes-sized upstream-idle bound from the budgets module and end the client stream with the typed transient verdict, while pi's own bound stays the lease. An upstream that closes is already caught by the proxy's close → pi's transient path this PR adds. Fixture for that follow-up: b03035f7's 28 s silence, replayed as a silent hang past the bound.

Co-Authored-By: coreplane-switchboard[bot] <318072483+coreplane-switchboard[bot]@users.noreply.github.com>
@coreplane-switchboard
coreplane-switchboard Bot force-pushed the plan/fix-the-remaining-half-o-9f6aca/u1 branch from b82be7f to d6369de Compare September 21, 2026 20:01

@coreplane-switchboard coreplane-switchboard Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM: The rebase preserves the reviewed patch, updates its contract expectation, and the two previously failing CI checks now pass.

Note

Approved · head d6369de · no findings

Full review

The rebased head resolves both prior CI findings; the bot gate and test shard now pass.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Auto-approved: coreplane-switchboard[bot] reviewed this PR and posted an LGTM verdict (see its review). This repository opted in through its REVIEW_BOT_LOGIN and REVIEW_BOT_ID variables.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant