Skip to content

Keep resumed threads in their own provider sessions - #3512

Merged
SawyerHood merged 9 commits into
bb/2580-codex-turn-start-correlation-thr_vdv2rx6c7wfrom
bb/3461-owned-sessions-stack
Sep 16, 2026
Merged

SawyerHood merged 9 commits into
bb/2580-codex-turn-start-correlation-thr_vdv2rx6c7wfrom
bb/3461-owned-sessions-stack

Conversation

@SawyerHood

@SawyerHood SawyerHood commented Sep 11, 2026

Copy link
Copy Markdown
Collaborator

Human comments

What was wrong

After the runtime identity mix-up in #3461, the server could resume a thread into whichever provider session its newest event named. Ordinary completion events and contaminated identity history could therefore redirect messages into another conversation. Current main still has this selector.

The independent safety review also reproduced a second path: editing a message can delete the original owner's identity event, making a contaminated later claimant appear to own that session.

What changed

Third layer of #3502#3503#3512. Review this layer against #3503; it retains the lower layers' Stop and Codex correlation behavior.

  • Resolve sessions only from identity events after the last completed context clear. Compare claims within the provider and host; unknown hosts are treated conservatively. An earlier foreign identity is skipped in favor of an owned identity. Earliest claims tied in the same millisecond are ambiguous, regardless of insertion order.
  • Refuse foreign-only, ambiguous, and invalid (null or empty authoritative handle) resume attempts with 409 provider_session_unavailable. No turn is dispatched or history appended. Queued sends record the refusal. /clear or bb thread clear starts a fresh session on the next send while retaining history.
  • Check ownership on edit checkpoints and anchored/tip fork sources. Refuse an edit that would erase the earliest shared ownership evidence, both before provider rewind preparation and atomically before suffix deletion. If another claimant arrives while a rewind is being staged, discard that staged session and retain the original history.
  • Record identities for empty-input starts. Require start/fork identity announcements in the public conformance kit and document unique, durable provider session handles. Scripted multi-process fixtures mint unique handles.
  • Regenerate the identity index migration after current main as 0125_silent_guardian. Preserve idempotent index creation for migration replay. Update pruning tests to keep their retention assertions while no longer treating ordinary event stamps as authoritative identities.
  • Resolve the real-provider integration manifest after test TMPDIR isolation so setup and workers use the same manifest.

No daemon wire fields change in this layer. Existing stored history is not rewritten on upgrade.

How you verified

Independent review covered the complete layer, #3461 and original investigation history, current-main changes, ownership queries/index plans, context clear, provider/host isolation, tied claims, legacy checkpoint lineage, and fork/rewind paths.

A new migrated-SQLite regression failed before the edit guard: removing the owner's identity changed the other thread from foreign to owned. Additional regressions fail before and pass after for null/empty identity handles; these cannot silently replace the conversation. Guard tests cover tied/strictly ordered claims, provider/host/unknown-host scoping, retained duplicate claims, clear-context retention, pre-rewind refusal, and a competing claim arriving before the atomic edit commit. Server tests assert no history mutation or replacement start after refusal, and successful staged-session discard.

Final head: 55097953348f0d583e34317375ed693d4d130716, based on #3503 cd67ddbb5735e4153e506bceec1a8c6a665dc2ef. Its tree exactly matches the reviewed local tree. Fresh forced Turbo checks ran on integrated implementation head e23319817a80042971fcf1e40036dda9b11ae5e6; the final follow-up changes only fixtures and formatting:

  • DB: 573 tests and typecheck passed, including the generated migration, replay, query plans, new edit guard and invalid-history regressions.
  • Codex: 317 tests and typecheck passed. The unit run excludes inherited account-pool environment variables because one missing-variable test assumes they are absent.
  • Server: typecheck passed. The full run passed 2,909 tests with one skipped; two failed. One new-main fork metadata fixture lacked an identity announcement and was corrected. The other, an unchanged machine-environment test, exceeded its five-second limit under full-suite load. All nine affected fixture tests then passed with default timeouts; the 57-test migration suite also passed after formatting. The final follow-up changes tests and formatting only.
  • Other unchanged surfaces were freshly verified before final lower-layer integration: protocol 282, SDK 291, Claude Code 364, ACP 319, Pi 167 (one skipped), echo 15, scripted echo 1, bb-guide 5, and server/bridge integration 82. Corresponding typechecks passed.
  • Real codex-cli 0.154.0: isolated session/workspace tests through the configured account pool. All 12 tests passed again with the final integrated implementation, serially: eight basic lifecycle and four cross-process resume tests. The final fixture-only commit does not change provider code. These are live provider calls, with no production BB process or imported store started.
  • The SDK npm version guard passes for unpublished 0.4.99. Changed-file formatting passes under repository configuration; generated snapshots were not manually edited.

Commands:

pnpm exec turbo run test typecheck --filter=@bb/db --filter=@bb/server --filter=bb-plugin-provider-codex --force --continue=always
pnpm exec turbo run test --filter=@bb/integration-tests --force
pnpm exec turbo run test:integration --filter=@bb/agent-runtime --force -- integration.provider-basic.test.ts integration.resume.test.ts -t '^codex' --maxConcurrency 1

Local runs used a writable npm cache. Tests were forced rather than relying on prior CI or cache results. The full server run is not represented as wholly green; the two failures and focused confirmations are reported above. Final-head CI remains the landing gate.

Limits

  • Ownership is inferred from retained records, not an independent durable ownership ledger. Purging all original-owner records still removes that protection.
  • Existing legacy fork/rewind checkpoints whose session was never announced remain usable when no conflicting claim exists. Missing announcement history is not proof of cross-thread ownership.
  • Falling back to an older owned session cannot recover context written into another rollout.
  • A late edit refusal leaves the source session and stored history intact. Discarding an unused staged rewind is best-effort; a provider cleanup failure can leave an unused staged session.
  • Third-party bridges must provide handles unique across that provider's sessions on a host.

Fixes #3461

AGENT GENERATED

@SawyerHood
SawyerHood added this pull request to stack #3504 September 11, 2026 18:50
@SawyerHood
SawyerHood force-pushed the bb/3461-owned-sessions-stack branch 4 times, most recently from d2863af to f9e8804 Compare September 11, 2026 22:59
@SawyerHood
SawyerHood force-pushed the bb/3461-owned-sessions-stack branch from f9e8804 to 7106c27 Compare September 11, 2026 23:05
stack merge was automatically disabled September 12, 2026 00:42

Pull Request is not mergeable

SawyerHood and others added 9 commits September 16, 2026 20:12
`getLastStoredProviderThreadId` took the newest `provider_thread_id` stamped
on any event of the thread, so one mis-stamped event, or a `thread/identity`
recorded after the server itself resumed into the wrong session, redirected
every later resume, archive forward, and interruption stamp into another
thread's provider session (#3461).

Resolution now reads only `thread/identity` events recorded since the
thread's last completed context clear, newest first, and classifies each
announced session against every same-provider, same-host claim:

- owned: no other thread claimed it, or this thread's earliest claim is
  strictly earlier than every other thread's.
- foreign: another thread claimed it strictly earlier. Resolution skips it
  and falls back to the thread's own earlier session, which repairs the
  persisted contamination the 2026-09-11 incident left behind.
- ambiguous: another thread's earliest claim falls in the same millisecond.
  Daemon event batches share one timestamp and rowid order is not durable,
  so resolution stops instead of picking a winner.

`resolveStoredProviderSessions` returns none, owned, foreign, or ambiguous,
so callers can refuse explicitly instead of treating an unavailable session
like a brand-new thread. `classifyStoredProviderThreadClaim` gives edit and
fork paths the same classification for a completion's stamp. A partial
index on identity rows backs the claim lookup.

Co-Authored-By: Claude Code <noreply@anthropic.com>
…er session

Sends, queued dispatch, turn submission, and goal clears resumed whatever
session id the stored history named. With a contaminated stamp that was
another thread's rollout (#3461); with only a foreign or tied identity, a
send silently started a new session without the thread's context.

Dispatch now goes through `requireDispatchableProviderThreadId`. An owned
session resumes and no session since the last context clear starts fresh.
A foreign or ambiguous session answers 409 `provider_session_unavailable`
with the session id and claimant threads in `details`, and the message
"Send /clear to start a new session; history is kept." Nothing is
dispatched and no history is rewritten; a queued row records the same text
as its failure reason. Message-edit rewinds and anchored or tip forks refuse
a completion or source whose session is foreign or ambiguous.

Empty-input fork starts record the daemon's start result as a real
`thread/identity` event, so the resolver sees the session the server itself
established. Test fixtures that established a session through a stamped
`turn/started` alone now announce it through `seedThreadIdentity`, matching
what every bridge emits.

Fixes #3461

Co-Authored-By: Claude Code <noreply@anthropic.com>
bb persists a provider session only from `thread/identity`, resumes only
sessions recorded that way, and now refuses a session another thread
announced first or in the same millisecond. The kit checked that
`thread/start` and `thread/fork` return `providerThreadId` on the result but
never that the session is announced, so a third-party bridge that skipped the
notification would pass conformance and start a new provider session on
every send.

`session/start-identity-announced` and `session/fork-identity-announced`
require a `thread/identity` notification or `thread.identity` delta naming
the returned session for that thread. Every in-repo and example bridge
already passes, and their expected-status maps pin both checks.

The protocol doc now states what resume already relied on: a
`providerThreadId` is a durable handle bb sends to a new bridge process after
a restart, so it must name one session among all of that provider's sessions
on the host and never come from a per-process counter. Real bridges mint
UUIDs, and the published echo example adds a random per-process nonce. The
scripted test bridge keeps readable `prov-N` counters for single-process
suites and gains `uniqueProviderThreadIds`, which the integration suite
enables because it runs several bridge processes against one server store.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Session resolution loaded every thread/identity event since the last
context clear and deduplicated them in JavaScript. Codex announces its
session on every turn, so the cost grew with turn count: 3.9 ms per lookup
on a production thread with 2,464 identity events, against 0.06 ms for the
previous LIMIT 1 query.

Resolution now reads the newest identity event (LIMIT 1), then finds the
session's earliest claim with a single statement whose ORDER BY created_at
LIMIT 1 subquery seeks the partial identity index. It reads further back
only when the newest session belongs to another thread. The ownership
semantics are unchanged: on a copy of the production store the result
matched the previous resolver for all 4,812 threads and 9,620 claim
checks, and a lookup takes about 0.25 ms regardless of history length.

The query-plan test now pins two statements, each using an index, for a
thread with a long identity history.

Co-Authored-By: Claude Code <noreply@anthropic.com>
@SawyerHood
SawyerHood force-pushed the bb/3461-owned-sessions-stack branch from c238f73 to 5509795 Compare September 16, 2026 20:20
@SawyerHood
SawyerHood merged commit b1e83d5 into main Sep 16, 2026
16 checks passed
@SawyerHood
SawyerHood deleted the bb/3461-owned-sessions-stack branch September 16, 2026 21:44
SawyerHood added a commit that referenced this pull request Sep 16, 2026
## Human comments

## What was wrong

When the server considered a thread idle or failed while its daemon
still held a root turn, Stop sent a release that deliberately retained
the turn. The daemon returned the same result as a completed release, so
Stop did nothing and subsequent sends kept failing as competing turns
(#3462). A competing-turn refusal could also fail the server run despite
a stored root turn still running.

## What changed

The daemon reports `activeTurnRetained` on a declined release and
identifies competing-turn refusals with `competing_turn`. Explicit Stop
reconciles the retained turn into the normal interrupt path. Concurrent
awaited stops share the complete operation, including an escalated
interrupt; machine-preservation callers requiring a stopped thread
receive any failure. Context clear cannot proceed while the thread
remains stopping. An explicit Stop also interrupts a turn that starts
while its release is pending.

A declined release no longer marks background commands stopped. A
regression covers an interrupt failure followed by a successful Stop
retry. Competing-turn refusals with a stored running root turn retain
the active run; negative controls still fail ordinary refusals and
refusals without a running root turn. Automatic release protection
remains intact.

Rebased onto main `c663ff1911496870026c779911ff874fb2a43f24`, preserving
the shared daemon Stop/storage-deletion helper, lifecycle ownership, and
writable queues during Stop. Protocol 211 → 212 and plugin SDK 0.4.98 →
0.4.99 cover the contract/declaration changes. SDK, thread guide, and
CLI skill document the behavior. The scripted Codex test server also
accepts main's new configuration arguments so this layer passes
independently.

Existing failure policy remains: an unsuccessful interrupt leaves the
thread `stopping`; the public Stop request can still return success, so
callers must inspect thread status before treating interruption as
confirmed. This is explicitly documented. `requireStopped` callers
reject on the command failure.

This is the bottom layer of #3502#3503#3512. #3503 repairs the
Codex response/notification correlation race; #3512 protects
provider-session ownership.

## How you verified

- Fresh Turbo tests/typechecks for server, daemon, runtime, daemon
contract, SDK, and templates. Runtime 331, contract 59, SDK 113, and
templates 55 passed. Server broad run passed 2,893 tests; three timeout
suites passed a focused rerun, and both inherited Codex fixture failures
passed after the argument parser fix. Stop/checkpoint suites passed 23
tests; the timeout/Stop rerun passed 108. Daemon passed 717 tests; its
two repository-discovery failures came from the filesystem sandbox's
synthetic `/tmp/.git` marker, and all 26 discovery tests passed outside
that sandbox.
- New retained-background-work regression fails before the settlement
guard and passes after it, including successful retry. Existing coverage
exercises idle/error retained turns, concurrent stops,
turn-start/release races, context clear, and escalated interrupt failure
for `requireStopped` callers.
- Real Codex through the configured account pool, fresh isolated BB
store and session home: with `sleep 90` running, injected the server's
erroneous `error` status, then invoked CLI Stop. The thread became idle
in under a second with an interrupted completion, and the next turn
answered `STOP_RECOVERY_CONFIRMED`. In a separate turn, injected idle
during a foreground wait: the competing send recorded `competing_turn`
without `system/error`, the original turn completed to idle, and a
follow-up answered `COMPETING_RECOVERY_CONFIRMED`. Test server and
daemon were shut down; no production store or process was changed. This
verifies provider behavior through normal pool routing, not a separate
scratch login.

Fixes #3462

> AGENT GENERATED

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Server resumes a thread into whichever provider thread its latest event names, with no cross-thread ownership check

1 participant