turn/completed can precede command completion and wedge later turns between active-writer and no-session errors

[pixexid]

Summary

A Codex-backed thread emitted turn/completed while a command-execution item from that same turn was still running. The command later emitted output and item/completed after the terminal turn event. Every subsequent new-turn request then failed in one of two contradictory states:

• thread <provider-thread> already has an active writer
• No active codex session for thread <bb-thread>

Stopping the broken runtime and retrying the exact input did not recover the thread. Provider retry refused because the input had not been accepted, and edit-message could not apply because the rejected request was agent-origin rather than an editable user turn.

Native ordering evidence

For one exact turn/provider-thread:

1. item/started for a long-running commandExecution.
2. turn/completed with status=completed.
3. item/commandExecution/outputDelta for the still-running command.
4. item/completed for that command, reporting success and a multi-minute duration.
5. A later client/turn/requested was rejected as already has an active writer.
6. A recovery new-turn request was rejected as No active codex session.
7. A further exact redelivery again rejected as already has an active writer.

This is a native event/state contradiction: the turn was publicly terminal before all of its writer-owned items terminalized, leaving BB and the provider bridge disagreeing about whether a session/writer existed.

Expected behavior

• Do not emit turn/completed until every command/tool/item belonging to that turn is terminal and no turn writer remains.
• If a provider sends premature completion, BB should delay/fail the terminal event or reconcile outstanding items deterministically.
• A terminal thread must not simultaneously report both active-writer and no-active-session states.
• Stop/restart should clear any orphaned writer ownership without deleting event history or replaying rejected input.
• Rejected agent-origin requests need a supported exact-redelivery recovery path; edit-message is not applicable.

Acceptance

1. A fixture delays command completion beyond provider turn completion; BB does not publish terminal completion first.
2. After true terminalization, one new-turn request is accepted exactly once.
3. Stop/restart clears stale writer ownership while preserving interruption/history.
4. Tests cover the contradictory already has an active writer / No active codex session sequence and prove it cannot persist.

No repository or canonical-state mutation is needed to reproduce; this report is based on bounded native event reads.

[SawyerHood]

---Auto-Generated---

Reproduction & root-cause report: https://get-bb.github.io/reports/issues/2327.html

• Verdict: REPRODUCED · root-cause confidence: high
• Root cause (short): Codex >= 0.146 holds a per-process flock writer lock on each loaded thread; bb's codex bridge reopens rollouts via thread/resume in a fresh app-server (after release, settings/auth rebuild, bridge restart, env move) without waiting for its previous child to exit or recognising a foreign writer, and treats Codex's "already has an active writer" refusal as an untyped bridge error (recovery: null) so the turn is rejected, the thread goes to error, stop/retry resumes the same locked rollout, and (pre-Provider plugins: one API for Codex, Claude Code, pi, and ACP agents #2325) the bridge forgot the thread while the runtime kept it, yielding "No active codex session" on the next turn.
• Proposed fix: (1) In the codex bridge's constructThreadSession and thread/stop{release}, await the old app-server connection's finalized exit before spawning/resuming the replacement so bb never races its own previous writer. (2) Classify Codex's "thread already has an active writer" (regex next to CODEX_ARCHIVED_SESSION_ERROR_PATTERN) into a typed retryable recovery hint (e.g. sessionBusy); have the runtime's actOnRejection retry the construction with bounded backoff and otherwise raise a typed error so the server records a specific system/error ("Codex thread is open in another program") and keeps th…
• Caveats: Revise pass: the verifier's major finding (4a did not run as written) was fixed by making codex-writer-lock.mjs create its cwd and CODEX_HOME, refuse ~/.codex, and warn when auth.json is missing; the step now lists the mkdir + cp auth.json prerequisite and was re-run from scratch at 494f665 (codex-writer-lock.log overwritten in place). 4b was re-run on a fresh dev instance (ports 17285/25285/332…

The report is self-contained: copy-paste repro steps with the test or script inline, expected vs actual output, screenshots where the bug is visual, and code permalinks at the base commit. It was independently re-executed by a second agent (see its Verification section).

AGENT GENERATED: by Claude Code

[Willhong]

Root-cause evidence from a live reproduction (2026-09-11, bb 0.42.0 from source, Codex CLI 0.154.0, macOS 25.6). The active-writer refusal did not start with premature turn/completed; it started with the host daemon binding one Codex thread id to two bb threads.

Observed sequence (bb DB events rows and ~/.codex/sessions rollouts on one machine, same environment env_3rm6y2khyw)

1. 08:00:24 thr_fh9dq7fvsu message edit (system/operation edit_message) staged a rewind; 08:00:26 the thread fork-started on Codex thread 01a08d8d.
2. 09:28:42 thr_r6xinwrds3 spawned on the same bridge and got Codex thread 01a08ddd (its thread/identity at 09:28:45.041).
3. 09:28:44.588 thr_fh9dq7fvsu item/started still carried providerThreadId: 01a08d8d; 09:28:46.250 the matching item/completed for the same item id daa50c5e42-i430 carried 01a08ddd. Every later thr_fh9dq7fvsu event, including its 09:32:30 turn/completed, was stamped 01a08ddd. The rollout file for 01a08d8d shows that turn completing normally at 09:32:30, so the events were routed to the right bb thread but stamped with the other thread's Codex id.
4. 10:12:46 a new turn on thr_fh9dq7fvsu resumed provider thread 01a08ddd (the server picks the latest stored provider_thread_id) and ran a turn inside thr_r6xinwrds3's rollout (user text visible in rollout-…-01a08ddd.jsonl at 01:12:51Z with no corresponding thr_r6xinwrds3 event).
5. 10:19:35 and 10:19:54 thr_r6xinwrds3 turn.submit → thread 01a08ddd-… already has an active writer (~/.bb/logs/host-daemon.38.log lines 1783–1784). lsof showed the lock ~/.codex/thread-writer-locks/01a08ddd-….lock held by the app-server child that thr_fh9dq7fvsu's session had spawned at 10:12:47.

Mechanism (permalinks at 8d32c61e)

• The Codex bridge announces identity twice: the direct thread/identity notification stamped with the bb thread id, and a thread.identity delta translated from Codex's thread/started (

  bb/plugins/provider-codex/src/delta-translation.ts

  Lines 1036 to 1043 in 8d32c61

  	return [
  	{ kind: "turn.open", providerTurnId: handledEvent.params.turn.id },
  	];
  	case "turn/completed": {
  	takeCodexRetryError(state, {
  	threadId: handledEvent.params.threadId,
  	turnId: handledEvent.params.turn.id,
  	});

  ). The assembler emits delta events with an empty thread id (

  bb/packages/provider-bridge-protocol/src/assembler/delta-assembler.ts

  Lines 1971 to 1978 in 8d32c61

  	threadId: UNSTAMPED_THREAD_ID,
  	providerThreadId: "",
  	providerId: options.providerId,
  	rawType: delta.rawType,
  	rawEvent: delta.raw,
  	scope: turnId === undefined ? threadScope() : turnScope(turnId),
  	...(parentToolCallId === undefined ? {} : { parentToolCallId }),
  	});

  ).
• emitTranslatedEvents records an identity for event.threadId when that id is hosted, otherwise it pops a FIFO of pending identity slots and ignores the notification's source thread (

  bb/packages/agent-runtime/src/runtime.ts

  Lines 1181 to 1208 in 8d32c61

  	function emitTranslatedEvents(args: EmitTranslatedEventsArgs): void {
  	for (const event of args.events) {
  	if (event.type !== "thread/identity" || !event.providerThreadId) {
  	continue;
  	}
  	if (args.proc.identity.threadIds.has(event.threadId)) {
  	recordProviderThreadIdentity(
  	args.proc,
  	event.threadId,
  	event.providerThreadId,
  	);
  	continue;
  	}
  	const bbThreadId =
  	threadIdentityRegistry.resolvePendingProviderThreadIdentity(
  	args.proc.identity,
  	);
  	if (bbThreadId) {
  	recordProviderThreadIdentity(
  	args.proc,
  	bbThreadId,
  	event.providerThreadId,
  	);
  	}
  	}

  ). A slot is pushed on every thread/start registration and is never consumed when the identity arrives through the direct notification; a rewind staging slot is never consumed because staging events are suppressed. So after one message edit the queue front was thr_fh9dq7fvsu, and thr_r6xinwrds3's delta identity was recorded onto it.

What this refutes: the "turn/completed before item/completed" ordering in the report is orthogonal. In this reproduction every turn terminalized in order; the contradiction between "active writer" and "no active session" comes from two bb threads owning one rollout.

Deterministic regression and fix in #3460 (runtime.codex-topology.test.ts "keeps a rewound thread's provider identity when another thread starts on the same bridge" fails on main with expected 'codex-fx-…-1' to be 'codex-fx-…-fork-1'). The server-side gap that let the corrupted stamp become the resume target is filed separately.

AGENT GENERATED