Fix cross-thread Codex identity binding and rollout writer handoffs - #3460
Open
Willhong wants to merge 5 commits into
Open
Fix cross-thread Codex identity binding and rollout writer handoffs#3460Willhong wants to merge 5 commits into
Willhong wants to merge 5 commits into
Conversation
…thread Delta-assembled thread/identity events carry an unstamped thread id, so the runtime ignored the notification's source thread and popped a FIFO of pending identity slots instead. Those slots were never consumed when the identity arrived through the direct bridge notification, and a staged rewind's slot was never consumed because staging events are suppressed. After one rewind, the next thread/start on the same bridge recorded its Codex thread id onto the rewound thread; both bb threads then resumed one rollout and the second resume failed with "already has an active writer". Resolve the identity target from the source thread first, consume a thread's pending slot whenever its identity is recorded, refuse to bind a provider thread that another hosted thread already owns, and refuse to resume a provider thread that another hosted thread already owns before the bridge touches the rollout.
…-serving children Since get-bb#3025 the bridge waits for a maintenance child to exit before it settles a maintenance request, so a fake that delays every child on SIGTERM let an in-flight rebuild finish before a concurrent discard could mark the replacement closing. A real maintenance child holds no rollout writer and exits promptly; apply the scripted delay only to children that started, resumed, or forked a thread.
This was referenced Sep 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Human comments
What was wrong
Two bb threads ended up bound to one Codex rollout, so the second thread's resume was refused with
thread <id> already has an active writer(#2327). The binding went wrong inside the host daemon, not in Codex: the Codex bridge announces a thread's identity twice, once through the directthread/identitynotification (stamped with the bb thread id) and once through thethread.identitydelta that the assembler emits for Codex'sthread/started(stamped with an empty thread id).emitTranslatedEventsignored the notification's source thread for the second event and popped a FIFO of "pending identity" slots instead. Those slots were never consumed when the identity arrived through the first path, and a staged rewind's slot was never consumed because staging events are suppressed. After one message edit, the nextthread/starton the same bridge recorded its Codex thread id onto the rewound thread. Every later event of that thread was stamped with the foreign id, the server resumed it into the other thread's rollout, and the rollout's owner then failed with the active-writer error.Investigation on a live machine:
thr_fh9dq7fvsu(edited at 08:00, provider thread01a08d8d) started stamping01a08dddtwo seconds afterthr_r6xinwrds3started with01a08dddon the same environment bridge; the rollout of01a08dddthen received a turn fromthr_fh9dq7fvsuat 10:12 andthr_r6xinwrds3was refused at 10:19.What changed
packages/agent-runtime/src/runtime.ts: athread/identityevent is bound to the notification's source thread first, then the event's own thread id, and only then a pending slot. An identity that would bind a provider thread another hosted thread already owns is dropped with a stderr line instead of being recorded.resumeThreadrefuses, before touching the bridge, to resume a provider thread that another hosted thread owns, so the daemon no longer collides on Codex's writer lock in that case.packages/agent-runtime/src/runtime-thread-identity.ts: recording a thread's identity consumes that thread's pending slot, so slots no longer accumulate.plugins/provider-codex/src/bridge: carries the three commits of Fix Codex rollout writer handoff races #2578 (author preserved) rebased onto currentmain. The writer-handoff serialization those commits added toconstructThreadSessionandreleaseSessionalready landed in Fix Codex archive undo synchronization #3025, so the bridge keepsmain's implementation; what this brings over is the bounded active-writer retry ladder with stderr diagnostics, the actionable guidance on a persistent writer, thewriter.lockfixture infake-codex-app-server.mjs,bridge-process.test-support.ts, and the writer-lock regressions.fake-codex-app-server.mjs: the scripted SIGTERM delay now applies only to children that served athread/start,thread/resume, orthread/fork. Since Fix Codex archive undo synchronization #3025 the bridge awaits a maintenance child's exit before settling a maintenance request, and a fake that delayed every child let an in-flight rebuild finish before a concurrent discard could mark the replacement closing. A real maintenance child holds no writer and exits promptly.No host-daemon wire contract changed, so
HOST_DAEMON_PROTOCOL_VERSIONis unchanged. #2328 (recovery storms on a dead session) is not addressed here. The server-side gap that let the corrupted stamp become the resume target is tracked in #3461.How you verified
New regressions, proven red before the fix:
runtime.codex-topology.test.ts"keeps a rewound thread's provider identity when another thread starts on the same bridge": with the old runtime,t1's provider thread id becamet2's (expected 'codex-fx-…-1' to be 'codex-fx-…-fork-1').runtime.codex-topology.test.ts"refuses to resume a provider thread that another hosted thread already owns".runtime-thread-identity.test.ts"consumes a thread's pending identity slot when its identity is recorded".bridge.writer-lock.test.tsfrom Fix Codex rollout writer handoff races #2578 (8 tests), including the concurrent-discard case that fails against the unrestricted fake delay.Commands run on this branch (based on current
main):pnpm exec turbo run test --filter=bb-plugin-provider-codex --filter=@bb/agent-runtime --filter=@bb/host-daemon --force(28/274, 22/321, 46/572 passing)pnpm exec turbo run typecheck --filter=bb-plugin-provider-codex --filter=@bb/agent-runtime --filter=@bb/host-daemonpnpm exec turbo run lint --filter=bb-plugin-provider-codex --filter=@bb/agent-runtimepnpm exec oxfmt --check plugins/provider-codex/src/bridge packages/agent-runtime/src(the only report is the pre-existingbridge.resume-hydration.test.ts, untouched here)Fixes #2327
Supersedes #2578