[architecture][memory] Make 32 concurrent tool-recall sessions protocol-safe and memory-bounded #820

Summary

OpenCodex should remain TypeScript/Bun-based, but it needs an explicit concurrency and memory architecture for the real workload we must support:

• 32 sustained independent agent sessions
• 64-session burst
• repeated model stream -> 1..N parallel tool/MCP calls -> tool results -> API recall
• no OpenCodex-imposed serialization below that target
• no loss of Codex / Responses / Chat Completions / Anthropic / MCP compatibility

This is not a proposal to raw-passthrough provider bytes. OpenCodex is a semantic protocol translator. The required change is to keep that translation while replacing unbounded or duplicated retention with bounded state, byte accounting, and stream-lifetime leases.

Audit baseline: dev@18545f87e92b856be180ba01e8b2c44d67163879.

This issue is related to the field symptoms in #314 and #509, but is not a duplicate. Those issues discuss observed Windows memory behavior and imperfect Bun counters. This issue defines the architecture, hypotheses, implementation slices, and acceptance harness needed to distinguish OpenCodex-owned retention from Bun allocator/runtime retention under multi-session tool recall.

Hard requirements

Concurrency contract

sustained independent sessions: 32
burst independent sessions:     64
global emergency active cap:     96
same-session active turns:        1
same-session queued turns:        0 by default

Different sessions must remain parallel. Only overlapping turns for the same logical session/thread may be rejected or serialized. Tool execution performed by the external client must not keep an OpenCodex model-turn lease occupied after the model stream reaches a terminal.

Protocol invariants that must not be removed

Any memory fix must preserve all of the following:

1. Codex namespace/MCP tools are flattened for chat providers and restored as { namespace, name } on the return path.
2. Codex custom/freeform tools such as apply_patch are exposed to chat models through a temporary { input: string } function schema, then restored as custom_tool_call, not function_call.
3. tool_search remains distinct from an ordinary function call.
4. call_id, Responses item_id, output_index, and Chat Completions tool indexes remain stable and correctly correlated.
5. The complete lifecycle remains valid:
   output_item.added -> arguments.delta* -> arguments.done -> output_item.done -> response terminal.
6. No-argument calls finalize as {} rather than an empty invalid JSON string.
7. Malformed or truncated arguments must not be reported as a completed usable call.
8. Reasoning signatures, redacted thinking, tool results, image content, continuation metadata, and provider-specific replay behavior remain intact.

Raw provider SSE passthrough would violate these contracts.

Important correction: the existing parallel-tool contract is deliberate

The repository already solved provider-interleaved OpenAI Chat tool deltas in devlog/_fin/260709_parallel_tool_calls/ and tests/openai-chat-parallel-stream.test.ts.

The current design is:

1. openai-chat keys provider fragments by index/id and buffers all pending calls while text/reasoning continues.
2. At a valid terminal boundary, it emits each completed call as one non-overlapping:
   tool_call_start -> one full tool_call_delta -> tool_call_end sequence.
3. bridgeToResponsesSSE() intentionally holds one currentToolCall, because the canonical AdapterEvent contract is sequential and non-overlapping.
4. Finish-less EOF with pending calls fails closed rather than fabricating a completed call.

Therefore a keyed, live, multi-call bridge is not a required P0 change and must not be introduced merely to reduce memory. It would reopen previously closed correctness hazards: text/reasoning barriers, item ordering, partial-call failure, and custom-tool unwrapping.

The measured optimization path should be:

• preserve the sequential canonical contract;
• replace fragment-by-fragment string concatenation with chunk arrays and byte counters where it matters;
• clear each adapter call as soon as its atomic sequence has been consumed;
• add per-call, per-turn, and process-wide argument-byte metrics/limits;
• consider an optional atomic tool_call_complete event only if profiling proves material duplicate reassembly between an adapter that already has a final snapshot and the bridge.

A keyed-live event redesign remains a separately reviewed alternative, not the default plan.

Current confirmed pressure points

1. tee() still creates two independently paced consumers

The native Responses passthrough path still has a client branch and a background inspection branch created from upstreamResponse.body.tee() in src/server/responses/core.ts.

The existing macOS RSS plan measured a large external-memory difference between rate-mismatched tee and one-reader variants. The remaining work is to make the one-reader path protocol-safe, runtime-qualified, compatible with required payload rewrites, and valid for the 32-session workload.

2. The inspector is not fully bounded on current dev

createSseInspector() currently:

• appends decoded chunks into an unterminated string without a frame-byte cap;
• parses the same payload in multiple consumers;
• retains every reconstructed response.output_item.done entry until completion;
• does not clear the reconstruction map immediately after the completed response is copied.

The repository already contains a concrete Phase 2 plan for a 1 MiB frame cap, 256 reconstructed items, single parse, and post-callback cleanup. Add a reconstructed-byte cap as well.

3. Tool conversion is correct, but its retained bytes are not explicitly bounded

Confirmed current behavior:

• OpenAI Chat buffers all parallel calls until a valid flush boundary.
• PendingToolCall.args is grown by string concatenation.
• The Responses bridge accumulates the active call again to create required .done and final item snapshots.
• custom/freeform input may be repeatedly decoded from a growing wrapper.
• Responses -> Chat outbound retains per-index argument strings until output_item.done/terminal to avoid duplicate append-only deltas.

These states are protocol-motivated. The open question is their measured byte cost, not whether they can simply be deleted.

4. Continuation bytes have a single-entry escape hatch

src/responses/state.ts has a 64 MiB aggregate accounting cap, but eviction currently stops while states.size > 1. One oversized final entry can remain above the cap. Forced store:false expansion also retains full expanded input for each response ID, which can grow approximately quadratically by chain length.

A final design needs a hard-cap policy plus explicit oversize/tombstone behavior, and likely parent-linked deltas and/or content-addressed large payloads so continuity never fails silently.

5. Cursor blobs are count-bounded, not byte-bounded

src/adapters/cursor/native-exec.ts stores content-addressed Uint8Array values in a process-global map with:

• 15-minute TTL
• 4,096-entry cap
• no per-blob byte cap
• no process-wide byte cap
• no active-request pin/lease accounting

Count-only eviction does not bound memory when one or more tool/history blobs are large. Selection/budgeting should happen before insertion, and live blobs need pins so byte eviction cannot remove data still needed by an active Cursor turn.

6. The eager relay is bounded per stream, but not globally or precisely

src/server/relay-eager.ts currently defaults to:

• 8 MiB client queue per stream;
• 15-second post-cancel drain;
• 32 MiB post-cancel drain bytes per stream;
• approximate queue accounting that resets queuedBytes to zero on pull().

At 32 sessions, configured per-stream limits can multiply into a large process-wide envelope. The target design needs process-wide byte accounting and either pull-paced reading or exact queued-byte release.

7. Active-turn tracking is not a coordinator

src/server/lifecycle.ts tracks only a Set<AbortController>. It does not model:

• logical session lanes;
• admitted/bootstrap/committed phases;
• global and weighted byte permits;
• provider/account active load;
• queue age;
• stream-lifetime ownership.

A permit must be released on terminal/cancel/error/shutdown, not when the handler returns a Response object.

8. Account selection is not active-load-aware

src/codex/pool-rotation.ts implements sticky/smooth weighted selection but does not include current active turns, TTFT EWMA, or transient-failure pressure. Under synchronized recall bursts, many sessions can select the same account before outcomes update.

Affinity should remain authoritative when continuation is account-scoped, but portable requests should prefer an eligible target with lower active saturation.

9. Bundled Bun 1.3.14 cannot be assumed safe for the target relay

The package pins Bun 1.3.14. Two upstream fixes directly affect the proposed shape:

• http: couple fetch() receive backpressure to JS body consumption (h1/h2/h3) oven-sh/bun#29831: fetch receive backpressure coupled to JS body consumption; merged after 1.3.14.
• Fix crash when a client aborts a streaming response under backpressure oven-sh/bun#32120: crash when a client aborts an async streaming response under backpressure; merged after 1.3.14, with a 20-concurrent-client regression test.

Also, oven-sh/bun#33368 remains open and documents why process.memoryUsage().heapUsed can exceed heapTotal dramatically on current Bun. heapUsed alone must not be used as the leak discriminator.

The release must pin an exact verified Bun.revision, not trust a semver guess or moving canary.

Expanded hypotheses and discriminating measurements

Target architecture

request
  -> logical session lane (one active turn per session)
  -> global active-turn permit + weighted request-memory permit
  -> affinity-aware, active-load-aware account selection
  -> bootstrap-only retry / circuit policy
  -> provider adapter
       -> provider-specific parallel-call assembly where required
       -> sequential canonical AdapterEvent contract
  -> protocol bridge with bounded current-item/full-response accounting
  -> one qualified upstream reader
       -> bounded inline inspector
       -> required payload rewrite
       -> downstream response
  -> idempotent terminal cleanup releases every lease/pin/reader

The one-reader invariant means one read of the upstream body, not zero transformation. Inspection and payload rewriting must be stages on the same semantic data path.

Proposed PR series

PR 1 — Acceptance harness and observe-only metrics

Add the real recall-burst harness before changing behavior:

• 32 sessions, 10 rounds each;
• 1..8 parallel function/MCP/custom/tool-search calls per round;
• provider-interleaved OpenAI Chat deltas while preserving the canonical sequential output contract;
• random tool latency 10..500 ms;
• barrier-synchronized recalls;
• slow consumers;
• 25% random cancellation;
• pre-first-byte resets, 503 outage, account 429;
• application-owned byte/readers/turn metrics and RSS samples.

No production behavior change in this PR.

PR 2 — Bounded, single-parse inspection

Implement the existing Phase 2 plan:

• 1 MiB unterminated frame cap with correct SSE resynchronization;
• one parse per payload;
• 256 reconstructed output items plus a reconstructed-byte cap;
• immediate map cleanup after synchronous continuation copy;
• no payload contents in overflow warnings.

PR 3 — Tool/output byte accounting without changing the canonical contract

• instrument adapter pending-tool bytes, bridge active-tool bytes, final output bytes, and outbound per-index bytes;
• replace hot-path args += fragment with chunks + join-once where A/B tests show lower peak/CPU;
• add explicit per-call/per-turn/process budgets and truthful oversize failure;
• make custom-tool JSON-string unwrapping incremental;
• clear adapter call buffers as each atomic sequence is consumed;
• optionally add an atomic tool_call_complete event only if it materially removes duplicate reassembly and passes the full protocol matrix.

PR 4 — Runtime qualification and protocol-safe one-reader relay

• add reproducible fixtures for Bun fetch receive backpressure and async-pull abort safety;
• record exact validated Bun.revision capabilities in the release manifest;
• keep legacy tee as an explicit compatibility fallback;
• compose inspection and required payload rewrites into one reader path;
• preserve synthetic failed-tail semantics;
• default post-cancel drain to zero, or a very small opt-in byte/time budget;
• add a process-wide stream byte budget.

PR 5 — Turn coordinator and session lanes

• replace controller-only tracking with metadata-rich TurnScope;
• one active turn per logical session, independent sessions parallel;
• global active and request-memory weighted permits;
• release only at stream terminal/cancel/error/shutdown;
• expose active/queued/by-phase/by-provider/by-account metrics.

PR 6 — Continuation, Cursor blob, and request-body byte budgets

• hard aggregate and per-entry bounds without the single-entry escape hatch;
• explicit oversize/tombstone behavior;
• parent-linked continuation deltas or content-addressed payloads;
• Cursor blob total bytes, per-blob limit, pins, expiry and eviction metrics;
• request cost admission before retaining large decoded bodies.

PR 7 — Bootstrap-only retry, circuit breaker, and active-aware account selection

• retry only before downstream commit;
• max two attempts / bounded elapsed time / full jitter;
• no retry after first downstream byte;
• provider+model transient circuit, account-scoped quota handling;
• affinity-safe active-load score.

Acceptance criteria

Correctness

• 32 sustained and 64 burst independent sessions complete without OpenCodex-imposed serialization below target.
• No call IDs, item IDs, indexes, namespaces, names, arguments, outputs, or session state cross-contaminate.
• Provider-interleaved fragments for at least 32 OpenAI Chat calls still emerge as the existing non-overlapping canonical sequences.
• Function, MCP namespace, custom/freeform, tool_search, image-bearing result, malformed args, no-arg call, and interrupted-call cases all preserve existing contracts.
• transport.postCommitRetryAttempts remains exactly zero.

Cleanup

After every wave, within the declared teardown window:

turns.active                    -> 0
turns.queued                    -> 0
stream.activeReaders            -> 0
stream.pendingBytes             -> idle baseline
stream.inspectorBytes           -> idle baseline
tool.pendingArgumentBytes       -> 0
tool.bridgeActiveBytes          -> 0
memory.requestCostInUse         -> idle baseline
memory.continuationBytes        -> configured bound
memory.cursorBlobPinnedBytes    -> idle baseline
budget waiters                  -> 0

Memory outcome

Success is not defined as RSS returning immediately to process-start values. Success requires:

1. all OpenCodex-owned byte/readers/lease counters return to the expected idle envelope;
2. repeated identical workload waves do not retain additional application state;
3. after warm-up, RSS does not maintain a continuously positive per-wave slope.

If application-owned state returns to baseline while RSS remains on a stable high plateau, classify the residual as runtime/allocator retention and attach the exact Bun revision and fixture result.

Explicit non-goals

• raw provider-byte passthrough for translated routes;
• reducing normal concurrency below 32 sessions;
• setting a default per-account concurrency of 1 or 2;
• treating Bun 1.3.14 as qualified for the new relay;
• using heapUsed alone to attribute Windows memory;
• replacing the existing sequential canonical tool-event contract without measured justification;
• a Go/Rust rewrite;
• one giant implementation PR.

Source pointers

OpenCodex pinned source and prior design records:

• https://github.com/lidge-jun/opencodex/blob/18545f87e92b856be180ba01e8b2c44d67163879/src/types.ts
• https://github.com/lidge-jun/opencodex/blob/18545f87e92b856be180ba01e8b2c44d67163879/src/responses/parser.ts
• https://github.com/lidge-jun/opencodex/blob/18545f87e92b856be180ba01e8b2c44d67163879/src/bridge.ts
• https://github.com/lidge-jun/opencodex/blob/18545f87e92b856be180ba01e8b2c44d67163879/src/adapters/openai-chat.ts
• https://github.com/lidge-jun/opencodex/blob/18545f87e92b856be180ba01e8b2c44d67163879/tests/openai-chat-parallel-stream.test.ts
• https://github.com/lidge-jun/opencodex/blob/18545f87e92b856be180ba01e8b2c44d67163879/devlog/_fin/260709_parallel_tool_calls/001_audit_round1.md
• https://github.com/lidge-jun/opencodex/blob/18545f87e92b856be180ba01e8b2c44d67163879/devlog/_fin/260709_parallel_tool_calls/010_phase1_stream_assembly.md
• https://github.com/lidge-jun/opencodex/blob/18545f87e92b856be180ba01e8b2c44d67163879/src/chat/outbound.ts
• https://github.com/lidge-jun/opencodex/blob/18545f87e92b856be180ba01e8b2c44d67163879/src/claude/outbound.ts
• https://github.com/lidge-jun/opencodex/blob/18545f87e92b856be180ba01e8b2c44d67163879/src/server/responses/core.ts
• https://github.com/lidge-jun/opencodex/blob/18545f87e92b856be180ba01e8b2c44d67163879/src/server/relay.ts
• https://github.com/lidge-jun/opencodex/blob/18545f87e92b856be180ba01e8b2c44d67163879/src/server/relay-eager.ts
• https://github.com/lidge-jun/opencodex/blob/18545f87e92b856be180ba01e8b2c44d67163879/src/server/lifecycle.ts
• https://github.com/lidge-jun/opencodex/blob/18545f87e92b856be180ba01e8b2c44d67163879/src/responses/state.ts
• https://github.com/lidge-jun/opencodex/blob/18545f87e92b856be180ba01e8b2c44d67163879/src/adapters/cursor/native-exec.ts
• https://github.com/lidge-jun/opencodex/blob/18545f87e92b856be180ba01e8b2c44d67163879/devlog/_plan/260731_macos_rss_retention/020_inspection_bounds.md
• https://github.com/lidge-jun/opencodex/blob/18545f87e92b856be180ba01e8b2c44d67163879/devlog/_plan/260731_macos_rss_retention/030_macos_single_reader.md

Runtime and comparative evidence:

• fetch().body piped through TransformStream does not propagate backpressure oven-sh/bun#28035
• http: couple fetch() receive backpressure to JS body consumption (h1/h2/h3) oven-sh/bun#29831
• Native segfault in RequestContext.onAbort when returning async ReadableStream + client aborts oven-sh/bun#32111
• Fix crash when a client aborts a streaming response under backpressure oven-sh/bun#32120
• process.memoryUsage: source heapTotal and heapUsed from the same heap oven-sh/bun#33368
• Unbounded memory growth: default InMemoryEventStore is never evicted and cannot be disabled punkpeye/mcp-proxy#72
• https://github.com/Portkey-AI/gateway/blob/669825cbe89ee51569918b8f78a9db486fd69dd4/src/handlers/streamHandler.ts
• https://github.com/musistudio/claude-code-router/blob/4a152d959c016b476220339e856c9f4f94624c42/packages/core/src/gateway/request/pipeline.ts

Definition of done

Close only after the protocol matrix and 32-session recall harness pass on every supported OS/runtime combination, the application-owned state is bounded and observable, and the bundled Bun revision is explicitly qualified for the selected default relay mode.