Design: report Linux pressure stalls and make shared-provider recovery scope explicit #25918

Problem

A Linux agent workload can remain alive under sustained reclaim while T3's HTTP server is responsive. An OOM-only diagnosis misses that state. With a shared provider such as the Codex app-server, restarting or killing the provider also affects other threads; a scope or PID must not be presented as a single-thread recovery target.

This issue asks for a pressure-aware diagnostic and recovery contract for shared providers. It builds on the resource-attribution proposal in #15449 and pressure-intervention proposal in #21288. The narrower question here is how to report and recover an alive but unresponsive shared runtime without implying that its processes belong exclusively to one conversation. This is a design request, not a claim that every provider stall is caused by memory pressure.

Incident evidence

A capture around 2026-10-25 21:56 UTC from 0.1.3-nightly.20261023.3592 showed:

• Host MemTotal=7,931,904 KiB; agents slice memory.high=4,061,134,848 and memory.max=6,091,702,272 bytes, matching the policy introduced by fix(server): agents run in their own systemd scopes so an OOM kill spares the server #17662.
• More than 138 million cumulative high events; group memory PSI full about 75% over five minutes; all 4 GiB of swap occupied, with host MemAvailable around 1.5 GiB.
• Slice max/oom counters zero and no kernel OOM kills in the preceding 24 hours. The cumulative high counter alone is not a current event rate.
• A 100,000-row in-memory review probe had run for nearly 22 hours, with about 2.57 GiB main Node RSS. Nine REA/npm helper pairs contributed about 1.29 GiB summed RSS and 1.35 GiB swap. RSS can count shared pages more than once; these figures are not a partition of cgroup usage or proof the helpers were unnecessary.
• Native Codex threads shared an app-server, while sampled T3 HTTP responses remained about 9-32 ms.

These are incident observations, not a deterministic reproduction, proof of a memory leak, or proof that pressure alone explains every stalled command. MemAvailable is host-wide and does not exempt a cgroup from its own soft threshold. Sustained working-set demand above a shared memory.high can keep its tasks reclaiming and throttled despite host headroom, without reaching memory.max or producing an OOM kill.

Source inspection at main 79f6cd30c7ed4772d5989d08ce26f64f161dfe49 confirms lazy cached scope initialization, the existing default limits, and continuing shared scopes that deliberately avoid assigning a tool OOM to one thread. Existing Codex interruption and command-terminal cleanup should be reused. This report does not establish a new failure of every current recovery path.

Proposed contract

1. Show the execution environment's effective shared-slice high/max limits, current memory and swap, memory PSI, and sampled event deltas. Distinguish sustained pressure, a confirmed OOM kill, provider control-request failure, and unavailable data. A quiet model or an old high counter is insufficient to declare a pressure stall. Coalesce warnings and show sample age and recovery, including in a bounded incident summary.
2. Present provider identity and the affected thread set separately from proven command ownership. A provider PID or shared scope is not exclusive ownership. Unknown attribution stays unknown; summed RSS must not be advertised as bytes a stop will free.
3. Offer the existing turn/command stop first where ownership is known, with a bounded, visible result. If the control channel is unresponsive or command cleanup cannot be verified, report that uncertainty instead of showing successful recovery. Diagnostics and recovery status should remain available without requiring a successful round trip through the stalled provider.
4. Any escalation to shared-runtime restart must explicitly show all affected threads and require an operator action. Recheck runtime identity and affected work before acting. Reconcile interrupted turns and pending control requests afterward, avoid silently replaying potentially completed tool side effects, and let subsequent work reconnect without restarting unrelated providers or the entire T3 server.

Before implementation, agree on the supported signals, sustained-pressure criteria, control deadlines, and restart semantics. Validate the outcomes for pressure without OOM, an unresponsive shared control channel, recovery before escalation, runtime replacement during an action, and unavailable telemetry. Keep collection cost and retained history bounded. A remote client must see the execution host's state; supported clients and read-only agent diagnostics should report the same condition.

Boundaries and related work

No automatic killing, default-limit removal, new scheduler, personal service configuration, or general-purpose process manager is proposed here. Preserve #17662's protection for T3, SSH and Tailscale. Configurable limits are already proposed separately in PR #22724 and host-managed policy in PR #24717; this issue does not request a competing settings implementation.

• feat(diagnostics): identify the agents and exec commands behind sustained CPU or memory load #15449 covers ownership and sustained-load diagnostics; When agents keep memory pressure high, stop one agent's command instead of stalling every agent #21288 discusses watchdog/admission/swap policies. This issue focuses on the shared-provider recovery outcome and its explicit blast radius.
• [Bug]: Codex sub-agents keep failing after the shared App Server is killed by SIGKILL #16502 concerns recovery after a shared Codex process has died. This case includes a process that remains alive under pressure.
• [Bug]: A two-minute stall of the shared OpenCode server under tool CPU load fails every OpenCode thread, though the server recovers #23962 concerns shared OpenCode stream recovery under CPU starvation; [Bug]: Terminal "timed out negotiating with the code-mode host" on every call while the VM's Codex app-server is backlogged #24322 reports code-mode negotiation timeouts during Codex backlog. Neither proves memory pressure in this incident.
• [Bug]: One unresponsive provider holds all four effect workers and stalls every thread #16636 tracks unresponsive provider controls exhausting orchestration workers; reuse that work rather than introducing another competing timeout mechanism.

No custom build was deployed and no pressure workload or recovery experiment was run for this report.