Repository navigation
[Bug][Windows] live Bun 1.4.0 listener stalls health checks and escapes child-exit-only recovery on 2.77.0 #6671
Description
Client or integration
Codex CLI
Area
Service lifecycle
Summary
On Windows with OpenCodex 2.77.0 and its bundled Bun 1.4.0, the process can remain alive and keep port10100 LISTEN while /healthz fails to answer within15seconds. Child-exit-only respawn therefore never activates. A same-day replacement process also had a transient status/health timeout around30minutes after startup. These observations establish a liveness/recovery gap, not the exact allocation or blocking stack.
Reproduction
The incident is intermittent, not a deterministic minimal reproduction yet:
- An installed proxy handled concurrent OAuth-backed Codex work. About40 jobs started around the incident; preceding successful requests included input-token counts up to793581. This is a token count, not a measured HTTP-body size.
- After about140minutes, the operator observed about4GB process memory and a /healthz timeout exceeding15seconds, while the process and listener remained alive. Killing the process and running service repair restored requests.
- In the replacement process, compare supported memory snapshots below. Retained continuation/app-owned bytes fell while external rose; the process-local60second watchdog had one interval around144seconds.
- Later under lighter load,96 probes all returned200 in approximately215–453ms. No claim that the timeout is continuously reproducible.
Version
@bitkyc08/opencodex2.77.0; bundled Bun1.4.0 revision34cbb9a40b4bd1bd767d134a7065e66c2432a676.
Operating system
Windows11 x64. Exact OS build was not recorded at the incident.
Provider and model
OAuth-backed Codex traffic. No provider-specific allocation cause is claimed.
Logs or error output
Same replacement process; scalar byte counters from ocx observe memory --json:
| Uptime seconds | RSS | external | responseState.totalBytes | appOwnedBytes.retainedBytes | active turns |
|---|---|---|---|---|---|
| 2207 | 699408384 | 429512405 | 81974176 | 88491604 | 12 |
| 2417 | 697417728 | 592831221 | not transcribed | not transcribed | 8 |
| 2717 | 634339328 | 677157512 | 12852131 | 21923148 | not transcribed |
| 3336 | 592379904 | 735611631 | 18141122 | 27673078 | 12 |
The replacement also reported253spillWrites and246spillWriteFailures at the first sample, including242timeout_memo_refusal classifications. Later it reported796successful spill writes, the same246failures, zero consecutive failures and healthy. Historical failures are not presented as an ongoing outage.
Provider debug was enabled; usage/injection/claude debug disabled. Process priority wasAboveNormal, so this does not reproduce the priorBelowNormal case.
Source observations:
- src/server/memory-watchdog.ts warns above4GiB but intentionally does not respawn; its timer shares the stalled event loop.
- Current Windows response spills await ACL subprocesses asynchronously. The old synchronous-ACL issue is not assumed to remain.
- request-decompress.ts still performs synchronous inflate, decoding, JSON.parse and JSON.stringify; its per-request byte cap is not an aggregate process cap. These are candidate pressure sources, not a captured incident stack.
Screenshots and supporting files
The original process was restarted before a heap/stack capture was taken. A separate Rust liveness observer is now being deployed to preserve Windows process-memory/CPU/health timing around any recurrence and restore service after three expired probes. That mitigation does not establish that the native retention is fixed.
Redacted configuration
{"port":10100,"platform":"win32","serviceBackend":"Task Scheduler","auth":"OAuth","routingPolicyProfiles":0}Checks
- I searched existing issues and documentation, including#314,[Bug] Bun.exe is consistently using up to 2GB of RAM #6135,[Bug][Windows] Continuation spill failures accumulate behind healthy readiness after #3011 (observed on 2.39.0) #3522,Windows service BelowNormal priority causes Offline health timeouts under CPU contention #3634 and the Windows memory troubleshooting page. The page's Bun1.3.14 discussion is not assumed to explain1.4.0. Reporter-confirmed recovery in#6135 does not prove this recurrence is the same cause.
- I removed secrets, tokens, account details, request credentials, prompts, request IDs, usernames and personal paths.
Activity
Ingwannu commentedon Oct 7, 2026
This report establishes a recovery blind spot: a process can remain alive/listening without answering health checks, so an exit-only supervisor cannot detect that condition. The source observation and reported delayed watchdog tick are consistent with event-loop starvation, but neither identifies the blocking stack nor proves an app-owned leak; the external/RSS/retained counters are different accounting views, and the original process was not captured.
Please keep a recurrence record to bounded scalar telemetry: exact Windows build and executable/Bun version, uptime, health probe start/end/status, CPU time, RSS/external/app-owned bytes, event-loop delay and active-turn count on the same timeline. Note whether the proxy exited or only the observer restarted it, and whether the replacement restored health. Avoid raw heap dumps, request bodies, credential files or full unredacted logs in this public issue.
An out-of-process health-recovery policy needs an explicit owner decision: failed probes during load are not proof that in-flight upstream work was not sent, and process recovery must not imply automatic replay of ambiguous turns. No stress workload, supervisor change or service restart was performed here; leaving this open for a reproducible pressure/liveness case.
mashfromband commentedon Oct 7, 2026
A naturally occurring recurrence on 2026-10-07 was captured by the external Rust observer, and the owner-authorized recovery restored a different process. The cause is still unconfirmed. This supervisor does not replay interrupted requests.
The affected PID was 71684 (creation identity held constant across the three failures; process start 12:01:01.824 JST). The installed package is still 2.77.0. Recent live-process diagnostics report bundled Bun 1.4.0, revision
34cbb9a40b4bd1bd767d134a7065e66c2432a676; the affected process was not asked for its runtime revision before termination. The current host reports Windows 11 Pro 25H2, build 26200.9457.These are Windows process counters, not Bun's external or app-owned counters. Those in-process counters and an event-loop stack were unavailable for this recurrence.
The failed probe starts were approximately 14:34:00.727, 14:34:30.939 and 14:35:00.951 JST. The process was still alive during those reads; it did not crash. The observer requested termination at 14:35:20.954, confirmed exit at 14:35:21.194, requested the existing service task at 14:35:22.051, and confirmed health from replacement PID 42664 at 14:35:33.537. Thus recovery took about 12.3 s from confirmed termination. A subsequent sample at 14:36:29.893 answered in 1 ms.
Persisted usage records with request timestamps in 14:32 and 14:33 contain 42 and 41 entries respectively (not a complete ingress count, and not payload byte measurements). The falling private bytes and roughly 3.36 CPU seconds consumed across the last 90 seconds of the old process do not establish a CPU-heavy JSON loop as the blocking cause.
A separate five-minute passive observation of a later process produced 291 health probes: 287 identity-verified 200s and four 1.5 s deadlines. Wait-chain snapshots around the slow/deadline samples showed no process-wait chain or detected cycle. These later snapshots are not the stack of the 14:34 stall, and WCT does not cover every wait primitive. Request decompression, storage waits and runtime/transport retention remain hypotheses. Isolated synthetic pressure measurements are being developed; there is no source-fix claim yet.
Isolated pressure controls (2026-10-07)
A Rust client reused the unchanged repository test child
native-main-owner-child.ts, with private fixture homes, a cleared/whitelisted child environment, and no real accounts. That fixture intercepts its synthetic upstream hosts, so these runs include inbound parsing, adapter/state handling and spill publication, but exclude native upstream fetch transport. The fixture uses the 2.78.0 checkout with the samerequest-decompress.tsSHA-256 as installed 2.77.0 (C35722F47F65343851D031BF1B3896DF25FCE79124A70FB3C0D24E276A073E5C) and the same bundled Bun 1.4.0 executable. Payloads were uncompressed JSON containing synthetic ASCII text, not recorded user traffic. Windows private/working-set values remain separate counters.All fixture health samples had matching process identity and HTTP 200. The large burst demonstrably creates transient multi-GB pressure and sub-second/roughly 1.4-second liveness delays, but it did not reproduce the 20-second failures. It does not prove that these were the original request sizes, a sustained leak, or the historical stall's cause.
The post-listen interval of the 32MiB×20 CPU profile contains byte-buffer growth/copying, request decode/parse/stringify, synchronous spill serialization/writes, and creation of asynchronous ACL subprocesses. Those frames identify work performed during this control; sampling weights alone do not establish the older wait or identify the owner of every retained allocation. The 8MiB×40 run did not publish a profile before its bounded stop fallback, so no profile result is claimed for that run. Each test child was stopped through its own held handle; the production listener was not used as the pressure target.
Plain HTTP adapter controls (2026-10-08)
The real-network control now uses a Rust loopback HTTP upstream instead of the helper's intercepted hosts. The effective provider and a complete preflight exchange are verified before pressure. This is plaintext HTTP through openai-chat, not ChatGPT OAuth, TLS, HTTP/2 or Codex passthrough; healthy results do not exclude a stall on those paths. No real account or credentials were copied. No production pressure or forced GC was used.
A preliminary 8MiB x12 run had eight upstream-reset refusals and cancelled on the first downstream SSE data, which could precede upstream output. It is not treated as a passing after-output cancellation test.
In a corrected six-wave control, each body was 8388668 bytes, with twelve clients per wave, alternating normal completion and closing after an upstream-only content marker. The upstream paused 2ms per16KiB read and held output for5s. All72 client HTTP statuses were200. Including preflight, all73 upstream bodies and first output chunks arrived:37 completed,36 stream writes saw connection resets, and no mid-upload failure, capacity refusal, other error or active connection remained. All864 identity-matched health probes returned200; max431ms. Peak Windows private bytes were2439884800 and working set1736638464. This artifact predates downstream terminal-event validation, so these are HTTP/upstream observations.
During about120s of idle observation, Bun external stayed near816.5-816.8MB and heapUsed near841.4-841.7MB, while registered retention was163644 bytes, continuation bytes12082, active turns0 and upstream sockets0. These counters are different; an idle plateau does not establish a persistent leak. A later release control also verifies successful response.completed versus response.failed:8MiB x4, two waves, all8 client HTTP200,4 successful completed terminals and4 marker closes; upstream5 completions including preflight and4 stream resets, no other errors/active sockets.
The durable spend journal is active in the installed2.77.0 process. Its checkpoint at03:42:43.652JST had278 scopes and16384 remembered sends. The surrounding identity-matched probes were0-174ms; this observed compaction did not reproduce a long stall. The installed and checkout spend-reservation-ledger.ts hashes match. Passive30-minute windows returned1635/1674 and1768/1776 identity-matched200s; the remaining probes reached the1.5s observer deadline. Those are not20s watchdog failures.
The optional Rust control is saved on the reporter's fork at commit9c7251842 (agent/todo1936-windows-cause-probe). Its four focused tests, Clippy and release passed; an independent model review has Must0. This is diagnostic source, not a cause-fix PR. The historical blocking stack is still unavailable and this issue remains open.
Synchronous model-list runtime-selection controls (2026-10-08)
There is now an isolated reproduction of a greater-than-20-second liveness failure in the model-list path. It identifies a concrete blocking path in both source versions, but does not establish that this path caused either historical production hang or the observed memory growth. No cause-fix PR or issue closure is claimed.
A Rust control uses the unchanged 2.78 test helper, cleared/whitelisted child environment, synthetic homes and an ephemeral loopback port. For the 2.77 comparison, a private scratch mirror contains a byte-identical helper with source/dependency junctions to the existing installation. CPU-profile module URLs resolve to installed 2.77 source. The fixture performs no writes to that installation; the links are not an ACL-based read-only boundary. The provider is configured as an unavailable loopback provider with no real credentials. Other egress is not blocked or audited.
After model-list warmup and a 16s memo-expiry wait, three fake version candidates each take about 7.5s: the first two return invalid version output and the third returns a valid version. Only one GET /v1/models is sent during measurement.
The two older artifacts do not record whether TimedOut occurred during connection or reading. Each retains the same process handle, PID and creation identity; the process was alive at the failed observation and subsequently returned matching health. Fake-event logs record sequential role 0/1/2 durations of 7501/7500/7500ms (2.78) and 7501/7500/7501ms (2.77). The prototype helper hashes are respectively 6EF2DCE9AA5FED3348FC0284B0EAC20AEA5C5EF4B71D0DE7AED04919ED758EC6 (spawn then Job assignment) and 7D7A6AD8E0A873449C262603A0779548DEE12859E0E9DFCF23723A2FD90EFC97 (suspended start before Job assignment).
The profiles attribute the synchronous chain to visibleNativeSlugs -> nativeOpenAiSlugs -> catalogNativeSlugs -> readCurrentCatalogOrCache -> loadBundledCodexCatalog -> resolveAndPersistCodexRuntime -> probeVersion -> execFileSync -> spawnSync. Installed-source lines include bundled.ts:239/518 and runtime.ts:468. The 2.77 native spawn frame has about 23.48s of sampled wall time; this is not consumed CPU time. The bundled loader resolves the runtime before looking up its 60s catalog memo, and the runtime memo is 15s with an 8s bound per candidate, not an aggregate bound.
A separate installed-source control has each fake version parent print/flush a valid version, create a descendant inheriting stdout, and exit. The writer is configured to live for 25s. Five version attempts occurred about 8s apart, including differently spelled candidates resolving to the same fake executable. This is consistent with bounded per-candidate failures accumulating while EOF is delayed, rather than an unbounded single-probe timeout. Writer termination is not logged, so it is not inferred from the start record alone.
That run's /v1/models returned 200 after 41209ms. Two sequential health probes failed at 20004ms and 20000ms with error_stage=read, while the same held process remained alive; later health matched and the child exited successfully. A read-stage timeout does not establish zero bytes received or the exact stall onset/end. Its profile has the same synchronous chain and about 40.92s of sampled native-spawn wall time. The prototype hash is 32B3E19EDB4CB202EEBDC4CBB78730E3B8EE1B7C0D664EB9FB2D4969A1282212.
The reusable optional Rust control is saved at commit 3478ef996. Three focused nextest contracts passed. Clippy/release passed after the final one-line write-stage labeling change, which does not affect the recorded read outcomes. Typecheck and privacy scan passed. Full Bun tests were not run: only an optional Rust diagnostic changed, and focused Rust checks plus actual isolated fixture exchanges were used under the documented shared-PC resource exception. The final source and README have independent-model Must 0. Historical correlation, sustained-retention ownership and a runtime fix remain open.
lidge-jun commentedon Oct 7, 2026
Held for an isolated reproduction. The current wrapper only recovers on child exit; a safe fix would be an opt-in external liveness recovery with bounded failures, ownership checks and no request replay.
Ingwannu commentedon Oct 8, 2026
The added model-list control is materially stronger than the earlier pressure-only evidence: you now have a single synthetic request, held process identity, delayed candidate probes and health-read deadlines, rather than only large-memory counters. It still does not attribute the historical production stall or retention growth to that path.
Source confirmation at current dev bbab02a: readCurrentCatalogOrCache calls loadBundledCodexCatalog on the default path; that loader resolves/persists the runtime before testing bundledCatalogCache, and the synchronous probeVersion uses execFileSync. So a warm catalog memo does not itself remove the expired-runtime-probe cost. Existing peekCodexRuntimeProcessCache/getCodexRuntimeSnapshot offer observational/asynchronous seams, but substituting either requires preserving selection identity, cache invalidation and unknown/cold-start behavior, not simply accepting a stale catalog for a different executable.
A narrow hot-path PR with injected candidates/clock and a nonblocking probe seam would be useful. Cover warm-catalog/expired-runtime memo, cold start, repeated slow/invalid candidates, selection replacement while refresh is pending, failed refresh and inherited-pipe EOF. Preserve runtime identity and bound aggregate work; keep process recovery/no-replay policy separate. I have not run the Rust/Windows controls or a live CLI probe here, and the Bun1.4.2 pin remains held under #6713's multi-platform runner crash; no runtime/production restart or memory-leak fix is claimed.
Current-dev follow-up — 2026-10-11
The narrow hot-path follow-up requested above has since landed: owner carry #6796 merged as
202b516b5a8300e2c27414e8bd7e7f3f58d36d52on 2026-10-09, and is an ancestor of current dev9b6f4b2632492b3df637dccc009d6a01ed47c85d.readCurrentCatalogOrCache()now usesbundledCodexCatalogSnapshot()rather than the synchronous loader; the snapshot serves only matching-selection evidence and starts an asynchronous single-flight refresh. The synchronous driver remains for explicit CLI work. This is current source confirmation, not a new Windows measurement.The landed patch includes selection/epoch fencing, failed-refresh retention/backoff, a 45-second shared-flight deadline, caller/shared cancellation separation, and owned subprocess TERM-to-KILL reaping.
tests/codex-integration/catalog-async-lifetime.test.tscontains the corresponding synthetic contracts, including READY-synchronized TERM-resistant children. The carry also links request abort to translator-budget finalization across Responses/Chat/Messages, with lifetime regressions. The PR author reported 472 focused passes; I have not re-executed or promoted those into my own/current-platform results. The bundled catalog implementation and its lifetime test are unchanged between the landed commit and this current dev.Accordingly, my earlier request to open that same narrow runtime PR is superseded; please do not duplicate #6796. This does not establish that the historical Windows listener stalls, sustained memory plateau/growth or the wrapper recovery policy are resolved. Those remaining observations still need a controlled report on a build containing the change, with runtime version and process identity preserved. No production pressure, native CLI probe, real account call, scan, restart or issue closure was performed here.
lidge-jun commentedon Oct 8, 2026
#6713 (merge
8d30945d50) moves the shipped Bun runtime from 1.4.0 to 1.4.2 to fix a WindowsBun.fetchstreaming regression (#6684). This issue also involves the bundled Bun 1.4.0 on Windows (a live listener stalling health checks), so the runtime change may affect it. That is unproven: nothing here attributes the stall to the fetch regression. I'm leaving this open. A re-check on the next release, which bundles Bun 1.4.2, would tell us whether the stall still occurs.fix(runtime): keep catalog probes async and reap aborted probe childr…
docs(devlog): state the current lease boundary in the #6671 design re…
fix(service): opt-in health sentinel recovers a hung but listening se…
docs(devlog): state the current lease boundary in the #6671 design re…
docs(structure): note the Windows handle binding clock-rollback caveat (
fix(service): opt-in health sentinel recovers a hung but listening se…
fix(service): require stable numeric health sentinel endpoints (#7031)
lidge-jun commentedon Oct 13, 2026
Closing this out from the service-reliability pass. The recovery gap this issue reports, where a live listener stops answering
/healthzand escapes exit-only restart, now has a fix ondev:serviceHealthRecovery.enabled: true). It is a separate process that probes/healthz. After about a minute of back-to-back timeouts it terminates the hung proxy, so the existing restart path takes over: the Task Scheduler wrapper:loop, launchd, systemd, or WinSW. On Windows the kill goes through a kernel handle held since startup, with exit code 1, which the wrapper restarts. Nothing is replayed, and recoveries are capped at three per hour.60752ec3ccee175f4bae5bdde0de4538a77ca3d5.The real Windows termination path is covered in CI by
tests/service/service-health-sentinel-process.test.ts, which kills a fixture that stops answering and asserts exit code 1 onwindows-latest. It also runs on Linux and macOS.The root cause of the historical 2026-10-07 stall is still unattributed. If it recurs on a build that includes these changes, please reopen and attach the bounded telemetry listed earlier in this thread, plus the
service-health-recovery.jsonentry. The remaining Linux PID-binding residual is tracked separately in #6987.