[Bug]: spawned Bun child processes stop producing output and never exit, on both macOS and Windows CI legs

Client or integration

CI only (GitHub Actions hosted macOS runners); not a client-facing path.

Area

CI and release

Summary

Sharded macOS test legs stop producing output mid-suite and are killed at their job budget with zero reported failures. Across four occurrences the signature is identical: a test logs PASS, then the job emits nothing at all for 16 to 18 minutes until the runner cancels it, and Bun and tee are orphan-killed. This has been present since at least the 2.58.0 release round and was recorded there as capacity cancellation, which the evidence below does not support. A related but distinct failure mode appears where a hook timeout exists to stop it: the same shard reports a beforeEach/afterEach hook timed out for this test at just over 30s instead of going silent.

Two consequences. Releases have been cut without complete macOS evidence while believing macOS reds were infrastructure. And because the silent form produces a cancellation rather than a failure, it is indistinguishable at a glance from the genuine capacity cancellations that also occur, so it has survived several rounds of triage.

macos control hitting its 30-minute budget is a separate problem with a different cause and is tracked in #4905. Do not merge the two.

Reproduction

Occurs on hosted macOS runners during the sharded platform-macos legs. It is not reproducible locally and no local verification was attempted for this report.

Silent form, four occurrences:

Run 35277719805, job 105392401547, macos 1/2, at dev a0f611d. Last output 21:52:22, a PASS in tests/codex-integration/codex-inject-write-lock.test.ts case a held lock makes real injection report busy and write nothing. Silence for 16m15s until cancellation at 22:08:37.
PR fix(reauth): preserve polling through cancellation HTTP races #4945, macos 2/2. Last PASS full ordering ignores empty entries and accepts raw upstream ids with slashes at 21:53:21.617. Silence for 17m33s until cancellation at 22:10:54.
Run 35247708168 attempt 2, job 105303640713, macos 2, during the 2.58.0 round. 2514 pass / 0 fail, then 17m32.9s of silence after a catalog picker test until the 20-minute cancellation.
Run 35254182109 attempt 1, job 105313855869, macos 2. 2512 pass / 0 fail, then 17m19.1s of silence after full ordering ignores empty entries and accepts raw upstream ids with slashes — the same locus as occurrence 2.

Hook-timeout form, two occurrences on unrelated branches:

Run 35279139497, job 105396939652, macos 2/2 at cca06b1 (PR test(cli): measure cold status setup before timed projections #4948, which changes only tests/cli/cli-status-json.test.ts). server combo failover 030 activation matrix > connect cancellation wins with 499, no backup, warning, or cooldown at 30066.66ms, hook timeout. 12807 pass / 12 skip / 1 fail.
Job 105398860800, macos 2/2 at 50ecf06 (PR fix(responses): enforce configured total sends during recovery #4947). Identical case, 30050.44ms, hook timeout. 12810 pass / 12 skip / 1 fail.

Neither branch touches tests/server/server-combo-failover-e2e.test.ts.

Version

2.59.0 (dev), and retrospectively 2.58.0.

Operating system

macOS, GitHub-hosted runners (Apple Silicon; /opt/homebrew in the job environment).

Provider and model

Not provider-specific.

Logs or error output

(fail) server combo failover 030 activation matrix > connect cancellation wins with 499, no backup, warning, or cooldown [30066.66ms]
  ^ a beforeEach/afterEach hook timed out for this test.
 12807 pass
 12 skip
 1 fail
Ran 12820 tests across 668 files. [649.49s]
##[error]macOS suite failed (exit 1).

Screenshots and supporting files

Not applicable.

Redacted configuration

Not applicable; this is a CI-environment defect with no user configuration involved.

⚠️ Possible Duplicate Issue(s)

[Bug] CI: Linux test shards intermittently hang ~15 minutes and are killed, leaving an orphan bun process #1302

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

Create Plan

🧪 Issue enrichment is currently in open beta.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

This is not macOS-specific, and the title now understates it. The same signature just appeared on Windows.

Run 35286915280, job 105421227726, windows 5/9, on branch codex/4943-history-preflight-wal-immutable-fallback (PR #4957). The case reset-credit auto-redeemer runtime (#822) > two processes reserve one durable id before either consume settles failed at 25294.71ms with timed out waiting for both consumes. The diagnostics are the part that matters:

a pid=4496 exit=null
stdout: 
stderr: 
b pid=5128 exit=null
stdout: 
stderr:

Both child processes were spawned and assigned PIDs. Neither exited. Neither wrote a single byte to stdout or stderr across the full 25-second window. Every other case in that file passed, several of them in well under two seconds.

That is the same phenomenon as the macOS occurrences above, seen through a test that happens to have its own deadline. On the sharded macOS legs a Bun child stops making progress and the job simply goes quiet until the runner kills it; here a Bun child never starts producing at all and a 25-second wait reports it. In both cases the process exists, produces nothing, and does not exit.

Two consequences for how this should be investigated. First, any hypothesis scoped to macOS runner behaviour is now insufficient, because it has to explain Windows too. Second, the reproducible handles are better than they looked: the combo-failover hook timeout and this reset-credit case both catch the condition within 30 seconds, whereas the silent macOS form costs 16 to 18 minutes per observation.

PR #4957 does not cause this. It touches src/codex/history-provider.ts, src/codex/history-state-open.ts and structure/codex-home.md, and the new open policy is reachable only from preflightCodexHistoryInjection, which the reset-credit children never enter.

Third distinct test, and it suggests the family is narrower than spawned child misbehaves.

Run 35286917745, windows 1/9, on branch codex/4897-teardown-owner-identity (PR #4960): management and data-plane credential separation > canceling an incomplete context body releases the actual listener turn failed at 15995.16ms. The other eight Windows shards in that run are green.

Put the three side by side:

server combo failover 030 activation matrix > connect cancellation wins with 499, no backup, warning, or cooldown — hook timeout at ~30s, twice, on two unrelated branches.
management and data-plane credential separation > canceling an incomplete context body releases the actual listener turn — ~16s, Windows.
reset-credit auto-redeemer runtime (#822) > two processes reserve one durable id before either consume settles — 25s, Windows, two children with PIDs and no output.

Two of the three are explicitly about cancelling an in-flight request and observing that the thing underneath it is released. Both name the release in the test title, which means the assertion is precisely that a teardown completes. That is a much more specific hypothesis than the one I have been carrying: not that child processes misbehave generally, but that a cancellation can leave something attached that never settles, and whatever is waiting on it waits forever.

The reset-credit case may still belong to the same family or may not. Its children produce nothing at all rather than hanging mid-teardown, so it could be a second bug that happens to share a symptom. I would rather that stay an open question than be forced into one explanation.

Worth testing against the macOS occurrences: the 16-to-18-minute silences follow a passing test, so the relevant question is what the NEXT file's setup or the previous file's teardown was doing at that moment, not what the last passing test did.

Scoped code-level finding: the connect-cancellation fixture cannot settle

I checked the current dev snapshot c9fd114cfac61570f249886a954800bd7dfa689e. There is a concrete fixture-lifecycle problem in the exact test named in the two macOS hook-timeout reports, independently of whether it explains the other silent-child occurrences.

Source at the reviewed revision:

const aStarted = deferred();
const a = serve(() => {
  aStarted.resolve();
  return new Promise<Response>(() => {});
});

The handler's promise has no resolver, abort handler, or test-owned release path. The test's finally restores console.warn only. The file-wide afterEach then awaits every server.stop(true) before flushResponseState() and the remaining cleanup. Thus this test deliberately leaves JavaScript handler work pending while teardown asks the server to finish.

Important boundary: this is a source-confirmed unresolved fixture, not yet a hosted macOS reproduction proving which teardown await stalls. It does not establish that the silent catalog/injection cases or the Windows reset-credit children have the same cause. Also, stop(true) is already present; recommending that flag alone is not a change to this code.

Narrow corrective direction

Replace only this fixture's never-settling handler with a test-owned deferred release:

const releaseA = deferred();
const a = serve(async () => {
  aStarted.resolve();
  await releaseA.promise;
  return new Response(null, { status: 204 });
});

Release it from the test's existing finally, after the cancellation assertions (also when an assertion throws), before global server teardown. Keep all existing assertions: 499 / client_cancelled, one cancellation receipt, no B request, no combo warning, and no cooldown. The release must not precede await pending, otherwise the fixture could accidentally make an uncancelled request complete and weaken the regression test. A bounded wait for fixture startup and explicit cleanup diagnostics would also make a failed setup distinguishable from a stop/flush hang; do not merely enlarge the 30-second hook budget.

Verification required before calling this a CI fix

At the pinned revision/Bun version, compare the original and releasable fixtures on hosted macOS, recording handler start/release and server.stop(true) start/completion separately from state flushing.
Run the named cancellation test and the complete tests/server/server-combo-failover-e2e.test.ts after the change, retaining every assertion above.
Re-run the normal macOS shards. Treat any remaining silent-child incident as still open, capturing previous-file teardown / next-file setup rather than blaming the last PASS line.

This comment does not claim a passing Bun/macOS rerun; none has been completed for this finding yet. #4905 remains the separate unsharded-control budget investigation and should not be closed by this fixture change.

ci: assemble and verify the scoped lidge-jun#4956 fixture patch on a …

Merge latest dev into the scoped lidge-jun#4956 fixture fix

Implemented the scoped fixture correction from the earlier comment in #5014 (draft).

Current head: d2eb664118408d281843c111cfd0dcdf6313fcdb, including dev 50d72a1d09053be7be243bdaede3d2aac09ca830. The five-file diff replaces only the named test's never-settling handler with a finally-released gate, bounds startup/cancellation observation, retains every cancellation assertion, and adds five helper lifecycle cases. No production path, test-skip policy, hook timeout, or file-size limit is relaxed.

The helper passed standalone TypeScript and five Node contract checks, including a negative control. These are not Bun/macOS verification. Exact-head hosted Linux/macOS/Windows validation is queued as diagnostic run 35303541528 on the contributor fork's Actions; the normal PR workflow is also requested on the contributor fork's PR 545. No completed platform pass or full-suite result is claimed.

Please keep this issue open for the other silent-child occurrences, and keep #4905 separate. The PR addresses a concrete fixture-lifecycle defect, not a demonstrated common root cause for every report.

A second test file shows the cold-spawn half of this, in a change that did not touch it.

Run on PR #5015, windows 2/9, job 105475642050: connected-client runtime probe scope > observes only the selected runtime and leaves full diagnostics available failed at 15339.29ms with expect(result.status).toBe(0) receiving null at tests/cli/cli-connect-readiness.test.ts:281. spawnSync returns a null status when the child is killed by its timeout rather than exiting, and that file passes timeout: INTERNAL_DEADLINE_MS, so a spawned child produced nothing for the full 15 seconds. 308 other cases in the same shard passed.

This is the same shape #4948 fixed in tests/cli/cli-status-json.test.ts: a spawnSync child whose cold module-graph startup is measured against a bound sized for warm children, where batch composition decides whether the file pays the cold cost. The remedy there was to move the cold start into an explicit setup outside the measured window rather than to raise the bound, and the same remedy is available here.

So the cold-spawn exposure is not specific to one file. Any test that spawns a real Bun child and measures it against a shared deadline carries it, and INTERNAL_DEADLINE_MS is read by more than twenty files. That is worth knowing before anyone reaches for the constant: raising it repeats the mistake tests/helpers/test-budget.ts already documents about SPAWN_BUDGET_MS.

Whether this is the same root cause as the silent 16-to-18-minute hangs or a separate cold-start problem that merely shares a symptom is still open. The distinction matters: a cold start is slow and finishes, a hang never does. These two files are slow-and-killed; the macOS shard silences and the reset-credit children are produce-nothing-and-never-exit.

test(server): release the combo connect-cancellation fixture during c…

test(ci): warm a spawned child module graph before it is timed (#5038)

Hosted reproduction and native stack evidence

The original #5248 tested source tree was replayed with Bun 1.4.0 and shard 1/2, changing only diagnostics and the fork-only launch wrapper. Diagnostic run 35679762449, job 106593967562 (hosted on the contributor fork's Actions) again hit the unchanged 20-minute budget.

The phase trace establishes that case 1 finished database cleanup, environment restoration and directory removal. Case 2 then entered setup:catalog-probe:start and never returned. At 60 seconds of silence the owned Bun process had one defunct direct child. This rules out the preceding fixture cleanup as the stopping point in this reproduction.

The native sample's main thread resolves to spawnSync / spawn_maybe_sync / SpawnSyncEventLoop::tick_with_timeout. This is function-level symbolication from the official Bun 1.4.0 ARM64 profile archive: its UUID 4C4C440C-5555-3144-A11D-99D1CCBEAE57 matches the sampled executable, its SHA-256 matches the published digest, and each mapped address lies within the symbol's address/size interval. Other sampled threads were parked. The evidence is a parent synchronous-wait failure with an already exited child, not proof that the child never started.

At the exact Bun release source, a timeout kills the child and closes readable pipes but the loop continues while subprocess activity remains; subsequent ticks can have no deadline. An unreflected exit therefore explains why the configured probe timeout does not necessarily return control to the test runner. The capture does not yet identify why the exit was lost.

A controlled follow-up (diagnostic run 35681504245, job 106599252387, on the contributor fork's Actions) changed only the general suite argument from tests to ./tests, keeping source, test assertions, Bun version, shard ratio and budgets unchanged. It passed 15,113 general tests and 194 serial tests, zero failures, within ten minutes. Both runs selected 748/1496 general files; the 173 files through the original stall occur in identical order. The passing run covered all 748 general files and the four serial files owned by shard 1.

The trigger hypothesis is Bun's directory descriptor retention in filter-mode discovery: upstream fix #40016. That hypothesis must be distinguished from the captured wait location: no FD count was captured. The rooted-discovery correction is now in #5483 at abdccbe59d68ca5dd5632a04d6440a6cb6a443a3, with exact-argv and unchanged failure-propagation contracts. Both hosted shards are validating that exact checkout as diagnostic run 35682404784 on the contributor fork's Actions, while maintainer security review is requested. No merge or issue closure is claimed.

Final validation: shard 1 at corrected head cb41ba3b4c (diagnostic run 35683370022, job 106604940668, on the contributor fork's Actions) passed 14,722 tests including serial files, zero failures/errors. Shard 2 at abdccbe59d (run 35682404784, job 106601967172) passed 14,002. The only intervening edit corrects the shard-1-owned classifier test's old command anchor; shard 2 is unchanged. The initial final-head shard-1 run honestly failed on that missed anchor before it was corrected. No lane retry policy was added.

The Windows reports are treated separately in #5484: fixture-local LOCALAPPDATA isolation and disposable-child ACL helper seams. Hosted Windows at exact head 5f9d1046a8 (diagnostic run 35682566884, job 106602459110, on the contributor fork's Actions) passed 39 tests / 250 assertions. A reservation-ID reuse negative control failed as expected and original source bytes were restored before the positive run. The remaining incomplete-body cancellation case (run 35682759908, job 106603069723) passed unchanged in 1.13 seconds under the 15-second ceiling; this is current non-reproduction, not proof of its historical cause. The combo-cancellation fixture fix #5014 is already merged.

Local full prepush was attempted twice and still exceeded its 900-second Windows suite budget, with other Windows failures. Focused validations above are not a claim of full local CI success. Both corrective PRs await required checks and maintainer review/integration. Keep #4956 open until that integration is complete.

Status on dev as of 2026-10-10, with one new occurrence.

Named fixtures: fixed and on dev. The connect-cancellation handler now releases through tests/helpers/held-response.ts (#5014, 668850d23f). The two-process reset-credit case drains both pipes from spawn, reaps both children and writes phase markers (#5509, 798c07c7a4). No source defect was found in the credential-separation cancel case.

New occurrence (Windows): run 38041407670, attempt 3, windows 6/9. The run was PR #6864, which changes none of these files. tests/clients/client-link-runtime.test.ts "an ended link recycles a connected sibling after listener cleanup" spawned a Bun child (tests/fixtures/client-sibling-recycle-child.ts) that wrote no readiness marker for 45.9 s. The test reported only timed out waiting for connected client readiness, with no child state or output. The same job passed on rerun (attempt 4).

#6893 gives that fixture the diagnostics the reset-credit case already has: it drains both pipes from spawn into bounded buffers and appends the child's pid, exit code and recent output to each wait timeout. It does not claim a cause. Bun reads spawned pipes eagerly, so an unread pipe is not established as the stall. This issue should stay open.

test: report the sibling-recycle child's state when it stalls (#6893)

Another case of the multi-file exit stall, on dev 7af092c, Cross-platform CI run 38162375606 (lane=all), test 1/4 attempts 1 and 2: shard 1/4 batch 5/57 timed out at 120 s. All 12 files in the batch reported every test passed (the last at 18:44:40), and the Bun process then failed to exit until it was killed at 18:46:32. The attribution sweep passed each file alone. The batch: tests/ci-workflows/{compatibility-version, docs-gui-screenshot-policy, docs-link-targets, exhaustive-deps-suppression, linux-desktop-packaged-ci, macos-serial-lanes, package-tree-integrity, package-tree-restart-ownership, release-notes, release-outcome-report, release-version-sources, repo-hygiene}.test.ts. It reproduced earlier the same day on PR run 38146205183 and passed on rerun, and the same batch passed in dev run 38160045923. Two candidates for the open handle: release-outcome-report.test.ts spawns bash with stderr: "pipe" and never drains or cancels that stream, and macos-serial-lanes.test.ts spawns detached bash groups with piped stdio. None of today's changes touched these files; only layout.json registrations moved, which changes batch composition.

diag(#4956): run shard 1 batches 1-5 in sequence

diag(#4956): same sequence repro on test runner Bun 1.4.3

Another Windows occurrence of the spawned-child class, recorded so it is not lost (no change requested here):

Run https://github.com/lidge-jun/opencodex/actions/runs/38222137968 (PR ci(macos): split the macOS suite into three shards under the same 20-minute cap #6983, head 736e7bf; that PR changes only the macOS shard matrix and its contract tests), job windows 4/9, batch 9/45.
tests/cli/cli-status-json.test.ts:109, status version skew projection: the cold warmColdSpawn("cli-index/status", …) child did not finish inside its deadline (timedOut true after 25417 ms). The shard failed on first occurrence as designed.
The same ocx status spawn path failed on dev dispatch run 38162375606 (windows 4/9, ocx status --json codexPlugins (spawned, read-only) at 20115 ms).

A separate Windows assertion failure in the same round (web-search pacing deadline, 504 on reset replay) is a different class and is filed as #6988.

fix(ci): run the test runner on Bun 1.4.3 so spawnSync stops losing c…

ci(macos): split the macOS suite into three shards under the same 20-…

diag(#4956): windows codex-journal batch A/B

Root cause found for the Linux shard 1/4 batch 5 stall (runs 38146205183 and 38162375606), with a fix in #6986.

What the logs actually show. The batch did not stall after every test had passed. In all three failed attempts it stopped inside tests/ci-workflows/release-version-sources.test.ts, in "the check CLI fails naming the drifted source and passes when they agree". That test calls Bun.spawnSync(..., { timeout: 15000, killSignal: "SIGKILL" }), and the call never returned for about 112 s.

Cause: Bun 1.4.0 spawnSync, fixed upstream in oven-sh/bun#44581 (first released in Bun 1.4.3). For the length of a call, spawnSync pointed the VM at its private wait loop. A GC finalizer that ran during one call released its poll on that loop. The loop is cached for the life of the process, so its poll count drifted to zero. A later call then never polled for its child and spun past its own timeout. In a multi-file --isolate process, what gets finalized depends on earlier files, which is why every file passed alone.

Evidence

Scratch hosted repro (branch codex/4956-diag-repro), running shard 1/4 batches 1–5 in sequence on Bun 1.4.0: 8 of 252 batch-5 runs stalled at the same test (run 38213764818). In every dump:
- the test process main thread is in state R with no wait channel, symbolized (bun-profile) as spawn_maybe_sync → tick_with_timeout → tick_tasks_only;
- the CLI child is a zombie whose parent is the test process;
- its pidfd is registered level-triggered in spawnSync's private epoll and readable, but never waited on.
The same sequence on Bun 1.4.3: 0 of 280 stalls and no crashes across 1,400 batches (run 38215466714). Running the 12-file batch alone never stalled in 450 runs (run 38213203939).
Upstream's mechanism reproduced locally with one finalized poll: Bun 1.4.0 hung past its own timeout in 10 of 12 runs; 1.4.3 passed 12 of 12.

What #6986 does. It moves testRunnerBun from 1.4.0 to 1.4.3; the shipped runtime stays 1.4.2. It adds a bounded regression test that fails on 1.4.0. It also fixes two things the 1.4.3 runner exposed: a test that hard-coded one runtime's spelling of a refused connection, and a real Claude picker relay defect on the shipped runtime (#6984).

What stays open here (Windows). On the exact-head run's first attempt, two Windows children timed out. Both passed when the failed jobs were rerun.

codex-cli-update-zero-effect: a Windows A/B (run 38242637613) shows the timeout on Bun 1.4.0 runners as well (1 of 45, plus an 11.5 s near-miss), against 0 of 45 on 1.4.3. In the 1.4.0 failure, the stalled process was the shipped-runtime grandchild (node_modules/bun/bin/bun.exe src/cli/index.ts … codex-cli-update attest), mostly idle with all threads waiting. That is this issue's silent-child shape, and it does not depend on the runner pin.
codex-journal: one 40 s child timeout. Its A/B (run 38245508335) found 0 failures in 90 runs across both versions, so the failure is not reproduced or attributed.

I'm leaving this issue open for those Windows observations.

fix(ci): run the test runner on Bun 1.4.3 so spawnSync stops losing c…

Fixed on dev by #6986 (commit 95bee2781f), landed through #7009 (merge 8f10f238e3). Ships in 2.84.0.

Navigation Menu

Repository navigation

[Bug]: spawned Bun child processes stop producing output and never exit, on both macOS and Windows CI legs #4956

Client or integration

Area

Summary

Reproduction

Version

Operating system

Provider and model

Logs or error output

Screenshots and supporting files

Redacted configuration

Scoped code-level finding: the connect-cancellation fixture cannot settle

Narrow corrective direction

Verification required before calling this a CI fix

Hosted reproduction and native stack evidence

4 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

[Bug]: spawned Bun child processes stop producing output and never exit, on both macOS and Windows CI legs #4956

Description

Client or integration

Area

Summary

Reproduction

Version

Operating system

Provider and model

Logs or error output

Screenshots and supporting files

Redacted configuration

Activity

coderabbitai commented on Sep 17, 2026

lidge-jun commented on Sep 17, 2026

lidge-jun commented on Sep 18, 2026

luvs01 commented on Sep 18, 2026

Scoped code-level finding: the connect-cancellation fixture cannot settle

Narrow corrective direction

Verification required before calling this a CI fix

luvs01 commented on Sep 18, 2026

lidge-jun commented on Sep 18, 2026

luvs01 commented on Sep 22, 2026

Hosted reproduction and native stack evidence

4 remaining items

lidge-jun commented on Oct 10, 2026

lidge-jun commented on Oct 11, 2026

lidge-jun commented on Oct 12, 2026

lidge-jun commented on Oct 12, 2026

lidge-jun commented on Oct 12, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions