Repository navigation
Batched test runner cannot run on macOS, and one-process runs give batch-size-dependent failure counts #5439
Description
Client or integration
Other / not listed
Area
Testing and CI
Summary
Two separate problems, one causing the other to go unnoticed.
1. scripts/ci/run-bun-test-batches.sh cannot run on macOS. It requires GNU
timeout (macOS ships none) and mapfile (a bash 4 builtin; macOS ships bash
3.2). Both fail before a single test executes — exit 69 with one line, or
line 174: mapfile: command not found. A macOS contributor cannot run the
batched runner at all, so the only available option is bun run test, which is
the method .github/workflows/ci.yml:309 explains is not safe.
2. Running the suite in one process gives a failure count that is a function
of batch size, not of the code. Same branch, same 1438 files, same commit:
| method | failures |
|---|---|
bun run test (one process) |
9, then wedged at 627 of 1435 files |
one process per tests/<domain> |
468 |
| batches of 12 | 73 |
| batches of 8 | 53 |
The 20 failures unique to batch-12 all pass in isolation
(codex-shim-autorestore 9/9, images/loop 52/52,
service-tier-capability 36/36). The proximate causes are cross-test global
state: tests/server/config.test.ts calls process.chdir; five files under
tests/server/ assign process.env.OPENCODEX_HOME without restoring it, which
trips the real-home guard (refusing to write the real OpenCodex home … from a test process); and the spend ledger is single-owner, producing 61
SpendLedgerOwnerErrors in one process.
CI already works around this with 12-file batches and a one-file-per-process
sweep. The request is to make that workaround runnable off Linux, and ideally
to fix the leaks so the count stops depending on the slicing.
Reproduction
- On macOS with no coreutils:
bash scripts/ci/run-bun-test-batches.sh 1/4
→GNU timeout is required to bound Bun test batches., exit 69. - With GNU timeout present but system bash:
→scripts/ci/run-bun-test-batches.sh: line 174: mapfile: command not found. bun test tests/serverin one process → 354 failures. Then
bun test tests/server/cancel-body-on-abort.test.tsalone → 13 pass, 0 fail.
Version
2.61.0 (fork of dev at 1687636)
Operating system
macOS 26.2, arm64 (M4 Pro), bash 3.2.57, Bun 1.4.1
Provider and model
Not provider-specific — test harness only.
Logs or error output
GNU timeout is required to bound Bun test batches.
scripts/ci/run-bun-test-batches.sh: line 174: mapfile: command not found
error: refusing to write the real OpenCodex home (/Users/<user>/.opencodex) from a test process.
Point OPENCODEX_HOME at a temp directory for this test, or inject persistence instead of
calling the global writer
Screenshots and supporting files
None.
Redacted configuration
Not configuration-dependent; reproduced on a clean checkout of dev.
Checks
- I searched existing issues before opening this one
- I can reproduce this on the latest version
Activity
FredAmartey commentedon Sep 22, 2026
Findings for the three files this issue lists as failing only in batches, run on current
devwith Bun 1.4.0 and, for the last one, at168763619with Bun 1.4.1 as reported:service-tier-capability: 26 test files pinOPENCODEX_HOMEand then delete it or never restore it, so later files in the same process lose the preload's sandbox and hit the real-home guard. fix(tests): restore the sandbox home after every test file #5570 restores it in all 26.images/loop: two causes. Five image test files stub modules withmock.moduleand never put them back, soloopfails on import afterz-fulfillordownload-cap-default, and the same stubs break files in other directories (fix(tests): put the real modules back after the image tests mock them #5605).tests/helpers/translator-budget.tsregisters its cleanupafterEachwhen it loads, so in a shared process only the first file that imports it gets that hook, andloopcounts 8 live budgets instead of 1 (fix(tests): dispose test translator budgets in every file that creates them #5607). With both,bun test tests/imagesin one process goes from 49 failures to none.codex-shim-autorestore: already fixed ondev. At168763619,codex-service-manager-probe-hardeningleftCODEX_HOMEandOPENCODEX_HOMEpointing into its deleted temp directory, and theocx codex-shim statuschild thatcodex-shim-autorestorespawns inherited them and exited 1. feat(desktop): native macOS usage panel and release regression fixes #5490 restores both variables. On currentdevthe file passes in every 12-file window around it, with Bun 1.4.0 and 1.4.1.One thing that explains why the counts move with the slicing: Bun runs a directory's files in the order the filesystem lists them, not in the order they are passed, so the
z-prefix onz-fulfillandz-handler-activationdoes not make them run last. #5482 fixes one more failure from the same family, a resolver mock that recursed into another file's mock.test(server): release the native-main block the stop-hardening case l…
test(server): start the replace-retry counter cases from zero
test(server): start the logs display-metrics cases from an empty requ…
fix(ci): bundle L6 — re-attestation timestamps, event-driven sideband…
lidge-jun commentedon Sep 24, 2026
Fixed on
dev. #5456 madescripts/ci/run-bun-test-batches.shrun on stock macOS (bash 3.2, no GNUtimeout), #5672 carried the first set of cross-test state leaks, and bundle #5740 (squash-merged as e535c65) fixed eleven more files that left state for later files in the same Bun process. Measured one-process runs on macOS after #5740:tests/server3008 pass / 0 fail (357 failures on dev before),tests/images220 / 0 (8 before). Please reopen with the counts if a batch size still changes the result.