Repository navigation
Flaky: two subprocess-spawning tests fail intermittently on a full-suite run #262
Description
Two tests failed once during a local full-suite run and passed on an immediate clean re-run, with no code change between the two runs. Filing so someone can look properly; I did not investigate beyond characterising them.
What was observed
Run A — pnpm --filter @taskless/cli test:
FAIL |cli| test/error-envelope.test.ts > standardized error envelope (--json) > auth login > emits INVALID_INPUT when --anonymous is set
FAIL |cli| test/verify-test-commands.test.ts > verify checks components without requiring tests > passes a rule that has no fixtures yet
Test Files 2 failed | 77 passed (79)
Tests 2 failed | 1271 passed (1273)
Run B — same command, immediately after, nothing changed:
Test Files 79 passed (79)
Tests 1273 passed (1273)
The suite was also green on several runs before and after. Observed on feat/delivery-fixtures-warning at c6474c2, but the two tests are unrelated to that branch's change (it touches the rule delivery write path; neither of these does).
I did not capture the assertion output. The run was being grepped for a pass/fail summary, so only the FAIL lines above were kept. That is the main gap in this report — a rerun that captures full output would be the first useful step.
What the two have in common
Both spawn the built CLI as a subprocess rather than importing it:
packages/cli/test/error-envelope.test.ts:11andpackages/cli/test/verify-test-commands.test.ts:12both resolvebinPathto../dist/index.js- both call it through
execFileand assert onexitCode/ parsed stdout
26 of 79 test files spawn dist/index.js. Under vitest's default parallelism that is a lot of concurrent node processes on a 10-core machine, which makes contention a plausible shape for an intermittent failure — a spawn that is slow enough to trip a timeout, or one that fails outright under load, would surface as a non-zero exit code and an unparseable envelope in exactly these assertions.
Candidate causes, none verified
- Subprocess contention / timeout under parallel load. Fits the shared trait and the fact that both failures were in spawning tests.
- A
dist/read while it is being written.dist/is a build artifact that nothing rebuilds automatically, and the whole suite reads one copy of it. Nothing was rebuilding during these runs as far as I know, so this is the weaker hypothesis, but it is the one with the worst failure mode if it is real. - Genuine order dependence between test files. Not investigated at all.
I would not rule any of these in or out on one observation.
Why it is worth chasing
A test that fails once in N runs and passes on retry trains people to re-run rather than read, which is how a real intermittent bug gets absorbed as noise. These two are also on the --json envelope and the verify exit-code paths, both of which are contract surfaces where an intermittent wrong exit code would matter.
Suggested first steps
- Re-run the full suite in a loop capturing complete output, to get the actual assertion failure rather than the summary line.
- If it reproduces, check whether
execFilecalls in the spawning tests carry a timeout, and what happens to the assertion when a spawn fails rather than the CLI exiting non-zero. - Consider whether the 26 spawning files need to be serialised against each other.
Not blocking the 0.11.1 release.
Activity
theCodeDrift commentedon Sep 6, 2026
Investigated. It did not reproduce in 8 consecutive full-suite runs (1297/1297 each). Rather than guess at a fix on one observation, #275 makes the next occurrence describe itself, which is the first step this issue asks for.
The mechanism that made the assertion output worthless
You flagged the missing assertion detail as the main gap in the report. Here is why it would not have helped much even if captured.
All 28 spawning test files share a
catchthat treats the rejection as the CLI having exited non-zero, readingerror.codeas an exit code.execFilereports a failed spawn and a non-zero exit through the same rejection shape:codeis a string errno on the first, so the helper returnsexitCode: "ENOENT"and:expect(result.exitCode).not.toBe(0)passes, because a string is not0stdoutis emptyparseEnvelopewithexpected 0 to be greater than 0, from inside a helperNothing in that message mentions spawning. Under fork pressure the errno is
EAGAINand the path is identical, which fits your candidate cause 1 and both failing tests being output-parsing ones.Two things that bound the urgency
files: ["dist"], sopackages/cli/test/is in no published package.v0.11.0tag; the two that failed were added 2026-05-10 and 2026-08-14.v0.10.2tov0.11.0was a one-day window that added none.Rejected: raising the timeout
There is no
execFiletimeout anywhere in either file, so the ceiling is vitest'stestTimeout: 20_000. Nothing shows it was reached, a CLI spawn is normally sub-second, and raising it makes a genuine hang slower to surface.Leaving this open
#275 does not fix the flake, so this stays open. If it recurs, the failure will now name the errno and the command and point back here: please file a fresh issue quoting that message and the full run output.
— AI Coding Agent
test(cli): make a failed spawn say so, instead of failing as a contra…
test(cli): stop asserting the winner of a 1ms race in the Vale timeou…
test(cli): stop asserting the winner of a 1ms race in the Vale timeou…
theCodeDrift commentedon Sep 10, 2026
Some progress on this, plus a correction: an unrelated flake was mistaken for this one, so the record is worth setting straight before anyone re-investigates.
Not this issue
A worktree left over from earlier work held 13 captured full-suite logs with 2 failures. Both were the same test, and it is not one of the two named here:
That is a third flaky test, in
vale-run.test.ts, with a different cause: a 1ms timeout budget racing Vale's process startup. It has since been fixed, and its surviving twin is fixed in #327. Neither is what this issue reports.This issue: not reproduced, and not disproved either
8 full-suite runs on current
main(3df3fe8), all 1412 passed, neither named test failing. Plus 25 isolated runs of the vale test. That is a non-reproduction, not a fix — the original sighting was a single occurrence, so 8 clean runs is weak evidence.The suggested first step checks out, and is worse than described
This issue asked whether the
execFilecalls in the spawning tests carry a timeout. Measured:timeouttoexecFileanywhere.grep -rn 'timeout:' packages/cli/test/*.test.tsreturns nothing.dist/index.js, up from the 26 reported here.So a spawn that hangs under contention has no bound, and the failure surfaces as an unparseable envelope or a wrong exit code — indistinguishable at the assertion from a genuine CLI failure, which is exactly the confusion this issue is about. That is consistent with candidate cause 1 (subprocess contention) and does not rule out 2 or 3.
Suggested next step, narrowed
Rather than looping the whole suite hoping to catch it: give the spawning helpers an explicit
execFiletimeout and assert on the spawn failure separately from the CLI's exit code. A bounded spawn turns an intermittent wrong-exit-code into a clear "the subprocess did not finish", which is both a real fix and a much better diagnostic if it recurs. Whether the 30 spawning files also need serialising against each other is a separate question, and I would want the bounded-spawn signal first before answering it.docs(cli): correct the issue citation and reconcile the flake counts
test(cli): give the vale timeout test a real margin
docs(cli): correct the issue citation and reconcile the flake counts
fix(test): run spawned CLIs from a per-run snapshot of dist/