What
Test fixtures that build a scratch directory from a per-process counter
collide between concurrent runs of the same binary, and the collision does not
present as a test failure — it presents as SIGBUS, exit 135, and no doctest
summary at all.
// tests/vllm/models/test_dots3_note_scaffold.cpp:104
static int counter = 0;
dir_ = std::filesystem::temp_directory_path() /
("dots3_note_cfg_" + std::to_string(counter++));
The counter is per-process, so two processes both produce dots3_note_cfg_0,
_1, … Each constructor rewrites a file the other has already mmapped through
SafetensorsFile::Open, and each destructor remove_all()s the other's
directory. Because doctest's stdout is block-buffered to a pipe, the report is
lost with the process: the crash reads as "no result", not as a failure.
Measured, not theorised
Reproduced during the review of #1847 (dots3-note W2): three rounds of two
concurrent runs of the same binary, 3/3 crashed —
A exit=0 B exit=135, A exit=0 B exit=135, A exit=135 B exit=0 — each with
Bus error (core dumped) and an empty summary on the crashed side. Both live
processes were observed sharing one /tmp/dots3_note_cfg_8 and one
/tmp/dots3_note_ckpt_1.
Disk pressure is not involved: the box was at 47 GB free and 61 GB RAM
available. That matters because the first observation of this crash was recorded
in .agents/specs/dots3-note.md §4.4 as happening "on a box that read 92% full
at that moment", which sent the next reader toward ENOSPC. The real cause is
concurrency, and it reproduces on demand with disk to spare.
Why this is worse than a flake
It corrupted a committed measurement. The dots3-note W2 mutation table has a
row (M11) whose recorded result came from this crash rather than from the
mutation, with a cause attributed to disk. A gate that crashes into silence is
indistinguishable from a gate that did not run, and this repository routinely has
two or more agents building in parallel worktrees on one box — so this is the
normal case, not an edge case.
For dots3-note specifically it is sharper still: under that row's spec §6.4
option B there is no oracle, so this test file is the only instrument the row
has.
Scope — house-wide, at least six files
grep -rln "static int counter" tests/vllm/models/ finds the same shape in:
test_dots3_note_scaffold.cpp, test_laguna_nvfp4_loader.cpp,
test_kimi_linear_scaffold.cpp, test_loader_unaligned_offsets.cpp,
test_ltx2_lora.cpp, test_minimax_h3_video_fold.cpp (and the review also named
test_kimi_linear_forward, test_kimi_linear_paged, test_indextts2_*,
test_minimax_music3_speech).
Any of them can be made to crash the same way by ctest -j>1 or by two agents
running the same target.
Fix
Make the path process-unique — mkdtemp, or PID plus counter — in a shared
fixture helper rather than re-implemented per file, so the next test that needs a
scratch directory inherits the safe version.
Worth pairing with a red-before test: today nothing pins that two concurrent runs
of the same test binary both succeed, which is precisely how this survived being
written six times.
The dots3-note row is repairing its own copy and correcting its §4.4 record under
#699; this issue owns the other files and the shared helper.
What
Test fixtures that build a scratch directory from a per-process counter
collide between concurrent runs of the same binary, and the collision does not
present as a test failure — it presents as SIGBUS, exit 135, and no doctest
summary at all.
The counter is per-process, so two processes both produce
dots3_note_cfg_0,_1, … Each constructor rewrites a file the other has already mmapped throughSafetensorsFile::Open, and each destructorremove_all()s the other'sdirectory. Because doctest's stdout is block-buffered to a pipe, the report is
lost with the process: the crash reads as "no result", not as a failure.
Measured, not theorised
Reproduced during the review of #1847 (dots3-note W2): three rounds of two
concurrent runs of the same binary, 3/3 crashed —
A exit=0 B exit=135,A exit=0 B exit=135,A exit=135 B exit=0— each withBus error (core dumped)and an empty summary on the crashed side. Both liveprocesses were observed sharing one
/tmp/dots3_note_cfg_8and one/tmp/dots3_note_ckpt_1.Disk pressure is not involved: the box was at 47 GB free and 61 GB RAM
available. That matters because the first observation of this crash was recorded
in
.agents/specs/dots3-note.md§4.4 as happening "on a box that read 92% fullat that moment", which sent the next reader toward ENOSPC. The real cause is
concurrency, and it reproduces on demand with disk to spare.
Why this is worse than a flake
It corrupted a committed measurement. The dots3-note W2 mutation table has a
row (
M11) whose recorded result came from this crash rather than from themutation, with a cause attributed to disk. A gate that crashes into silence is
indistinguishable from a gate that did not run, and this repository routinely has
two or more agents building in parallel worktrees on one box — so this is the
normal case, not an edge case.
For dots3-note specifically it is sharper still: under that row's spec §6.4
option B there is no oracle, so this test file is the only instrument the row
has.
Scope — house-wide, at least six files
grep -rln "static int counter" tests/vllm/models/finds the same shape in:test_dots3_note_scaffold.cpp,test_laguna_nvfp4_loader.cpp,test_kimi_linear_scaffold.cpp,test_loader_unaligned_offsets.cpp,test_ltx2_lora.cpp,test_minimax_h3_video_fold.cpp(and the review also namedtest_kimi_linear_forward,test_kimi_linear_paged,test_indextts2_*,test_minimax_music3_speech).Any of them can be made to crash the same way by
ctest -j>1or by two agentsrunning the same target.
Fix
Make the path process-unique —
mkdtemp, or PID plus counter — in a sharedfixture helper rather than re-implemented per file, so the next test that needs a
scratch directory inherits the safe version.
Worth pairing with a red-before test: today nothing pins that two concurrent runs
of the same test binary both succeed, which is precisely how this survived being
written six times.
The dots3-note row is repairing its own copy and correcting its §4.4 record under
#699; this issue owns the other files and the shared helper.