Skip to content

Test temp dirs built from a per-process counter collide between concurrent runs: SIGBUS, exit 135, and NO doctest summary #1860

Description

@localai-bot

What

Test fixtures that build a scratch directory from a per-process counter
collide between concurrent runs of the same binary, and the collision does not
present as a test failure — it presents as SIGBUS, exit 135, and no doctest
summary at all
.

// tests/vllm/models/test_dots3_note_scaffold.cpp:104
static int counter = 0;
dir_ = std::filesystem::temp_directory_path() /
       ("dots3_note_cfg_" + std::to_string(counter++));

The counter is per-process, so two processes both produce dots3_note_cfg_0,
_1, … Each constructor rewrites a file the other has already mmapped through
SafetensorsFile::Open, and each destructor remove_all()s the other's
directory. Because doctest's stdout is block-buffered to a pipe, the report is
lost with the process: the crash reads as "no result", not as a failure.

Measured, not theorised

Reproduced during the review of #1847 (dots3-note W2): three rounds of two
concurrent runs of the same binary, 3/3 crashed
A exit=0 B exit=135, A exit=0 B exit=135, A exit=135 B exit=0 — each with
Bus error (core dumped) and an empty summary on the crashed side. Both live
processes were observed sharing one /tmp/dots3_note_cfg_8 and one
/tmp/dots3_note_ckpt_1.

Disk pressure is not involved: the box was at 47 GB free and 61 GB RAM
available. That matters because the first observation of this crash was recorded
in .agents/specs/dots3-note.md §4.4 as happening "on a box that read 92% full
at that moment", which sent the next reader toward ENOSPC. The real cause is
concurrency, and it reproduces on demand with disk to spare.

Why this is worse than a flake

It corrupted a committed measurement. The dots3-note W2 mutation table has a
row (M11) whose recorded result came from this crash rather than from the
mutation, with a cause attributed to disk. A gate that crashes into silence is
indistinguishable from a gate that did not run, and this repository routinely has
two or more agents building in parallel worktrees on one box — so this is the
normal case, not an edge case.

For dots3-note specifically it is sharper still: under that row's spec §6.4
option B there is no oracle, so this test file is the only instrument the row
has.

Scope — house-wide, at least six files

grep -rln "static int counter" tests/vllm/models/ finds the same shape in:
test_dots3_note_scaffold.cpp, test_laguna_nvfp4_loader.cpp,
test_kimi_linear_scaffold.cpp, test_loader_unaligned_offsets.cpp,
test_ltx2_lora.cpp, test_minimax_h3_video_fold.cpp (and the review also named
test_kimi_linear_forward, test_kimi_linear_paged, test_indextts2_*,
test_minimax_music3_speech).

Any of them can be made to crash the same way by ctest -j>1 or by two agents
running the same target.

Fix

Make the path process-unique — mkdtemp, or PID plus counter — in a shared
fixture helper rather than re-implemented per file, so the next test that needs a
scratch directory inherits the safe version.

Worth pairing with a red-before test: today nothing pins that two concurrent runs
of the same test binary both succeed, which is precisely how this survived being
written six times.

The dots3-note row is repairing its own copy and correcting its §4.4 record under
#699; this issue owns the other files and the shared helper.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions