test_cpu_kernel_bench asserts calls_per_sample > 1, which is a property of the RUNNER's speed and reds under ThreadSanitizer

[localai-bot]

tests/scripts/test_cpu_kernel_bench.py:42 asserts

self.assertGreater(got["timing"]["calls_per_sample"], 1)

and that is not a property of the code under test. It is a property of how fast the machine is.

The mechanism

examples/cpu_kernel_bench/main.cpp:663-670 calibrates by doubling:

size_t calls_per_sample = 1;
for (;;) {
  const auto begin = Clock::now();
  for (size_t i = 0; i < result.calls_per_sample; ++i) fixture.Run();
  const auto end = Clock::now();
  const double ns = ...;
  if (ns >= std::max(2.0e6, 1000.0 * overhead) || result.calls_per_sample >= (1U << 20)) break;
  result.calls_per_sample *= 2;
}

It starts at 1 and doubles until ONE SAMPLE takes at least 2 ms. So calls_per_sample == 1 is the correct output whenever a single call already fills the 2 ms budget. Under ThreadSanitizer instrumentation the kernel runs several times slower, one call clears 2 ms on the first iteration, the loop breaks immediately, and the assertion fails on the calibrator working exactly as designed.

Observed

sanitize-cpu (thread) on PR #1856, job 97470564992:

596/606 Test #596: test_cpu_kernel_bench_cli ...***Failed 0.17 sec
FAIL: test_json_schema_and_deterministic_checksum
  File "tests/scripts/test_cpu_kernel_bench.py", line 42
    self.assertGreater(got["timing"]["calls_per_sample"], 1)
AssertionError: 1 not greater than 1

Zero WARNING: ThreadSanitizer lines in the whole job — no race was detected. The sanitizer lane failed on a timing assertion, not on a finding.

Not attributable to the pull request that surfaced it

PR #1856 changes zero files matching kernel_bench, and zero C++ files at all — its diff is .agents/, two scripts/ltx25-* files and one tests/scripts/ suite. The assertion is byte-identical on main. Two of the last three completed main runs had no failing jobs, so the lane is normally green; this is latent and fires when the runner is slow enough, which under tsan is most of the time it is unlucky.

What the assertion should be

Do not weaken it to >= 1, which asserts nothing — the field is initialised to 1. Assert the PROPERTY the calibrator guarantees instead:

• calls_per_sample is a power of two (it only ever doubles), and
• the reported sample duration met the floor: median_ns >= max(2.0e6, 1000 * overhead) or calls_per_sample == 1 << 20 (the cap).

That holds on a fast machine and a slow one, and it still reds a calibrator that stopped early, which is what the current line is presumably reaching for.

Owner

Not owned by an active row. Filed from #1743 / PR #1856 work, where it appeared as an unexplained sanitize-cpu (thread) red on a change containing no C++. Whoever owns examples/cpu_kernel_bench should take it; it needs no GPU and no lease.

[localai-bot]

Second sighting, and a deterministic local reproduction that does not need a slow runner.

Observed again on PR #1994 (sanitize-cpu (thread), run 32990521407, job 98246846510), byte-identical to the report above: tests/scripts/test_cpu_kernel_bench.py:42, AssertionError: 1 not greater than 1, and zero WARNING: ThreadSanitizer lines in the job. That PR's C++ diff is one file, src/vllm/entrypoints/openai/serving_chat.cpp, nowhere near the bench.

Reproducing it on a fast box

The threshold is per-call latency, so you can cross it by raising the shape instead of by finding a slow machine. On a TSan build (-DVLLM_CPP_SANITIZE=thread), one binary, one commit:

shape	ns per call	calls_per_sample	line 42
n=8 k=256 (the test's own shape)	67,792	8	PASS
n=8 k=1024	1,149,534	2	PASS
n=16 k=1024	1,170,215	1	FAIL
n=32 k=1024	5,969,152	1	FAIL
n=64 k=2048	9,525,998	1	FAIL

setarch -R ./build-tsan/examples/vllm-cpu-kernel-bench \
  --dtype q4_k --m 1 --n 16 --k 1024 \
  --warmup 1 --iterations 2 --counters off --format json

"calls_per_sample":1 with "valid":true — the calibrator working as designed, and line 42 red.

That gives whoever takes this a red-before case for the replacement assertion proposed above, without needing to reproduce a hosted runner's speed.

One local-only wrinkle

Running the suite under TSan on a recent kernel aborts before it reaches the assertion:

FATAL: ThreadSanitizer: unexpected memory mapping 0x5c5870d03000-0x5c5870d49000

That is TSan's ASLR incompatibility, not this defect. setarch -R in front of the command clears it. Worth a line in the fix, since anyone reproducing locally hits it first and it looks like a different failure.

Measured 2026-08-26, AMD Ryzen 9 9950X3D, gcc 13.3.0, -DVLLM_CPP_SANITIZE=thread -DVLLM_CPP_CUDA=OFF.