record(SERVER-STALL-1737): the zero-byte hang reproduces off-GPU, and its proposed fix is a divergence (#1737)

.agents/issue-index.md

@@ -635,6 +635,7 @@ rather than merged. `scripts/check-agent-record.py` gates both.
 | [#1724](https://github.com/mudler/vllm.cpp/issues/1724) | `LTX25-DIT-ATTN-FA2-HD128` | **`LaunchDenseFA2Bf16`'s head-dim `else throw` is unreachable while the admissibility guard above it stands, so no test enters it and no mutation can turn it red.** `cuda_flash_attn_fa2.cu::LaunchDenseFA2Bf16` now ends its dispatch `if (d == 64) { ... } else if (d == 128) { ... } else { throw ... }`. The arm is RIGHT and the fresh review of [PR #1704](https://github.com/mudler/vllm.cpp/pull/1704) asked for it: the head dim is a TEMPLATE parameter, so the two compiled instantiations are two call sites and not one call with an argument, and the shape it replaced put the 128 call in a bare `else` on the reasoning that a head dim with no instantiation would be a LINK error. That reasoning is wrong -- the set the function serves is decided by the `d != 64 && d != 128` guard earlier in the same function, not by the linker -- so widening that guard alone, the exact first edit a head_dim-192 rung makes, links fine and sends 192 into the 128 kernel, which reads 128 of its 192 channels and returns a SILENTLY TRUNCATED answer. The two guards are ORDERED, which is what leaves the arm without coverage: no input reaches it, no test can enter it without first making the very edit it guards, and deleting the arm or inverting its condition leaves every case green. NOT a regression and NOT a capability that landed dead -- the same input previously produced a silently truncated answer with no diagnostic, so the arm strictly replaces silence with a named refusal. The owning suite `test_ops_attention_dense_fa2` is CUDA-gated in all 12 of its cases -- each returns early on `!HasCuda()` -- so doctest reports 12 cases and ZERO assertions on a CPU build, so it gives the arm no coverage on the authoring host either. OWES: the first commit that widens the `d != 64 && d != 128` guard -- for head_dim 192, for f32, or for any new rung -- owes a RED-FIRST case proving the throw fires for a head dim the widened guard admits and the launcher has no instantiation for, taken BEFORE the instantiation that makes the throw unreachable again is added; that widening is the only moment at which the guarantee is both reachable and provable. NOT fixed in flow, because the only change that makes it testable is the guard widening this issue conditions, which is a separate rung with its own row. Under `## Owed` in [`ltx25-dit-attn-fa2-hd128.md`](specs/ltx25-dit-attn-fa2-hd128.md) | bug |
 | [#1734](https://github.com/mudler/vllm.cpp/issues/1734) | `LTX25-DIT-ATTN-FA2-HD128` | **`scripts/ltx25-dit-attn-fa2-hd128-ab.sh` prints an EMPTY `memavail low-water:` for every arm, and the cause is the writer rather than the reducer that prints it.** Found by this row's own A/B on `dgx:gpu0` (rc job `91e0b5d9-b7f7-4b69-bf3f-d593aa25f871`, artifacts `/mnt/nas_share/rc/ltx25-fa2hd128/out/20260822T203535Z/`). At line 367, `n=$(grep -c 'last=' "$log" 2>/dev/null \|\| echo 0)` emits TWO lines when the count is zero, because `grep -c` prints `0` AND exits 1, so the `\|\| echo 0` arm fires as well. Line 369 then writes that value into a tab-separated record, which lands on disk split across two lines: `watch-flash.tsv` is 85 lines with `NF=3`, 85 with `NF=2` and 16 with `NF=4` (186 total), and `watch-fa2.tsv` reads 85 / 85 / 6. The 85 pairs are exactly the 5-second polls taken during the ~7 minutes of model load, before any `last=` line existed; once the log has a match `grep -c` exits 0 and the record is well formed, which is why a minority of lines are intact. TWO consequences. The reducer at line 390 reads `$4` over a file where `$4` is empty on 170 of 186 lines, and an empty string sorts first under `sort -n`, so `head -1` returns it -- that is the reported symptom, and the reducer is independently fragile because it strips the `memavail_gib=` prefix off `$4` alone and so depends on a field position it cannot rely on. The sample cap's own test `[ "${n:-0}" -ge "$WANT_SAMPLES" ]` also receives the two-line value for those 85 iterations; harmless here because the cap cannot fire before a sample exists, but a non-integer reaching an integer comparison inside the guard that stops a job on a shared box is not something to leave standing. TOUCHES NO NUMBER in the measurement: both arms report `stopped_by=sample-cap`, which is the direct evidence that neither was stopped by memory pressure, and re-derived with a prefix-stripping match instead of a positional one the low-water is **40.3 GiB on both arms** against `MEM_FLOOR_GIB=12.0`, so the run stayed 3.36x above its own floor. NOT fixed in the record commit that found it, because that commit is `.agents/`-only by scope and a `scripts/` edit owes a red-first case over a fixture `watch-*.tsv` asserting the record is one line and the low-water reduces to the expected value. Under `## Owed` in [`ltx25-dit-attn-fa2-hd128.md`](specs/ltx25-dit-attn-fa2-hd128.md) | bug |
 | [#1649](https://github.com/mudler/vllm.cpp/issues/1649) | `GATE-WINDOWS-PORTABILITY-TARGET-SCOPE` | **`check-windows-portability.py` read `/w` anywhere in `CMakeLists.txt` as a project-wide negation of `/W4 /WX`, so the vendored BoringSSL targets' PRIVATE `/w` red `windows-msvc-cpu` and `windows-msvc-vulkan` on main and on every PR.** `check()` set `warnings = cmake` -- the whole file -- concatenated `cmake/CompilerWarnings.cmake` and did a flat token search, which cannot tell a global `add_compile_options(/w)` from a `target_compile_options(<vendored> PRIVATE /w)`; only the first negates the policy. The refusal lands before any translation unit is read, so the job carries no `error C####`. SECOND red, which #1649 did not record: the same defect fails this checker's own suite -- `test_real_tree_msvc_warning_policy_reaches_the_cxx_compile` with `negation='/w'` on main at `8540a2755` (78 tests, 1 failure) -- so the tree asserted the contradiction in two places at once and one fix clears both. FIXED IN FLOW: `without_foreign_target_compile_options` blanks `target_compile_options` spans whose target PROVABLY names only targets this project never declares (`add_library`/`add_executable` first arguments, with `foreach` bindings resolved); anything unresolved stays in scope, and `cmake/CompilerWarnings.cmake` is kept whole because it applies the policy through an unresolvable function parameter. The widening is bounded by three guard properties proved discriminating by MUTATION -- forcing `_target_is_foreign` to `return True` reds exactly those three and nothing else (82 tests, 3 failures), tree restored byte-for-byte. 82 tests OK; the real tree now prints `Windows portability contract OK`. RESIDUAL, stated not hidden: a negation reaching a project target through a `set()` binding rather than a `foreach()` one is still not caught; no such construct exists in the tree today. Recorded under [`gate-windows-portability-target-scope.md`](specs/gate-windows-portability-target-scope.md) `## 4. Risks / decisions` | bug |
+| [#1737](https://github.com/mudler/vllm.cpp/issues/1737) | — | **A request the scheduler can never admit hangs forever with zero bytes, and our advertised `max_model_len` is one block larger than the servable KV.** Found chasing the 2026-08-22 comment on [#931](https://github.com/mudler/vllm.cpp/issues/931), whose own cause (the SSE keepalive) is fixed and closed — this is a SECOND defect with the same client-visible signature, and unlike that report it reproduces OFF-GPU. CPU build of `vllm-server`, `facebook/opt-125m`, `--num-blocks 4 --block-size 16`, one non-streaming `/v1/completions` of 6 prompt + 56 output tokens: `curl: (28) ... with 0 bytes received` after 150 s, no status line and no error body, while `/health` answers 200 and `/metrics` reads `num_requests_waiting 1`, `num_requests_running 0`, `kv_cache_usage_perc 0.0`, `num_preemptions_total 1`. `VT_SERVER_VERBOSE=1` shows 48,285 consecutive `core-step end model_executed=0` heartbeats, 48,284 of them carrying an admission refusal — the CPU equivalent of the `0 %, 11.43 W` GPU reading in the #931 comment. MECHANISM, measured with a scratch scheduler trace since reverted: `ResolveMaxModelLen` (`model_loader.cpp:1508`) reconstructs `available` as `num_blocks * bytes_per_block` and advertises `max_model_len = num_blocks * block_size`, while only `(num_blocks - 1) * block_size` is allocatable because block 0 is the reserved null block (`block_pool.cpp:53-57`, `:453`); a request inside the advertised context is admitted, decodes until the pool is exhausted, preempts ITSELF as the only request in flight, and is then refused re-admission on every subsequent step forever. Boundary is exact: at `--num-blocks 2` (advertised 32, servable 16) 17 total tokens return 200 and 18 hang. NOT FIXED HERE, and the reason is the in-flow rule's own criterion: the silent-wedge half is shared with vLLM (`sched/scheduler.py:934-938` at the pinned `555967922` breaks out of the same loop and leaves the request waiting), so changing it is a DIVERGENCE needing a decision; and the off-by-one half tightens an assertion `tests/vllm/entrypoints/test_loaded_engine_dense.cpp:635-660` currently pins, so it needs its own row, spec and fresh review. It is NOT established that this is what the #931 comment hit — that report used a 22-token request against a 27B checkpoint, nowhere near any advertised context — but the SHAPE transfers, and so does a one-command discriminator that needs no new code: `VT_SERVER_VERBOSE=1` and read the heartbeat during the stall. Owed by [`server-concurrency-failures.md`](specs/server-concurrency-failures.md) `## Owed` | bug |
 | [#961](https://github.com/mudler/vllm.cpp/issues/961) | `BACKEND-CUDA-SM110` | `tests/tools/test_online_gate_startup.py:259` guards `shellcheck` absence with a check that cannot fire, so an absent instrument reads as a code verdict: `test_serve_low_tools` raises `FileNotFoundError: 'shellcheck'` instead of skipping. Filed by the sm_110 baseline lane ([#955](https://github.com/mudler/vllm.cpp/issues/955)), where the leased `thor:gpu0` worker carries no `shellcheck`, so the baseline names the failure as a known entry rather than an sm_110 fact. Indexed late: the issue was opened 2026-08-15 and its index row was lost with the unmerged repair of PR [#956](https://github.com/mudler/vllm.cpp/pull/956) | bug |
 | [#962](https://github.com/mudler/vllm.cpp/issues/962) | `BACKEND-CUDA-SM110` | NVFP4 marlin disagrees with itself on sm_110: `test_ops_moe_grouped:1262` fails `CHECK(bitdiff == 0)` on `NVFP4 block8-vs-block16 M=8 K=4096 N=4096 bitdiff=15/32768`, against a kernel configure reports as `marlin-nvfp4: ENABLED for [110]`. A live kernel defect on an ENABLED feature, not an absent one, and the only substantive standing sm_110 finding in the [#955](https://github.com/mudler/vllm.cpp/issues/955) baseline. Indexed late, same cause as the row above | bug |
 | [#1363](https://github.com/mudler/vllm.cpp/issues/1363) | `BACKEND-CUDA-SM110` | Thor reports 30 GiB of swap, all free, measured inside `rc run -d thor:gpu0`, so the "zero swap" half of the box's `THIS BOX REBOOTS INSTEAD OF OOM-KILLING` warning is stale. `vm.overcommit_memory=1` is unchanged and the three 2026-08-11 reboots were observed, so the hazard stands and is not relaxed; unresolved are whether the swap is the host's or a container view, whether it changes the failure mode at all, and when it appeared | bug |

.agents/specs/server-concurrency-failures.md

@@ -441,3 +441,107 @@ loud failure, which is the protection #577 asked for and did not get.
   refuted as that cause: the pool was 36 against an offered concurrency of 8. It
   is a real mirror gap and needs its own change, because the next person raising
   `--max-num-seqs` for throughput silently raises the HTTP ceiling with it.
+- [#1737](https://github.com/mudler/vllm.cpp/issues/1737) — **a request the
+  scheduler can never admit hangs forever with zero bytes, and our advertised
+  `max_model_len` is one block larger than the servable KV.** Found on
+  2026-08-22 while chasing the sharper stall the 2026-08-22 comment on
+  [#931](https://github.com/mudler/vllm.cpp/issues/931) reported on `dgx:gpu0`,
+  which that reporter could not pursue because the box is bounded by
+  [#1647](https://github.com/mudler/vllm.cpp/issues/1647). **#931's own cause is
+  fixed and closed; this is a second defect with the same client-visible
+  signature**, and it reproduces OFF-GPU: a CPU build of `vllm-server` on
+  `facebook/opt-125m` with `--num-blocks 4 --block-size 16` and one
+  non-streaming `/v1/completions` request of 6 prompt + 56 output tokens
+  returns `curl: (28) ... with 0 bytes received` after 150 s, while `/health`
+  answers 200 and `/metrics` reads `num_requests_waiting 1`,
+  `num_requests_running 0`, `kv_cache_usage_perc 0.0` and
+  `num_preemptions_total 1`. `VT_SERVER_VERBOSE=1` shows 48,285 consecutive
+  `core-step end model_executed=0` heartbeats — the CPU equivalent of the
+  `0 %, 11.43 W` GPU reading in the #931 comment. **Mechanism, measured:** the
+  auto-fit advertises `max_model_len = num_blocks x block_size` because
+  `ResolveMaxModelLen` (`model_loader.cpp:1508`) reconstructs `available` from
+  the full block count, while only `(num_blocks - 1) x block_size` is
+  allocatable — block 0 is the reserved null block (`block_pool.cpp:53-57`,
+  `:453`). A request inside the advertised context is therefore admitted,
+  decodes until the pool is exhausted, preempts ITSELF (it is the only request
+  in flight), and can never be re-admitted. **Two halves, and they separate.**
+  The silent-wedge half is shared with vLLM, whose v1 scheduler breaks out of
+  the same loop and leaves the request waiting (`sched/scheduler.py:934-938` at
+  the pinned `555967922`), so changing it is a DIVERGENCE and needs a decision
+  rather than a patch. **The off-by-one half was recorded here as "ours", and
+  that is WITHDRAWN — see the correction below.** Neither is fixed here: the
+  arithmetic change tightens an assertion
+  `tests/vllm/entrypoints/test_loaded_engine_dense.cpp:635-660` currently pins
+  (`params.num_blocks = 1` asserting `max_model_len == 32`), which the in-flow
+  rule excludes and which needs its own row, spec and fresh review. **It is NOT
+  established that this is what the #931 comment hit** — that report used a
+  22-token request against a 27B checkpoint, nowhere near any advertised
+  context. What transfers is the SHAPE and a one-command discriminator that
+  needs no new code: restart that server with `VT_SERVER_VERBOSE=1` and read the
+  heartbeat during the stall, because `model_executed=0` against a non-zero
+  `unfinished` count puts the search on the KV pool rather than on `AsyncLLM`.
+- **CORRECTION to the entry above, 2026-08-23: the off-by-one is NOT ours, and
+  the fix it proposes is a DIVERGENCE from vLLM.** The entry above calls the
+  advertised-context overhang "ours" and proposes correcting `ResolveMaxModelLen`
+  to `(num_blocks - 1)`. Read at the exact parity pin `5559679229bc` in a local
+  checkout of the oracle, upstream carries the identical arithmetic:
+  `kv_cache_utils.py:2144-2158` reconstructs `available_memory = override x
+  bytes_per_block` from the FULL block count whenever `num_gpu_blocks_override`
+  is set, which is byte-for-byte what `model_loader.cpp:1505-1507` does;
+  `kv_cache_coordinator.py:90-95` then builds `BlockPool(num_gpu_blocks=
+  kv_cache_config.num_blocks, ...)` with NO `+ 1`; and `block_pool.py:188-191`
+  pops block 0 out of the free queue as the null block exactly as
+  `block_pool.cpp:53-57` does. Upstream's allocatable capacity is therefore
+  `num_blocks - 1` while its advertised `max_model_len` assumes `num_blocks`,
+  which is the same one-block overhang. The self-preemption half matches too:
+  `sched/scheduler.py:607-609` preempts the sole request, breaks, and leaves it
+  waiting. **BOTH halves are mirrored upstream behaviour**, so subtracting the
+  null block is a product decision about diverging from the reference, not an
+  in-flow bug fix, and it must not be taken as an implementation task. The
+  `## Owed` entry above is left in place rather than rewritten, because the
+  shape of the error — reading our own code against a hang and concluding the
+  arithmetic was ours without reading the oracle's — is the lesson.
+
+- **The hang REPRODUCES independently off-GPU, and one hypothesis was falsified
+  by measurement rather than dropped.** Fresh CPU build (`-DVLLM_CPP_CUDA=OFF`,
+  RelWithDebInfo, x86_64) on `facebook/opt-125m`, three legs, same binary:
+  a control at `--num-blocks 64 --block-size 16` with `max_tokens 16` returns
+  **HTTP 200** in 13.2 s with `finish_reason="length"`; the wedge at
+  `--num-blocks 4 --block-size 16` with `max_tokens 56` returns
+  `curl: (28) Operation timed out after 90002 milliseconds with 0 bytes
+  received`, `http_status=000 size=0`, while `/health` answers 200 throughout
+  and the server logs only its 11 startup lines plus
+  `INFO auto-fit max_model_len: reduced from 2048 to 64 to fit the KV cache
+  (4 blocks x 16 tokens)`. That confirms the entry above on a second machine and
+  a second build. **The falsified hypothesis:** upstream's serving layer hard
+  clamps `max_tokens` to `max_model_len - input_length` — `get_max_tokens`
+  (`entrypoints/serve/utils/api_utils.py:170-206`) returns the `min` over the
+  model headroom, the request value and the operator ceiling, and
+  `completion/serving.py:163-181` feeds it to `to_sampling_params`, which at
+  `completion/protocol.py:358` assigns it UNCONDITIONALLY rather than as a
+  fallback — while `serving_completion.cpp:241` and `serving_chat.cpp:767` call
+  `to_sampling_params()` with no argument at all and our `InputProcessor` fills
+  `max_tokens` only when it is unset (`input_processor.cpp:268-271`, `:304`,
+  `:340`). The predicted consequence was that an over-long `max_tokens` would
+  wedge the same way. **It does not.** Measured at `--num-blocks 8 --block-size
+  16 --max-model-len 64` with `max_tokens 5000`: **HTTP 200** in 62.8 s with
+  `finish_reason="length"` and 58 generated tokens. Our engine stops at
+  `max_model_len` by another route, so the mechanism differs from upstream while
+  the client-visible result matches, and there is no defect here. The adjacent
+  operator-ceiling arm of that same upstream function is separately open as
+  [#544](https://github.com/mudler/vllm.cpp/issues/544).
+
+- **The wedge also survives SIGTERM, which the record above does not mention.**
+  Observed 2026-08-23 on the same CPU reproduction. After the client's 90 s
+  timeout the harness sent SIGTERM to the wedged server; it logged
+  `server: shutting down on signal 15` and then **did not exit**. Measured 53
+  minutes later: still alive, `State: S`, 23 threads, ~5.2% average CPU since
+  launch, and it needed SIGKILL. The two healthy legs of the same harness — the
+  `--num-blocks 64` control and the `--max-model-len 64` / `max_tokens 5000` leg
+  — both exited on the identical SIGTERM, so the harness returned from its
+  `wait` promptly and left no process behind. That is a one-variable contrast
+  with n=1 on each arm, not a controlled experiment, and the shutdown path was
+  not traced; what is established is that the signal was received and
+  acknowledged and the process still never terminated. It raises the
+  operational cost of the wedge above a hung client: a supervisor that restarts
+  on SIGTERM will hang too, and only a SIGKILL escalation clears it.