spec(ENG-EXPERT-STREAM-DEVICE): W0h — pre-register the experiment that decides whether the CUDA arm is WORSE or only different (#1736)

.agents/issue-index.md

@@ -640,6 +640,7 @@ rather than merged. `scripts/check-agent-record.py` gates both.
 | [#1725](https://github.com/mudler/vllm.cpp/issues/1725) | `BACKEND-CUDA-SM110` | `kMatmulFp8Cutlass` and `kMatmulFp8BlockScaled` fall through to the portable CPU tier and SEGFAULT on sm_110 (`test_ops_fp8_cutlass:191`, `test_ops_matmul_fp8_block_cuda:345`, measured at `0764ded2b`). The sm_110 baseline attributed them to [#960](https://github.com/mudler/vllm.cpp/issues/960), which was CLOSED COMPLETED three days earlier by `d607fec4c` -- that fix covered `QuantFp8Static` only, while these two ops are registered from TUs `CMakeLists.txt:1790-1791` builds solely for `VT_CUTLASS_FP8_ARCHS`. `cuda_matmul_fp8_block_cutlass.cu:56-58` asserts they refuse by name instead, which the measurement contradicts; `cffe59b02` has since rewritten that dispatch, and no CI lane can see either way because `cutlass-fp8` is ENABLED on the GB10 gate host | bug |
 | [#1731](https://github.com/mudler/vllm.cpp/issues/1731) | `ENG-RECORD-CONFLICT-SURFACES` | `check-agent-record` is RED on `main`: `#1649` is listed twice in `.agents/issue-index.md`, at `:592` (added by `a7bb3130b`, the lane that FILED it) and `:632` (added by `2f2a70925`, the lane that FIXED it). Neither is wrong and neither could see the other -- `merge=union` combines two appends silently, so a duplicate is the ordinary outcome when filing and fixing happen on different branches, and the checker only notices once both have landed on main, where every later PR inherits the red. Not repaired in flow: the index preamble forbids editing or deleting a row, and the two bodies carry different facts (`:632` has a second red and the mutation evidence, `:592` has the attribution to `a50c57d69`), so choosing which survives is a judgement | bug |
 | [#1733](https://github.com/mudler/vllm.cpp/issues/1733) | `ENG-RECORD-CONFLICT-SURFACES` | CLOSED as a duplicate of [#1731](https://github.com/mudler/vllm.cpp/issues/1731), filed seventeen minutes after it against the same red, and recorded here rather than left unindexed because its one MEASURED claim is FALSE and an append-only row cannot be corrected in place later. Both issues report that `.agents/issue-index.md` lists [#1649](https://github.com/mudler/vllm.cpp/issues/1649) twice on `main` at `038ff61e5`, at `:592` from `a7bb3130b` (the lane that FILED it) and at `:632` from `2f2a70925` (the lane that FIXED it). #1731 is the earlier filing, already carries an index row, and is the record. #1733 adds one thing #1731 does not, and it is wrong: "the repair is measured and it is NOT blocked", on the evidence that removing one of the two rows in a worktree and running both checkers gives `agent record OK: ENGINE=170 MODEL=377 ...` and `OK: issue index append-only`. The second half is an artefact of the instrument. `scripts/check-issue-index-append-only.py:50-51` diffs `merge-base(origin/main, HEAD)..HEAD`, which reads COMMITS, so an UNCOMMITTED deletion is invisible to it. Measured on `row/FIX-ISSUE-INDEX-1649-DUP` at base `038ff61e5`: deleting `:592` in the WORKING TREE alone returns `OK: issue index append-only` at rc 0 with `git diff --numstat 038ff61e5..HEAD -- .agents/issue-index.md` EMPTY, and committing the byte-identical deletion turns the same checker rc 1 with a `removed:` line naming the row. #1733's own quoted `agent record OK: ENGINE=170 MODEL=377` is the tell, because that is the working-tree reading and the committed tree cannot produce it while the duplicate stands. So the duplicate IS base-reachable, the two checkers ARE in genuine contradiction on this tree, and the repair is the argued exception #1731's row anticipated rather than the free edit #1733 reported | bug |
+| [#1736](https://github.com/mudler/vllm.cpp/issues/1736) | `ENG-EXPERT-STREAM-DEVICE` | **On `Qwen3.8-2.4T-A95B UD-Q1_0` the `--device cuda` arm and the `--device cpu` arm emit different tokens, and nothing measures whether CUDA is WORSE or only DIFFERENT.** W0g excluded the router gate weights, the embedding table and the W0f host alias, and named no cause. Three things keep the question open: the CUDA continuation degenerates into a mechanical recursion after the 8 tokens the arms share, which a coin flip between two equally good tokens does not produce; every comparison so far is arm-against-arm with no oracle, so "they differ" cannot say which arm is wrong; and the growth-rate argument for "partly systematic" does not survive a proper fit -- a least-squares fit of `log(divergence)` on `log(block + 1)` over all eight recorded points gives an exponent of 0.651 +/- 0.066, interval [0.489, 0.813], which INCLUDES the 0.5 a random walk predicts and EXCLUDES the 1.0 a systematic error predicts, so the two-point 24x-against-9.5x reading overstated it. Scoped as wave **W0h** of `ENG-EXPERT-STREAM-DEVICE`: feed BOTH arms the identical token sequence through the ABI logits processor (`include/vllm.h` v8, applied at `src/vllm/v1/sample/sampler.cpp:441`) and measure the negative log likelihood each assigns to held-out text, which is a quality statement needing no oracle. The decision rule is PRE-REGISTERED before any measurement, the oracle arm is `llama-cpp-unsloth` at `36fe8e1cc` (`gateable = no`, owed by [#933](https://github.com/mudler/vllm.cpp/issues/933)), and the in-tree `VT_CPU_REF=1` switch is excluded on arithmetic (every tensor to `kExpandBf16` at `gguf_keep_quant.cpp:157` is 4.37 to 4.87 TiB against a 119.631 GiB box). No product code, no speed claim, G0-SPEED stays VOID. Spec [`cuda-arm-degradation-experiment.md`](specs/cuda-arm-degradation-experiment.md) | verification |
 | [#1746](https://github.com/mudler/vllm.cpp/issues/1746) | `V1-LOGITSPROC-HOST-ADDRESSABLE` | **`apply_logits_processors` hands every ABI logits-processor callback a raw `cudaMalloc` pointer on CUDA/GB10, because it gates the staging bounce on `Backend::UnifiedMemory()` where the question is `Backend::DeviceMemoryIsHostAddressable()`.** `src/vllm/v1/sample/logits_processor/builtin.cpp` sets `host = logits.data` when the WIDE predicate holds, and `CudaBackend` answers it `pageable_memory_access && integrated`, which is true on GB10 over allocations `CudaBackend::Alloc` takes from `cudaMalloc`; CUDA never overrides the narrow predicate, so it keeps the base `false` from `include/vt/backend.h`, whose own comment says a backend must opt in "because being wrong here hands a device pointer to a host memcpy and segfaults". This is the [#844](https://github.com/mudler/vllm.cpp/issues/844) / [#1435](https://github.com/mudler/vllm.cpp/issues/1435) / [#960](https://github.com/mudler/vllm.cpp/issues/960) class in a second location, and `src/vt/op_provider.cpp` warns about it in the same tree. The CPU suite could not see it: on `Device{kCPU,0}` both predicates are true and the pointer really is host memory, so the wrong one reads correct. FIXED IN FLOW: the predicate narrows, the `else` staging arm is unchanged, and a new own-executable test carries the GB10 pair on a fake backend — `UnifiedMemory()` true, `DeviceMemoryIsHostAddressable()` false — entering through `Sampler::forward` rather than through the function, so the reachability mutation of the one production call site turns it red. A second case pins that a backend answering both predicates true keeps the zero-copy in-place wrap, so the fix does not become "always stage". Spec [`logits-processor-host-addressable.md`](specs/logits-processor-host-addressable.md) | bug |
 | [#1748](https://github.com/mudler/vllm.cpp/issues/1748) | — | **`CpuBackend` never opts in to `DeviceMemoryIsHostAddressable()`, so every reader of that predicate gets the conservative `false` on the one device where it is trivially true.** `src/vt/cpu/cpu_backend.cpp` overrides `UnifiedMemory()` to `true` and leaves the narrow predicate at the `include/vt/backend.h` default, although `CpuBackend::Alloc` returns ordinary aligned host memory, `Copy` is `std::memcpy`, and the class comment two lines above says "Host and device memory are the SAME allocation here". Found while grounding [#1746](https://github.com/mudler/vllm.cpp/issues/1746). Three readers: `ReferenceTierEligible` and `ReferenceTierRefusalReason` (`src/vt/op_provider.cpp`) never reach it, because both return earlier on `device == DeviceType::kCPU` — which is why the wrong answer stayed invisible; the direct-upload adoption in `src/vllm/model_executor/models/qwen3_5_weights.cpp` (two sites) returns early on it and therefore never adopts on CPU, and whether adoption is even meaningful there is NOT established and is part of what this issue owes; and `apply_logits_processors` reads it from #1746 onward, so CPU takes a staging bounce of `[n, vocab]` f32 down and back per step, charged only to a request that registered a processor. NOT fixed in flow: the one-line override flips the qwen3_5 residency path and needs its own red-before test and its own measurement, and a crash-class correctness repair must not carry an unmeasured residency change. Owed under [`logits-processor-host-addressable.md`](specs/logits-processor-host-addressable.md) `## Owed` | bug |
 | [#1751](https://github.com/mudler/vllm.cpp/issues/1751) | `LTX25-DIT-ATTN-FA2-HD128` | **`VLLM_LTX2_DIT_FLASH_ATTN` matches the naive arm on a PREFIX and falls through to the FA-2 default for every value it does not recognise, so a typo cannot refuse.** In `src/vllm/model_executor/models/ltx2_device.cpp` the DiT self-attention dispatch reads the three-way A/B knob with two different rules: the naive arm tests `arm[0] == '0'`, so `0x`, `07`, `0flash` and any other string starting with `0` all select `vt::Attention`, while the flash arm uses `strcmp(arm, "flash")` and matches only the exact value. Everything that matches neither -- a typo such as `falsh`, a case difference such as `FLASH`, a trailing space, `naive`, `1`, or an empty string -- lands in the bare `else` and runs `vt::AttentionDenseFa2`, the default, with no diagnostic. WHY IT IS A CORRECTNESS PROBLEM: `flash` is the DENOMINATOR of the 2.74x ratio recorded in [`ltx25-dit-attn-fa2-hd128.md`](specs/ltx25-dit-attn-fa2-hd128.md) section 8, so a mistyped denominator arm does not fail -- it runs the numerator's kernel a second time and yields ~1.00x, which is also exactly what "no speedup" looks like, so the number cannot report its own failure. That is the same shape `847e22f80` already repaired once on this knob, where the `flash` arm was protected by nothing executable. NOT EXPOSED IN THE RECORDED MEASUREMENTS: `assert_arm_op` in `scripts/ltx25-dit-attn-fa2-hd128-ab.sh` reads the `VT_OP_PROVIDER_STATS=1` announcement and exits 47 on a mismatch, so both arms of section 8.2 asserted their rung rather than assuming it. A human operator who sets the environment variable by hand, in a shell or a service unit, gets no such check and sees a run that looks entirely normal. THE FIX is an exact three-way parse -- `strcmp` against `0` and `flash`, unset for the default -- with a refusal that names the variable and lists what it accepts, in the shape AGENTS.md requires of an unimplemented arm. NOT FIXED IN FLOW, because a product edit to a dispatch that decides which kernel runs owes a RED-FIRST case asserting the refusal fires for an unrecognised value, taken before the parse changes; that is a separate rung with its own row, and the change that found this one is `.agents/`-only by scope. Found by the fresh review of [PR #1704](https://github.com/mudler/vllm.cpp/pull/1704). Under `## Owed` in [`ltx25-dit-attn-fa2-hd128.md`](specs/ltx25-dit-attn-fa2-hd128.md) | bug |