diff --git a/.agents/issue-index.md b/.agents/issue-index.md index abd014ed9..096d71e7a 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -670,6 +670,7 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#1812](https://github.com/mudler/vllm.cpp/issues/1812) | `KERNEL-ACCEL-PROVIDER-DECLINE-EXACT` | **The two exact `declines == 1` routing assertions in `tests/vt/test_ops_attention_cross.cpp` are the only thing in the tree that pins the CUDA half of [#1584](https://github.com/mudler/vllm.cpp/issues/1584)'s repair, and the run CI performs cannot see it move.** Measured on `thor:gpu0` (sm_110) on 2026-08-23 while discharging [#1692](https://github.com/mudler/vllm.cpp/issues/1692), `rc` jobs `d452b91f` and `43a27be9` at `bacb71109`. Two mutations, each rebuilt and rerun on the device: reintroducing the #1584 double count in the seam (`GetOpFallbackUncounted` delegating with `/*count=*/true`), and reverting this row's whole CUDA edit (`GetOpFallbackUncounted` -> `GetOpFallback` in `src/vt/cuda/cuda_attention_cross.cu`). **Both leave the full `test_ops_attention_cross` GREEN at 20 cases / 156 assertions**, and both turn the two cases RED with `CHECK( 2 == 1 )` under a per-case `-tc=` filter. `tests/CMakeLists.txt` registers ONE ctest entry per suite, so the second row of that pair is the consequence stated plainly: the CUDA call site can be reverted to the counting resolver and every gate in this repository stays green. **Why the full run is blind was measured, not reasoned:** the reachability mutation reds `test_ops_attention_cross.cpp:326` and `:334` first -- plain `attention-cross:` cases that predate the blocked provider, assert nothing about `declines`, and run geometries `BlockedShape` rejects, so each of them resolves `BlockedFallback()`'s function-local static OUTSIDE any counted window and a later `ResetOpProviderStats` erases the extra increment. The suite warms itself, exactly as #1555's deleted `WarmDeclineOnce` warmed it by hand. The file's own comment is accurate -- exact "standalone and under `-tc=`" -- but nothing executes them that way, and `.agents/specs/op-provider-decline-exact.md` §9 said the opposite until this measurement corrected it in place. Candidate repairs: per-case ctest entries with a trailing wildcard (the depth-decoder name contains a comma and `-tc=` splits on commas, so a bare filter matches nothing and prints `0 cases ran` + `SUCCESS!`) and an asserted `test cases: 1`; or a way to reset the static between cases, which changes production code to suit a gate. Listed as O3 under `## Owed` in [op-provider-decline-exact.md](specs/op-provider-decline-exact.md) | bug | | [#1804](https://github.com/mudler/vllm.cpp/issues/1804) | `MODEL-MM-dots3-note-dots3-note-for-causal-lm` | **`.agents/specs/dots3-note.md` §4 item 6 read `is_neox_style=False` as belonging to the dots3-note SLIDING rope ONLY, and it belongs to both MLA ropes** — the sentence would have sent a W3 implementer to rotate the 13 full-attention layers split-half NeoX. `Dots3NoteSlidingAttention` does pass `is_neox_style=False` literally (`model.py:408` @ vLLM `origin/main` `c205726108df54bb6fbf15b19e725a4a3add2b18`), which is the half W0 read; `Dots3NoteFullAttention` (`model.py:219`) inherits the SAME hard-coded value from `deepseek_v2.py`::`DeepseekV2MLAAttention.__init__` (`:1093-1098`). So the two geometries do NOT differ on the RoPE layout at all — they differ on the THETA, `swa_rope_theta` 5e4 on 33 layers against `rope_theta` 8e7 on 13. The polarity that DOES flip is the INDEXER's, and that is §4 trap 2's point rather than item 6's: `deepseek_v2.py:1148` sets the indexer rope to `is_neox_style = not indexer_rope_interleave`, so at DeepSeek-V3.2's absent-key default the indexer runs NeoX beside an MLA rope that is GPT-J, and `indexer_rope_interleave = True` (`configs/dots3_note.py:23`) is what makes dots3-note's two agree. Being wrong in this direction is SILENT: the same 64 coordinates are rotated either way, so nothing changes shape and nothing throws — the §4 defect class exactly, on a row that spec §6.4 says has no oracle anywhere to catch it. FIXED IN FLOW on `row/MODEL-MM-dots3-note-W1`: §4 item 6 corrected in place and it says what it used to say (`main` is never rewritten), `ParseDots3NoteParams` resolves `rope_is_neox_style = false` on BOTH geometries with the two citations beside it, and `tests/vllm/models/test_dots3_note_scaffold.cpp` asserts both plus the indexer's agreement with them. The assertion was captured RED against the NeoX reading, on an arm that compiled and ran, before the corrected value existed | bug | | [#1802](https://github.com/mudler/vllm.cpp/issues/1802) | `BACKEND-CUDA-SM110` | **Re-measuring the sm_110 `ctest` baseline at `6756f9131` (2026-08-23, job `8bf39567-9334-4f7e-aa27-43a2aa867bb7`, artifacts `/mnt/nas_share/rc/thor-w05-955/out/`) found six names red that were green at `0764ded2b`, none with an owner**, plus two standing entries whose recorded cause no longer holds. Arrivals, all mode `Failed`: `test_cuda_ops` (`:106` `CHECK(bad == 0)` -> `6 == 0` and `7 == 0`, silu_and_mul vs CPU; also red on GB10 at 439/440 against 438/440 here, [#907](https://github.com/mudler/vllm.cpp/issues/907)), `test_backend_cross_device` (`:2063` MoeSiluMul and `:2601` MoE combine/gate vs the CPU oracle, 80205/80207), `test_llama_embedding_fold` (`:254` engine vs direct at `epsilon(1e-5)`), `test_mtp_depth` (`:738` `CHECK(st.capture_shapes == 0)`), `test_qwen3_dflash2_draft` (`:2574` `CHECK(r.generate_threw.empty())`), and `test_ops_attention_dense_fa2` (`:692` `CHECK(Mismatches(on, ref) > 0)` -> `0 > 0`, which reads as a TEST arch-assumption: the case asserts the knob-ON path DIFFERS from the scalar reference, and with `fa2` DISABLED for `[110]` they are the same kernel). Four of the six are CUDA-vs-CPU numeric comparisons off by about one ulp, so they MAY be one sm_110 difference wearing four names -- a hypothesis from the logs, not a measurement. Also folded in: `test_gguf_device_fit_reach`, red and unattributed since 2026-08-15 (`:463`, moved from `:278`, assertion unchanged); and `test_serve_low_tools`, whose `(name, mode)` pair never moved while its CAUSE changed completely -- `73ada0df8` (#1661/#1662) fixed the [#961](https://github.com/mudler/vllm.cpp/issues/961) guard so it skips, the string `shellcheck` appears nowhere in the run's log, and the failure is now four `tests/tools/test_dflash2_speed_harness.py` `ShellDriverTest` cases of 517. The CONTROL was rerun and now falsifies the old conclusion: the baseline reads `FAILED (failures=3, errors=1, skipped=1)` and the control with `shellcheck` 0.9.0 installed reads `FAILED (failures=3, errors=1)` -- the same four cases, differing only in the vanished skip. **That vanished `skipped=1` is a second proof**: the one test that skipped in the baseline is the `shellcheck` guard itself, so with the binary present it stopped skipping and PASSED, showing the instrument was the only thing the install changed. #961 was CLOSED COMPLETED on 2026-08-23 acting on that prompt -- `73ada0df8` fixed its guard while referencing the sibling filing #1661/#1662, which had left #961 orphaned rather than resolved. Not in scope here, because they have owners and did not worsen: the four FA-2 refusals, `test_platform`/`test_op_parity` hardcoding GB10, `test_linear_method` and the three `qwen3_5_gdn_spec_routing` tests (#907), the FP8 pair ([#1725](https://github.com/mudler/vllm.cpp/issues/1725), which improved `SEGFAULT` -> `Failed`), and [#962](https://github.com/mudler/vllm.cpp/issues/962), which reproduced byte-identically at `bitdiff=15/32768`. Table and diff in [`environment.md`](environment.md); the re-measure debt itself is [#955](https://github.com/mudler/vllm.cpp/issues/955) | bug | +| [#1829](https://github.com/mudler/vllm.cpp/issues/1829) | `MODEL-MM-dots3-note` | **`src/vllm/model_executor/models/dots3_note.h:282` declares `[[noreturn]] static ForwardLogits ForwardDevice(...)` -- `[[noreturn]]` on a non-void return type -- and MSVC's C4646 plus warnings-as-errors turns that into `error C2220` at `dots3_note.cpp(606,31)`, so the whole `vllm` project fails to COMPILE on Windows.** GCC and Clang accept the declaration silently, so `build-test-cpu`, `build-test-cpu-arm64`, `build-newest-gcc` and `verify (cpu)` are all green on the same commit. Found on the `windows-msvc-cpu` job of #1821, whose own diff is four files and none of them this one; the declaration arrived with `849a7dd73` (#1805) which is an ancestor of `af320abb2`, so it is INHERITED and every pull request branched from `main` since carries it. It landed unseen because `windows-msvc-cpu` and `windows-msvc-vulkan` are PULL-REQUEST-ONLY jobs with no `main` baseline, which is the second half of the defect: `main` receives no Windows verdict at all. **NOT #584** -- that is the runtime `exit -1073740791` STATUS_STACK_BUFFER_OVERRUN in `test_openai_api_server.exe`, and a reader who stops at the job name will wave this real break through as the known one. Not repaired in flow by #1821: `ForwardDevice` overrides into a registry hook and cannot simply become `void`, so the fix is a semantic decision in an actively-developed file the `MODEL-MM-dots3-note` row owns, and #1821 has no MSVC to verify one against | bug | | [#1809](https://github.com/mudler/vllm.cpp/issues/1809) | `GATE-CI-ENFORCEMENT-FLOOR` | **`commit-protocol-tag` and `documentation-checkpoint` have been red on `main` since 2026-08-13 and cannot recover on their own: their walk bases on the head of the last SUCCESSFUL push run, so one unrepairable commit freezes the base and every later push re-walks the same violations over a range one commit wider.** Re-measured 2026-08-23 at `bacb71109`: `LAST_GREEN` is `fafa16f0f` of 2026-08-13T01:53, the range is **499** first-parent commits with **0** merges, and **41 distinct commits** in it can never be repaired because they are on `main` and adding a trailer or a task-branch arrival needs a rewrite `AGENTS.md` forbids — **35** fail `check-commit-trailers.py --range` (of which **20** carry no `FOLLOWING_AGENTS_PROTOCOL` marker at all and the other **15** carry it 1 to 9 times in a form the strict contract rejects, so the two counts describe the same population under different contracts, not different walks) and **6** disjoint ones fail `check-role-discipline.py` for a repository change that reached `main` with no task branch; `check-now-current.py` passes over the whole range and forgives nothing. The `LAST_GREEN` base is RIGHT about what it was built for and is not reverted: `github.event.before` skips a cancelled run's commits and nothing re-covers them, which is what lets the push lane be latest-only ([#822](https://github.com/mudler/vllm.cpp/issues/822), [#863](https://github.com/mudler/vllm.cpp/issues/863)), and cancelled runs are common here ([#1285](https://github.com/mudler/vllm.cpp/issues/1285)). FIXED IN FLOW with a recorded **enforcement floor** in `scripts/ci-enforcement-floor.txt` — one commit the walk never goes behind — taken as the newer of `LAST_GREEN` and the floor by ANCESTRY (`git merge-base --is-ancestor`, never by commit date, which is author-controlled and can go backwards). Losslessness survives because the floor is a lower clamp on an otherwise unchanged base: while the floor sits behind `LAST_GREEN`, which is the steady state, the resolved base is byte-identical to today's, and the only window that skips anything is `LAST_GREEN..floor` right after an advance — bounded, reviewed, and enumerated in the spec. The four byte-similar copies of the selection in `.github/workflows/ci.yml` collapse into `scripts/ci-walk-base.py`, which is the point of the extraction: of those four inline blocks exactly ONE was ever replayed by a test (`test_main_baseline.py::AgentRecordDiffRangeTests`, on `agent-record`'s body, under a shim that stubs every `python3` call and therefore pins which checker ran with which range string rather than the base rule), and the other three were executed by nothing, so the property most likely to break silently now has a test that replays a cancelled-run sequence against a real throwaway repository with the naive `before` base as its positive control. A per-commit exemption list was REJECTED — `AGENTS.md` has no waiver registry, the mechanism would have to be built three more times in checkers that have no exemption concept, and it never shrinks the walk, so the range grows by one per merge forever. This NARROWS enforcement by 41 commits and says so: enforcing on an immutable landed commit is a permanent red rather than a gate, because no contributor action can clear it, and a permanent red is read by nobody ([#1722](https://github.com/mudler/vllm.cpp/issues/1722)'s effect). No assertion is deleted and no checker is modified; only the base of the walk moves, and a new violation after the floor still reds, proved by a scratch-commit mutation rather than by reading the diff. Spec [ci-enforcement-floor.md](specs/ci-enforcement-floor.md) | bug | | [#1823](https://github.com/mudler/vllm.cpp/issues/1823) | `BACKEND-ATTN-REGISTRY` | **`FlashAttentionBackend::supports_compute_capability` is upstream's NVIDIA `capability >= (8,0)` (`include/vllm/v1/attention/backend.h:401-404`, `flash_attn.py:200-202`), and `MetalPlatform::get_device_capability` (`src/vllm/platforms/metal.cpp:35-38`) answers with the MTLGPUFamilyApple GENERATION — deliberately, with a comment saying `has_device_capability(N, 0)` means "Apple family >= N". `validate_configuration` at `src/vllm/v1/attention/backend.cpp:197` compares the two whenever `capability.present()`, so an SM-version bar is applied to an Apple family number.** Apple family 9 on the M4 gate box clears it by coincidence; a GitHub `macos-15` runner reports lower, FLASH_ATTN — the ONLY entry in `MetalPlatform::get_attn_backend_priority()` — is refused, and `SelectAttentionBackendName` throws. MEASURED 2026-08-23 on the first execution of `test_metal_backend` since `7020de936` (v0.0.2, 2026-08-11), run [32668677681](https://github.com/mudler/vllm.cpp/actions/runs/32668677681) at `744abeca3`: `tests/vt/test_metal_backend.cpp:170` ERROR `"No valid attention backend for device type 2 from {FLASH_ATTN: [compute capability not supported]}"`, suite 26 cases / 25 passed / **1 failed** / 3 skipped, 112336 assertions, `Status: FAILURE!`. **Red since `369ea7fd4` (2026-08-19), which is NOT an ancestor of `7020de936`** — four days in the exposure window [#1765](https://github.com/mudler/vllm.cpp/issues/1765) measured, with no lane able to run the case. **The prose above `backend.cpp:197` states the premise that is false**: it argues `DeviceCapability::present()` is already false for every platform that cannot answer, and Metal answers in a different unit; `vulkan.cpp` and `tenstorrent.cpp` need the same check before any repair. **Not fixed in flow**: every candidate repair (drop the predicate for non-SM platforms, stop reporting `present()` on Metal — which `test_metal_backend.cpp:153-154` asserts on purpose — or make the backend device-type aware) changes what selection means for kCPU/kMETAL/kVULKAN/kTENSTORRENT, so it takes the normal row, spec and fresh-review path. Until then `macos-metal-mlx` is RED on `main`, deliberately not hidden behind `continue-on-error` or a skip. Listed under `## Found in flow, filed, not fixed here` in [attn-validate-configuration.md](specs/attn-validate-configuration.md) | bug | | [#1819](https://github.com/mudler/vllm.cpp/issues/1819) | `ENG-MM-INPUT-PIPELINE` | **`scripts/mm/tower_skip_rss.sh` configured both build directories with `-DVLLM_CPP_BUILD_EXAMPLES=OFF` and then ran `ninja -C "$d" -j 4 vllm-server`, and `vllm-server` is an `examples/` target** -- the `OUTPUT_NAME` of `server` (`examples/CMakeLists.txt:91,108`), in a directory the root `CMakeLists.txt:2828` adds only under `if(VLLM_CPP_BUILD_EXAMPLES)`. Reproduced with the harness's own flags: the configure returns 0 and `ninja` answers `unknown target 'vllm-server'`, so the run `exit 4`s at arm A before any RSS exists. The block landed on `main` in `bacb71109` (#1364) and had never been executed. **Nothing could catch it**: `tests/scripts/test_tower_skip_rss_report.py` covers `--report-only`, `--check-source` and `--stage-check` -- every path needing no checkpoint -- while the configure, the build, `run_arm`, the `/health` poll and the kill/wait only ever run on a leased box, so the suite was 41/41 green over a harness that could not build its own binary. FIXED IN FLOW: `-DVLLM_CPP_BUILD_EXAMPLES=ON` (measured: `ninja -j 4 vllm-server` then returns 0 and writes `/examples/vllm-server`, the one file of that name in the tree and the path `docs/USAGE.md:54,95,128,204` names), the binary is NAMED rather than found by a `find` piped into `head -1`, and a new `--dry-run` prints the `cmake`/`ninja`/`run_arm` invocations out of the same variables the run issues them from and asserts that CMake defines the requested target under those flags -- statically, so it builds nothing and runs in CI, plus a live `ninja -t targets` prong on an already-configured tree that skips BY NAME when there is none. `DryRunTests` runs it against the script as committed, which is the case that reds on this defect, and against scratch copies with the flag flipped OFF, the flag dropped, and the binary path pointed away from where CMake writes it | bug | @@ -680,3 +681,4 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#1824](https://github.com/mudler/vllm.cpp/issues/1824) | `SPEC-DFLASH2` | **A configured speculator forces synchronous scheduling; upstream keeps async ON for the Eagle-type family (dflash included).** `model_loader.cpp` resolves `async_scheduling_enabled_` to false whenever `resolved_spec_config_` is present, a deferral recorded at SPEC-MTP I5d ([mtp-spec-decode.md](specs/mtp-spec-decode.md) §2.8's not-ported list). Upstream's polarity at the pin (`vllm/config/vllm.py:1064-1112`) disables async only for a method OUTSIDE `EagleModelTypes ∪ NgramGPUTypes ∪ {"dspark"}` — and `"dflash"` (which DFlash2 rides) and every MTP type are Eagle-type. At c1 spec decode (~360 steps / 2048 tokens) every host-side scheduling cost is serialized into each step, the largest named host-side divergence in the [#1574](https://github.com/mudler/vllm.cpp/issues/1574) gap. W7 under `SPEC-DFLASH2` ports the draft-in-output flow (AsyncScheduler `-1` placeholders, worker-side fill, `update_draft_token_ids_in_output`, the `async_tokens_to_discard` rollback guard) and flips the enable to upstream's method predicate; the GPU TPOT A/B stays owed to the operator. Spec [`spec-decode-async-scheduling.md`](specs/spec-decode-async-scheduling.md) | feature | | [#1754](https://github.com/mudler/vllm.cpp/issues/1754) | `GATE-CI-SITE-HUGO-LANE` | **`agent-record` is RED on `main` and on every pull request cut from it: `test_rendered_benchmark_index_links_resolve_to_emitted_pages` shells out to `hugo` and no job in `ci.yml` installs it, so `subprocess.run` raises `FileNotFoundError` before any assertion runs and `unittest` reports an ERROR.** Introduced by [#1714](https://github.com/mudler/vllm.cpp/pull/1714) at `1db7e59cf`; the scheduled baseline at `deb53c6a3` was green on this job and `08c81a892` and every scheduled run after it is red. FIXED IN FLOW by installing the renderer on the lane -- the same `peaceiris/actions-hugo@v3` action, the same `extended: true`, and the same `0.146.3` pin the `gh-pages` job publishes on -- plus `hugo version` as the step's first line and `tests/scripts/test_ci_site_lane.py` holding the two pins equal and the setup step present and ordered before the render. A SKIP GUARD WAS REJECTED AS THE FIX and is deliberately not in this change: it retires the red by arranging for the assertion to run nowhere, which [PR #1726](https://github.com/mudler/vllm.cpp/pull/1726) demonstrates -- its `agent-record` is SUCCESS having rendered no site. The guard is still wanted for a developer box without Hugo and is separately owned by [#1722](https://github.com/mudler/vllm.cpp/issues/1722) / #1726, which edits a disjoint file, so the two compose. [#1764](https://github.com/mudler/vllm.cpp/issues/1764) names this red as one of its three and is not closed by this change | bug | | [#1828](https://github.com/mudler/vllm.cpp/issues/1828) | `GATE-CI-SITE-HUGO-LANE` | **`tests/scripts/test_check_site.py` asserts a literal `10` rendered benchmark detail links, which is a stored count of `docs/benchmarks/*.md` inside another file, and #1754's lane fix is what ARMS it**: until Hugo was installed the case never reached that line in CI. Measured on `d60692c89` with Hugo 0.146.3+extended: 10 slugs, 10 table hrefs, no duplicated target and no unlinked slug, so the relationship is a bijection and the literal is derivable at read time -- with its own non-vacuity floor, since `sorted(x) == sorted(y)` is satisfied by two empty sets. NOT FIXED IN FLOW: deriving it changes what the case asserts rather than how it spells a number, which `AGENTS.md` routes through the normal row, spec and fresh-review path, and the file already has an open pull request against the same case. Listed under `## Owed` in [gate-ci-site-hugo-lane.md](specs/gate-ci-site-hugo-lane.md) | bug | +| [#1839](https://github.com/mudler/vllm.cpp/issues/1839) | `MUSIC3-DEPTH-DEVICE` | **The engine's call to `Music3SelectDepthArm` (`minimax_music3_speech.cpp:638`) is reachable but not gated, and [#1131](https://github.com/mudler/vllm.cpp/issues/1131) no longer covers it: #1131 named both device-arm twins and row `MUSIC3-DIT-ARM-REACH` closes it with only the DiT half.** Deleting the two-line call leaves `test_minimax_music3_ar` 37/37 · 640/640 and `test_minimax_music3_speech` 9/9 · 223/223 green ([`minimax-music3.md`](specs/minimax-music3.md) §19.5 carries the mutation and the binary hashes). Two things stop an existing gate from seeing it, and the SECOND is the one that matters: `--speech-device 1` is refused by name on a CPU-only build before a queue exists, AND §19.6's "device path TAKEN" leg rides `test_minimax_music3_ar`, whose observable `Music3DepthDeviceForwardCount()` is a counter §19.5 itself records as unreachable from production — its only readers are the tests written for it (`test_minimax_music3_ar.cpp:1325,1351,1583,1589,1753,1758`). The instrument that WOULD answer the call site is `ar.depth_staging`, emitted at `minimax_music3_llm.cpp:582` and read by nothing. NOT FIXED IN FLOW and the reason is precise: closing it needs the shipped engine on a real accelerator against the 28.5 GB checkpoint inside an `rc` lease, which is a second GPU leg and a second gate file, not a repair to the row in flight. It is closable by exactly `MUSIC3-DIT-ARM-REACH`'s method — a `gpu;checkpoint;music3`-labelled parity gate entering through `include/vllm.h` with `device = 1`, exiting 77 without its preconditions, asserting `ar.depth_staging` `calls == 1` with the host bucket absent — and that row's `thor:gpu0` job `f63f60e8-957a-4062-92f8-54e5bbb49d92` already FIRED `ar.depth_staging` once without asserting it, so the instrument is known live on the real path. Owed under `## Owed` in [`minimax-music3.md`](specs/minimax-music3.md) §19.7 | bug | diff --git a/.agents/specs/minimax-music3.md b/.agents/specs/minimax-music3.md index b2005df8d..2aba45ac2 100644 --- a/.agents/specs/minimax-music3.md +++ b/.agents/specs/minimax-music3.md @@ -4686,13 +4686,28 @@ Every item below is owned by row `MUSIC3-DEPTH-DEVICE` and names the issue that tracks it, per `.agents/reachability.md` and `AGENTS.md` `## Nothing lands dead`. * **The engine's call to `Music3SelectDepthArm` is reachable but not gated** - ([#1131](https://github.com/mudler/vllm.cpp/issues/1131), row + ([#1839](https://github.com/mudler/vllm.cpp/issues/1839), row `MUSIC3-DEPTH-DEVICE`). `--speech-device 1` reaches it and no CI gate can: deleting the two-line call leaves `test_minimax_music3_ar` 37/37 · 640/640 and `test_minimax_music3_speech` 9/9 · 223/223 green, because the engine needs the 28.5 GB checkpoint and a real device. §19.5 carries the mutation and the binary hashes. The *rule* it calls is gated on both sides of its condition, so what is - owed is the call, not the logic. It closes with the `thor:gpu0` legs in §19.6. + owed is the call, not the logic. + + **Retracked from [#1131](https://github.com/mudler/vllm.cpp/issues/1131) to + #1839, because #1131 closed with only its DiT half.** #1131 named both twins; + row `MUSIC3-DIT-ARM-REACH` closed the DiT one on `thor:gpu0` and closes #1131 + with it, which would have left this entry pointing at a closed issue. + **§19.6's "device path TAKEN" leg does not close it either**, and that is the + part worth stating plainly: that leg rides `test_minimax_music3_ar`, whose + observable is `Music3DepthDeviceForwardCount()` — a counter §19.5 itself + records as unreachable from production, whose only readers are the tests + written for it. The instrument that would answer the call site is + `ar.depth_staging` (`minimax_music3_llm.cpp:582`), and no test reads it. It is + closable by exactly `MUSIC3-DIT-ARM-REACH`'s method — a labelled parity gate + entering through `include/vllm.h` and asserting `ar.depth_staging` — and that + row's own `thor:gpu0` run fired the span once without asserting it, so the + instrument is known live on the real path. * **`scripts/check-fusion-consistency.py` is satisfied by a COMMENT** ([#1351](https://github.com/mudler/vllm.cpp/issues/1351), row `MUSIC3-DEPTH-DEVICE`). Replacing the `layers::UnquantizedMlpGateUpMethod` call diff --git a/.agents/specs/music3-dit-arm-reachability.md b/.agents/specs/music3-dit-arm-reachability.md index e8f3978ee..13127ce33 100644 --- a/.agents/specs/music3-dit-arm-reachability.md +++ b/.agents/specs/music3-dit-arm-reachability.md @@ -113,6 +113,11 @@ The focused gate is `./build/tests/test_minimax_music3_acoustic`. Adjacent suites run for regression: `test_minimax_music3_speech`, `test_minimax_music3_ar`, `test_music3_profile`. +Wave 3 adds two tree gates, both already on the preflight and CI lanes: +`scripts/check-test-registration.py`, which now pins the CTest label selection, +and `tests/scripts/test_check_test_registration.py`, which carries the four +mutations that hold it. + ## Risks and decisions **D1. The fixture is a fabricated denoise, not the shipped one.** Reduced DiT @@ -143,6 +148,10 @@ Every mutation reports compile status **before** any test result, the binary that never applied both read as a passing test. The tree is restored byte-for-byte and the restored binary is hashed back to the baseline. +**MET on both legs.** Wave 1 hashed all four CPU binaries back +(`RESTORE VERIFIED: True`, six times). Wave 2's CUDA binary hashes back too -- +`ed268392...` baseline and restored, `ce8bd1f8...` mutated -- see `### Restore`. + Filled in under `## Outcome` when taken. ## Owed @@ -150,16 +159,14 @@ Filled in under `## Outcome` when taken. Owned by row `MUSIC3-DIT-ARM-REACH`, per `.agents/reachability.md` and `AGENTS.md` `## Nothing lands dead`. -* **The engine's one-line call to `Music3SelectDitArm` is reachable but not - gated** ([#1131](https://github.com/mudler/vllm.cpp/issues/1131), row - `MUSIC3-DIT-ARM-REACH`). `--speech-device 1` reaches it and no CI gate can: - the engine needs the 28.5 GB checkpoint and a real accelerator, and - `SpeechEngineDeviceType` refuses device 1 on a CPU-only build before a queue - exists. The **rule** it calls is now gated on both sides of its condition and - both of #1131's own mutations now red; what is owed is the call itself. This is - the identical residual `minimax-music3.md` §19.7 carries for the depth arm, and - it closes with a `thor:gpu0` leg under an `rc` lease, not on a CPU runner. - **#1131 stays open for it.** +* ~~**The engine's one-line call to `Music3SelectDitArm` is reachable but not + gated**~~ **CLOSED** by wave 2 below, on `thor:gpu0` under `rc` lease + `f63f60e8-957a-4062-92f8-54e5bbb49d92`, 2026-08-23. The gate is + `tests/parity/test_minimax_music3_device_arm_real.cpp`, and deleting the call + reds it: `REQUIRE(staging != nullptr)`, 1 case / 0 passed / 1 failed, + `Status: FAILURE!`. What made the residual real was never in doubt and is now + measured: with the call gone the run took the host arm, `denoise.dit_host` + 196.786 s against `denoise.dit_device` 0.527 s, for 0.24 s of audio. * **The selector's `release_host` pass-through on the DEVICE path is not gated** ([#1131](https://github.com/mudler/vllm.cpp/issues/1131), row `MUSIC3-DIT-ARM-REACH`). MEASURED, not assumed: mutation M4b below replaces @@ -172,6 +179,172 @@ Owned by row `MUSIC3-DIT-ARM-REACH`, per `.agents/reachability.md` and selection case). The 9.7 GB peak the flag exists for is a property of the real checkpoint on Jetson Thor and belongs to `minimax-music3.md` §14. +## The GPU leg — closing the engine's own call site (wave 2) + +Everything above ran on a CPU-only build and left one thing unreached: the +engine's two-line call to `Music3SelectDitArm` at +`src/vllm/model_executor/models/minimax_music3_speech.cpp:701`. Mutation M5 +deleted it and every suite stayed green. This wave closes that, and it closes it +the only way it can be closed — by running the shipped engine on a real +accelerator against the real checkpoint, inside an `rc` lease. + +### What is being gated, precisely + +Not the arm, not the kernels, not the numbers. **The line that turns the arm +on.** A change that deleted it would return a 2.4B fp32 DiT to the host loops, +produce a correct song many hours late, and red nothing in the tree. + +### The gate + +`tests/parity/test_minimax_music3_device_arm_real.cpp`, one executable, one +`ctest` entry named `test_minimax_music3_device_arm_real`. + +**It enters through `include/vllm.h`.** `vllm_speech_engine_load` with +`vllm_speech_model_params.device = 1`, then `vllm_synthesize`. That is a +production entry point by `AGENTS.md` `## Nothing lands dead`'s own list, and its +reach does not depend on any other surface agreeing with it. +`examples/minimax_music3_gen` is a thin client of exactly these two calls. The +server's `/v1/audio/speech` route is **not**, and saying so avoids a claim +nothing measured: `ApiServer::handle_audio_speech` +(`src/vllm/entrypoints/openai/api_server.cpp:612`) goes through an internal +`synthesizer_` seam rather than through the C ABI, so it is a second production +path onto the same engine and not a client of this one. Nothing in the +gate constructs `Music3SpeechEngine`, reaches into `Music3DenoiseChunks`, or +builds an arm by hand. Those are wave 1's subjects and they are already gated. + +**Which arm ran is read off three instruments, and never off the audio.** The two +arms agree by design, so output equality answers nothing: + +| instrument | what it proves | expected | +|---|---|---| +| `acoustic.dit_staging` | `Music3SelectDitArm` **was called and took the device branch** — the span is inside the function, after the CPU early-return | present, `calls == 1` | +| `denoise.dit_device` | the production denoise loop **selected** the device branch | present, `calls == steps x windows` | +| `dit.pack` | the device forward's **body executed** — it lives inside `DitForwardDevice`, so a mislabelled bucket cannot fake it | present, `calls == 2 x steps x windows` | +| `denoise.dit_host` | the control that makes the three above mean something | **absent** | + +`windows` is not a constant the test asserts against itself: it is read from the +engine's own `denoise.windows` counter, which is the length of the chunk vector +the loop returned. `steps` is the request's. So the count assertion is arithmetic +over two independently produced quantities rather than agreement with whatever +was found. + +**The three are not independent of each other on the call-site mutation, and +claiming they were would be the easy overstatement.** `denoise.dit_device` is +emitted under `on_device = device_arm.engaged()` +(`minimax_music3_speech.cpp:250,346`), and an arm is engaged only through fields +`Music3SelectDitArm` sets, so deleting the engine's call reds all three at once. +`acoustic.dit_staging` is the **most direct** of them -- its span sits inside the +selector past the CPU early return, so it answers the call site and nothing else +-- and the other two corroborate it rather than answer it independently. + +They do separate on **other** defects, which is why all three are asserted. +`acoustic.dit_staging` present with `denoise.dit_device` absent is an arm staged +and then dropped by the loop, which is wave 1's M1. `dit.pack` below +`2 x steps x windows` with the label unchanged is a per-classifier-free-guidance-branch +fallback to the host forward: at `steps = 2`, `windows = 1` that reads `dit.pack` +2 against 4 while `denoise.dit_host` stays absent and the other two instruments +stay exactly correct, which is wave 1's M6 in its partial form. + +The waveform is checked for being finite and non-degenerate. That is a control +against an arm that threw halfway and left a plausible buffer behind, and the +case says so; it is not a numerical gate and no tolerance here is a claim. + +### What runs it, and what happens where it cannot + +**It is GPU-only and checkpoint-only by nature, and it is labelled so.** The +`ctest` entry carries `LABELS "gpu;checkpoint;music3"`, so +`ctest -L gpu` selects it. `-LE gpu` is NOT the complement a reader would assume +and `docs/USAGE.md` says so rather than implying it: this is the only labelled +test in the tree, and the six other checkpoint-gated suites +(`test_minimax_music3_{ar,llm,acoustic,quant,e2e}_real`, +`test_muse_glimmer_real_weights`) carry no label at all and, unlike this one, do +not exit 77 — they print a `SKIP` line and return, so CTest reports them +**Passed**. + +**And the label itself fails open, which for this row is in scope rather than +beside it.** Measured by the fresh review on CMake 3.28.3: with the label +renamed, `ctest -L gpu` prints `No tests were found!!!` and returns **0**. The +row's own GPU leg recorded `ctest_L_gpu_rc=0` as its "THE LABEL IS REAL" prong, +and that value is also what a zero-test selection returns — only the printed +`Total Tests: 1` separated them, and nothing asserted it. A row whose whole +subject is a gate that measures nothing while printing green cannot ship its own +discovery mechanism in that state. + +So the selection is pinned in `scripts/check-test-registration.py`, the one +production layer here that already asks CMake and CTest what exists instead of +parsing CMake text. `REQUIRED_LABEL_SELECTIONS` maps `gpu` to the **exact** set +`{test_minimax_music3_device_arm_real}`, read from +`ctest --show-only=json-v1` over a configured tree, and the expectation is a +literal in the checker — never read back out of `tests/CMakeLists.txt`, because +a checker that reads its expectation from the file it checks is a tautology. +The diagnostic names both sides of the comparison and says that an empty +selection is the dangerous case, per `.agents/verification.md` `## Make the +instrument say what it is measuring`. + +Four mutations hold it, in `LabelSelectionMutationTests` of +`tests/scripts/test_check_test_registration.py` (M44-M47 of that suite's fixed +manifest): the label renamed, the `set_tests_properties` call deleted, a second +test taking the label, and the labelled registration removed outright. The first +of those was additionally run against the **shipped** `tests/CMakeLists.txt`, +and the evidence is in `## Outcome, wave 3` below. + +**Why not a checker of its own.** A second checker would need its own preflight +and CI wiring and its own mutation suite for a pin that is one dictionary entry; +`check-test-registration.py` already configures the tree once and already owns +"the registration promise became vacuous", which is exactly what a label +selecting nothing is. The CPU-only configure it already performs sees this test, +because the entry is registered unconditionally — no accelerator is needed to +read what `-L gpu` would select. + +It **skips loudly and exits 77** — never 0 — when either precondition is absent: + +* no accelerator, decided by calling + `multimodal::SpeechEngineDeviceType(1, "minimax-music3")` and catching its + refusal, which is the same resolution the engine itself performs; +* no checkpoint, from `VLLM_CPP_MUSIC3_CHECKPOINT` or + `${CHECKPOINT_ROOT}/minimax-music3`. + +`vllm_cpp_add_test` registers `SKIP_RETURN_CODE 77`, so CTest reports **Skipped** +rather than Passed. A doctest case that returned early would print +`assertions: 0` and `Status: SUCCESS!`, which is the trap this repository has hit +twice. + +**A gate nobody runs is the defect being fixed, so where it runs is recorded +rather than implied.** No CI runner can execute it: CI has no accelerator and no +28.5 GB checkpoint. It runs inside an `rc` lease on a fleet device, and the +recipe is in `docs/USAGE.md` beside the other checkpoint-gated Music3 gates. + +### Risks and decisions, wave 2 + +**D5. The intra-DiT spans are armed, and that perturbs timing.** `dit.pack` needs +`VLLM_CPP_MUSIC3_DIT_SPANS=1`, which inserts a `Backend::Synchronize` at every +bracket inside the device forward. This gate makes no timing claim, so the +perturbation costs it nothing — and it buys the one instrument that cannot be +faked by a mislabelled bucket. `minimax-music3.md` §21.3's measurement of that +perturbation is unaffected and is not re-derived here. + +**D6. The request is the smallest one that still produces a window.** +`audio_duration_s = 0.24`, `num_inference_steps = 2`, `seed = 7`. A routing +assertion needs a window, not a song. The reduced request is why the mutation leg +is affordable: with the call deleted the run takes the HOST arm, and a host arm +at full duration is the thirty-hour failure this row exists to prevent. + +**D7. No fabricated device, and no call designed to fail.** Wave 1 excluded +`kCUDA` from its fabricated-queue list because a refusal latches a sticky CUDA +error the next unrelated kernel reports as its own. This wave makes no such call: +every path it takes is a path a user takes. + +### Evidence this wave owes + +The lease id, the device, the worker's boot id, `findmnt` for the path the +checkpoint was actually read from, and `SRC_BYTES == DST_BYTES` for the staging. +Compile status before any verdict, `git diff --stat` and the applied hunk count +for the mutation, and `test cases:` / `assertions:` / `Status:` plus the `[SKIP]` +count for every suite quoted. + +The acceptance criterion is one line: **with the engine's call to +`Music3SelectDitArm` deleted, this gate reds.** + ## Stop conditions Stop and report rather than widening scope if: the reachability mutation stays @@ -265,3 +438,280 @@ existing spans.** #1309 already landed `Music3DepthDeviceForwardCount()` and no production run reads it; the `dit.*` spans and the `denoise.dit_*` buckets are instruments the engine's own `profile::Report` prints, so they answer the reachability question without adding a symbol whose only reader is a test. + +## Outcome, wave 2 — the GPU leg + +Taken inside an `rc` lease on `thor:gpu0`, job +`f63f60e8-957a-4062-92f8-54e5bbb49d92`, 2026-08-23 21:54Z-22:23Z. Worker pod +`rc-worker-kk96r`, boot id `e2112cac-660b-434e-911d-33cbd29b9176`, unchanged +across the whole run. NVIDIA Thor, `compute_cap 11.0`, driver 595.78, 14 +aarch64 cores, 125 GB. Toolkit installed by the job: nvcc 13.0.88 +(`cuda-toolkit-13-0`). Build: Release, `-DVLLM_CPP_CUDA=ON +-DVLLM_CPP_CUDA_ARCHITECTURES=110 -DVLLM_CPP_TRITON=OFF`, named targets at +`-j 8`, `configure_rc=0`, `compile_rc=0` in 247 s. + +Tree under test: `bc61ce5182446475485f54504a742d9e38d1326a` on +`row/MUSIC3-DIT-ARM-REACH`, cloned inside the worker and asserted against the +requested SHA before anything was built (`got_sha` equal, `clean_tree=yes`). + +**Staging, because a gate that reads 28.5 GB over CIFS measures the share.** +`/workspace` is `//192.168.68.102/Data[/rc]`, `cifs`. The checkpoint was copied +to `/tmp/m3reach-ckpt` on the worker's own overlay in 1158 s, and the copy was +verified by bytes rather than by existence: +`SRC_BYTES == DST_BYTES == 28517617303`, a hard failure otherwise. The load the +gate then paid was `load.ar_weights` 8.145 s and `load.acoustic_weights` +2.466 s, against the 780 s + 249 s this lane measured over CIFS. + +### The gate, green + +| suite | test cases | assertions | Status | `[SKIP]` lines | +|---|---|---|---|---| +| `test_minimax_music3_device_arm_real` | 1 / 1 passed | 22 / 22 passed | SUCCESS! | 0 | +| `test_minimax_music3_acoustic` | 40 / 40 passed | 403 / 403 passed | SUCCESS! | 0 | +| `test_minimax_music3_speech` | 9 / 9 passed | 223 / 223 passed | SUCCESS! | 0 | +| `test_minimax_music3_ar` | 37 / 37 passed | 649 / 649 passed | SUCCESS! | 0 | +| `test_music3_profile` | 7 / 7 passed | 50 / 50 passed | SUCCESS! | 0 | + +CTest reported `1/1 Test #70: test_minimax_music3_device_arm_real ... Passed +17.66 sec`, and `ctest -N -L gpu` selected exactly that one test, which is the +label doing its job rather than being declared. + +`test_minimax_music3_acoustic` is 403 assertions here against 345 + 41 + 9 = 395 +on a CPU-only build: the `the DEVICE-resident DiT matches upstream on CUDA` case +runs 8 assertions on this box that it skips on a runner. That case is §14's and +this row does not touch it; it is named because a reader comparing the two totals +would otherwise have to guess. + +### WHICH ARM RAN, from the run's own instruments + +`device 1 resolves to 'cuda'` and `vllm_speech_engine_device()` returned 1, so +what follows is the granted device arm and not a request echoed back. + +| bucket | calls | seconds | what it proves | +|---|---|---|---| +| `acoustic.dit_staging` | 1 | 1.219 | the ENGINE CALLED `Music3SelectDitArm` and the selector took the device branch | +| `denoise.dit_device` | 2 | 0.527 | the production denoise loop SELECTED the device branch | +| `dit.pack` | 4 | 0.000 | `DitForwardDevice`'s BODY executed | +| `denoise.dit_host` | ABSENT | -- | the control | + +The counts are arithmetic over quantities the run produced: `request.steps` 2 +and `denoise.windows` 1, so `denoise.dit_device` is `steps x windows = 2` and +`dit.pack` is `2 x steps x windows = 4`, one bracket spanning both +classifier-free-guidance branches. The other fifteen intra-DiT spans agree with +the same arithmetic (`dit.norm1` and the eight other per-layer spans at 144 = +36 layers x 4 forwards). The request was `audio_duration_s = 0.24`, +`num_inference_steps = 2`, `seed = 7`, resolving to `request.max_frames` 6 and +`ar.frames` 6. The waveform control: 10240 samples x 2 ch at 44100 Hz, peak +0.00397296, every value finite. + +### THE ACCEPTANCE CRITERION — mutation M7 + +`.agents/reachability.md` `## The reachability mutation`: delete the production +call site and rerun the focused gate. The call is two lines; deleting them +outright would not compile, so the deletion keeps the declaration and +default-constructs the arm, which is the same thing the tree would contain if +the call had never been written. + +```diff + Music3DitDeviceWeights staged_dit; +- const Music3DenoiseDeviceArm arm = Music3SelectDitArm( +- queue_, config_.transformer, acoustic.dit, /*release_host=*/true, &staged_dit); ++ const Music3DenoiseDeviceArm arm; // MUTATION #1131: the engine's CALL deleted ++ (void)staged_dit; +``` + +`mutation_applied=1` (the replace asserts its own match count), `hunk_count=1`, +`git diff --stat` 1 file changed / 2 insertions / 2 deletions, and +**`compile_rc=0` reported BEFORE any verdict** -- a mutation that fails to build +reads as a passing test, which is why the rc is printed first and why M4 of +wave 1 is recorded as NO VERDICT rather than dropped. + +| id | mutation | compile_rc | `test_minimax_music3_device_arm_real` | +|---|---|---|---| +| M7 | the ENGINE'S OWN CALL to `Music3SelectDitArm`, deleted | 0 | **RED** -- 1 case / 0 passed / 1 failed, 9 assertions / 8 passed / 1 failed, `Status: FAILURE!`; CTest `***Failed 211.54 sec`. `REQUIRE(staging != nullptr)` at `:267` | + +**And what the red is made of, which is the point.** With the call gone the run +did not fail, break, or produce different audio. It produced +`denoise.dit_host` **196.786 s, 93.43% of the run**, where the shipped tree +produced `denoise.dit_device` **0.527 s** in the same bucket position -- on +**0.24 seconds** of audio, from a change that alters no number anywhere. Two +different ratios live in that paragraph and they are named separately so a +reader is not left deriving one from the other: the DENOISE BUCKET ratio is +196.786 / 0.527 = **373x**, and the CTest WALL ratio for the whole gate is +211.54 s mutated against 17.66 s shipped = **12x**, because the run also loads a +28.5 GB checkpoint and synthesises either way. Neither is a performance claim +(see `### What this leg did NOT do`). That is #1131's whole argument, executed +rather than asserted: the arms agree by design, so nothing that reads the output +can see this, and until this gate existed nothing in the tree could. + +M5 of wave 1 -- the same deletion, on a CPU-only build -- stayed green. It is +unchanged and remains recorded above; what changed is that a gate now exists +that the deletion reds. + +### Restore + +`git checkout --` on the one file, then `touch`. `git status --porcelain` empty, +`HEAD` still `bc61ce518`, rebuild `compile_rc=0`, and the gate re-run green: +1 / 1 cases, 22 / 22 assertions, `SUCCESS!`, CTest `Passed 13.84 sec`. + +**The restored binary hashes back to the baseline.** Three `sha256` lines of +`tests/test_minimax_music3_device_arm_real` are in job +`f63f60e8-957a-4062-92f8-54e5bbb49d92`'s log, and they are two distinct values, +not three: + +| stage | `sha256` | +|---|---| +| baseline, after the first build | `ed268392882246f8cf7ae78549086f3044be62bfb1f2269d7ae4b287bcec397c` | +| mutated, after the M7 build | `ce8bd1f894dec7c5a06bce5ff14d8bd88984b765c0a75df5233ebda1bbfa8130` | +| restored, after the restore build | `ed268392882246f8cf7ae78549086f3044be62bfb1f2269d7ae4b287bcec397c` | + +So `restored == baseline`, and the CUDA link on this toolchain reproduced +byte-for-byte across a rebuild. The fresh review reproduced the same answer +independently at head `718546680` on `thor:gpu0` in a second job with a second +build directory: baseline `ba9b62cb...` == restored `ba9b62cb...`, printed as +`HASH VERDICT: RESTORED == BASELINE`. + +**An earlier draft of this section said the opposite, and it is worth recording +why rather than only deleting it.** It compared the restored hash against the +MUTATED one instead of against the baseline, read the inequality as +irreproducibility, and generalised it into "this build links CUDA objects and +its link is not bit-reproducible". That sentence is deleted rather than +softened. It is the sentence a later row would have cited to waive a hash +restore on any CUDA build, and nothing measured it. The restore here is proven +by all three of the pair it claimed plus the hash: an empty +`git status --porcelain` against the exact SHA, a green re-run, and +`restored == baseline`. + +### What this leg did NOT do + +It did not measure speed. `denoise.dit_host` 196.786 s against +`denoise.dit_device` 0.527 s is the shape of the defect, on one geometry, with +the intra-DiT spans armed and therefore with a `Backend::Synchronize` at every +bracket inside the device forward. It is not a ratio anybody may quote: +`.agents/benchmarking.md` governs those and this run took no clock window, no +idle-host control and no A/B. + +It did not touch the second `## Owed` entry, the selector's `release_host` +pass-through on the device path. That flag IS now exercised end to end -- the +engine passes `true` and `acoustic.dit_staging` fired -- but nothing in this gate +would red if it were replaced by `false`, because the gate asserts routing and +not residency. The entry stands as written. + +It observed, without asserting, that `ar.depth_staging` fired once in the same +run. That is `minimax-music3.md` §19.7's twin residual for the depth arm, and it +is reachable by exactly this method. This row does not claim it. + +## Outcome, wave 3 — the fresh review's findings, repaired + +The fresh review at head `718546680` returned **FAIL**. It reproduced the +acceptance criterion on `thor:gpu0` in its own job and its own build directory +and confirmed the red is `REQUIRE(staging != nullptr)` rather than a timeout, so +the gate itself is sound. Every finding was in the record, and one of them was a +measured statement that is false. + +### F1 — the row stated the opposite of what it measured + +`### Restore` claimed the restored CUDA binary does not hash back to the +baseline, and generalised that into "this build links CUDA objects and its link +is not bit-reproducible". Both halves are wrong. Job +`f63f60e8-957a-4062-92f8-54e5bbb49d92`'s log carries three `sha256` lines for +`tests/test_minimax_music3_device_arm_real` and they hold **two** values: +`ed268392...` after the baseline build, `ce8bd1f8...` after the M7 build, and +`ed268392...` again after the restore build. The row compared **restored against +mutated** and read the inequality as irreproducibility. The review reproduced +the correct answer independently at this head — `ba9b62cb...` baseline against +`ba9b62cb...` restored, printed as `HASH VERDICT: RESTORED == BASELINE`. + +The generalisation is **deleted rather than softened**, and the reason is that it +travels: it is the sentence a later row would cite to waive a hash restore on any +CUDA build, and it already reached one reviewer brief as a binding instruction. +If some translation unit here genuinely does not link reproducibly, that is a +separate measured claim for whoever measures it. The `## Evidence this row owes` +line — "the restored binary is hashed back to the baseline" — is therefore MET on +both legs rather than waived on one. + +### F2 — the `gpu` label failed open, and the label is this row's own recipe + +Measured by the review on CMake 3.28.3: rename the label and `ctest -L gpu` +prints `No tests were found!!!` and returns **0**. Pinned in +`scripts/check-test-registration.py` — see `### What runs it, and what happens +where it cannot` for the design and for why it lives in that checker rather than +in a new one. + +**The mutation, on the shipped `tests/CMakeLists.txt` rather than on a fixture.** +`LABELS "gpu;checkpoint;music3"` renamed to `LABELS "device;checkpoint;music3"`, +`mutation_applied=1`, `git diff --stat` 1 file changed / 1 insertion / 1 +deletion for that file, 1 hunk: + +| stage | `scripts/check-test-registration.py` | +|---|---| +| shipped | **rc 0** — `OK: ... the configured tree matches the pinned label selection (-L gpu -> 1 [test_minimax_music3_device_arm_real]) ...` | +| label renamed | **rc 1** — `ERROR: ctest -L gpu selects 0 test(s) []; REQUIRED_LABEL_SELECTIONS ... pins 1 [test_minimax_music3_device_arm_real]` | +| restored | **rc 0**, and `git hash-object tests/CMakeLists.txt` is `9a74e857a5a8b51e23a80d23e3b73f3c3af9f0f1` before and after, with `git status --porcelain` empty for that path | + +`tests/scripts/test_check_test_registration.py` gains +`LabelSelectionMutationTests`: a passing baseline plus M44 (label renamed), M45 +(`set_tests_properties` deleted), M46 (a second test takes the label) and M47 +(the labelled registration removed outright). M46 is why the pin is an exact set +and not a floor. The fixed mutation manifest and its production-pinned digest are +updated in the same change, because the checker refuses a suite whose inventory +has drifted from the manifest. + +### F3 — `docs/USAGE.md` described a convention that exists at n = 1 + +The page offered `-L gpu` as "the device gates" and `-LE gpu` as "everything +else". Neither is true: this is the only labelled test in the tree, and the six +other checkpoint-gated suites carry no label and do not exit 77 — they print a +`SKIP` line, return, and CTest reports them **Passed**. The section now names the +one labelled gate, names the six unlabelled suites and their Passed-on-skip +behaviour, and tells the reader to read `Total Tests:` rather than the exit +status. + +### F4 — the depth twin got its own issue before this row's `Closes` fires + +#1131 named both device-arm twins. This row closes the DiT one, so +`minimax-music3.md` §19.7's un-struck depth entry would have been left pointing +at a closed issue. Filed as +[#1839](https://github.com/mudler/vllm.cpp/issues/1839), retracked in §19.7 and +appended to `.agents/issue-index.md`. `Closes #1131` stands: the DiT arm +genuinely is closed. + +The depth twin is genuinely ungated, and the reason is worth carrying: §19.6's +"device path TAKEN" leg rides `test_minimax_music3_ar`, whose observable +`Music3DepthDeviceForwardCount()` is a counter §19.5 itself records as +unreachable from production. `ar.depth_staging` — the instrument that would +answer the call site — is emitted at `minimax_music3_llm.cpp:582` and read by +nothing. This row's own `thor:gpu0` run fired it once, so it is known live. + +### F5 — an arithmetic non-sequitur beside two measured numbers + +"a 12x wall-clock difference" sat immediately after "196.786 s ... 0.527 s", +which is 373x. The 12x is the unstated CTest wall ratio, 211.54 / 17.66 = 11.98. +No number changed; both ratios are now named with the quantity each belongs to, +and neither is a performance claim. + +### F6 — "thin clients" was loose for the server + +`/v1/audio/speech` goes through `ApiServer::handle_audio_speech` +(`api_server.cpp:612`) and an internal `synthesizer_` seam, not the C ABI. It is +a second production path onto the same engine, not a client of this one. The +example genuinely does drive `vllm.h`. Reachability is unaffected, because +`include/vllm.h` is independently a listed production entry point. + +### The instrument-independence claim, narrowed + +The review noted that `denoise.dit_device` is emitted under +`on_device = device_arm.engaged()` (`minimax_music3_speech.cpp:250,346`), so it +would red on the call-site deletion by itself. The three instruments are +therefore **not** independent on M7: `acoustic.dit_staging` is the most direct +and the other two corroborate. They do separate on other defects, and the two +that matter are now named in `### What is being gated, precisely`. + +### What wave 3 did NOT do + +It did not re-run the device leg. The review reproduced the acceptance criterion +on hardware at this head and no finding disputes a measured value from +`f63f60e8-957a-4062-92f8-54e5bbb49d92`; a second lease would spend a fleet +device to re-derive a number two jobs already agree on. No number in +`## Outcome, wave 2` is changed by wave 3 — the only value corrected is the +hash comparison, which was a reading of the log rather than a measurement. diff --git a/docs/USAGE.md b/docs/USAGE.md index 366a97176..77e168227 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -382,6 +382,52 @@ VLLM_CPP_MUSIC3_PROFILE=1 ./build/vllm_music3_vocoder_conv_ab --lengths=86 `VLLM_CPP_CPU_THREADS` selects the pool size for both, and both print the thread count they actually got beside the count that was asked for. +## Run a gate that needs a GPU and a checkpoint + +Most of the suite runs anywhere. `test_minimax_music3_device_arm_real` cannot: +it needs an accelerator **and** a 28.5 GB checkpoint, so no +continuous-integration runner can execute it. It carries the CTest label +`gpu;checkpoint;music3` so that it is selectable by name rather than by whoever +remembers it exists, and a missing precondition makes it exit 77, which CTest +reports as **Skipped** rather than Passed. + +```sh +ctest --test-dir build -L gpu -N # list it; expect `Total Tests: 1` +ctest --test-dir build -L gpu -V # run it +``` + +**Read the count, not the exit status.** `ctest -L