test(MUSIC3-DIT-ARM-REACH): gate the engine's own call to the DiT arm selector on a real device (#1131)

.agents/specs/minimax-music3.md

@@ -4686,13 +4686,28 @@ Every item below is owned by row `MUSIC3-DEPTH-DEVICE` and names the issue that
 tracks it, per `.agents/reachability.md` and `AGENTS.md` `## Nothing lands dead`.
 
 * **The engine's call to `Music3SelectDepthArm` is reachable but not gated**
-  ([#1131](https://github.com/mudler/vllm.cpp/issues/1131), row
+  ([#1839](https://github.com/mudler/vllm.cpp/issues/1839), row
   `MUSIC3-DEPTH-DEVICE`). `--speech-device 1` reaches it and no CI gate can:
   deleting the two-line call leaves `test_minimax_music3_ar` 37/37 · 640/640 and
   `test_minimax_music3_speech` 9/9 · 223/223 green, because the engine needs the
   28.5 GB checkpoint and a real device. §19.5 carries the mutation and the binary
   hashes. The *rule* it calls is gated on both sides of its condition, so what is
-  owed is the call, not the logic. It closes with the `thor:gpu0` legs in §19.6.
+  owed is the call, not the logic.
+
+  **Retracked from [#1131](https://github.com/mudler/vllm.cpp/issues/1131) to
+  #1839, because #1131 closed with only its DiT half.** #1131 named both twins;
+  row `MUSIC3-DIT-ARM-REACH` closed the DiT one on `thor:gpu0` and closes #1131
+  with it, which would have left this entry pointing at a closed issue.
+  **§19.6's "device path TAKEN" leg does not close it either**, and that is the
+  part worth stating plainly: that leg rides `test_minimax_music3_ar`, whose
+  observable is `Music3DepthDeviceForwardCount()` — a counter §19.5 itself
+  records as unreachable from production, whose only readers are the tests
+  written for it. The instrument that would answer the call site is
+  `ar.depth_staging` (`minimax_music3_llm.cpp:582`), and no test reads it. It is
+  closable by exactly `MUSIC3-DIT-ARM-REACH`'s method — a labelled parity gate
+  entering through `include/vllm.h` and asserting `ar.depth_staging` — and that
+  row's own `thor:gpu0` run fired the span once without asserting it, so the
+  instrument is known live on the real path.
 * **`scripts/check-fusion-consistency.py` is satisfied by a COMMENT**
   ([#1351](https://github.com/mudler/vllm.cpp/issues/1351), row
   `MUSIC3-DEPTH-DEVICE`). Replacing the `layers::UnquantizedMlpGateUpMethod` call

docs/USAGE.md

@@ -382,6 +382,52 @@ VLLM_CPP_MUSIC3_PROFILE=1 ./build/vllm_music3_vocoder_conv_ab --lengths=86
 `VLLM_CPP_CPU_THREADS` selects the pool size for both, and both print the
 thread count they actually got beside the count that was asked for.
 
+## Run a gate that needs a GPU and a checkpoint
+
+Most of the suite runs anywhere. `test_minimax_music3_device_arm_real` cannot:
+it needs an accelerator **and** a 28.5 GB checkpoint, so no
+continuous-integration runner can execute it. It carries the CTest label
+`gpu;checkpoint;music3` so that it is selectable by name rather than by whoever
+remembers it exists, and a missing precondition makes it exit 77, which CTest
+reports as **Skipped** rather than Passed.
+
+```sh
+ctest --test-dir build -L gpu -N        # list it; expect `Total Tests: 1`
+ctest --test-dir build -L gpu -V        # run it
+```
+
+**Read the count, not the exit status.** `ctest -L <label>` prints
+`No tests were found!!!` and still returns 0 when the label selects nothing, so
+a renamed or dropped label reads as a clean run of a gate that never executed.
+
+**`-L gpu` is not a taxonomy of the device gates**, and `-LE gpu` is not
+"everything else". Exactly one test in this tree carries a label today, and it
+is this one. The other checkpoint-gated suites --
+`test_minimax_music3_ar_real`, `_llm_real`, `_acoustic_real`, `_quant_real`,
+`_e2e_real` and `test_muse_glimmer_real_weights` -- carry no label, and unlike
+this one they do not exit 77: without a checkpoint they print a `SKIP` line and
+return normally, so **CTest reports them Passed**. For those, read the
+transcript rather than the CTest verdict.
+
+It drives the C ABI with `device = 1` and asserts, from the engine's own profile
+buckets, that the 2.4B flow-matching transformer ran on the accelerator rather
+than on the host reference loops. The two arms agree numerically by design, so
+the audio cannot answer that question and the gate never asks it to.
+
+```sh
+# Inside an `rc` lease on a fleet device -- never over `ssh`.
+# Stage the checkpoint to LOCAL disk first: read over the shared CIFS mount it
+# is the dominant cost of the run.
+export VLLM_CPP_MUSIC3_CHECKPOINT=/local/disk/minimax-music3
+ctest --test-dir build -R test_minimax_music3_device_arm_real -V
+```
+
+Without `VLLM_CPP_MUSIC3_CHECKPOINT` the gate falls back to
+`${CHECKPOINT_ROOT}/minimax-music3`, and without either it skips and says so.
+It needs a build configured with an accelerator backend; on a CPU-only build
+`--speech-device 1` is refused by name before a queue exists, and the gate
+skips with that refusal quoted.
+
 ## First-line troubleshooting
 
 - Run the executable with `--help` and confirm that you are using the expected

scripts/check-test-registration.py

@@ -6,6 +6,17 @@
 This tree gate pins the small set of tests whose review explicitly requires a
 non-vacuous registration guard.  It also verifies that the shared helper still
 creates an executable *and* registers that executable with CTest.
+
+It pins CTest LABELS for the same reason, and the reason is measured rather than
+assumed.  A gate whose preconditions no runner has is invoked by label -- the
+documented recipe is ``ctest -L gpu`` -- and ``ctest -L`` prints
+``No tests were found!!!`` and returns **0** when the label selects nothing
+(CMake 3.28.3).  So a renamed or deleted label turns a documented gate into a
+command that measures nothing while reporting success, which is the exact defect
+class the labelled gate itself exists to close.  ``REQUIRED_LABEL_SELECTIONS``
+holds the expected selection as a literal in this file, never read back out of
+``tests/CMakeLists.txt``, because a checker that reads its expectation from the
+file it checks is a tautology.
 """
 
 from __future__ import annotations
@@ -29,13 +40,21 @@
 MUTATION_SUITE = ROOT / "tests/scripts/test_check_test_registration.py"
 MUTATION_MANIFEST = ROOT / "tests/scripts/check_test_registration_mutations.txt"
 MUTATION_MANIFEST_SHA256 = (
-    "46ac35fc533e345aa7735aeffde9d49524598998bdca1b784bd872d21a012803"
+    "40377fb90253d514a326bde1785b9867e848eebda0ff248b95f2bea9d3b5362b"
 )
 
 REQUIRED_TESTS = {
     "test_device_selection": "vllm/entrypoints/test_device_selection.cpp",
 }
 
+# label -> the EXACT set of CTest test names it may select.  Exact, not a floor:
+# a floor cannot see a second test that quietly joins a lane whose whole purpose
+# is that a human runs it deliberately inside a lease.  Adding a labelled gate is
+# a one-line addition here and is meant to be a deliberate record.
+REQUIRED_LABEL_SELECTIONS = {
+    "gpu": ("test_minimax_music3_device_arm_real",),
+}
+
 def _without_line_comments(text: str) -> str:
     """Remove ``#`` comments while preserving quoted ``#`` characters."""
 
@@ -290,6 +309,93 @@ def _configured_contract_errors(
     return errors
 
 
+def _test_labels(test: dict[str, object]) -> set[str]:
+    """Return the CTest LABELS of one ``--show-only=json-v1`` test entry."""
+
+    properties = test.get("properties", [])
+    if not isinstance(properties, list):
+        return set()
+    labels: set[str] = set()
+    for prop in properties:
+        if not isinstance(prop, dict) or prop.get("name") != "LABELS":
+            continue
+        value = prop.get("value")
+        if isinstance(value, str):
+            labels.add(value)
+        elif isinstance(value, list):
+            labels.update(entry for entry in value if isinstance(entry, str))
+    return labels
+
+
+def _label_selection_errors(
+    tests: dict[str, dict[str, object]], selections: dict[str, tuple[str, ...]]
+) -> list[str]:
+    """Compare the configured LABELS against this file's literal pin.
+
+    The diagnostic names BOTH sides of the comparison in words.  ``ctest -L``
+    reports success over an empty selection, so a reader who is handed only a
+    return code cannot tell a passing lane from an absent one.
+    """
+
+    errors: list[str] = []
+    for label, expected in sorted(selections.items()):
+        selected = sorted(
+            name for name, test in tests.items() if label in _test_labels(test)
+        )
+        wanted = sorted(expected)
+        if selected == wanted:
+            continue
+        errors.append(
+            f"ctest -L {label} selects {len(selected)} test(s) "
+            f"[{', '.join(selected) if selected else '<none>'}]; "
+            f"REQUIRED_LABEL_SELECTIONS in scripts/check-test-registration.py pins "
+            f"{len(wanted)} [{', '.join(wanted) if wanted else '<none>'}]. "
+            "Compared the LABELS property of the CONFIGURED tree against the literal "
+            "pin in this checker, not against tests/CMakeLists.txt text. "
+            "An empty selection is the dangerous case: ctest -L returns 0 over it"
+        )
+    return errors
+
+
+def _configured_label_errors(
+    source_dir: Path,
+    build_dir: Path,
+    selections: dict[str, tuple[str, ...]],
+    extra_args: list[str] | None = None,
+) -> list[str]:
+    """Ask CMake/CTest which tests a label selects, then compare with the pin."""
+
+    configured = _configure(source_dir, build_dir, extra_args)
+    if configured.returncode != 0:
+        return [
+            "CMake configure failed while proving CTest label selection; "
+            "no label could be read, which fails closed"
+        ]
+    configuration, _ = _codemodel_targets(build_dir)
+    return _label_selection_errors(_ctest_tests(build_dir, configuration), selections)
+
+
+def label_errors(
+    cmake_text: str, selections: dict[str, tuple[str, ...]] | None = None
+) -> list[str]:
+    """Return violations of the CTest label-selection contract."""
+
+    if selections is None:
+        selections = REQUIRED_LABEL_SELECTIONS
+    with tempfile.TemporaryDirectory(prefix="vllm-label-unit-") as temporary:
+        root = Path(temporary)
+        (root / "CMakeLists.txt").write_text(cmake_text, encoding="utf-8")
+        for source in {
+            *REQUIRED_TESTS.values(),
+            "vllm/entrypoints/other.cpp",
+            "parity/test_minimax_music3_device_arm_real.cpp",
+        }:
+            path = root / source
+            path.parent.mkdir(parents=True, exist_ok=True)
+            path.write_text("int label_guard_dummy;\n", encoding="utf-8")
+        return _configured_label_errors(root, root / "build", selections)
+
+
 def registration_errors(
     cmake_text: str, required: dict[str, str] | None = None
 ) -> list[str]:
@@ -734,26 +840,34 @@ def check_tree(root: Path = ROOT) -> list[str]:
     if missing:
         return [f"required registration-guard input is missing: {path}" for path in missing]
 
+    configure_args = [
+        "-DVLLM_CPP_CUDA=OFF",
+        "-DVLLM_CPP_HIP=OFF",
+        "-DVLLM_CPP_VULKAN=OFF",
+        "-DVLLM_CPP_METAL=OFF",
+        "-DVLLM_CPP_MLX=OFF",
+        "-DVLLM_CPP_TRITON=OFF",
+        "-DVLLM_CPP_BUILD_TESTS=ON",
+        "-DVLLM_CPP_BUILD_EXAMPLES=OFF",
+        "-DVLLM_CPP_SERVER=OFF",
+        "-DCMAKE_BUILD_TYPE=Release",
+    ]
     with tempfile.TemporaryDirectory(prefix="vllm-registration-tree-") as temporary:
+        build_dir = Path(temporary) / "build"
         registration = _configured_contract_errors(
             root,
-            Path(temporary) / "build",
+            build_dir,
             {
                 target: f"tests/{source}"
                 for target, source in REQUIRED_TESTS.items()
             },
-            [
-                "-DVLLM_CPP_CUDA=OFF",
-                "-DVLLM_CPP_HIP=OFF",
-                "-DVLLM_CPP_VULKAN=OFF",
-                "-DVLLM_CPP_METAL=OFF",
-                "-DVLLM_CPP_MLX=OFF",
-                "-DVLLM_CPP_TRITON=OFF",
-                "-DVLLM_CPP_BUILD_TESTS=ON",
-                "-DVLLM_CPP_BUILD_EXAMPLES=OFF",
-                "-DVLLM_CPP_SERVER=OFF",
-                "-DCMAKE_BUILD_TYPE=Release",
-            ],
+            configure_args,
+        )
+        # The labelled gate is registered unconditionally, so the CPU-only
+        # configure above already knows about it: no accelerator is needed to
+        # read what `ctest -L gpu` would select.
+        registration += _configured_label_errors(
+            root, build_dir, REQUIRED_LABEL_SELECTIONS, configure_args
         )
     integrity = mutation_suite_integrity_errors(
         paths["tests/scripts/test_check_test_registration.py"].read_text(encoding="utf-8"),
@@ -772,8 +886,13 @@ def main() -> int:
         for error in errors:
             print(f"ERROR: {error}", file=sys.stderr)
         return 1
+    selection = ", ".join(
+        f"-L {label} -> {len(names)} [{', '.join(sorted(names))}]"
+        for label, names in sorted(REQUIRED_LABEL_SELECTIONS.items())
+    )
     print(
-        "OK: required regression tests have executable + CTest registration "
+        "OK: required regression tests have executable + CTest registration, "
+        f"the configured tree matches the pinned label selection ({selection}), "
         "and the guard is wired into preflight/CI."
     )
     return 0

tests/CMakeLists.txt

@@ -277,6 +277,32 @@ target_include_directories(test_minimax_music3_e2e_real PRIVATE
 # under a saturated `ctest -jN` the server's accept thread starves and the
 # client sees a connection failure rather than the defect it was looking for.
 set_tests_properties(test_minimax_music3_e2e_real PROPERTIES RUN_SERIAL ON)
+
+# THE ENGINE'S OWN CALL to the DiT arm selector (#1131, row MUSIC3-DIT-ARM-REACH).
+# Wave 1 of that row gated the RULE from a CPU runner; what stayed unreached was
+# the two-line CALL in `Music3SpeechEngine::Synthesize`, which needs an
+# accelerator and the 28.5 GB checkpoint and therefore cannot run on anything CI
+# owns. This entry is that gate. It drives `include/vllm.h` -- the production
+# entry point -- with `device = 1` and asserts WHICH ARM RAN off the engine's own
+# profile buckets, never off the audio, because the two arms agree by design.
+#
+# LABELLED, because a gate whose preconditions no runner has must be selectable
+# and excludable by name rather than by whoever remembers it exists:
+#   ctest -L gpu    selects it   (inside an `rc` lease; see docs/USAGE.md)
+#   ctest -LE gpu   excludes it
+# Without a device or a checkpoint it EXITS 77 and CTest reports **Skipped** --
+# `vllm_cpp_add_test` registers SKIP_RETURN_CODE 77 -- so it can never be read as
+# a pass. RUN_SERIAL because it stages ~10 GB of fp32 DiT onto the accelerator.
+vllm_cpp_add_test(test_minimax_music3_device_arm_real
+  parity/test_minimax_music3_device_arm_real.cpp)
+# Reaches src/ for `music3_profile.h`, the INTERNAL instrument the engine's own
+# `profile::Report` prints. Deliberately not a public header: it is how this gate
+# reads which arm ran, and it is not part of the ABI.
+target_include_directories(test_minimax_music3_device_arm_real PRIVATE
+  ${CMAKE_SOURCE_DIR}/src)
+set_tests_properties(test_minimax_music3_device_arm_real PROPERTIES
+  RUN_SERIAL ON
+  LABELS "gpu;checkpoint;music3")
 # ARCH-ONE-SURFACE ROW 2 fold gate: the video seam == the replicated pre-fold
 # pipeline == the goldens the PRE-fold minimax-h3-gen binary rendered at the
 # branch base (frames + WAV byte-identical, mux argv pinned).