Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions .agents/specs/minimax-music3.md
Original file line number Diff line number Diff line change
Expand Up @@ -4708,6 +4708,21 @@ tracks it, per `.agents/reachability.md` and `AGENTS.md` `## Nothing lands dead`
entering through `include/vllm.h` and asserting `ar.depth_staging` — and that
row's own `thor:gpu0` run fired the span once without asserting it, so the
instrument is known live on the real path.

**DISCHARGED by row `MUSIC3-DEPTH-ARM-REACH`, spec
[`music3-depth-arm-reachability.md`](music3-depth-arm-reachability.md), which
closes #1839.** `tests/parity/test_minimax_music3_depth_arm_real.cpp` enters
through `include/vllm.h` with `device = 1`, is labelled `gpu;checkpoint;music3`
with that selection pinned by name in `scripts/check-test-registration.py`,
exits 77 without a device or checkpoint, and asserts `ar.depth_staging`
`calls == 1` with `ar.depth_host` absent. The host bucket that entry asked for
now exists: `Music3DepthStage`'s append lambda emits `ar.depth_device` and
`ar.depth_host`, the depth twin of `denoise.dit_device` / `denoise.dit_host`,
so every instrument the gate reads is one the engine's own `profile::Report`
prints rather than a counter written for a test. Deleting the engine's call
reds the gate on `thor:gpu0`; that row's `## Outcome` carries the job, the
hashes and the failing assertion. **This entry is closed; the one below is
not.**
* **`scripts/check-fusion-consistency.py` is satisfied by a COMMENT**
([#1351](https://github.com/mudler/vllm.cpp/issues/1351), row
`MUSIC3-DEPTH-DEVICE`). Replacing the `layers::UnquantizedMlpGateUpMethod` call
Expand Down
438 changes: 438 additions & 0 deletions .agents/specs/music3-depth-arm-reachability.md

Large diffs are not rendered by default.

43 changes: 24 additions & 19 deletions docs/USAGE.md
Original file line number Diff line number Diff line change
Expand Up @@ -384,48 +384,53 @@ thread count they actually got beside the count that was asked for.

## Run a gate that needs a GPU and a checkpoint

Most of the suite runs anywhere. `test_minimax_music3_device_arm_real` cannot:
it needs an accelerator **and** a 28.5 GB checkpoint, so no
continuous-integration runner can execute it. It carries the CTest label
`gpu;checkpoint;music3` so that it is selectable by name rather than by whoever
remembers it exists, and a missing precondition makes it exit 77, which CTest
reports as **Skipped** rather than Passed.
Most of the suite runs anywhere. Two tests cannot:
`test_minimax_music3_device_arm_real` and `test_minimax_music3_depth_arm_real`
each need an accelerator **and** a 28.5 GB checkpoint, so no
continuous-integration runner can execute either. Both carry the CTest label
`gpu;checkpoint;music3` so that they are selectable by name rather than by
whoever remembers they exist, and a missing precondition makes them exit 77,
which CTest reports as **Skipped** rather than Passed.

```sh
ctest --test-dir build -L gpu -N # list it; expect `Total Tests: 1`
ctest --test-dir build -L gpu -V # run it
ctest --test-dir build -L gpu -N # list them; expect `Total Tests: 2`
ctest --test-dir build -L gpu -V # run them
```

**Read the count, not the exit status.** `ctest -L <label>` prints
`No tests were found!!!` and still returns 0 when the label selects nothing, so
a renamed or dropped label reads as a clean run of a gate that never executed.

**`-L gpu` is not a taxonomy of the device gates**, and `-LE gpu` is not
"everything else". Exactly one test in this tree carries a label today, and it
is this one. The other checkpoint-gated suites --
"everything else". Exactly two tests in this tree carry a label today and they
are these. The other checkpoint-gated suites --
`test_minimax_music3_ar_real`, `_llm_real`, `_acoustic_real`, `_quant_real`,
`_e2e_real` and `test_muse_glimmer_real_weights` -- carry no label, and unlike
this one they do not exit 77: without a checkpoint they print a `SKIP` line and
these two they do not exit 77: without a checkpoint they print a `SKIP` line and
return normally, so **CTest reports them Passed**. For those, read the
transcript rather than the CTest verdict.

It drives the C ABI with `device = 1` and asserts, from the engine's own profile
buckets, that the 2.4B flow-matching transformer ran on the accelerator rather
than on the host reference loops. The two arms agree numerically by design, so
the audio cannot answer that question and the gate never asks it to.
Both drive the C ABI with `device = 1` and assert, from the engine's own profile
buckets, which arm ran -- `test_minimax_music3_device_arm_real` for the 2.4B
flow-matching transformer and `test_minimax_music3_depth_arm_real` for the
0.646B RVQ depth decoder. They are separate entries because the two arms are
selected at two separate call sites on one `--speech-device 1` switch, which is
how one of them drifts. Each arm agrees numerically with its host reference by
design, so the audio cannot answer the question and neither gate asks it to.

```sh
# Inside an `rc` lease on a fleet device -- never over `ssh`.
# Stage the checkpoint to LOCAL disk first: read over the shared CIFS mount it
# is the dominant cost of the run.
export VLLM_CPP_MUSIC3_CHECKPOINT=/local/disk/minimax-music3
ctest --test-dir build -R test_minimax_music3_device_arm_real -V
ctest --test-dir build -R test_minimax_music3_depth_arm_real -V
```

Without `VLLM_CPP_MUSIC3_CHECKPOINT` the gate falls back to
`${CHECKPOINT_ROOT}/minimax-music3`, and without either it skips and says so.
It needs a build configured with an accelerator backend; on a CPU-only build
`--speech-device 1` is refused by name before a queue exists, and the gate
Without `VLLM_CPP_MUSIC3_CHECKPOINT` both gates fall back to
`${CHECKPOINT_ROOT}/minimax-music3`, and without either they skip and say so.
They need a build configured with an accelerator backend; on a CPU-only build
`--speech-device 1` is refused by name before a queue exists, and each gate
skips with that refusal quoted.

## First-line troubleshooting
Expand Down
8 changes: 6 additions & 2 deletions scripts/check-test-registration.py
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ class the labelled gate itself exists to close. ``REQUIRED_LABEL_SELECTIONS``
MUTATION_SUITE = ROOT / "tests/scripts/test_check_test_registration.py"
MUTATION_MANIFEST = ROOT / "tests/scripts/check_test_registration_mutations.txt"
MUTATION_MANIFEST_SHA256 = (
"40377fb90253d514a326bde1785b9867e848eebda0ff248b95f2bea9d3b5362b"
"9c35a1373af09bab9bb9fb65e1ee7bb15e7e831c379b00487eda5235a2bbcf9a"
)

REQUIRED_TESTS = {
Expand All @@ -52,7 +52,10 @@ class the labelled gate itself exists to close. ``REQUIRED_LABEL_SELECTIONS``
# is that a human runs it deliberately inside a lease. Adding a labelled gate is
# a one-line addition here and is meant to be a deliberate record.
REQUIRED_LABEL_SELECTIONS = {
"gpu": ("test_minimax_music3_device_arm_real",),
"gpu": (
"test_minimax_music3_depth_arm_real",
"test_minimax_music3_device_arm_real",
),
}

def _without_line_comments(text: str) -> str:
Expand Down Expand Up @@ -389,6 +392,7 @@ def label_errors(
*REQUIRED_TESTS.values(),
"vllm/entrypoints/other.cpp",
"parity/test_minimax_music3_device_arm_real.cpp",
"parity/test_minimax_music3_depth_arm_real.cpp",
}:
path = root / source
path.parent.mkdir(parents=True, exist_ok=True)
Expand Down
12 changes: 12 additions & 0 deletions src/vllm/model_executor/models/minimax_music3_llm.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -466,11 +466,23 @@ std::vector<float> Music3DepthStage(const std::vector<float>& last_hidden_condit
// THE PRODUCTION SELECTION (#1309, spec §19.5). Deleting this lambda's device
// branch is the reachability mutation, and the composed-stage gate goes RED on
// it — which is the leg #1131 records as missing for the DiT arm.
//
// The two counters say WHICH branch ran, and they are the depth twin of
// `denoise.dit_device` / `denoise.dit_host` (#1839). They exist because the
// two arms agree numerically by design, so no property of the codes, the
// hidden states or the audio can separate them, and because the only other
// observable that could — `Music3DepthDeviceForwardCount()` — is read by no
// production run, which makes it an instrument for a class rather than for a
// capability. `profile::Count` writes a `seconds = -1` pure counter, so it
// joins no leaf sum and perturbs no split; with the profile off it returns on
// one predicted branch and reads no clock.
const auto append = [&](const std::vector<float>& embeds) {
if (device_arm.engaged()) {
profile::Count("ar.depth_device", 1);
return DepthDecoderAppendDevice(*device_arm.queue, config, *device_arm.depth, embeds,
/*batch=*/2, &device_cache);
}
profile::Count("ar.depth_host", 1);
return DepthDecoderAppend(embeds, /*batch=*/2, config, weights.depth, ArCompute::kBFloat16,
&cache);
};
Expand Down
30 changes: 29 additions & 1 deletion tests/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -193,7 +193,11 @@ target_include_directories(test_minimax_music3_loader PRIVATE
# The full-scale bf16 companion is tests/parity/test_minimax_music3_ar_real.cpp.
vllm_cpp_add_test(test_minimax_music3_ar vllm/models/test_minimax_music3_ar.cpp)
target_include_directories(test_minimax_music3_ar PRIVATE
${CMAKE_CURRENT_SOURCE_DIR}/vllm/models)
${CMAKE_CURRENT_SOURCE_DIR}/vllm/models
# ...and src/ for `music3_profile.h`, the INTERNAL instrument that carries the
# `ar.depth_device` / `ar.depth_host` buckets this suite drives on both sides
# (#1839). Deliberately not a public header.
${CMAKE_SOURCE_DIR}/src)
# The FULL-SCALE companion: the REAL bf16 checkpoint driven on the REAL inputs,
# against the committed oracle capture (frame_hiddens / condition_chunk0 /
# rvq_codes). Checkpoint-gated on MUSIC3_CHECKPOINT or CHECKPOINT_ROOT; every
Expand Down Expand Up @@ -303,6 +307,30 @@ target_include_directories(test_minimax_music3_device_arm_real PRIVATE
set_tests_properties(test_minimax_music3_device_arm_real PROPERTIES
RUN_SERIAL ON
LABELS "gpu;checkpoint;music3")

# THE ENGINE'S OWN CALL to the DEPTH arm selector (#1839, row
# MUSIC3-DEPTH-ARM-REACH). The twin of the entry above, and a separate entry for
# a reason: #1131 named BOTH device-arm twins, the DiT one closed with it, and
# the depth call site at `minimax_music3_speech.cpp:638` stayed unreached --
# deleting it left `test_minimax_music3_ar` 37/37 and `test_minimax_music3_speech`
# 9/9 green. Two call sites on one `--speech-device 1` switch is how one of them
# drifts, so each carries its own gate rather than one gate asserting both.
#
# Same labels, same 77-on-missing-precondition contract, same lease. `ctest -L gpu`
# now selects TWO tests, and that count is pinned by name in
# `scripts/check-test-registration.py` because `ctest -L` returns 0 over an empty
# selection. RUN_SERIAL because it stages the depth decoder and the DiT onto the
# accelerator and loads 28.5 GB to do it.
vllm_cpp_add_test(test_minimax_music3_depth_arm_real
parity/test_minimax_music3_depth_arm_real.cpp)
# Reaches src/ for `music3_profile.h`, the INTERNAL instrument the engine's own
# `profile::Report` prints. Deliberately not a public header: it is how this gate
# reads which arm ran, and it is not part of the ABI.
target_include_directories(test_minimax_music3_depth_arm_real PRIVATE
${CMAKE_SOURCE_DIR}/src)
set_tests_properties(test_minimax_music3_depth_arm_real PROPERTIES
RUN_SERIAL ON
LABELS "gpu;checkpoint;music3")
# ARCH-ONE-SURFACE ROW 2 fold gate: the video seam == the replicated pre-fold
# pipeline == the goldens the PRE-fold minimax-h3-gen binary rendered at the
# branch base (frames + WAV byte-identical, mux argv pinned).
Expand Down
Loading
Loading