From 82b8ed0b47148dde8bd7a7f32553b44239aef071 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 22 Aug 2026 15:10:05 +0000 Subject: [PATCH 01/19] spec(LTX25-DIT-ATTN-FLASH): the criterion for the pixel A/B, written before the renders are taken (#1612) The routing change landed on `main` as `90e8c3c85` with no comparison of what the model RENDERS. #1612 names the hole and this is the design that fills it, committed first so the acceptance criterion cannot be read off the numbers it judges. A criterion chosen after the fact is a retrofit however good the numbers turn out to be. New section 10. Three things in it are not obvious and are the reason it is this long. **bf16 predicts that the frames WILL differ, and the prediction is registered rather than discovered.** The two ops run the same f32 online softmax and differ only in association. Two orders of an `n`-term f32 sum differ by about `sqrt(n) * 2^-24`, which is `6.7e-07` over head_dim 128 and `2.9e-06` over the 2352-key axis. bf16's spacing is `2^-7`, four orders coarser, so the rounded store flips on `8.6e-05` to `3.7e-04` of elements -- and the video stream carries `4.62e8` attention outputs per forward. One forward therefore injects between `4.0e4` and `1.7e5` single-ULP flips, and a render is 120 forwards inside a nonlinear sampler. Bit-identical frames are NOT the expected outcome, and a tight a-priori pixel bound is not derivable from that analysis: whether those flips damp or amplify over 120 steps is empirical. So the bound comes from a convention the video-coding field already agreed on and from the render's own scale, and section 10.2 says so rather than deriving a number it cannot. **Three renders, not two, and the third carries the argument.** #1612 asks for a same-binary pair. A pair cannot attribute its own delta: cuBLAS reduction splits and anything else nondeterministic across 120 forwards plus a VAE decode sit inside it with no way out. The third render is `flash` AGAIN -- same binary, same seed -- and it measures the noise floor. Control zero means every bit of the delta is the swapped op; control comparable to the delta means the swap changed nothing the box does not change by itself. Both are answers and neither is available from two renders. **The fourth threshold is derived rather than borrowed.** V1 to V3 are conventions: one 8-bit level, 40 dB PSNR, SSIM 0.99 on the WORST frame. V4 is the arm-to-arm luma difference divided by arm A's own mean adjacent-frame MAD, capped at 0.10 -- the two arms must differ by less than a tenth of one frame of this video's own motion. Both terms move together, so it needs no re-argument at another geometry or prompt, which is what lets #1551 take the same criterion for the FA-2 arm. Measured on the recorded 20260820 baseline's own frames: a one-pixel global shift reads 0.624 and a +/-1 dither reads 0.0026, so the threshold sits between them and nearer the dither. Section 10.5 states how each outcome is read BEFORE there is one, including that a failing check is a finding about a change already on `main` rather than a reason to widen a threshold; section 9 gains that stop condition explicitly. Section 10.6 names what this does not measure: PPM is 8-bit, it is one trajectory, and the 20260820 NAS render is a cross-check and never the control because `a50c57d69` is a different binary lineage. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-opus-5 [Claude Code] --- .agents/specs/ltx25-dit-attn-flash.md | 207 +++++++++++++++++++++++++- 1 file changed, 205 insertions(+), 2 deletions(-) diff --git a/.agents/specs/ltx25-dit-attn-flash.md b/.agents/specs/ltx25-dit-attn-flash.md index ae1565c3a..f03ec70d2 100644 --- a/.agents/specs/ltx25-dit-attn-flash.md +++ b/.agents/specs/ltx25-dit-attn-flash.md @@ -7,7 +7,14 @@ Issue: [#1549](https://github.com/mudler/vllm.cpp/issues/1549). ## Now -`ACTIVE`. The diagnosis is confirmed against the tree and the change is scoped +`ACTIVE`. The routing change landed as `90e8c3c85` (#1557). What is live now is +**§10, the pixel A/B** ([#1612](https://github.com/mudler/vllm.cpp/issues/1612)): +the swap is on `main` with no comparison of what it RENDERS, and §10 is committed +before the renders are taken so the acceptance criterion cannot be read off the +numbers it judges. The same lease settles the speed A/B that §8 still carries as +`PENDING`. + +The diagnosis is confirmed against the tree and the change is scoped to **one production call site**. It was once scoped to that call site plus a shared-memory cap repair in `LaunchAttentionDenseFlash`; that repair is **reverted and out of scope** — see §4.3 for why, and for who owns the bound @@ -514,6 +521,9 @@ never a synonym for "probably fine". | bf16 head_dim 128 tile fits without an opt-in | arithmetic, `cuda_ops.cu:3338` | **PASS** — `2 * kFlashBc(64) * 128 * sizeof(bf16)` = 32,768 B against the 49,152 B every architecture gives without an opt-in. This is the whole of what the swapped shape needs from §4.3, and it is a property of the code, so no device is owed for it | | the flash op's advertised head_dim bound | not this row | **NOT A GATE HERE** — §4.3's cap-raise is reverted, so `LaunchAttentionDenseFlash` and `tests/vt/test_ops_attention.cpp` are byte-identical to `main`. Owned by [#1578](https://github.com/mudler/vllm.cpp/pull/1578), which merges first | | A/B, same binary, both arms | `dgx:gpu0` under an `rc` lease | **PENDING** — the flash arm is measured at 7.680 s median (n=19); the worker was lost before the naive arm, so no pair exists (§7.1) | +| pixel A/B at production geometry | `dgx:gpu0` under an `rc` lease, `scripts/ltx25-dit-attn-flash-pixel-ab.sh` | **PENDING** — criterion registered in §10.4; result in §10.7 | +| run-to-run control (`flash` twice) | the same lease | **PENDING** — §10.3; without it no arm-to-arm delta is attributable to the kernel | +| the comparison tool discriminates | `tests/scripts/test_ltx25_render_compare.py` | **PENDING until §10.7** — a dither passes and a one-pixel shift fails all four checks (§10.4) | | full preflight | `scripts/agent-preflight.sh` | **PASS at HEAD** — and it was NOT before: `documentation-checkpoint` was red on two of this branch's own commits (see below) | | `documentation-checkpoint` | CI, and locally over the branch range | **PASS at HEAD, RED before it, and the red was THIS BRANCH's** — `2aa78c69b` and `2f39a9426` each recorded a measurement in `.agents/benchmark-record.md` without writing `docs/STATUS.md` (and `docs/BENCHMARKS.md` for the second). The control on the main-only range `4c193bd55..5d548d003` is rc 0, so it was not inherited. Both commits were replaced by one that writes all three surfaces together when the branch was rebuilt, and the checker is re-run at each head rather than trusted to have stayed fixed — a job that has stopped appearing in a failing set is not the same fact as a job that passes | | `build-newest-gcc` | CI | **PASS, and now green on `main` too** — it was red on `main` on `::getpid` in `test_qwen3_dflash2_gguf.cpp:547`, a file this change does not touch; [#1581](https://github.com/mudler/vllm.cpp/pull/1581) fixed it and this branch carries that fix through the merge. A red here after the merge is therefore this row's, not inherited | @@ -544,10 +554,203 @@ because it passed. - `NEEDS_DECISION` if the measured speedup is far from §7's prediction. - The A/B is reported as **pending an external resource** if `dgx:gpu0` is not free. It is never taken on another box, and never replaced by an estimate. +- **The pixel thresholds in §10.4 are never widened to admit the change.** A + failing check is a finding about a change already on `main`, filed as its own + issue with what diverged and by how much (§10.5), and it is never repaired by + moving a number in `scripts/ltx25-render-compare.py`. + +## 10. The pixel A/B — what the model RENDERS, designed before it is read + +Issue: [#1612](https://github.com/mudler/vllm.cpp/issues/1612). This section is +written and committed **before the renders are taken**, because a criterion read +off the numbers it is meant to judge is not a criterion. + +### 10.1 Why this section has to exist at all + +Every other model in this tree leans on a token gate. Greedy decode gives a +discrete output, and either the tokens match the oracle or they do not. **A +diffusion render has no such output.** The DiT emits latents, the VAE emits +pixels, and nothing in that chain is a symbol that can be compared for equality +against a reference by construction. + +So when §5 says the swap is **not bit-identical on CUDA**, the usual net is +absent. What remains is the reduced-dimension host-vs-device parity case +(`8.94e-08` video, `4.47e-08` audio against `2e-5`), and §7.1 already states its +limit in the same breath as its result: it bounds the *arithmetic* change — a +length-`head_dim` sum reassociated — at the fixture's dimensions. Production is +bf16, head_dim 128, 2352 keys, 32 heads, 48 layers, 120 forwards. Nothing has +measured that. + +### 10.2 What bf16 predicts, which is that the frames WILL differ + +This is a prediction registered in advance, not a result. + +`vt::Attention` and `vt::AttentionDenseFlash` run the same f32 online softmax +and differ only in association (`include/vt/ops.h:3304-3306`). Two summation +orders of an `n`-term f32 sum differ by roughly `sqrt(n) * u` in the +random-walk regime, with `u = 2^-24 = 5.96e-08` the f32 unit roundoff: + +| axis reassociated | `n` | relative deviation | +|---|---|---| +| the head_dim dot product, which §5 names | 128 | `~6.7e-07` | +| the key-axis online accumulation | 2352 | `~2.9e-06` | + +The DiT then **stores that result to bf16**, whose spacing is `2^-7 = 7.81e-03` +relative — four orders of magnitude coarser. A perturbation of relative size +`d` moves the rounded bf16 value only when the exact value sits within `d` of a +rounding boundary, so the per-element flip probability is `d / 7.81e-03`: +between `8.6e-05` and `3.7e-04`. + +The video stream carries `2352 * 32 * 128 = 9.63e6` attention output elements +per layer and `4.62e8` over 48 layers, so **one forward injects between 4.0e4 +and 1.7e5 single-ULP bf16 flips**, and a render is 120 forwards inside a +nonlinear sampler that feeds each step's output into the next. + +**Three things follow, and the third is the one that shapes the gate.** + +1. **Bit-identical frames are not the expected outcome.** Predicting them and + then finding a difference would make any threshold read as a retrofit. +2. **The floor is not zero.** "Within bf16 noise" has to mean something other + than equality, because equality is not what the arithmetic predicts. +3. **A tight a-priori pixel bound is NOT derivable.** The sampler is nonlinear + and iterative; whether ~1e5 ULP flips per step damp or amplify over 120 steps + is an empirical property of this model at this geometry, not something the + error analysis above can be pushed to answer. Anyone who claims to derive one + is deriving it from an assumption of contraction that nothing here has + measured. + +So the bound is not derived from the arithmetic. It is derived from two things +that exist independently of this experiment: a convention the video-coding field +already agreed on, and the render's own scale. + +### 10.3 The design: three renders, and the third is the whole argument + +One binary, built once, in one `rc` lease on `dgx:gpu0`, on one staged +checkpoint set, at `768x448/49f` (2352 tokens), seed `20260820`, with the exact +70-word prompt of the recorded 20260820 baseline. + +| # | render | knob | what it is | +|---|---|---|---| +| 1 | `flash` | `VLLM_LTX2_DIT_FLASH_ATTN=1` | the arm #1549 shipped | +| 2 | `naive` | `VLLM_LTX2_DIT_FLASH_ATTN=0` | the arm it replaced | +| 3 | `flash-ctl` | `VLLM_LTX2_DIT_FLASH_ATTN=1` | **flash again**, same binary, same seed | + +**(3) is not a spare. It is the control, and without it the experiment does not +answer its own question.** A difference between (1) and (2) is only attributable +to the kernel if the box produces the same render twice when nothing changes. +cuBLAS reduction splits, allocator-dependent kernel selection and any +nondeterminism anywhere in 120 forwards plus a VAE decode would otherwise sit +inside the measured delta with no way to separate them. So: + +- **control == 0** (bit-identical): the noise floor is exactly zero, and every + bit of the flash-vs-naive delta is the swapped op. The strongest attribution + available. +- **control ~= treatment**: the swap changed nothing the machine does not change + by itself. The strongest possible *null* result, and it can only be stated + because the control was taken. +- **control > 0 and < treatment**: the delta is partly kernel, and the control + is the floor the thresholds must be read against. + +Every one of those is an answer. None is available from two renders, which is +what #1612 asked for and what this section deliberately exceeds. + +**Order: flash, naive, flash-ctl.** The naive arm is ~6x the wall clock and it +is the one whose loss leaves no A/B at all, so it is taken while the box is +known good rather than last. The control goes last because it is the only one +recoverable cheaply: the harness caches the binary keyed on the source sha, so a +follow-up lease reaches a render in minutes rather than re-spending the build. + +**Both arms are instrumented identically and neither is stack-sampled.** §7.1 +established that `runguard.py --stack-period 12` cost the recorded 47.84 s +denominator ~3.2% by `ptrace`-stopping every thread. This harness runs a +`MemAvailable` watchdog and nothing else, so the speed pair it produces needs no +sampler correction and finally replaces the cross-run 6.03-6.23x range with a +same-binary ratio. + +**Routing is proved per arm, two-sided, from that arm's own log.** +`VT_OP_PROVIDER_STATS=1` makes each op announce itself once when it resolves. +The flash arm must show `op=21 device=1` **and no** `op=18 device=1`; the naive +arm the reverse. A one-sided count cannot tell a routed call from an added one, +and "the knob was exported" is not evidence that the branch was taken. + +### 10.4 The registered acceptance criterion + +Committed as the defaults of `scripts/ltx25-render-compare.py`, so the numbers +are read by a tool that already holds the thresholds rather than compared to +them by hand afterwards. + +| # | check | threshold | where it comes from | +|---|---|---|---| +| V1 | mean \|delta\|, 8-bit RGB | `<= 1.0` level | one level is the quantisation step of the artefact itself; a mean below it says the average pixel is within the PPM's own resolution | +| V2 | worst-frame PSNR | `>= 40 dB` | the video-coding "visually lossless" convention. This experiment did not choose it | +| V3 | worst-frame SSIM | `>= 0.99` | Wang et al. 2004, 11x11 Gaussian sigma=1.5 on luma. 0.98 is the usual transparency line; this is stricter, and it is the WORST frame rather than the mean | +| V4 | mean \|delta\| on luma / arm A's mean adjacent-frame MAD | `<= 0.10` | **the self-calibrating one** | +| A1 | audio PSNR vs full scale | `>= 40 dB` | same convention as V2 | +| A2 | audio Pearson r | `>= 0.999` | a waveform that has drifted in time fails this while PSNR can still look tolerable | + +**V4 is the bound that is derived rather than borrowed, so it carries the +argument.** The denominator is the render's own frame-to-frame step: how much +one frame differs from the next, in the same 8-bit luma units as the numerator. +`0.10` therefore says *the two arms differ by less than a tenth of one frame of +this video's own motion*. It needs no re-argument at another geometry or another +prompt, because both terms move together — which is exactly what a constant +cannot do, and why the FA-2 arm (#1551) can take this same criterion. + +**The scale is executable, not asserted.** Measured on the recorded 20260820 +baseline's own frames, and pinned in `tests/scripts/test_ltx25_render_compare.py` +on synthetic fixtures so it is a gate rather than a claim: + +| perturbation of a real frame | mean \|d\| | PSNR | SSIM | V4 ratio | verdict | +|---|---|---|---|---|---| +| none | 0 | inf | 1.000000 | 0.000 | bit-identical | +| +/-1 LSB on 3% of samples | 0.0199 | 65.1 dB | 0.99992 | 0.0026 | **PASS**, all four | +| one pixel of global horizontal shift | 5.183 | 28.1 dB | 0.8705 | 0.624 | **FAIL**, all four | + +So the thresholds sit between a dither and a single pixel of motion, nearer the +dither: V4 at `0.10` refuses anything above a sixth of a one-pixel global shift. +A criterion that admitted the shift row would not be a criterion. + +### 10.5 Reading the result, stated before there is one + +- **All checks pass and the control is 0** — the swap changes the render by less + than a tenth of its own motion step, the difference is entirely the kernel's, + and the verdict is **within bf16 noise**. The measured values, with their + headroom, are then what belongs in the record. +- **All checks pass and the control is comparable to the delta** — the verdict is + **indistinguishable from run-to-run nondeterminism**, which is stronger, and + the row additionally owes an issue for the nondeterminism itself, because a + render that is not reproducible is its own defect. +- **Any check fails** — the verdict is **visibly different**, and that is a + finding about a change already on `main`, not a failure of this work. It owes + an issue naming what diverged and by how much, and it does not owe a widened + threshold. Widening a gate to admit a change is the failure this protocol + exists to prevent, and §9's stop conditions already say so for the numeric + gate. + +### 10.6 What this deliberately does not measure + +- **PPM is 8-bit.** The comparison is on the artefact the pipeline writes, which + is already quantised from the VAE's float output. A difference below `1/255` + relative is invisible to it. That is the right resolution for the question + "does it render the same video" and the wrong one for "how large is the + latent-space deviation"; the second is the host-vs-device case's job and it is + answered at the fixture's dimensions in §7.1. +- **One prompt, one seed, one geometry.** A sampler is chaotic and one trajectory + is one trajectory. This bounds the swap on the trajectory production actually + ran and recorded, and it does not claim a bound over the prompt distribution. +- **The 20260820 NAS baseline is a cross-check, never the control.** It was built + from `a50c57d69`, an ancestor of the swap, so it is a different binary lineage: + everything else that landed on `main` in between sits inside any delta measured + against it. It is compared anyway, because how far two naive renders drift + across builds bounds how much of the A/B delta could be something other than + the kernel — but the same-binary pair is the evidence and this is context. ## Owed -- **There is NO numeric or pixel comparison at production geometry.** The swap +- **There is NO numeric or pixel comparison at production geometry.** + **§10 is the design that discharges this, and it is committed before the + renders are taken.** The result lands in §10.7. Until it does, the statement + below is still the honest one. The swap is not bit-identical on CUDA (§5), and the only numeric gate that exists is the reduced-dimension host-vs-device case — `8.94e-08` / `4.47e-08` against `2e-5` — which bounds the ARITHMETIC change and not the change at head_dim 128 with From b224efe3456da921ce99647fc2527541abc4c815 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 22 Aug 2026 15:10:48 +0000 Subject: [PATCH 02/19] test(#1612): a pixel comparison for a model that cannot have a token gate, and the harness that renders both arms Section 10 of `.agents/specs/ltx25-dit-attn-flash.md`, committed in the previous commit, says what will be measured and against what. This is the instrument. **`scripts/ltx25-render-compare.py`** reads two render directories and reports byte identity first, then max and mean `|delta|` over 8-bit RGB with the full histogram, PSNR, SSIM (Wang et al. 2004, the ORIGINAL 11x11 Gaussian sigma=1.5 on luma, not scikit-image's 7x7 uniform default -- the two disagree in the third decimal and this gate reads that far), the temporal ratio, and the audio pair. It holds the section 10.4 thresholds as its DEFAULTS, so the numbers are read by a tool that already carries the criterion rather than compared to one by hand afterwards. `--control` takes a third directory and reports it separately, never folded into the verdict: it is the scale the verdict is read against, and a tool that mixed the two would hide the attribution it exists to supply. Its PPM and WAV readers are written out rather than delegated. The only image library certain to be present in a leased worker is none, and the parser is strict rather than accommodating: a maxval other than 255 would silently change what every threshold means, so it refuses. **`tests/scripts/test_ltx25_render_compare.py`** pins both halves of the discrimination on fabricated frames, with no NAS and no GPU. A `+/-1` dither on 3% of samples passes all four video checks; ONE PIXEL of global horizontal shift fails all four. Four, not one -- a criterion that caught a shift on a single axis would be one threshold with three decorations. The fixtures are textured and moving on purpose: a flat sequence makes SSIM degenerate and a one-pixel shift invisible, so a test built on one would pass while proving nothing. Two mutations, applied to a scratch copy and restored byte-for-byte (`sha256 a188c50d...` before and after): M1 the four video thresholds widened to 100 / 10 dB / 0.50 / 10.0. The shift case goes RED and it is the ONLY failure, so the other twelve are not resting on those numbers. The run's own output prints `<= 100.0` and `>= 0.5`, which is how the mutation is proved to have applied to a file `git diff` cannot see because it is new. M2 V4's denominator replaced by the constant 10.0. The normalisation case goes RED with `0.2 not greater than 0.2` -- a constant makes a slow and a fast render read identically, which is the defect V4 exists to avoid. **`scripts/ltx25-dit-attn-flash-pixel-ab.sh`** renders `flash`, `naive` and `flash-ctl` from ONE binary in ONE lease at `768x448/49f`, seed 20260820, on the recorded baseline's own prompt. It differs from its sibling `ltx25-dit-attn-flash-ab.sh` in the thing that matters: that one caps each arm at 13 forwards for a median and writes NO FRAMES, which is why #1612 exists. Neither arm here is stack-sampled, because `runguard.py --stack-period 12` ptrace-stops every thread and cost the recorded 47.84 s denominator ~3.2%; the speed pair this produces therefore needs no sampler correction. Routing is proved per arm and TWO-SIDED from that arm's own log -- the flash arm must show `op=21` and no `op=18`, the naive arm the reverse -- because a one-sided count cannot tell a routed call from an added one, and "the knob was exported" is not evidence that the branch was taken. Nothing here is reachable from a production entry point and nothing claims to be: these are measurement tools for a change that already landed. The capability under test is `ltx2_device.cpp`'s call site, which #1549 wired and proved reached. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-opus-5 [Claude Code] --- scripts/ltx25-dit-attn-flash-pixel-ab.sh | 442 ++++++++++++++++++++ scripts/ltx25-render-compare.py | 451 +++++++++++++++++++++ tests/scripts/test_ltx25_render_compare.py | 302 ++++++++++++++ 3 files changed, 1195 insertions(+) create mode 100755 scripts/ltx25-dit-attn-flash-pixel-ab.sh create mode 100755 scripts/ltx25-render-compare.py create mode 100644 tests/scripts/test_ltx25_render_compare.py diff --git a/scripts/ltx25-dit-attn-flash-pixel-ab.sh b/scripts/ltx25-dit-attn-flash-pixel-ab.sh new file mode 100755 index 000000000..913825d3d --- /dev/null +++ b/scripts/ltx25-dit-attn-flash-pixel-ab.sh @@ -0,0 +1,442 @@ +#!/bin/bash +# LTX25-DIT-ATTN-FLASH (#1612) -- the PIXEL A/B, and the speed A/B it also settles. +# +# `scripts/ltx25-dit-attn-flash-ab.sh` is this file's sibling and answers a +# different question. It caps each arm at 13 forwards, which is enough for a +# per-forward median and produces NO FRAMES. #1549 shipped on that: a kernel +# swap that is explicitly not bit-identical on CUDA, gated only by a +# reduced-dimension host-vs-device parity case, with nothing at all said about +# what the model RENDERS. A diffusion model has no token gate, so there is no +# discrete output to fall back on. This file renders both arms to completion and +# compares the pixels. +# +# THREE RENDERS, and the third is the point. +# +# 1. flash VLLM_LTX2_DIT_FLASH_ATTN=1 the arm that ships today +# 2. naive VLLM_LTX2_DIT_FLASH_ATTN=0 the arm #1549 replaced +# 3. flash-ctl VLLM_LTX2_DIT_FLASH_ATTN=1 flash AGAIN, same binary, same seed +# +# Without (3) an arm-to-arm difference cannot be attributed to the kernel. Two +# runs of one configuration measure what the BOX does on its own -- cuBLAS split +# reductions, allocator-dependent kernel selection, anything nondeterministic +# anywhere in a 120-forward denoise plus a VAE decode. If (3) is bit-identical +# to (1) the noise floor is exactly zero and every bit of the flash-vs-naive +# delta is the swapped op. If (3) differs from (1) by as much as (2) does, the +# swap changed nothing the machine does not change by itself. Either reading is +# an answer; neither is available from two renders. +# +# ORDER: flash, naive, flash-ctl. The naive arm is ~6x the wall clock of a flash +# arm and it is the one whose loss leaves no A/B at all, so it is taken second, +# while the box is known good, rather than last. The control is last because it +# is the only one recoverable in a short follow-up lease: the build cache below +# is keyed on the source sha, so a resumed run reaches a render in minutes. +# +# The 20260821T092516Z attempt lost the worker at forward 20 with no memory +# trace and no guard, so it could not say afterwards what had happened. This one +# writes a MemAvailable trace per arm and stops an arm that crosses a floor. +# It does NOT run under `runguard.py --stack-period`: that sampler ptrace-stops +# every thread and cost the recorded 47.84 s denominator ~3.2% (spec section +# 7.1). Both arms here are instrumented identically and neither is sampled, so +# the ratio needs no correction. +set -u +T0=$(date +%s) +say() { echo "[pixab +$(( $(date +%s) - T0 ))s] $*"; } +W=/workspace/ltx25-attnflash +FULL=/workspace/ltx25-fullmodel # checkpoints and the text-encoder config +SRC=/root/src-pixab +BLD=/root/build-pixab +CK=/root/ckpt +OUT=$W/pixel-ab/$(date -u +%Y%m%dT%H%M%SZ) +mkdir -p "$OUT" "$CK" +export DEBIAN_FRONTEND=noninteractive +say "OUT=$OUT" +{ + echo "rc_job=${RC_JOB_ID:-unknown}" + echo "harness_sha256=$(sha256sum "$0" 2>/dev/null | awk '{print $1}')" + echo "started=$(date -Is)" +} > "$OUT/PROVENANCE" + +# A LIVENESS LINE, and nothing more. The build redirects to a file and a single +# 42 GB checkpoint copy takes minutes, so this job can produce no stdout at all +# for the better part of an hour, and a lease with an idle timeout would kill a +# healthy run. It prints ONLY the elapsed time: it reports no count and no +# progress, because a line emitted on a fixed cadence cannot distinguish work +# from a hang and must not be read as if it could. The render loop below prints +# the forward count separately, and that one CAN stop advancing. +( while :; do sleep 120; echo "[pixab-alive +$(( $(date +%s) - T0 ))s]"; done ) & +HEARTBEAT=$! +trap 'kill $HEARTBEAT 2>/dev/null' EXIT + +say "=== [0] the box ===" +uname -m; nproc; free -g | head -2 +nvidia-smi --query-gpu=name,driver_version,memory.total --format=csv 2>&1 | head -3 +df -h / /root /workspace 2>&1 | head -6 + +say "=== [1] tools ===" +# ffmpeg matters beyond the mp4: ltx2-gen exits 127 from its absence AFTER every +# frame and the wav are on disk (#1149), so its absence reads as a failed render +# even though the artifacts this file compares are already written. numpy is +# what scripts/ltx25-render-compare.py needs and it is the only dependency it +# has; the PPM and WAV readers there are written out precisely so that no image +# or audio library has to be present in a leased worker. +apt-get update -qq >/root/apt.log 2>&1 +apt-get install -y -qq ffmpeg python3-numpy >>/root/apt.log 2>&1 +echo "apt_rc=$?" +for t in ffmpeg ffprobe python3; do printf '%s=' "$t"; command -v "$t" || echo MISSING; done +python3 -c 'import numpy;print("numpy",numpy.__version__)' 2>&1 | tail -1 + +say "=== [A] CUDA toolkit, resolved BEFORE anything probes it ===" +CUDATK=/root/cudatk +STAGED=/workspace/a3/cuda-staged +need_ok() { [ -x "$1/bin/nvcc" ] && [ -f "$1/targets/sbsa-linux/lib/libcublasLt.so" ]; } +TKLIB="" +for r in /usr/local/cuda /usr/local/cuda-13.0 "$CUDATK"; do + need_ok "$r" && TKLIB="$r" && say "complete toolkit: $r" && break +done +if [ -z "$TKLIB" ]; then + for r in /usr/local/cuda /usr/local/cuda-13.0; do + [ -x "$r/bin/nvcc" ] || continue + LD="$r/targets/sbsa-linux/lib" + [ -d "$LD" ] && [ -w "$LD" ] || continue + for lib in libcublasLt libcublas; do + f=$(ls "$STAGED"/targets/sbsa-linux/lib/$lib.so.13* 2>/dev/null | grep -v static | head -1) + [ -n "$f" ] || continue + say "repairing $r: adding $(basename "$f")" + cp -f "$f" "$LD"/ && ( cd "$LD" && b=$(basename "$f"); ln -sf "$b" "$lib.so.13"; ln -sf "$b" "$lib.so" ) + done + need_ok "$r" && TKLIB="$r" && break + done +fi +if [ -z "$TKLIB" ] && [ -d "$STAGED/bin" ]; then + if ! need_ok "$CUDATK"; then + say "staging a COMPLETE toolkit locally (~4.9G)" + mkdir -p "$CUDATK"; cp -a "$STAGED"/. "$CUDATK"/ 2>/dev/null + # The share serves file_mode=0664: every binary copied off it arrives + # NON-EXECUTABLE, and an nvcc that cannot run reads as an nvcc that is absent. + find "$CUDATK/bin" "$CUDATK/nvvm/bin" -type f -exec chmod 0755 {} + 2>/dev/null + # CIFS stores no symlinks; find_package wants libX.so and libX.so.MAJOR. + ( cd "$CUDATK/targets/sbsa-linux/lib" 2>/dev/null || exit 0 + for f in *.so.*; do + case "$f" in *.a) continue;; esac + b=${f%%.so.*}; rest=${f#*.so.}; maj=${rest%%.*} + [ -e "$b.so.$maj" ] || ln -sf "$f" "$b.so.$maj" + [ -e "$b.so" ] || ln -sf "$f" "$b.so" + done ) + fi + need_ok "$CUDATK" && TKLIB="$CUDATK" +fi +[ -n "$TKLIB" ] || { echo "FATAL: no COMPLETE CUDA toolkit"; exit 38; } +# cmake finds nvcc via PATH, not CUDAToolkit_ROOT alone. +export PATH="$TKLIB/bin:$PATH" CUDAToolkit_ROOT="$TKLIB" +say "CUDAToolkit_ROOT=$TKLIB" +nvcc --version | tail -2 + +say "=== [B] source, and the PRECONDITION that it carries both arms ===" +rm -rf "$SRC"; mkdir -p "$SRC" +tar xzf "$W/pixab-src.tar.gz" -C "$SRC" || { echo "FATAL: cannot unpack source"; exit 31; } +WANT_SHA=$(cat "$W/pixab-src.sha" 2>/dev/null) +echo " built_from=$WANT_SHA" +echo "source_sha=$WANT_SHA" >> "$OUT/PROVENANCE" +# BOTH sides, because a half-applied tree satisfies either alone: the swapped op +# without the knob makes the naive arm a second flash arm, and the knob without +# the swap makes the flash arm a second naive one. Either way the A/B renders one +# configuration three times and still prints three columns. +NEWOP=$(grep -c 'vt::AttentionDenseFlash' "$SRC/src/vllm/model_executor/models/ltx2_device.cpp") +KNOB=$(grep -c 'VLLM_LTX2_DIT_FLASH_ATTN' "$SRC/src/vllm/model_executor/models/ltx2_device.cpp") +echo " AttentionDenseFlash call sites: $NEWOP (want >= 1)" +echo " A/B knob sites: $KNOB (want >= 1)" +[ "$NEWOP" -ge 1 ] || { echo "FATAL: #1549 is NOT in this source tree"; exit 40; } +[ "$KNOB" -ge 1 ] || { echo "FATAL: the A/B knob is NOT in this source tree; both arms would be one arm"; exit 41; } +CMP="$SRC/scripts/ltx25-render-compare.py" +[ -s "$CMP" ] || { echo "FATAL: the comparison tool is not in this source tree"; exit 43; } + +say "=== [C] cutlass (resolved, never fetched) ===" +CUT="" +for c in /cutlass /workspace/cutlass /root/cutlass; do + [ -f "$c/include/cutlass/cutlass.h" ] && CUT="$c" && break +done +TB=/workspace/cutlass-v4.5.0.tar.gz +if [ -z "$CUT" ] && [ -f "$TB" ]; then + say "unpacking staged cutlass" + mkdir -p /root/cutlass && tar xzf "$TB" -C /root/cutlass && CUT=/root/cutlass +fi +[ -n "$CUT" ] && [ -f "$CUT/include/cutlass/cutlass.h" ] || { echo "FATAL: no CUTLASS tree"; exit 36; } +say "CUTLASS_DIR=$CUT" + +say "=== [D] configure + build (-j 4, per the GB10 recipe) ===" +# The binary is REUSED only when the cached source sha matches the tarball's, so +# a resumed run can never render a tree other than the one it claims. That is +# the failure this guard prevents, not the rebuild. +CACHE="$W/pixab-bin" +BIN=/root/pixabbin; mkdir -p "$BIN" +SKIP_BUILD=0 +if [ -s "$CACHE/ltx2-gen" ] && [ -s "$CACHE/libvllm.so.0.0.3" ] && \ + [ -n "$WANT_SHA" ] && [ "$(cat "$CACHE/SRC_SHA" 2>/dev/null)" = "$WANT_SHA" ]; then + say "REUSING the staged binary: cached SRC_SHA matches $WANT_SHA" + cp -f "$CACHE/ltx2-gen" "$BIN"/ && chmod 0755 "$BIN/ltx2-gen" + cp -f "$CACHE/libvllm.so.0.0.3" "$BIN"/ && chmod 0755 "$BIN/libvllm.so.0.0.3" + cp -f "$CACHE/ltx2_gemma4_text_config.json" "$BIN"/ 2>/dev/null || \ + cp -f "$FULL/bin/ltx2_gemma4_text_config.json" "$BIN"/ + ( cd "$BIN" && ln -sf libvllm.so.0.0.3 libvllm.so.0 && ln -sf libvllm.so.0.0.3 libvllm.so ) + [ -s "$CACHE/test_ltx2_device" ] && cp -f "$CACHE/test_ltx2_device" "$BIN"/ && chmod 0755 "$BIN/test_ltx2_device" + SKIP_BUILD=1 +fi +if [ "$SKIP_BUILD" = 0 ]; then + cmake -S "$SRC" -B "$BLD" -G Ninja -DCMAKE_BUILD_TYPE=Release -DVLLM_CPP_CUDA=ON \ + -DVLLM_CPP_CUTLASS_DIR="$CUT" -DCUDAToolkit_ROOT="$TKLIB" > /root/cfg.log 2>&1 + CFG=$?; echo "CFG_RC=$CFG" + grep -iE 'CUDA target arch|cutlass|flashattention' /root/cfg.log | head -8 + if [ "$CFG" != 0 ]; then + awk '/CMake (Error|Warning)/,/^$/' /root/cfg.log | head -60; tail -40 /root/cfg.log + cp -f /root/cfg.log "$OUT/configure-fail.log"; echo "FATAL: configure failed"; exit 33 + fi + cp -f /root/cfg.log "$OUT/configure.log" + # Unconstrained parallelism has OOM-rebooted this box. + ninja -C "$BLD" -j 4 ltx2-gen test_ltx2_device > /root/build.log 2>&1 + B=$?; echo "BUILD_RC=$B" + echo " compile_errors=$(grep -ciE ' error: ' /root/build.log)" + tail -15 /root/build.log + cp -f /root/build.log "$OUT/build.log" + [ "$B" = 0 ] || { echo "FATAL: build failed"; exit 34; } + GEN=$(find "$BLD" -name ltx2-gen -type f | head -1) + LIB=$(find "$BLD" -name 'libvllm.so.0.0.3' | head -1) + [ -n "$GEN" ] && [ -n "$LIB" ] || { echo "FATAL: build artefacts not found"; exit 35; } + cp -f "$GEN" "$BIN"/ && chmod 0755 "$BIN/ltx2-gen" + cp -f "$LIB" "$BIN"/ && chmod 0755 "$BIN/libvllm.so.0.0.3" + ( cd "$BIN" && ln -sf libvllm.so.0.0.3 libvllm.so.0 && ln -sf libvllm.so.0.0.3 libvllm.so ) + cp -f "$SRC/tests/vllm/models/ltx2_gemma4_text_config.json" "$BIN"/ 2>/dev/null || \ + cp -f "$FULL/bin/ltx2_gemma4_text_config.json" "$BIN"/ + T=$(find "$BLD" -name test_ltx2_device -type f | head -1) + [ -n "$T" ] && cp -f "$T" "$BIN"/ && chmod 0755 "$BIN/test_ltx2_device" + mkdir -p "$CACHE" + cp -f "$BIN/ltx2-gen" "$CACHE"/ && cp -f "$BIN/libvllm.so.0.0.3" "$CACHE"/ + cp -f "$BIN/ltx2_gemma4_text_config.json" "$CACHE"/ 2>/dev/null + [ -s "$BIN/test_ltx2_device" ] && cp -f "$BIN/test_ltx2_device" "$CACHE"/ + echo "$WANT_SHA" > "$CACHE/SRC_SHA" + say "staged the binary for a resumed run" +fi +BINSHA=$(sha256sum "$BIN/ltx2-gen" | awk '{print $1}') +say "ONE BINARY, all three renders: sha256=$BINSHA" +{ echo "binary_sha256=$BINSHA"; echo "binary_built=$([ "$SKIP_BUILD" = 1 ] && echo cache || echo in-lease)"; } >> "$OUT/PROVENANCE" +export LD_LIBRARY_PATH="$BIN:$TKLIB/targets/sbsa-linux/lib:${LD_LIBRARY_PATH:-}" +# sha256 and ldd both pass on a file with no execute bit. Ask the binary itself. +"$BIN/ltx2-gen" --help >/dev/null 2>&1 || { echo "FATAL: ltx2-gen will not exec (126 = no exec bit, 127 = missing lib)"; ldd "$BIN/ltx2-gen" | head; exit 25; } +say "EXECUTABLE_OK" + +say "=== [E] checkpoints, staged to LOCAL disk ===" +# Measured: 589-1446 s per load over CIFS at 34-83 MiB/s, against ~32 s from +# local disk. Three renders pay that three times, so the ~9 minute copy is not +# an optimisation, it is most of the difference between fitting in a lease and +# not. Each file is matched on EXACT BYTE SIZE, so a half-written stage is +# refused rather than loaded. +declare -A WANT=( + [ltx-2.5-22b-dev-transformer-bf16.safetensors]=42018190584 + [ltx-2.5-video-vae-conv-bf16.safetensors]=1452269922 + [ltx-2.5-audio-vae-bf16.safetensors]=364866540 + [gemma4-12b-with-proj-nvfp4-torchao.safetensors]=7423624178 +) +FREE_K=$(df -k --output=avail /root | tail -1) +NEED_K=$(( (42018190584 + 1452269922 + 364866540 + 7423624178) / 1024 + 8388608 )) +say "local free ${FREE_K}K, need ${NEED_K}K" +CKUSE=$CK +if [ "$FREE_K" -le "$NEED_K" ]; then + say "NOT staging (insufficient local disk); reading weights over CIFS" + CKUSE=$FULL/ckpt +else + for f in "${!WANT[@]}"; do + s=$FULL/ckpt/$f; d=$CK/$f; want=${WANT[$f]} + got=$(stat -c %s "$s" 2>/dev/null || echo 0) + [ "$got" = "$want" ] || { echo "FATAL: source $f is $got bytes, want $want"; exit 23; } + if [ -s "$d" ] && [ "$(stat -c %s "$d")" = "$want" ]; then say " already staged $f"; continue; fi + t=$SECONDS; cp "$s" "$d" || { echo "FATAL: cannot stage $f"; exit 23; } + [ "$(stat -c %s "$d")" = "$want" ] || { echo "FATAL: short stage of $f"; exit 23; } + say " staged $f $want bytes in $((SECONDS-t))s" + done +fi +# THE FULL/DEV TRANSFORMER, 42,018,190,584 B. Never the distilled one: nothing +# validates checkpoint class (#1137) and it would render plausibly in the wrong +# regime. +say "checkpoints from $CKUSE" +echo "checkpoint_dir=$CKUSE" >> "$OUT/PROVENANCE" + +say "=== [F] CORRECTNESS FIRST: the CUDA unit gate, before any render ===" +# AGENTS.md: establish the correctness gate before accepting a performance +# result. `assertions: 0` is a skip wearing a pass and a thrown case shows up +# only on the `Status:` line, so both are printed rather than a grep for ok. +if [ -x "$BIN/test_ltx2_device" ]; then + "$BIN/test_ltx2_device" > "$OUT/test_ltx2_device.log" 2>&1 + echo " test_ltx2_device_RC=$?" + grep -E 'assertions:|test cases:|Status:|SKIP' "$OUT/test_ltx2_device.log" | tail -8 +else + echo " MISSING test_ltx2_device" +fi + +say "=== [G] the three renders ===" +# The prompt, seed and geometry of the recorded 49-frame baseline render +# (out/20260820T223701Z/768x448-49f/render.log line 1), copied byte-for-byte so +# that this pair is additionally comparable to it. The primary evidence is the +# same-binary pair below; the older render was built from a50c57d69, which is an +# ANCESTOR of the swap and therefore a different binary lineage, so it is a +# cross-check and never the control. +PROMPT='A golden retriever shakes water from its coat on a sunlit lawn, droplets flying outward in a bright arc around its head and shoulders, wet fur rippling and separating into strands from shoulders to tail, ears flapping, muscles moving under the coat. Crisp midday light, shallow depth of field, vivid green grass behind. The dog barks once, water patters onto the grass, and a light breeze moves through the trees.' +FRAMES=${FRAMES:-49}; WW=${WW:-768}; HH=${HH:-448}; SEED=${SEED:-20260820} +TOK=$(( (WW/32) * (HH/32) * (((FRAMES-1)/8) + 1) )) +MEM_FLOOR_GIB=${MEM_FLOOR_GIB:-8.0} +say "geometry ${WW}x${HH}/${FRAMES}f = $TOK video tokens, seed $SEED" +say "MemAvailable floor ${MEM_FLOOR_GIB} GiB (the recorded baseline's low-water was 40.13 GiB)" +{ + echo "geometry=${WW}x${HH}/${FRAMES}f" + echo "video_tokens=$TOK" + echo "seed=$SEED" + echo "prompt_sha256=$(printf '%s' "$PROMPT" | sha256sum | awk '{print $1}')" +} >> "$OUT/PROVENANCE" + +render() { # $1 = label, $2 = knob value, $3 = hard timeout seconds + local label=$1 knob=$2 tmo=$3 + local d="$OUT/$label"; mkdir -p "$d" + local log="$d/render.log" + say "--- render $label (VLLM_LTX2_DIT_FLASH_ATTN=$knob, hard cap ${tmo}s) ---" + : > "$log" + # EVERY RENDER STATES ITS OWN INVOCATION on line 1 of its own log, the way the + # recorded baseline's render.log does and the way the withdrawn 7.680 s arm did + # not. An arm whose log cannot say what it ran is not evidence, whatever number + # it contains. + { + echo "[arm] label=$label knob=$knob tmo=${tmo}s" + echo "[arm] harness=$0 sha256=$(sha256sum "$0" 2>/dev/null | awk '{print $1}')" + echo "[arm] binary=$BIN/ltx2-gen sha256=$BINSHA src_sha=$WANT_SHA" + echo "[arm] geometry=${WW}x${HH}/${FRAMES}f tokens=$TOK seed=$SEED ckpt=$CKUSE" + echo "[arm] prompt=<<$PROMPT>>" + echo "[arm] cmd: $BIN/ltx2-gen --pipeline-kind one_stage --dit $CKUSE/ltx-2.5-22b-dev-transformer-bf16.safetensors" \ + "--video-vae $CKUSE/ltx-2.5-video-vae-conv-bf16.safetensors --audio-vae $CKUSE/ltx-2.5-audio-vae-bf16.safetensors" \ + "--checkpoint-class full --encoder $CKUSE/gemma4-12b-with-proj-nvfp4-torchao.safetensors" \ + "--encoder-config $BIN/ltx2_gemma4_text_config.json --prompt " \ + "--frames $FRAMES --width $WW --height $HH --seed $SEED --device cuda" \ + "--workdir $d --out $d/video.mp4" + } >> "$log" + # VT_OP_PROVIDER_STATS=1 makes each op announce itself once when it resolves: + # op=18 is kAttention, op=21 is kAttentionDenseFlash, device=1 is kCUDA. + # Without it the only evidence of which arm ran is the wall clock, which is one + # of the things being measured. + export VLLM_LTX2_DIT_FLASH_ATTN="$knob" + VT_OP_PROVIDER_STATS=1 timeout -s INT "$tmo" stdbuf -oL -eL "$BIN/ltx2-gen" \ + --pipeline-kind one_stage \ + --dit "$CKUSE/ltx-2.5-22b-dev-transformer-bf16.safetensors" \ + --video-vae "$CKUSE/ltx-2.5-video-vae-conv-bf16.safetensors" \ + --audio-vae "$CKUSE/ltx-2.5-audio-vae-bf16.safetensors" \ + --checkpoint-class full \ + --encoder "$CKUSE/gemma4-12b-with-proj-nvfp4-torchao.safetensors" \ + --encoder-config "$BIN/ltx2_gemma4_text_config.json" \ + --prompt "$PROMPT" \ + --frames "$FRAMES" --width "$WW" --height "$HH" --seed "$SEED" \ + --device cuda --workdir "$d" --out "$d/video.mp4" >> "$log" 2>&1 & + local pid=$! + # MEMORY FLOOR ONLY. No sample cap: the point of this run is a COMPLETED + # render, and a cap is what left the previous attempt with no frames. No stack + # sampler either: it would ptrace-stop the process and move the very per-forward + # times the same run is reducing. + local stopped_by="none" tick=0 + while kill -0 "$pid" 2>/dev/null; do + local n avail + n=$(grep -c 'last=' "$log" 2>/dev/null | head -1) + avail=$(awk '/^MemAvailable:/{printf "%.1f", $2/1048576}' /proc/meminfo 2>/dev/null) + echo "$(date -u +%H:%M:%S) $label forwards=$n memavail_gib=$avail" >> "$d/watch.tsv" + # HEARTBEAT ON STDOUT, every ~2 minutes. The engine writes to its own log, so + # without this the job produces NOTHING on stdout for up to two hours during + # the naive render, and a lease with an idle timeout would kill a healthy run + # that is doing exactly what it was asked to do. It doubles as progress: a + # forward count that stops advancing is visible before the deadline, not + # after it. + tick=$((tick + 1)) + [ $((tick % 12)) = 1 ] && say " [$label] forward $n, MemAvailable ${avail} GiB" + if [ -n "$avail" ] && awk -v a="$avail" -v f="$MEM_FLOOR_GIB" 'BEGIN{exit !(a/dev/null; break + fi + sleep 10 + done + wait "$pid" 2>/dev/null + local rc=$? + unset VLLM_LTX2_DIT_FLASH_ATTN + local nf; nf=$(ls "$d"/frame_*.ppm 2>/dev/null | wc -l) + say "render $label exit=$rc stopped_by=$stopped_by frames=$nf" + # THE TWO-SIDED ROUTING PROOF, per arm, from that arm's own log. One-sided + # counting cannot tell a routed call from an added one: the flash arm must show + # op=21 AND NOT op=18, and the naive arm the reverse. LTX's cross-attentions use + # op=19 in both, which is why it is printed rather than asserted on. + echo "--- op-provider selections (18 kAttention / 19 kAttentionCross / 21 kAttentionDenseFlash, device=1 CUDA) ---" | tee -a "$d/ARM" + grep -E 'op-provider.*op=(18|19|20|21) device=1' "$log" | sort -u | sed 's/^/ /' | tee -a "$d/ARM" + local n18 n21 + n18=$(grep -cE 'op-provider.*op=18 device=1' "$log") + n21=$(grep -cE 'op-provider.*op=21 device=1' "$log") + echo " op18_naive=$n18 op21_flash=$n21" | tee -a "$d/ARM" + case "$knob" in + 0) [ "$n18" -ge 1 ] && [ "$n21" = 0 ] && echo " ROUTING_OK=naive" || echo " ROUTING_BAD=naive (want op18>=1 op21==0)";; + *) [ "$n21" -ge 1 ] && [ "$n18" = 0 ] && echo " ROUTING_OK=flash" || echo " ROUTING_BAD=flash (want op21>=1 op18==0)";; + esac | tee -a "$d/ARM" + # Per-forward MEDIAN from the engine's own `last=` lines. Never the governor, + # which has reported 1.00 s, 69.1 s, 162 s and 396.9 s for this one quantity. + grep -ohE 'last=[0-9.]+s' "$log" | sed 's/last=//;s/s$//' > "$d/samples.txt" + sort -n "$d/samples.txt" | awk -v L="$label" ' + {a[NR]=$1; s+=$1} + END{ if(!NR){print " " L ": NO SAMPLES"; exit} + m = (NR%2) ? a[(NR+1)/2] : (a[NR/2]+a[NR/2+1])/2; + printf " %s: n=%d median=%.3fs mean=%.3fs min=%.3fs max=%.3fs\n", L, NR, m, s/NR, a[1], a[NR] }' | tee -a "$d/ARM" + echo " memavail low-water: $(awk -F'\t' '{gsub(/memavail_gib=/,"",$4); print $4}' "$d/watch.tsv" 2>/dev/null | sort -n | head -1) GiB" | tee -a "$d/ARM" + echo " frames=$nf audio=$([ -s "$d/audio.wav" ] && stat -c %s "$d/audio.wav" || echo 0)" | tee -a "$d/ARM" +} + +render flash 1 "${TMO_FLASH:-3600}" +render naive 0 "${TMO_NAIVE:-10800}" +render flash-ctl 1 "${TMO_FLASH:-3600}" + +say "=== [H] the speed pair, same binary, same lease, neither arm sampled ===" +python3 - "$OUT/flash/samples.txt" "$OUT/naive/samples.txt" "$OUT/flash-ctl/samples.txt" <<'PY' +import sys, statistics +def med(p): + try: v=[float(x) for x in open(p).read().split()] + except OSError: v=[] + return v, (statistics.median(v) if v else None) +f,fm = med(sys.argv[1]); n,nm = med(sys.argv[2]); c,cm = med(sys.argv[3]) +for lab,v,m in (("flash",f,fm),("naive",n,nm),("flash-ctl",c,cm)): + print(f" {lab}: n={len(v)} median={m}") +if fm and nm: + print(f" SPEEDUP (naive median / flash median) = {nm/fm:.3f}x") +else: + print(" INCOMPLETE: an arm produced no samples; report that, do not impute") +PY + +say "=== [I] the pixel comparison ===" +# The tool is the one committed in this same source tree, run from the tree, so +# the thresholds it applies are the ones the spec derives and not a copy that +# drifted. Its exit status is the gate: 0 pass, 1 a threshold failed, 2 an input +# could not be read. A 2 is never a pass. +python3 "$CMP" \ + --a "$OUT/naive" --b "$OUT/flash" --control "$OUT/flash-ctl" \ + --label-a naive --label-b flash --label-control flash-ctl \ + --json "$OUT/pixel-compare.json" 2>&1 | tee "$OUT/pixel-compare.txt" +echo "PIXEL_COMPARE_RC=${PIPESTATUS[0]}" + +say "=== [J] the cross-check against the recorded 20260820 baseline ===" +# A different binary lineage (a50c57d69, an ancestor of the swap), so this is +# never the control and never the A/B. It answers one narrow question: whether a +# naive render is reproducible ACROSS builds, which bounds how much of any delta +# above could be everything-else-on-main rather than the kernel. +OLD=$FULL/out/20260820T223701Z/768x448-49f +if [ -d "$OLD" ]; then + python3 "$CMP" --a "$OLD" --b "$OUT/naive" \ + --label-a baseline-20260820 --label-b naive \ + --json "$OUT/cross-check.json" 2>&1 | tee "$OUT/cross-check.txt" + echo "CROSS_CHECK_RC=${PIPESTATUS[0]}" +else + echo " the recorded baseline is not on this share; cross-check SKIPPED" +fi + +say "=== [K] artefacts ===" +for d in "$OUT"/flash "$OUT"/naive "$OUT"/flash-ctl; do + [ -d "$d" ] || continue + printf " %-24s frames=%s audio=%s mp4=%s\n" "$(basename "$d")" \ + "$(ls "$d"/frame_*.ppm 2>/dev/null | wc -l)" \ + "$([ -s "$d/audio.wav" ] && echo yes || echo no)" \ + "$([ -s "$d/video.mp4" ] && echo yes || echo no)" +done +say "DONE OUT=$OUT" diff --git a/scripts/ltx25-render-compare.py b/scripts/ltx25-render-compare.py new file mode 100755 index 000000000..8d25ff5ec --- /dev/null +++ b/scripts/ltx25-render-compare.py @@ -0,0 +1,451 @@ +#!/usr/bin/env python3 +"""Compare two LTX-2.5 renders pixel-for-pixel and sample-for-sample. + +A diffusion render has no token gate. There is no discrete output to hold +against a reference, so the correctness net every other model in this tree +leans on does not exist here. This is the substitute: two renders of the same +prompt, seed, geometry, checkpoint and binary, differing only in the knob under +test, compared on the bytes they actually wrote. + +It exists because `LTX25-DIT-ATTN-FLASH` (#1549) moved the DiT self-attention +from `vt::Attention` to `vt::AttentionDenseFlash`, the two are NOT bit-identical +on CUDA, and nothing measured what that did to a picture (#1612). The same +question is owed for the FA-2 arm (#1551), whose divergence is larger, so this +tool takes the arm labels as arguments and hard-codes neither. + +WHAT IT MEASURES, and why each one is here rather than a fourth statistic: + + identity Byte equality of the frame files, then array equality. If the + two arms are bit-identical there is nothing further to argue + and every threshold below is vacuous. Report it FIRST so a + reader never mistakes a passing bound for an unread one. + + |delta| max, mean and the full histogram over 8-bit RGB. The histogram + is not decoration: "within numerical noise" predicts a mass + concentrated at 0 and 1, and a bimodal tail is the shape of a + structural difference wearing a small mean. + + PSNR The video-coding convention. 40 dB on 8-bit is the usual + "visually lossless" line and it is a threshold this experiment + did not choose for itself. + + SSIM Wang et al. 2004, the ORIGINAL 11x11 Gaussian sigma=1.5 window + on luma, not scikit-image's 7x7 uniform default. Stated because + the two disagree in the third decimal and this gate reads that + far. + + temporal The self-calibrating one, and the only bound here derived from + the render rather than from a convention. Mean absolute + difference between ADJACENT FRAMES of arm A is the video's own + frame-to-frame step. An arm-to-arm difference far below it is + smaller than the motion the render is made of. A ratio, unlike + a constant, does not need to be re-argued at another geometry. + + audio The DiT drives both streams, so a video-only comparison would + leave half the change unmeasured. + +USAGE + ltx25-render-compare.py --a --b [--control ] \ + [--label-a naive] [--label-b flash] [--json out.json] + +`--control` is a THIRD render of arm A's own configuration. It measures the +noise floor: run-to-run nondeterminism of the same binary and the same knob. +Without it, an arm-to-arm delta cannot be attributed to the knob rather than to +the machine. With it, the attribution is arithmetic -- if the control is zero, +every bit of the A-vs-B delta is the knob; if the control is the same size as +the delta, the knob changed nothing the box does not change on its own. + +Exit 0 when every threshold passes, 1 when one fails, 2 when the inputs cannot +be read. A missing input is never a pass. +""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import math +import os +import struct +import sys +import wave + +import numpy as np + +# --- the registered thresholds ------------------------------------------------ +# These are DEFAULTS, and they are written here rather than passed at the call +# site so that the criterion is committed to the repository before any number is +# read against it. `.agents/specs/ltx25-dit-attn-flash.md` section 10.4 derives +# each one. Overriding one on the command line is legitimate for a different +# arm pair (#1551) and is recorded in the JSON as an override. +DEFAULT_MAX_MEAN_ABS = 1.0 # 8-bit levels, mean over every pixel/channel +DEFAULT_MIN_PSNR_DB = 40.0 # visually-lossless convention +DEFAULT_MIN_SSIM = 0.99 # per-frame minimum, not the mean +DEFAULT_MAX_TEMPORAL_RATIO = 0.10 # arm delta vs the render's own motion step +DEFAULT_MIN_AUDIO_PSNR_DB = 40.0 +DEFAULT_MIN_AUDIO_CORR = 0.999 + + +# --- PPM ---------------------------------------------------------------------- +def read_ppm(path: str) -> np.ndarray: + """Read a binary P6 PPM into an (H, W, 3) uint8 array. + + Written out rather than delegated because the only image library certain to + be present in a leased worker is the one that ships with numpy, which is + none. The parser is strict: a maxval other than 255 changes the meaning of + every threshold below, so it refuses instead of rescaling silently. + """ + with open(path, "rb") as fh: + data = fh.read() + if not data.startswith(b"P6"): + raise ValueError(f"{path}: not a binary P6 PPM (starts {data[:2]!r})") + # Header tokens: P6 width height maxval, with '#' comments allowed anywhere. + tokens: list[bytes] = [] + i = 2 + while len(tokens) < 3: + while i < len(data) and data[i : i + 1].isspace(): + i += 1 + if data[i : i + 1] == b"#": + while i < len(data) and data[i : i + 1] not in (b"\n", b"\r"): + i += 1 + continue + j = i + while j < len(data) and not data[j : j + 1].isspace(): + j += 1 + tokens.append(data[i:j]) + i = j + i += 1 # exactly one whitespace byte after maxval, per the format + w, h, maxval = (int(t) for t in tokens) + if maxval != 255: + raise ValueError(f"{path}: maxval {maxval}, expected 255") + need = w * h * 3 + px = data[i : i + need] + if len(px) != need: + raise ValueError(f"{path}: truncated, {len(px)} of {need} pixel bytes") + return np.frombuffer(px, dtype=np.uint8).reshape(h, w, 3) + + +def frame_paths(d: str) -> list[str]: + names = sorted(n for n in os.listdir(d) if n.startswith("frame_") and n.endswith(".ppm")) + return [os.path.join(d, n) for n in names] + + +def sha256_file(path: str) -> str: + h = hashlib.sha256() + with open(path, "rb") as fh: + for chunk in iter(lambda: fh.read(1 << 20), b""): + h.update(chunk) + return h.hexdigest() + + +# --- SSIM --------------------------------------------------------------------- +def _gauss1d(size: int = 11, sigma: float = 1.5) -> np.ndarray: + r = np.arange(size, dtype=np.float64) - (size - 1) / 2.0 + k = np.exp(-(r ** 2) / (2.0 * sigma ** 2)) + return k / k.sum() + + +def _blur(x: np.ndarray, k: np.ndarray) -> np.ndarray: + """Separable convolution with reflect padding, numpy only.""" + pad = len(k) // 2 + xp = np.pad(x, ((pad, pad), (0, 0)), mode="reflect") + out = np.zeros_like(x) + for i, w in enumerate(k): + out += w * xp[i : i + x.shape[0], :] + xp = np.pad(out, ((0, 0), (pad, pad)), mode="reflect") + out2 = np.zeros_like(x) + for i, w in enumerate(k): + out2 += w * xp[:, i : i + x.shape[1]] + return out2 + + +def luma(rgb: np.ndarray) -> np.ndarray: + """Rec.601 luma, the plane SSIM is conventionally computed on.""" + f = rgb.astype(np.float64) + return 0.299 * f[..., 0] + 0.587 * f[..., 1] + 0.114 * f[..., 2] + + +def ssim(a: np.ndarray, b: np.ndarray) -> float: + """Wang et al. 2004 mean SSIM on luma, 11x11 Gaussian sigma=1.5, L=255.""" + k = _gauss1d() + c1 = (0.01 * 255.0) ** 2 + c2 = (0.03 * 255.0) ** 2 + mu_a, mu_b = _blur(a, k), _blur(b, k) + mu_a2, mu_b2, mu_ab = mu_a * mu_a, mu_b * mu_b, mu_a * mu_b + s_a = _blur(a * a, k) - mu_a2 + s_b = _blur(b * b, k) - mu_b2 + s_ab = _blur(a * b, k) - mu_ab + num = (2 * mu_ab + c1) * (2 * s_ab + c2) + den = (mu_a2 + mu_b2 + c1) * (s_a + s_b + c2) + return float(np.mean(num / den)) + + +def psnr_from_mse(mse: float, peak: float = 255.0) -> float: + if mse <= 0.0: + return math.inf + return 20.0 * math.log10(peak) - 10.0 * math.log10(mse) + + +# --- video -------------------------------------------------------------------- +def compare_video(dir_a: str, dir_b: str, label_a: str, label_b: str) -> dict: + pa, pb = frame_paths(dir_a), frame_paths(dir_b) + if not pa or not pb: + raise SystemExit(f"FATAL: no frames ({label_a}: {len(pa)}, {label_b}: {len(pb)})") + if len(pa) != len(pb): + raise SystemExit(f"FATAL: frame count differs ({len(pa)} vs {len(pb)})") + + res: dict = {"label_a": label_a, "label_b": label_b, "frames": len(pa), "per_frame": []} + + identical_files = 0 + total_sq = 0.0 + total_abs = 0.0 + total_n = 0 + hist = np.zeros(256, dtype=np.int64) + prev_a: np.ndarray | None = None + adjacent_mads: list[float] = [] + global_max = 0 + + for idx, (fa, fb) in enumerate(zip(pa, pb)): + ha, hb = sha256_file(fa), sha256_file(fb) + same_file = ha == hb + identical_files += int(same_file) + A, B = read_ppm(fa), read_ppm(fb) + if A.shape != B.shape: + raise SystemExit(f"FATAL: frame {idx} shape {A.shape} vs {B.shape}") + d = np.abs(A.astype(np.int16) - B.astype(np.int16)) + hist += np.bincount(d.reshape(-1), minlength=256).astype(np.int64) + mx = int(d.max()) + global_max = max(global_max, mx) + mean_abs = float(d.mean()) + mse = float((d.astype(np.float64) ** 2).mean()) + total_sq += mse * d.size + total_abs += float(d.sum()) + total_n += d.size + la, lb = luma(A), luma(B) + s = ssim(la, lb) + if prev_a is not None: + adjacent_mads.append(float(np.abs(la - prev_a).mean())) + prev_a = la + res["per_frame"].append( + { + "index": idx, + "file_a": os.path.basename(fa), + "sha_equal": same_file, + "max_abs": mx, + "mean_abs": mean_abs, + "psnr_db": psnr_from_mse(mse), + "ssim": s, + "differing_pixels": int((d.sum(axis=2) > 0).sum()), + "pixels": int(d.shape[0] * d.shape[1]), + } + ) + + agg_mse = total_sq / total_n + res["identical_frame_files"] = identical_files + res["bit_identical"] = identical_files == len(pa) + res["max_abs"] = global_max + res["mean_abs"] = total_abs / total_n + res["psnr_db"] = psnr_from_mse(agg_mse) + res["rmse"] = math.sqrt(agg_mse) + res["ssim_mean"] = float(np.mean([f["ssim"] for f in res["per_frame"]])) + res["ssim_min"] = float(np.min([f["ssim"] for f in res["per_frame"]])) + res["psnr_min_db"] = float(np.min([f["psnr_db"] for f in res["per_frame"]])) + res["delta_histogram"] = {str(v): int(c) for v, c in enumerate(hist) if c} + res["samples"] = int(total_n) + # The self-calibrating denominator: arm A's own frame-to-frame step, on luma, + # in the same 8-bit units as mean_abs above. + res["adjacent_frame_mad_a"] = float(np.mean(adjacent_mads)) if adjacent_mads else None + if res["adjacent_frame_mad_a"]: + # mean_abs is over RGB, adjacent MAD over luma; recompute the numerator on + # luma so the ratio divides like with like rather than nearly-like. + res["temporal_ratio"] = None # filled by the caller, which has the luma delta + return res + + +def compare_video_luma_delta(dir_a: str, dir_b: str) -> float: + """Mean |delta| on LUMA, the numerator of the temporal ratio.""" + pa, pb = frame_paths(dir_a), frame_paths(dir_b) + tot, n = 0.0, 0 + for fa, fb in zip(pa, pb): + la, lb = luma(read_ppm(fa)), luma(read_ppm(fb)) + tot += float(np.abs(la - lb).sum()) + n += la.size + return tot / n + + +# --- audio -------------------------------------------------------------------- +def read_wav(path: str) -> tuple[np.ndarray, int]: + with wave.open(path, "rb") as w: + n, ch, sw, sr = w.getnframes(), w.getnchannels(), w.getsampwidth(), w.getframerate() + raw = w.readframes(n) + if sw != 2: + raise ValueError(f"{path}: sample width {sw}, expected 2 (16-bit PCM)") + a = np.frombuffer(raw, dtype=" dict: + if not (os.path.exists(a_path) and os.path.exists(b_path)): + return {"present": False, "reason": "one or both wav files absent"} + A, sr_a = read_wav(a_path) + B, sr_b = read_wav(b_path) + out: dict = {"present": True, "sample_rate_a": sr_a, "sample_rate_b": sr_b, + "frames_a": int(A.shape[0]), "frames_b": int(B.shape[0]), + "sha_equal": sha256_file(a_path) == sha256_file(b_path)} + if A.shape != B.shape or sr_a != sr_b: + out["comparable"] = False + return out + out["comparable"] = True + d = np.abs(A - B) + peak = 32768.0 + mse = float((d ** 2).mean()) + out["max_abs_lsb"] = float(d.max()) + out["mean_abs_lsb"] = float(d.mean()) + out["max_abs_fs"] = float(d.max() / peak) + out["rms_diff_fs"] = float(math.sqrt(mse) / peak) + out["psnr_db"] = psnr_from_mse(mse, peak=peak) + out["bit_identical"] = bool(out["sha_equal"] and d.max() == 0) + fa, fb = A.reshape(-1), B.reshape(-1) + if fa.std() > 0 and fb.std() > 0: + out["pearson_r"] = float(np.corrcoef(fa, fb)[0, 1]) + else: + out["pearson_r"] = None + return out + + +# --- main --------------------------------------------------------------------- +def main() -> int: + ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument("--a", required=True, help="arm A render directory (the reference)") + ap.add_argument("--b", required=True, help="arm B render directory (the change under test)") + ap.add_argument("--control", default=None, help="a repeat of arm A: the run-to-run noise floor") + ap.add_argument("--label-a", default="a") + ap.add_argument("--label-b", default="b") + ap.add_argument("--label-control", default="control") + ap.add_argument("--audio-name", default="audio.wav") + ap.add_argument("--json", default=None) + ap.add_argument("--max-mean-abs", type=float, default=DEFAULT_MAX_MEAN_ABS) + ap.add_argument("--min-psnr-db", type=float, default=DEFAULT_MIN_PSNR_DB) + ap.add_argument("--min-ssim", type=float, default=DEFAULT_MIN_SSIM) + ap.add_argument("--max-temporal-ratio", type=float, default=DEFAULT_MAX_TEMPORAL_RATIO) + ap.add_argument("--min-audio-psnr-db", type=float, default=DEFAULT_MIN_AUDIO_PSNR_DB) + ap.add_argument("--min-audio-corr", type=float, default=DEFAULT_MIN_AUDIO_CORR) + args = ap.parse_args() + + for d in (args.a, args.b) + ((args.control,) if args.control else ()): + if not os.path.isdir(d): + print(f"FATAL: not a directory: {d}", file=sys.stderr) + return 2 + + report: dict = { + "thresholds": { + "max_mean_abs": args.max_mean_abs, + "min_psnr_db": args.min_psnr_db, + "min_ssim": args.min_ssim, + "max_temporal_ratio": args.max_temporal_ratio, + "min_audio_psnr_db": args.min_audio_psnr_db, + "min_audio_corr": args.min_audio_corr, + }, + "inputs": {"a": os.path.abspath(args.a), "b": os.path.abspath(args.b), + "control": os.path.abspath(args.control) if args.control else None}, + } + + v = compare_video(args.a, args.b, args.label_a, args.label_b) + luma_delta = compare_video_luma_delta(args.a, args.b) + v["mean_abs_luma"] = luma_delta + if v["adjacent_frame_mad_a"]: + v["temporal_ratio"] = luma_delta / v["adjacent_frame_mad_a"] + report["video"] = v + + report["audio"] = compare_audio( + os.path.join(args.a, args.audio_name), os.path.join(args.b, args.audio_name) + ) + + if args.control: + c = compare_video(args.a, args.control, args.label_a, args.label_control) + c["mean_abs_luma"] = compare_video_luma_delta(args.a, args.control) + report["control_video"] = c + report["control_audio"] = compare_audio( + os.path.join(args.a, args.audio_name), os.path.join(args.control, args.audio_name) + ) + + # --- verdict -------------------------------------------------------------- + checks: list[tuple[str, bool, str]] = [] + if v["bit_identical"]: + checks.append(("video.bit_identical", True, "every frame file sha256-equal")) + else: + checks.append( + ("video.mean_abs", v["mean_abs"] <= args.max_mean_abs, + f"{v['mean_abs']:.6f} <= {args.max_mean_abs}") + ) + checks.append( + ("video.psnr_min_db", v["psnr_min_db"] >= args.min_psnr_db, + f"{v['psnr_min_db']:.3f} >= {args.min_psnr_db}") + ) + checks.append( + ("video.ssim_min", v["ssim_min"] >= args.min_ssim, + f"{v['ssim_min']:.6f} >= {args.min_ssim}") + ) + if v.get("temporal_ratio") is not None: + checks.append( + ("video.temporal_ratio", v["temporal_ratio"] <= args.max_temporal_ratio, + f"{v['temporal_ratio']:.6f} <= {args.max_temporal_ratio}") + ) + else: + checks.append(("video.temporal_ratio", False, "no adjacent-frame denominator")) + + a = report["audio"] + if not a.get("present"): + checks.append(("audio.present", False, a.get("reason", "absent"))) + elif not a.get("comparable"): + checks.append(("audio.comparable", False, "shape or sample rate differs")) + elif a.get("bit_identical"): + checks.append(("audio.bit_identical", True, "wav sha256-equal")) + else: + checks.append(("audio.psnr_db", a["psnr_db"] >= args.min_audio_psnr_db, + f"{a['psnr_db']:.3f} >= {args.min_audio_psnr_db}")) + checks.append(("audio.pearson_r", (a["pearson_r"] or 0.0) >= args.min_audio_corr, + f"{a['pearson_r']} >= {args.min_audio_corr}")) + + ok = all(c[1] for c in checks) + report["checks"] = [{"name": n, "pass": p, "detail": d} for n, p, d in checks] + report["verdict"] = "PASS" if ok else "FAIL" + + # --- print ---------------------------------------------------------------- + print(f"=== {args.label_a} vs {args.label_b} ===") + print(f"frames {v['frames']}") + print(f"bit-identical frames {v['identical_frame_files']}/{v['frames']}") + print(f"max |delta| (8-bit) {v['max_abs']}") + print(f"mean |delta| RGB {v['mean_abs']:.6f}") + print(f"mean |delta| luma {v['mean_abs_luma']:.6f}") + print(f"RMSE {v['rmse']:.6f}") + print(f"PSNR aggregate {v['psnr_db']:.3f} dB (worst frame {v['psnr_min_db']:.3f} dB)") + print(f"SSIM mean {v['ssim_mean']:.6f} (worst frame {v['ssim_min']:.6f})") + print(f"adjacent-frame MAD (A) {v['adjacent_frame_mad_a']}") + print(f"temporal ratio {v.get('temporal_ratio')}") + print(f"|delta| histogram {dict(list(report['video']['delta_histogram'].items())[:12])}") + if "control_video" in report: + c = report["control_video"] + print(f"--- control ({args.label_a} vs {args.label_control}): the noise floor ---") + print(f"bit-identical frames {c['identical_frame_files']}/{c['frames']}") + print(f"max |delta| {c['max_abs']} mean {c['mean_abs']:.6f}") + print(f"PSNR {c['psnr_db']:.3f} dB SSIM min {c['ssim_min']:.6f}") + print("--- audio ---") + for k in ("present", "comparable", "bit_identical", "max_abs_lsb", "max_abs_fs", + "rms_diff_fs", "psnr_db", "pearson_r"): + if k in a: + print(f"{k:22s} {a[k]}") + print("--- checks ---") + for n, p, d in checks: + print(f" [{'PASS' if p else 'FAIL'}] {n}: {d}") + print(f"VERDICT {report['verdict']}") + + if args.json: + with open(args.json, "w") as fh: + json.dump(report, fh, indent=2, sort_keys=True) + print(f"wrote {args.json}") + return 0 if ok else 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tests/scripts/test_ltx25_render_compare.py b/tests/scripts/test_ltx25_render_compare.py new file mode 100644 index 000000000..12d1c973b --- /dev/null +++ b/tests/scripts/test_ltx25_render_compare.py @@ -0,0 +1,302 @@ +#!/usr/bin/env python3 +"""The pixel comparison's own discrimination proof. + +`.agents/specs/ltx25-dit-attn-flash.md` section 10, #1612. + +`scripts/ltx25-render-compare.py` is the substitute for a token gate on a model +that cannot have one: LTX-2.5 renders pixels, not symbols, so "the output is +the same" has to be a measurement rather than an equality. A tool that answers +that question is only worth its verdict if it FAILS on a difference that matters +and PASSES on one that does not, and neither half is provable by reading it. + +So both halves are pinned here, on fabricated frames, with no NAS and no GPU: + + Identity two identical renders read as bit-identical, and every + threshold is then reported as vacuous rather than as passed. + Dither +/-1 on 3% of samples -- the shape section 10.2 predicts from + bf16 rounding -- passes all four video checks with headroom. + Structure ONE PIXEL of global horizontal shift fails ALL FOUR. That is + the calibration the section 10.4 threshold table quotes, and a + criterion that admitted it would not be a criterion. + Audio a scaled waveform fails, because the DiT drives both streams + and a video-only comparison leaves half the change unmeasured. + Refusal an unreadable input exits 2. A missing input is never a pass. + +The fixtures are TEXTURED rather than flat. A flat image makes SSIM degenerate +and makes a one-pixel shift invisible, so a test built on one would pass while +proving nothing -- the shape this file exists to refuse. +""" + +from __future__ import annotations + +import json +import struct +import subprocess +import sys +import tempfile +import unittest +import wave +from pathlib import Path + +import numpy as np + +ROOT = Path(__file__).resolve().parents[2] +TOOL = ROOT / "scripts/ltx25-render-compare.py" + +EXIT_PASS = 0 +EXIT_FAIL = 1 +EXIT_UNREADABLE = 2 + +W, H, FRAMES = 96, 64, 6 + + +def write_ppm(path: Path, arr: np.ndarray) -> None: + h, w, _ = arr.shape + path.write_bytes(b"P6\n%d %d\n255\n" % (w, h) + arr.astype(np.uint8).tobytes()) + + +def write_wav(path: Path, samples: np.ndarray, rate: int = 48000) -> None: + with wave.open(str(path), "wb") as f: + f.setnchannels(2) + f.setsampwidth(2) + f.setframerate(rate) + f.writeframes(samples.astype(" list[np.ndarray]: + """A textured, MOVING sequence: SSIM and the temporal denominator both need one.""" + d.mkdir(parents=True, exist_ok=True) + yy, xx = np.mgrid[0:H, 0:W] + base = ( + 127 + + 60 * np.sin(xx / 4.0) + + 40 * np.cos(yy / 3.0) + + rng.integers(-20, 21, (H, W)) + ) + frames = [] + for i in range(FRAMES): + shifted = np.roll(base, motion * i, axis=1) + rgb = np.stack( + [shifted, np.roll(shifted, 5, axis=0), np.roll(shifted, -5, axis=1)], axis=2 + ) + arr = np.clip(rgb, 0, 255).astype(np.uint8) + write_ppm(d / f"frame_{i:06d}.ppm", arr) + frames.append(arr) + t = np.arange(4800) + wav = np.stack([8000 * np.sin(t / 20.0), 8000 * np.sin(t / 31.0)], axis=1) + write_wav(d / "audio.wav", wav) + return frames + + +def run(*args: str) -> tuple[int, str, dict | None]: + with tempfile.NamedTemporaryFile(suffix=".json", delete=False) as jf: + jpath = jf.name + p = subprocess.run( + [sys.executable, str(TOOL), *args, "--json", jpath], + capture_output=True, + text=True, + ) + try: + report = json.loads(Path(jpath).read_text()) + except (OSError, json.JSONDecodeError): + report = None + return p.returncode, p.stdout + p.stderr, report + + +def checks_of(report: dict) -> dict[str, bool]: + return {c["name"]: c["pass"] for c in report["checks"]} + + +class Discrimination(unittest.TestCase): + def setUp(self) -> None: + self.tmp = tempfile.TemporaryDirectory() + root = Path(self.tmp.name) + rng = np.random.default_rng(20260822) + self.a = root / "a" + self.frames = make_render(self.a, rng) + + # SAME bytes, a second directory: two renders that agree. + self.same = root / "same" + make_render(self.same, np.random.default_rng(20260822)) + + # DITHER: +/-1 on 3% of samples. Section 10.2 predicts single-ULP bf16 + # flips on 8.6e-05 to 3.7e-04 of attention outputs; 3% at the 8-bit + # artefact is deliberately far MORE perturbation than that, so a pass + # here is a pass with room. + self.dither = root / "dither" + self.dither.mkdir() + drng = np.random.default_rng(7) + for i, arr in enumerate(self.frames): + noise = (drng.random(arr.shape) < 0.03) * drng.integers(-1, 2, arr.shape) + write_ppm(self.dither / f"frame_{i:06d}.ppm", + np.clip(arr.astype(np.int16) + noise, 0, 255)) + (self.dither / "audio.wav").write_bytes((self.a / "audio.wav").read_bytes()) + + # STRUCTURE: one pixel of global horizontal shift. + self.shift = root / "shift" + self.shift.mkdir() + for i, arr in enumerate(self.frames): + write_ppm(self.shift / f"frame_{i:06d}.ppm", np.roll(arr, 1, axis=1)) + (self.shift / "audio.wav").write_bytes((self.a / "audio.wav").read_bytes()) + + def tearDown(self) -> None: + self.tmp.cleanup() + + def test_identical_renders_read_as_bit_identical(self) -> None: + rc, out, rep = run("--a", str(self.a), "--b", str(self.same)) + self.assertEqual(rc, EXIT_PASS, out) + self.assertTrue(rep["video"]["bit_identical"], out) + self.assertEqual(rep["video"]["max_abs"], 0) + self.assertEqual(rep["video"]["identical_frame_files"], FRAMES) + # A bit-identical pair reports THAT, and does not report four thresholds + # as passed. A reader must never mistake a vacuous bound for a read one. + names = checks_of(rep) + self.assertIn("video.bit_identical", names) + self.assertNotIn("video.psnr_min_db", names) + self.assertTrue(rep["audio"]["bit_identical"]) + self.assertEqual(rep["verdict"], "PASS") + + def test_dither_passes_every_video_check(self) -> None: + rc, out, rep = run("--a", str(self.a), "--b", str(self.dither)) + self.assertEqual(rc, EXIT_PASS, out) + v, names = rep["video"], checks_of(rep) + self.assertFalse(v["bit_identical"]) + for name in ("video.mean_abs", "video.psnr_min_db", "video.ssim_min", + "video.temporal_ratio"): + self.assertTrue(names[name], f"{name} failed on a dither: {out}") + # The delta is confined to 0 and 1: the shape "numerical noise" predicts. + self.assertEqual(set(v["delta_histogram"]) - {"0", "1"}, set()) + self.assertLess(v["mean_abs"], 0.1) + self.assertGreater(v["psnr_min_db"], 50.0) + self.assertGreater(v["ssim_min"], 0.99) + + def test_one_pixel_shift_fails_all_four_video_checks(self) -> None: + """The discrimination proof. All four, not one: a criterion that caught + a global shift on only one axis would be one threshold with three + decorations.""" + rc, out, rep = run("--a", str(self.a), "--b", str(self.shift)) + self.assertEqual(rc, EXIT_FAIL, out) + names = checks_of(rep) + for name in ("video.mean_abs", "video.psnr_min_db", "video.ssim_min", + "video.temporal_ratio"): + self.assertFalse(names[name], f"{name} PASSED on a one-pixel shift: {out}") + self.assertEqual(rep["verdict"], "FAIL") + + def test_temporal_ratio_is_normalised_by_arm_a_motion(self) -> None: + """V4's denominator is the render's own frame-to-frame step, so the same + absolute difference must read SMALLER against a faster-moving render. + Without that, V4 is a constant wearing a ratio's name.""" + root = Path(self.tmp.name) + slow, fast = root / "slow", root / "fast" + make_render(slow, np.random.default_rng(3), motion=1) + make_render(fast, np.random.default_rng(3), motion=12) + for src, dst in ((slow, root / "slow_s"), (fast, root / "fast_s")): + dst.mkdir() + for p in sorted(src.glob("frame_*.ppm")): + arr = np.frombuffer( + p.read_bytes().split(b"255\n", 1)[1], dtype=np.uint8 + ).reshape(H, W, 3) + write_ppm(dst / p.name, np.clip(arr.astype(np.int16) + 2, 0, 255)) + (dst / "audio.wav").write_bytes((src / "audio.wav").read_bytes()) + _, _, slow_rep = run("--a", str(slow), "--b", str(root / "slow_s")) + _, _, fast_rep = run("--a", str(fast), "--b", str(root / "fast_s")) + self.assertAlmostEqual(slow_rep["video"]["mean_abs"], + fast_rep["video"]["mean_abs"], places=6) + self.assertGreater(slow_rep["video"]["temporal_ratio"], + fast_rep["video"]["temporal_ratio"]) + + def test_audio_divergence_fails_even_when_the_video_matches(self) -> None: + """The DiT drives both streams. A comparison that only reads pixels + would call a broken audio path identical.""" + root = Path(self.tmp.name) + bad = root / "bad_audio" + bad.mkdir() + for p in sorted(self.a.glob("frame_*.ppm")): + (bad / p.name).write_bytes(p.read_bytes()) + with wave.open(str(self.a / "audio.wav"), "rb") as f: + n, rate = f.getnframes(), f.getframerate() + raw = f.readframes(n) + s = np.frombuffer(raw, dtype=" None: + """The control never enters the verdict: it is the scale the verdict is + read against, and a tool that folded it into the pass/fail would hide + exactly the attribution it exists to supply.""" + rc, out, rep = run("--a", str(self.a), "--b", str(self.dither), + "--control", str(self.same)) + self.assertEqual(rc, EXIT_PASS, out) + self.assertIn("control_video", rep) + self.assertTrue(rep["control_video"]["bit_identical"], out) + self.assertNotIn("control", " ".join(c["name"] for c in rep["checks"])) + + +class Refusal(unittest.TestCase): + def test_missing_directory_exits_two(self) -> None: + with tempfile.TemporaryDirectory() as t: + a = Path(t) / "a" + make_render(a, np.random.default_rng(1)) + rc, out, _ = run("--a", str(a), "--b", str(Path(t) / "absent")) + self.assertEqual(rc, EXIT_UNREADABLE, out) + + def test_frame_count_mismatch_is_fatal(self) -> None: + with tempfile.TemporaryDirectory() as t: + a, b = Path(t) / "a", Path(t) / "b" + make_render(a, np.random.default_rng(1)) + make_render(b, np.random.default_rng(1)) + next(iter(sorted(b.glob("frame_*.ppm")))).unlink() + rc, out, _ = run("--a", str(a), "--b", str(b)) + self.assertNotEqual(rc, EXIT_PASS, out) + self.assertIn("frame count differs", out) + + def test_ppm_reader_refuses_a_maxval_it_cannot_scale(self) -> None: + """Every threshold below is in 8-bit levels. A 16-bit PPM would make + each one mean something else, so it is refused rather than rescaled.""" + with tempfile.TemporaryDirectory() as t: + p = Path(t) / "x.ppm" + p.write_bytes(b"P6\n2 2\n65535\n" + b"\0" * 24) + mod = _load_tool() + with self.assertRaises(ValueError): + mod.read_ppm(str(p)) + + def test_ppm_reader_refuses_a_truncated_file(self) -> None: + with tempfile.TemporaryDirectory() as t: + p = Path(t) / "x.ppm" + p.write_bytes(b"P6\n8 8\n255\n" + b"\0" * 10) + mod = _load_tool() + with self.assertRaises(ValueError): + mod.read_ppm(str(p)) + + +def _load_tool(): + import importlib.util + + spec = importlib.util.spec_from_file_location("ltx25_render_compare", TOOL) + mod = importlib.util.module_from_spec(spec) + spec.loader.exec_module(mod) + return mod + + +class Metrics(unittest.TestCase): + def test_ssim_of_a_frame_with_itself_is_one(self) -> None: + mod = _load_tool() + rng = np.random.default_rng(5) + a = rng.integers(0, 256, (H, W)).astype(np.float64) + self.assertAlmostEqual(mod.ssim(a, a), 1.0, places=9) + + def test_psnr_of_zero_error_is_infinite_not_a_large_number(self) -> None: + mod = _load_tool() + self.assertEqual(mod.psnr_from_mse(0.0), float("inf")) + + def test_psnr_matches_its_definition(self) -> None: + mod = _load_tool() + # A uniform error of exactly 1 level: 20*log10(255) = 48.1308 dB. + self.assertAlmostEqual(mod.psnr_from_mse(1.0), 48.13080361, places=6) + + +if __name__ == "__main__": + unittest.main() From 8fde7c91f62bf682174aa3bd3daa0c6f016d554e Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 22 Aug 2026 15:22:24 +0000 Subject: [PATCH 03/19] fix(#1612): two identically broken renders read as a perfect match, and now they do not MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Found in my own criterion before it was ever run, which is the only reason it is a commit rather than a wrong number. Every measurement §10.4 held was a DIFFERENCE. A difference cannot tell two good renders from two identically broken ones. Two all-black renders differ by zero, score infinite PSNR and SSIM `1.000000`, and would have read as the STRONGEST POSSIBLE PASS the table can produce -- on a change whose whole question is whether the picture survived a kernel swap. A run that exited 0 having written frames that were all one colour has happened in this repository, so this is a recorded failure mode and not a hypothesis. C0 closes it. Each arm is judged on its own content, before anything is subtracted, and it is checked FIRST: frames were written, no frame is near-uniform (per-frame variance under 1.0 squared 8-bit levels, which is a standard deviation of one level), every frame hash is distinct, and no adjacent pair has zero motion. These are the checks `ltx25-fullmodel/job/verify_render.py` makes; they are recomputed inside the committed tool rather than shelled out to it, because that file lives on a mutable path on a share -- the same reason §7.1 gives for committing the harness. The report now prints what each arm rendered above what they differ by, so a reader meets the absolute numbers first. Three cases, and the first is the one that motivated the change: two all-black renders bit_identical stays TRUE and is reported honestly, and the verdict is FAIL on four C0 checks. a frozen render textured but 49 identical frames: `not_uniform` PASSES and `motion` and `distinct_frames` FAIL, which is why both checks exist rather than one. a healthy pair all four C0 checks pass on both arms. Mutation M3, applied to a scratch copy and restored byte-for-byte (`sha256 876654cd...` before and after): the C0 loop is emptied. The all-black pair then prints `VERDICT PASS`, and the suite goes red with one failure and two KeyErrors -- the errors being the checks themselves having vanished from the report. 16 of 16 green with the loop restored. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-opus-5 [Claude Code] --- .agents/specs/ltx25-dit-attn-flash.md | 15 +++- scripts/ltx25-render-compare.py | 91 ++++++++++++++++++++++ tests/scripts/test_ltx25_render_compare.py | 74 ++++++++++++++++++ 3 files changed, 179 insertions(+), 1 deletion(-) diff --git a/.agents/specs/ltx25-dit-attn-flash.md b/.agents/specs/ltx25-dit-attn-flash.md index f03ec70d2..253320b67 100644 --- a/.agents/specs/ltx25-dit-attn-flash.md +++ b/.agents/specs/ltx25-dit-attn-flash.md @@ -523,7 +523,7 @@ never a synonym for "probably fine". | A/B, same binary, both arms | `dgx:gpu0` under an `rc` lease | **PENDING** — the flash arm is measured at 7.680 s median (n=19); the worker was lost before the naive arm, so no pair exists (§7.1) | | pixel A/B at production geometry | `dgx:gpu0` under an `rc` lease, `scripts/ltx25-dit-attn-flash-pixel-ab.sh` | **PENDING** — criterion registered in §10.4; result in §10.7 | | run-to-run control (`flash` twice) | the same lease | **PENDING** — §10.3; without it no arm-to-arm delta is attributable to the kernel | -| the comparison tool discriminates | `tests/scripts/test_ltx25_render_compare.py` | **PENDING until §10.7** — a dither passes and a one-pixel shift fails all four checks (§10.4) | +| the comparison tool discriminates | `tests/scripts/test_ltx25_render_compare.py` | **PENDING until §10.7** — a dither passes, a one-pixel shift fails all four V checks, and two all-black renders fail C0 while reading as a perfect match on every V (§10.4) | | full preflight | `scripts/agent-preflight.sh` | **PASS at HEAD** — and it was NOT before: `documentation-checkpoint` was red on two of this branch's own commits (see below) | | `documentation-checkpoint` | CI, and locally over the branch range | **PASS at HEAD, RED before it, and the red was THIS BRANCH's** — `2aa78c69b` and `2f39a9426` each recorded a measurement in `.agents/benchmark-record.md` without writing `docs/STATUS.md` (and `docs/BENCHMARKS.md` for the second). The control on the main-only range `4c193bd55..5d548d003` is rc 0, so it was not inherited. Both commits were replaced by one that writes all three surfaces together when the branch was rebuilt, and the checker is re-run at each head rather than trusted to have stayed fixed — a job that has stopped appearing in a failing set is not the same fact as a job that passes | | `build-newest-gcc` | CI | **PASS, and now green on `main` too** — it was red on `main` on `::getpid` in `test_qwen3_dflash2_gguf.cpp:547`, a file this change does not touch; [#1581](https://github.com/mudler/vllm.cpp/pull/1581) fixed it and this branch carries that fix through the merge. A red here after the merge is therefore this row's, not inherited | @@ -681,6 +681,7 @@ them by hand afterwards. | # | check | threshold | where it comes from | |---|---|---|---| +| C0 | each arm, ON ITS OWN: frames written, no near-uniform frame, every frame hash distinct, no zero-motion pair | all four, per arm | **the hole every difference-only comparison has** | | V1 | mean \|delta\|, 8-bit RGB | `<= 1.0` level | one level is the quantisation step of the artefact itself; a mean below it says the average pixel is within the PPM's own resolution | | V2 | worst-frame PSNR | `>= 40 dB` | the video-coding "visually lossless" convention. This experiment did not choose it | | V3 | worst-frame SSIM | `>= 0.99` | Wang et al. 2004, 11x11 Gaussian sigma=1.5 on luma. 0.98 is the usual transparency line; this is stricter, and it is the WORST frame rather than the mean | @@ -688,6 +689,18 @@ them by hand afterwards. | A1 | audio PSNR vs full scale | `>= 40 dB` | same convention as V2 | | A2 | audio Pearson r | `>= 0.999` | a waveform that has drifted in time fails this while PSNR can still look tolerable | +**C0 is not a formality, and this criterion did not have it at first.** Every +other line in the table is a DIFFERENCE, and a difference cannot tell two good +renders from two identically broken ones. Two all-black renders differ by zero, +score infinite PSNR and SSIM `1.000000`, and would read as the strongest +possible pass this table can produce. A run that exited 0 having written frames +that were all one colour has happened in this repository, so that is a recorded +failure mode rather than a hypothesis. C0 is computed per arm before anything is +subtracted, and it is checked first. `verify_render.py` makes the same checks, +and they are recomputed inside `scripts/ltx25-render-compare.py` rather than +shelled out to, because that file lives on a mutable path on a share and this +one is committed per revision — the same reason §7.1 gives for the harness. + **V4 is the bound that is derived rather than borrowed, so it carries the argument.** The denominator is the render's own frame-to-frame step: how much one frame differs from the next, in the same 8-bit luma units as the numerator. diff --git a/scripts/ltx25-render-compare.py b/scripts/ltx25-render-compare.py index 8d25ff5ec..1f24ca784 100755 --- a/scripts/ltx25-render-compare.py +++ b/scripts/ltx25-render-compare.py @@ -186,6 +186,55 @@ def psnr_from_mse(mse: float, peak: float = 255.0) -> float: return 20.0 * math.log10(peak) - 10.0 * math.log10(mse) +# --- what each arm rendered, on its own ------------------------------------- +def arm_content(d: str) -> dict: + """Absolute content of ONE arm, independent of the other. + + THE HOLE THIS CLOSES. Every other measurement in this file is a DIFFERENCE, + and a difference cannot tell two good renders from two identically broken + ones. Two all-black renders differ by zero, score infinite PSNR and SSIM + 1.0, and would read as the strongest possible pass. A run that exited 0 + having written frames that were all one colour has happened in this + repository, so this is a recorded failure mode and not a hypothetical. + + So each arm is also judged on its own: does it contain a picture, are the + frames distinct, and does anything move. These are the checks + `ltx25-fullmodel/job/verify_render.py` makes, computed here instead of + shelled out to, because that file lives on a mutable path on a share and + this one is committed per revision. + """ + paths = frame_paths(d) + out: dict = {"dir": os.path.abspath(d), "frames": len(paths)} + if not paths: + out["verdict"] = "NO FRAMES" + return out + means, variances, hashes, adjacent = [], [], set(), [] + prev = None + for p in paths: + a = read_ppm(p) + f = a.astype(np.float64) + means.append(float(f.mean())) + variances.append(float(f.var())) + hashes.add(sha256_file(p)) + lu = luma(a) + if prev is not None: + adjacent.append(float(np.abs(lu - prev).mean())) + prev = lu + out["pixel_mean"] = float(np.mean(means)) + out["per_frame_var_min"] = float(np.min(variances)) + out["per_frame_mean_min"] = float(np.min(means)) + out["per_frame_mean_max"] = float(np.max(means)) + out["distinct_frame_hashes"] = len(hashes) + out["adjacent_frame_mad_mean"] = float(np.mean(adjacent)) if adjacent else 0.0 + out["adjacent_frame_mad_min"] = float(np.min(adjacent)) if adjacent else 0.0 + out["zero_motion_pairs"] = int(sum(1 for m in adjacent if m == 0.0)) + # A frame whose variance is under this carries no picture. 1.0 in squared + # 8-bit levels is a standard deviation of one level: below that, every pixel + # in the frame is the same colour to within the artefact's own resolution. + out["near_uniform_frames"] = int(sum(1 for v in variances if v < 1.0)) + return out + + # --- video -------------------------------------------------------------------- def compare_video(dir_a: str, dir_b: str, label_a: str, label_b: str) -> dict: pa, pb = frame_paths(dir_a), frame_paths(dir_b) @@ -350,6 +399,16 @@ def main() -> int: "control": os.path.abspath(args.control) if args.control else None}, } + # WHAT EACH ARM RENDERED, before anything is subtracted. Reported first + # because a difference of zero between two broken renders is the strongest + # possible pass on every other line in this report. + report["content"] = { + args.label_a: arm_content(args.a), + args.label_b: arm_content(args.b), + } + if args.control: + report["content"][args.label_control] = arm_content(args.control) + v = compare_video(args.a, args.b, args.label_a, args.label_b) luma_delta = compare_video_luma_delta(args.a, args.b) v["mean_abs_luma"] = luma_delta @@ -371,6 +430,28 @@ def main() -> int: # --- verdict -------------------------------------------------------------- checks: list[tuple[str, bool, str]] = [] + # C0 FIRST, and it is not a formality. Everything after this line is a + # DIFFERENCE, and every difference check passes vacuously when both arms are + # equally broken. An arm that rendered nothing, rendered one colour, or + # rendered the same frame 49 times fails HERE, where the failure is legible, + # rather than passing silently as a perfect match. + for label in (args.label_a, args.label_b): + c = report["content"][label] + if not c.get("frames"): + checks.append((f"content.{label}.frames", False, "no frames written")) + continue + checks.append((f"content.{label}.frames", True, f"{c['frames']} frames")) + checks.append((f"content.{label}.not_uniform", + c["near_uniform_frames"] == 0, + f"near-uniform frames {c['near_uniform_frames']} == 0 " + f"(min per-frame variance {c['per_frame_var_min']:.3f})")) + checks.append((f"content.{label}.distinct_frames", + c["distinct_frame_hashes"] == c["frames"], + f"{c['distinct_frame_hashes']} distinct of {c['frames']}")) + checks.append((f"content.{label}.motion", + c["zero_motion_pairs"] == 0 and c["adjacent_frame_mad_mean"] > 0.0, + f"zero-motion pairs {c['zero_motion_pairs']}, " + f"mean adjacent MAD {c['adjacent_frame_mad_mean']:.4f}")) if v["bit_identical"]: checks.append(("video.bit_identical", True, "every frame file sha256-equal")) else: @@ -412,6 +493,16 @@ def main() -> int: report["verdict"] = "PASS" if ok else "FAIL" # --- print ---------------------------------------------------------------- + print("=== what each arm rendered, before anything is subtracted ===") + for label, c in report["content"].items(): + if not c.get("frames"): + print(f"{label:12s} NO FRAMES") + continue + print(f"{label:12s} frames={c['frames']} distinct={c['distinct_frame_hashes']} " + f"mean={c['pixel_mean']:.3f} min_var={c['per_frame_var_min']:.1f} " + f"near_uniform={c['near_uniform_frames']} " + f"adj_mad={c['adjacent_frame_mad_mean']:.4f} " + f"zero_motion_pairs={c['zero_motion_pairs']}") print(f"=== {args.label_a} vs {args.label_b} ===") print(f"frames {v['frames']}") print(f"bit-identical frames {v['identical_frame_files']}/{v['frames']}") diff --git a/tests/scripts/test_ltx25_render_compare.py b/tests/scripts/test_ltx25_render_compare.py index 12d1c973b..8a63c135e 100644 --- a/tests/scripts/test_ltx25_render_compare.py +++ b/tests/scripts/test_ltx25_render_compare.py @@ -235,6 +235,80 @@ def test_control_arm_is_reported_separately(self) -> None: self.assertNotIn("control", " ".join(c["name"] for c in rep["checks"])) +class IdenticallyBroken(unittest.TestCase): + """The hole every difference-only comparison has. + + Two all-black renders differ by zero, score infinite PSNR and SSIM 1.0, and + would read as the strongest possible pass. A run that exited 0 having + written frames that were all one colour has happened in this repository, so + this is a recorded failure mode and not a hypothetical. Each arm is + therefore judged on its own content BEFORE anything is subtracted. + """ + + def setUp(self) -> None: + self.tmp = tempfile.TemporaryDirectory() + self.root = Path(self.tmp.name) + + def tearDown(self) -> None: + self.tmp.cleanup() + + def _flat(self, d: Path, value: int = 0) -> None: + d.mkdir(parents=True, exist_ok=True) + arr = np.full((H, W, 3), value, dtype=np.uint8) + for i in range(FRAMES): + write_ppm(d / f"frame_{i:06d}.ppm", arr) + t = np.arange(4800) + write_wav(d / "audio.wav", np.stack([8000 * np.sin(t / 20.0)] * 2, axis=1)) + + def test_two_all_black_renders_do_not_read_as_a_perfect_match(self) -> None: + a, b = self.root / "a", self.root / "b" + self._flat(a) + self._flat(b) + rc, out, rep = run("--a", str(a), "--b", str(b), "--label-a", "x", "--label-b", "y") + # The difference really is nothing, and the tool says so honestly. + self.assertTrue(rep["video"]["bit_identical"], out) + # And it still FAILS, because neither arm rendered a picture. + self.assertEqual(rc, EXIT_FAIL, out) + names = checks_of(rep) + self.assertFalse(names["content.x.not_uniform"], out) + self.assertFalse(names["content.y.not_uniform"], out) + self.assertFalse(names["content.x.motion"], out) + self.assertFalse(names["content.x.distinct_frames"], out) + + def test_a_frozen_render_fails_on_motion_even_when_it_has_a_picture(self) -> None: + """Textured but identical frames: a picture with nothing moving. The + variance check passes and the motion check is what catches it, which is + why both exist.""" + a = self.root / "a" + make_render(a, np.random.default_rng(11)) + frozen = self.root / "frozen" + frozen.mkdir() + first = sorted(a.glob("frame_*.ppm"))[0].read_bytes() + for i in range(FRAMES): + (frozen / f"frame_{i:06d}.ppm").write_bytes(first) + (frozen / "audio.wav").write_bytes((a / "audio.wav").read_bytes()) + rc, out, rep = run("--a", str(a), "--b", str(frozen), + "--label-a", "good", "--label-b", "frozen") + self.assertEqual(rc, EXIT_FAIL, out) + names = checks_of(rep) + self.assertTrue(names["content.frozen.not_uniform"], out) + self.assertFalse(names["content.frozen.motion"], out) + self.assertFalse(names["content.frozen.distinct_frames"], out) + self.assertTrue(names["content.good.motion"], out) + + def test_a_healthy_pair_passes_every_content_check(self) -> None: + a, b = self.root / "a", self.root / "b" + make_render(a, np.random.default_rng(13)) + make_render(b, np.random.default_rng(13)) + rc, out, rep = run("--a", str(a), "--b", str(b), "--label-a", "p", "--label-b", "q") + self.assertEqual(rc, EXIT_PASS, out) + names = checks_of(rep) + for label in ("p", "q"): + for check in ("frames", "not_uniform", "distinct_frames", "motion"): + self.assertTrue(names[f"content.{label}.{check}"], + f"content.{label}.{check} failed on a healthy render: {out}") + + class Refusal(unittest.TestCase): def test_missing_directory_exits_two(self) -> None: with tempfile.TemporaryDirectory() as t: From e6ee5ec20d869e06b74d910928eccaaeb3234499 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 22 Aug 2026 16:16:16 +0000 Subject: [PATCH 04/19] fix(#1612): a broken render exited 1, and ten mutations of "Wang et al. 2004" left the suite green MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `scripts/ltx25-render-compare.py` is the substitute for a token gate on a model that cannot have one, so the review of its landing asked the only question that matters about an instrument: which of the things it claims can it actually detect. Six answers were wrong, and this repairs them. EXIT 2 MEANT ONE THING AND SAID ANOTHER. The docstring promised "2 when the inputs cannot be read. A missing input is never a pass", and only a missing top-level directory did that. An empty arm directory, a frame-count mismatch, a truncated frame and a 16-bit frame all exited 1 — the status that says THE TWO RENDERS DIFFER. A broken render was therefore indistinguishable from a divergent one, which matters most exactly when it is least noticed: exit 1 is the expected answer for a while. Every unreadable-input condition now raises `UnreadableInput` and leaves `main()` at 2 with no JSON, and a threshold failure still exits 1. FRAMES WERE PAIRED BY POSITION AND NOTHING CHECKED THE NAMES. `sorted()` against `sorted()` puts index i against index i, and equal counts are not equal frames. An arm that lost `frame_000000.ppm` and gained a later one now pairs every frame against its neighbour: on the fixtures that reads as mean |d| 37.8 and a FAIL, a large and entirely spurious finding. The names must correspond. `content..frames = False` WAS UNREACHABLE. `compare_video` raised on a zero-frame arm before the checks loop ran, so one of the four C0 sub-checks §10.4 registers could never fire. Resolved as a refusal rather than a check: an arm that wrote nothing has no defined video statistic, and a check that cannot be False is a decoration in a table whose whole value is that each row can. The spec's C0 row now reads three per arm. An absent `audio.wav` stays a failed check, deliberately — the video comparison is fully defined without it. THE CONTROL WAS COMPARED AGAINST WHICHEVER ARM THE TOOL CHOSE. `--control-of {a,b}` now states which arm the control repeats, the JSON records it, and the control block prints the sentence. The harness's inverted call site is fixed in its own commit; a silent convention a caller can invert is what made that possible. §10.5 SELECTED BETWEEN TWO OPPOSITE VERDICTS AND NOTHING COMPUTED THE SELECTOR. The tool printed the control's numbers for a reader to eyeball. It now reports `control_ratio`: the control's mean |delta| over the treatment's, on luma, with the expected division by zero named rather than crashed. Reported, never checked, because it chooses between two readings of a pass. EVERY PART OF THE SSIM WAS UNPINNED. sigma 1.5 to 3.0 or 0.5, the window 11 to 7 or 3, C1 or C2 to 0.5 or 0, the second separable pass on the wrong axis or deleted, and the luma weights red-only, flat or R/B-swapped: ten mutations, ten greens. V3 names "Wang et al. 2004, 11x11 Gaussian sigma=1.5 on luma" because scikit-image's 7x7 uniform default disagrees in the third decimal and this gate reads to the sixth, so that name was doing work nothing verified. scikit-image is absent here and stays absent, so it is pinned by properties and by values computed at this revision: the taps, the impulse response as `outer(k, k)`, its equality along both axes, the fitted sigma, the Rec.601 triple, and three fixture SSIMs to eight decimals. All ten mutations are now red. A2 COULD NOT FAIL ON ANYTHING THE SUITE OWNED. The only divergent-audio fixture was a 0.5x scale, and Pearson r against a scaled copy is exactly 1.0, so `DEFAULT_MIN_AUDIO_CORR` at -1.0 was green and deleting the check was green. The new fixture is what §10.4 says A2 exists for — a waveform drifted in time. One sample on a 20-sample radian scale is 0.05 rad: 41.28 dB, which A1 passes, and r = 0.99875, which A2 fails. The two checks disagree, which is the measured fact and the reason both exist. Also: absent audio is pinned as a failed check; `read_ppm` is pinned on a non-square fixture, since every previous fixture was square-ish and could not see a transposed header; the unused `import struct` is gone from both files; and the test module docstring lists the six groups that exist rather than five that did. The six `DEFAULT_*` thresholds are byte-identical. Their evidentiary value is that they were committed before any number was read against them, and no repair here needed one moved. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-opus-5 [Claude Code] --- scripts/ltx25-render-compare.py | 211 ++++++++-- tests/scripts/test_ltx25_render_compare.py | 439 ++++++++++++++++++++- 2 files changed, 595 insertions(+), 55 deletions(-) diff --git a/scripts/ltx25-render-compare.py b/scripts/ltx25-render-compare.py index 1f24ca784..85eee50ca 100755 --- a/scripts/ltx25-render-compare.py +++ b/scripts/ltx25-render-compare.py @@ -46,17 +46,36 @@ USAGE ltx25-render-compare.py --a --b [--control ] \ - [--label-a naive] [--label-b flash] [--json out.json] - -`--control` is a THIRD render of arm A's own configuration. It measures the -noise floor: run-to-run nondeterminism of the same binary and the same knob. -Without it, an arm-to-arm delta cannot be attributed to the knob rather than to -the machine. With it, the attribution is arithmetic -- if the control is zero, -every bit of the A-vs-B delta is the knob; if the control is the same size as -the delta, the knob changed nothing the box does not change on its own. + [--control-of a|b] [--label-a naive] [--label-b flash] [--json out.json] + +`--control` is a THIRD render that repeats ONE of the two arms with nothing +changed. It measures the noise floor: run-to-run nondeterminism of the same +binary and the same knob. Without it, an arm-to-arm delta cannot be attributed +to the knob rather than to the machine. With it, the attribution is arithmetic +-- if the control is zero, every bit of the A-vs-B delta is the knob; if the +control is the same size as the delta, the knob changed nothing the box does not +change on its own. + +`--control-of` says WHICH ARM the control repeats, and it exists because the +answer used to be a silent convention. The control was always compared against +arm A, stated only here, and the harness called this tool with the control +repeating arm B -- so the "noise floor" was a second copy of the treatment +comparison, guaranteed to read about the same size as the delta it was supposed +to calibrate, and the null verdict below would have been published whatever the +kernel did. The argument is now explicit, it is recorded in the JSON as +`control_of`, and the control block prints the arm it was read against. + +`control_ratio` is the number section 10.5 selects on: the control's mean +absolute difference over the treatment's, on luma, in the same units. It is +REPORTED and never checked, because it chooses between two readings of a passing +result rather than between passing and failing. It is `null`, with a reason, +when the treatment is bit-identical and the denominator is therefore zero. Exit 0 when every threshold passes, 1 when one fails, 2 when the inputs cannot -be read. A missing input is never a pass. +be read. A missing input is never a pass, and it is never an exit 1 either: a +1 says "the two renders differ", which is a reading of an experiment that +happened, and a broken render that reported it would be indistinguishable from +the finding this tool exists to make. """ from __future__ import annotations @@ -66,12 +85,26 @@ import json import math import os -import struct import sys import wave import numpy as np +# --- the three exit statuses, which are three different statements ------------ +EXIT_PASS = 0 # every threshold passed +EXIT_FAIL = 1 # a threshold failed: the renders differ, and by how much +EXIT_UNREADABLE = 2 # nothing was compared, because an input could not be read + + +class UnreadableInput(ValueError): + """An input this tool cannot read, as opposed to one it read and refused. + + It subclasses `ValueError` because `read_ppm` raised that before this class + existed and callers outside `main()` still catch it. Everything that raises + it leaves `main()` at `EXIT_UNREADABLE` with no JSON written: a report with + no comparison in it is worse than no report. + """ + # --- the registered thresholds ------------------------------------------------ # These are DEFAULTS, and they are written here rather than passed at the call # site so that the criterion is committed to the repository before any number is @@ -98,7 +131,7 @@ def read_ppm(path: str) -> np.ndarray: with open(path, "rb") as fh: data = fh.read() if not data.startswith(b"P6"): - raise ValueError(f"{path}: not a binary P6 PPM (starts {data[:2]!r})") + raise UnreadableInput(f"{path}: not a binary P6 PPM (starts {data[:2]!r})") # Header tokens: P6 width height maxval, with '#' comments allowed anywhere. tokens: list[bytes] = [] i = 2 @@ -117,16 +150,26 @@ def read_ppm(path: str) -> np.ndarray: i += 1 # exactly one whitespace byte after maxval, per the format w, h, maxval = (int(t) for t in tokens) if maxval != 255: - raise ValueError(f"{path}: maxval {maxval}, expected 255") + raise UnreadableInput(f"{path}: maxval {maxval}, expected 255") need = w * h * 3 px = data[i : i + need] if len(px) != need: - raise ValueError(f"{path}: truncated, {len(px)} of {need} pixel bytes") + raise UnreadableInput(f"{path}: truncated, {len(px)} of {need} pixel bytes") return np.frombuffer(px, dtype=np.uint8).reshape(h, w, 3) def frame_paths(d: str) -> list[str]: + """Every `frame_*.ppm` in one arm, in name order, and NEVER an empty list. + + A directory with no frames is a render that did not happen, so it is refused + here rather than carried forward as a comparison with zero terms. It used to + reach a `content..frames = False` check that could never fire -- the + zero-frame arm raised out of `compare_video` first -- and exit 1, which is + the status that says the two renders DIFFER. + """ names = sorted(n for n in os.listdir(d) if n.startswith("frame_") and n.endswith(".ppm")) + if not names: + raise UnreadableInput(f"{d}: no frame_*.ppm files, so this arm rendered nothing") return [os.path.join(d, n) for n in names] @@ -203,15 +246,21 @@ def arm_content(d: str) -> dict: shelled out to, because that file lives on a mutable path on a share and this one is committed per revision. """ - paths = frame_paths(d) + paths = frame_paths(d) # refuses an empty arm; see EXIT_UNREADABLE above out: dict = {"dir": os.path.abspath(d), "frames": len(paths)} - if not paths: - out["verdict"] = "NO FRAMES" - return out means, variances, hashes, adjacent = [], [], set(), [] prev = None + shape = None for p in paths: a = read_ppm(p) + # ONE geometry per arm. Without this the adjacent-frame difference below + # raises a numpy broadcast error, which is not an exit status this tool + # defines and would leave the run with a traceback instead of a verdict. + if shape is None: + shape = a.shape + elif a.shape != shape: + raise UnreadableInput( + f"{p}: frame shape {a.shape} differs from {shape} earlier in {d}") f = a.astype(np.float64) means.append(float(f.mean())) variances.append(float(f.var())) @@ -238,10 +287,26 @@ def arm_content(d: str) -> dict: # --- video -------------------------------------------------------------------- def compare_video(dir_a: str, dir_b: str, label_a: str, label_b: str) -> dict: pa, pb = frame_paths(dir_a), frame_paths(dir_b) - if not pa or not pb: - raise SystemExit(f"FATAL: no frames ({label_a}: {len(pa)}, {label_b}: {len(pb)})") if len(pa) != len(pb): - raise SystemExit(f"FATAL: frame count differs ({len(pa)} vs {len(pb)})") + raise UnreadableInput(f"frame count differs ({len(pa)} vs {len(pb)})") + # PAIRED BY NAME, not merely by position. `sorted()` against `sorted()` puts + # index i of one arm against index i of the other, and equal counts are not + # equal frames: an arm that lost `frame_000000.ppm` and gained a later one + # pairs every frame against its neighbour, and then reports the render's own + # frame-to-frame motion as the arm-to-arm delta. That is a large, plausible + # and entirely spurious number, and nothing downstream can tell it from a + # real divergence. + na = [os.path.basename(p) for p in pa] + nb = [os.path.basename(p) for p in pb] + if na != nb: + i = next(i for i, (x, y) in enumerate(zip(na, nb)) if x != y) + only_a = sorted(set(na) - set(nb))[:4] + only_b = sorted(set(nb) - set(na))[:4] + raise UnreadableInput( + f"frame names do not correspond at index {i}: " + f"{label_a} has {na[i]}, {label_b} has {nb[i]}; " + f"only in {label_a}: {only_a or 'none'}; only in {label_b}: {only_b or 'none'}" + ) res: dict = {"label_a": label_a, "label_b": label_b, "frames": len(pa), "per_frame": []} @@ -260,7 +325,7 @@ def compare_video(dir_a: str, dir_b: str, label_a: str, label_b: str) -> dict: identical_files += int(same_file) A, B = read_ppm(fa), read_ppm(fb) if A.shape != B.shape: - raise SystemExit(f"FATAL: frame {idx} shape {A.shape} vs {B.shape}") + raise UnreadableInput(f"frame {idx} shape {A.shape} vs {B.shape}") d = np.abs(A.astype(np.int16) - B.astype(np.int16)) hist += np.bincount(d.reshape(-1), minlength=256).astype(np.int64) mx = int(d.max()) @@ -324,11 +389,14 @@ def compare_video_luma_delta(dir_a: str, dir_b: str) -> float: # --- audio -------------------------------------------------------------------- def read_wav(path: str) -> tuple[np.ndarray, int]: - with wave.open(path, "rb") as w: - n, ch, sw, sr = w.getnframes(), w.getnchannels(), w.getsampwidth(), w.getframerate() - raw = w.readframes(n) + try: + with wave.open(path, "rb") as w: + n, ch, sw, sr = w.getnframes(), w.getnchannels(), w.getsampwidth(), w.getframerate() + raw = w.readframes(n) + except wave.Error as exc: # a file that is present and is not a wav + raise UnreadableInput(f"{path}: not a readable wav ({exc})") from exc if sw != 2: - raise ValueError(f"{path}: sample width {sw}, expected 2 (16-bit PCM)") + raise UnreadableInput(f"{path}: sample width {sw}, expected 2 (16-bit PCM)") a = np.frombuffer(raw, dtype=" int: ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) ap.add_argument("--a", required=True, help="arm A render directory (the reference)") ap.add_argument("--b", required=True, help="arm B render directory (the change under test)") - ap.add_argument("--control", default=None, help="a repeat of arm A: the run-to-run noise floor") + ap.add_argument("--control", default=None, + help="a repeat of ONE arm, unchanged: the run-to-run noise floor") + ap.add_argument("--control-of", choices=("a", "b"), default="a", + help="which arm --control repeats, and therefore which arm it is " + "compared against (default: a)") ap.add_argument("--label-a", default="a") ap.add_argument("--label-b", default="b") ap.add_argument("--label-control", default="control") @@ -381,10 +453,27 @@ def main() -> int: ap.add_argument("--min-audio-corr", type=float, default=DEFAULT_MIN_AUDIO_CORR) args = ap.parse_args() + # ONE place turns an unreadable input into the status that says so. Every + # refusal below raises rather than returning a number, so a new one cannot + # be added that quietly reports `the renders differ` instead. + try: + return _compare(args) + except UnreadableInput as exc: + print(f"FATAL: {exc}", file=sys.stderr) + print(f"VERDICT UNREADABLE (exit {EXIT_UNREADABLE}): nothing was compared", + file=sys.stderr) + return EXIT_UNREADABLE + except OSError as exc: + print(f"FATAL: cannot read an input: {exc}", file=sys.stderr) + print(f"VERDICT UNREADABLE (exit {EXIT_UNREADABLE}): nothing was compared", + file=sys.stderr) + return EXIT_UNREADABLE + + +def _compare(args: argparse.Namespace) -> int: for d in (args.a, args.b) + ((args.control,) if args.control else ()): if not os.path.isdir(d): - print(f"FATAL: not a directory: {d}", file=sys.stderr) - return 2 + raise UnreadableInput(f"not a directory: {d}") report: dict = { "thresholds": { @@ -397,6 +486,7 @@ def main() -> int: }, "inputs": {"a": os.path.abspath(args.a), "b": os.path.abspath(args.b), "control": os.path.abspath(args.control) if args.control else None}, + "control_of": args.control_of if args.control else None, } # WHAT EACH ARM RENDERED, before anything is subtracted. Reported first @@ -421,12 +511,41 @@ def main() -> int: ) if args.control: - c = compare_video(args.a, args.control, args.label_a, args.label_control) - c["mean_abs_luma"] = compare_video_luma_delta(args.a, args.control) + # THE ARM THE CONTROL REPEATS, named by the caller. A control is only a + # noise floor when it is compared against the arm it is a repeat OF. + ctl_dir = args.a if args.control_of == "a" else args.b + ctl_label = args.label_a if args.control_of == "a" else args.label_b + c = compare_video(ctl_dir, args.control, ctl_label, args.label_control) + c["mean_abs_luma"] = compare_video_luma_delta(ctl_dir, args.control) report["control_video"] = c report["control_audio"] = compare_audio( - os.path.join(args.a, args.audio_name), os.path.join(args.control, args.audio_name) + os.path.join(ctl_dir, args.audio_name), os.path.join(args.control, args.audio_name) ) + # THE NUMBER SECTION 10.5 SELECTS ON, computed rather than eyeballed. + # It is reported and never checked: it chooses between two readings of a + # PASSING result -- "the delta is entirely the kernel's" against "the + # delta is what the box does on its own" -- and a pass/fail cannot + # express a choice between two answers that are both answers. + num, den = c["mean_abs_luma"], v["mean_abs_luma"] + num_rgb, den_rgb = c["mean_abs"], v["mean_abs"] + undefined = None + if den <= 0.0: + # The EXPECTED division by zero, and it has a name: a bit-identical + # treatment. Section 10.2 predicts this experiment will not see one, + # so a reader meets this line only when something else went wrong. + undefined = (f"the {args.label_a} vs {args.label_b} delta is zero, " + f"so the ratio has no denominator") + report["control_ratio"] = { + "repeats": args.control_of, + "repeats_label": ctl_label, + "control_mean_abs_luma": num, + "treatment_mean_abs_luma": den, + "ratio_mean_abs_luma": (num / den) if den > 0.0 else None, + "control_mean_abs_rgb": num_rgb, + "treatment_mean_abs_rgb": den_rgb, + "ratio_mean_abs_rgb": (num_rgb / den_rgb) if den_rgb > 0.0 else None, + "undefined": undefined, + } # --- verdict -------------------------------------------------------------- checks: list[tuple[str, bool, str]] = [] @@ -435,12 +554,13 @@ def main() -> int: # equally broken. An arm that rendered nothing, rendered one colour, or # rendered the same frame 49 times fails HERE, where the failure is legible, # rather than passing silently as a perfect match. + # + # THREE checks per arm, not four. "frames written" used to be a fourth, and + # it could never be False: `frame_paths` refuses an empty arm at + # EXIT_UNREADABLE long before this loop runs. A row that cannot fail is a + # decoration in a table whose entire value is that every row can. for label in (args.label_a, args.label_b): c = report["content"][label] - if not c.get("frames"): - checks.append((f"content.{label}.frames", False, "no frames written")) - continue - checks.append((f"content.{label}.frames", True, f"{c['frames']} frames")) checks.append((f"content.{label}.not_uniform", c["near_uniform_frames"] == 0, f"near-uniform frames {c['near_uniform_frames']} == 0 " @@ -495,9 +615,6 @@ def main() -> int: # --- print ---------------------------------------------------------------- print("=== what each arm rendered, before anything is subtracted ===") for label, c in report["content"].items(): - if not c.get("frames"): - print(f"{label:12s} NO FRAMES") - continue print(f"{label:12s} frames={c['frames']} distinct={c['distinct_frame_hashes']} " f"mean={c['pixel_mean']:.3f} min_var={c['per_frame_var_min']:.1f} " f"near_uniform={c['near_uniform_frames']} " @@ -517,10 +634,24 @@ def main() -> int: print(f"|delta| histogram {dict(list(report['video']['delta_histogram'].items())[:12])}") if "control_video" in report: c = report["control_video"] - print(f"--- control ({args.label_a} vs {args.label_control}): the noise floor ---") + r = report["control_ratio"] + arm = "A" if r["repeats"] == "a" else "B" + # SAY WHICH ARM, in words, in the block itself. The convention used to be + # silent and the harness inverted it. + print(f"--- control ({args.label_control}): the noise floor ---") + print(f"the control {args.label_control} repeats arm {arm} " + f"({r['repeats_label']}), so it is compared against {r['repeats_label']}") print(f"bit-identical frames {c['identical_frame_files']}/{c['frames']}") print(f"max |delta| {c['max_abs']} mean {c['mean_abs']:.6f}") print(f"PSNR {c['psnr_db']:.3f} dB SSIM min {c['ssim_min']:.6f}") + if r["ratio_mean_abs_luma"] is None: + print(f"control/treatment undefined: {r['undefined']}") + else: + print(f"control/treatment {r['ratio_mean_abs_luma']:.6f} on luma " + f"({r['control_mean_abs_luma']:.6f} / " + f"{r['treatment_mean_abs_luma']:.6f}), " + f"{r['ratio_mean_abs_rgb']:.6f} on RGB") + print(" REPORTED, never checked: section 10.5 reads it") print("--- audio ---") for k in ("present", "comparable", "bit_identical", "max_abs_lsb", "max_abs_fs", "rms_diff_fs", "psnr_db", "pearson_r"): @@ -535,7 +666,7 @@ def main() -> int: with open(args.json, "w") as fh: json.dump(report, fh, indent=2, sort_keys=True) print(f"wrote {args.json}") - return 0 if ok else 1 + return EXIT_PASS if ok else EXIT_FAIL if __name__ == "__main__": diff --git a/tests/scripts/test_ltx25_render_compare.py b/tests/scripts/test_ltx25_render_compare.py index 8a63c135e..dd4545178 100644 --- a/tests/scripts/test_ltx25_render_compare.py +++ b/tests/scripts/test_ltx25_render_compare.py @@ -11,16 +11,28 @@ So both halves are pinned here, on fabricated frames, with no NAS and no GPU: - Identity two identical renders read as bit-identical, and every - threshold is then reported as vacuous rather than as passed. - Dither +/-1 on 3% of samples -- the shape section 10.2 predicts from - bf16 rounding -- passes all four video checks with headroom. - Structure ONE PIXEL of global horizontal shift fails ALL FOUR. That is - the calibration the section 10.4 threshold table quotes, and a - criterion that admitted it would not be a criterion. - Audio a scaled waveform fails, because the DiT drives both streams - and a video-only comparison leaves half the change unmeasured. - Refusal an unreadable input exits 2. A missing input is never a pass. + Discrimination two identical renders read as bit-identical and every + threshold is then vacuous rather than passed; a +/-1 + dither on 3% of samples -- the shape section 10.2 predicts + from bf16 rounding -- passes all four video checks with + headroom; ONE PIXEL of global horizontal shift fails ALL + FOUR, which is the calibration section 10.4 quotes and a + criterion that admitted it would not be a criterion. + IdenticallyBroken C0: two all-black renders differ by zero and score the + strongest pass every V check can give, so each arm is + judged on its own content first. + Refusal an input that cannot be read exits 2, and a threshold + failure still exits 1. A missing input is never a pass and + never a difference either. + Audio A1 and A2 disagree on a waveform that drifted in time, + which is the case A2 exists for and the only case that can + read it; absent audio is a failed check and never a pass. + Control which arm the control repeats is an argument, and the + control-to-treatment ratio section 10.5 selects on is + computed rather than eyeballed. + SsimIsTheOneItNames V3 names Wang et al. 2004 at 11x11 sigma=1.5 on Rec.601 + luma, and every part of that name is pinned by a property + or a value: scikit-image is absent here and stays absent. The fixtures are TEXTURED rather than flat. A flat image makes SSIM degenerate and makes a one-pixel shift invisible, so a test built on one would pass while @@ -30,7 +42,6 @@ from __future__ import annotations import json -import struct import subprocess import sys import tempfile @@ -304,12 +315,32 @@ def test_a_healthy_pair_passes_every_content_check(self) -> None: self.assertEqual(rc, EXIT_PASS, out) names = checks_of(rep) for label in ("p", "q"): - for check in ("frames", "not_uniform", "distinct_frames", "motion"): + for check in ("not_uniform", "distinct_frames", "motion"): self.assertTrue(names[f"content.{label}.{check}"], f"content.{label}.{check} failed on a healthy render: {out}") + # C0 registers THREE checks per arm, not four. "frames written" was a + # fourth that could never be False -- an arm with no frames is refused + # at exit 2 before any check is built -- and a check that cannot fail is + # a decoration in a table whose whole value is that each row can. + self.assertNotIn("content.p.frames", names) + self.assertNotIn("content.q.frames", names) class Refusal(unittest.TestCase): + """Exit 2 is a SEPARATE verdict from exit 1, and the separation is the point. + + Exit 1 means "the pixels differ", which is a statement about a render that + happened. Exit 0 and exit 1 are both readings of a completed experiment. An + input that cannot be read is not a reading at all, and a broken render that + reported the same status as a divergent one would be indistinguishable from + the finding this tool exists to make -- at exactly the moment nobody is + looking closely, because a 1 is the expected answer for a while. + + The docstring promised this and only a missing top-level DIRECTORY did it. + An empty arm, a frame-count mismatch, frames that do not correspond, a + truncated frame and a 16-bit frame all exited 1 and wrote no JSON. + """ + def test_missing_directory_exits_two(self) -> None: with tempfile.TemporaryDirectory() as t: a = Path(t) / "a" @@ -317,15 +348,106 @@ def test_missing_directory_exits_two(self) -> None: rc, out, _ = run("--a", str(a), "--b", str(Path(t) / "absent")) self.assertEqual(rc, EXIT_UNREADABLE, out) - def test_frame_count_mismatch_is_fatal(self) -> None: + def test_an_empty_arm_directory_exits_two(self) -> None: + """A render that wrote nothing. Every video statistic is undefined, so + there is no comparison to fail: this is a refusal, not a verdict.""" + with tempfile.TemporaryDirectory() as t: + a, b = Path(t) / "a", Path(t) / "b" + make_render(a, np.random.default_rng(1)) + b.mkdir() + rc, out, rep = run("--a", str(a), "--b", str(b)) + self.assertEqual(rc, EXIT_UNREADABLE, out) + self.assertIn("no frame", out.lower()) + self.assertIsNone(rep, "a refused run must not write a verdict") + + def test_frame_count_mismatch_exits_two(self) -> None: with tempfile.TemporaryDirectory() as t: a, b = Path(t) / "a", Path(t) / "b" make_render(a, np.random.default_rng(1)) make_render(b, np.random.default_rng(1)) next(iter(sorted(b.glob("frame_*.ppm")))).unlink() - rc, out, _ = run("--a", str(a), "--b", str(b)) - self.assertNotEqual(rc, EXIT_PASS, out) + rc, out, rep = run("--a", str(a), "--b", str(b)) + self.assertEqual(rc, EXIT_UNREADABLE, out) self.assertIn("frame count differs", out) + self.assertIsNone(rep, "a refused run must not write a verdict") + + def test_frames_that_do_not_correspond_by_name_exit_two(self) -> None: + """The pairing is `sorted()` against `sorted()`. Equal counts are not + equal frames: an arm that dropped frame 000000 and gained a frame 000049 + pairs every index against its neighbour and reports the render's own + motion as the arm-to-arm delta -- a large, plausible, entirely spurious + number.""" + with tempfile.TemporaryDirectory() as t: + a, b = Path(t) / "a", Path(t) / "b" + make_render(a, np.random.default_rng(1)) + make_render(b, np.random.default_rng(1)) + first = sorted(b.glob("frame_*.ppm"))[0] + first.rename(b / "frame_000099.ppm") + rc, out, rep = run("--a", str(a), "--b", str(b)) + self.assertEqual(rc, EXIT_UNREADABLE, out) + self.assertIn("frame_000099.ppm", out) + self.assertIsNone(rep, "a refused run must not write a verdict") + + def test_a_truncated_frame_exits_two(self) -> None: + with tempfile.TemporaryDirectory() as t: + a, b = Path(t) / "a", Path(t) / "b" + make_render(a, np.random.default_rng(1)) + make_render(b, np.random.default_rng(1)) + p = sorted(b.glob("frame_*.ppm"))[2] + p.write_bytes(p.read_bytes()[: len(p.read_bytes()) // 2]) + rc, out, rep = run("--a", str(a), "--b", str(b)) + self.assertEqual(rc, EXIT_UNREADABLE, out) + self.assertIn("truncated", out) + self.assertIsNone(rep, "a refused run must not write a verdict") + + def test_a_sixteen_bit_frame_exits_two(self) -> None: + with tempfile.TemporaryDirectory() as t: + a, b = Path(t) / "a", Path(t) / "b" + make_render(a, np.random.default_rng(1)) + make_render(b, np.random.default_rng(1)) + p = sorted(b.glob("frame_*.ppm"))[1] + p.write_bytes(b"P6\n%d %d\n65535\n" % (W, H) + b"\0" * (W * H * 6)) + rc, out, rep = run("--a", str(a), "--b", str(b)) + self.assertEqual(rc, EXIT_UNREADABLE, out) + self.assertIn("maxval", out) + self.assertIsNone(rep, "a refused run must not write a verdict") + + def test_a_geometry_mismatch_between_arms_exits_two(self) -> None: + with tempfile.TemporaryDirectory() as t: + a, b = Path(t) / "a", Path(t) / "b" + make_render(a, np.random.default_rng(1)) + make_render(b, np.random.default_rng(1)) + p = sorted(b.glob("frame_*.ppm"))[0] + write_ppm(p, np.zeros((H // 2, W, 3), dtype=np.uint8) + 40) + rc, out, rep = run("--a", str(a), "--b", str(b)) + self.assertEqual(rc, EXIT_UNREADABLE, out) + self.assertIsNone(rep, "a refused run must not write a verdict") + + def test_a_threshold_failure_is_still_exit_one(self) -> None: + """The other half of the separation. Widening exit 2 to cover a + divergent render would erase the finding instead of the ambiguity.""" + with tempfile.TemporaryDirectory() as t: + a, b = Path(t) / "a", Path(t) / "b" + frames = make_render(a, np.random.default_rng(1)) + b.mkdir() + for i, arr in enumerate(frames): + write_ppm(b / f"frame_{i:06d}.ppm", np.roll(arr, 1, axis=1)) + (b / "audio.wav").write_bytes((a / "audio.wav").read_bytes()) + rc, out, rep = run("--a", str(a), "--b", str(b)) + self.assertEqual(rc, EXIT_FAIL, out) + self.assertEqual(rep["verdict"], "FAIL") + + def test_ppm_reader_reads_width_before_height(self) -> None: + """A square fixture cannot see this, and every fixture in this file was + square-ish. The header is `P6 W H maxval` and the array is (H, W, 3).""" + with tempfile.TemporaryDirectory() as t: + p = Path(t) / "x.ppm" + px = bytes(range(7 * 3 * 3)) + p.write_bytes(b"P6\n7 3\n255\n" + px) + arr = _load_tool().read_ppm(str(p)) + self.assertEqual(arr.shape, (3, 7, 3)) + self.assertEqual(tuple(int(v) for v in arr[0, 0]), (0, 1, 2)) + self.assertEqual(tuple(int(v) for v in arr[1, 0]), (21, 22, 23)) def test_ppm_reader_refuses_a_maxval_it_cannot_scale(self) -> None: """Every threshold below is in 8-bit levels. A 16-bit PPM would make @@ -346,6 +468,166 @@ def test_ppm_reader_refuses_a_truncated_file(self) -> None: mod.read_ppm(str(p)) +class Audio(unittest.TestCase): + """A1 and A2 are two checks and they were pinned by one fixture. + + The only divergent-audio fixture was a 0.5x scale, and Pearson r of a + waveform against a scaled copy of itself is EXACTLY 1.0. So A2 could not + fail on anything this suite owned: `DEFAULT_MIN_AUDIO_CORR` moved from 0.999 + to -1.0 stayed green, and deleting the check entirely stayed green. Section + 10.4 says A2 exists for "a waveform that has drifted in time", so the + fixture that reads it is a waveform that has drifted in time. + """ + + def _video_only_pair(self, root: Path) -> tuple[Path, Path]: + a, b = root / "a", root / "b" + make_render(a, np.random.default_rng(17)) + b.mkdir() + for p in sorted(a.glob("frame_*.ppm")): + (b / p.name).write_bytes(p.read_bytes()) + return a, b + + def test_a_time_shifted_waveform_fails_correlation_while_psnr_passes(self) -> None: + """The two checks DISAGREE on this fixture, and that is the measured + fact rather than a hoped-for one: one sample of shift on a 20-sample + radian scale is 0.05 rad, which is 41.28 dB (A1 passes, floor 40) and + r = 0.99875 (A2 fails, floor 0.999). A tool holding only A1 would call + this stream identical.""" + with tempfile.TemporaryDirectory() as t: + root = Path(t) + a, b = self._video_only_pair(root) + n = np.arange(4800) + write_wav(a / "audio.wav", + np.stack([8000 * np.sin(n / 20.0), 8000 * np.cos(n / 20.0)], axis=1)) + write_wav(b / "audio.wav", + np.stack([8000 * np.sin((n + 1) / 20.0), + 8000 * np.cos((n + 1) / 20.0)], axis=1)) + rc, out, rep = run("--a", str(a), "--b", str(b)) + self.assertEqual(rc, EXIT_FAIL, out) + self.assertTrue(rep["video"]["bit_identical"], out) + names = checks_of(rep) + self.assertTrue(names["audio.psnr_db"], + f"A1 was expected to PASS on a one-sample drift: {out}") + self.assertFalse(names["audio.pearson_r"], + f"A2 was expected to FAIL on a one-sample drift: {out}") + self.assertGreater(rep["audio"]["psnr_db"], 40.0) + self.assertLess(rep["audio"]["psnr_db"], 42.0) + self.assertLess(rep["audio"]["pearson_r"], 0.999) + self.assertGreater(rep["audio"]["pearson_r"], 0.998) + + def test_absent_audio_is_a_failed_check_and_never_a_pass(self) -> None: + """Two arms whose video agrees bit-for-bit and that wrote no audio at + all. `present` must be False and the verdict must be FAIL: the DiT + drives both streams, so a silent half is an unmeasured half.""" + with tempfile.TemporaryDirectory() as t: + root = Path(t) + a, b = self._video_only_pair(root) + (a / "audio.wav").unlink() + rc, out, rep = run("--a", str(a), "--b", str(b)) + self.assertEqual(rc, EXIT_FAIL, out) + self.assertTrue(rep["video"]["bit_identical"], out) + self.assertFalse(rep["audio"]["present"], out) + self.assertNotIn("comparable", rep["audio"]) + self.assertNotIn("bit_identical", rep["audio"]) + self.assertFalse(checks_of(rep)["audio.present"], out) + self.assertEqual(rep["verdict"], "FAIL") + + +class Control(unittest.TestCase): + """WHICH ARM the control repeats is now stated, not conventional. + + The tool computed the control as `compare_video(a, control, ...)` and said + so only in a docstring. The harness passed `--a naive --b flash --control + flash-ctl`, so the "run-to-run noise floor" was a SECOND naive-vs-flash + comparison. It necessarily read about the same size as the treatment, and + section 10.5's second branch would then have published "indistinguishable + from run-to-run nondeterminism" whatever the kernel did. A silent convention + a caller can invert is not a convention. + """ + + def setUp(self) -> None: + self.tmp = tempfile.TemporaryDirectory() + root = Path(self.tmp.name) + self.a = root / "a" + frames = make_render(self.a, np.random.default_rng(21)) + # b differs from a everywhere by ONE level, which every threshold + # admits, so the pair is a PASS and the control is read on a passing + # result -- the only case section 10.5 has two readings for. `ctl` is a + # byte-for-byte repeat of b. + self.b, self.ctl = root / "b", root / "ctl" + for d in (self.b, self.ctl): + d.mkdir() + for i, arr in enumerate(frames): + write_ppm(d / f"frame_{i:06d}.ppm", np.clip(arr.astype(np.int16) + 1, 0, 255)) + (d / "audio.wav").write_bytes((self.a / "audio.wav").read_bytes()) + + def tearDown(self) -> None: + self.tmp.cleanup() + + def test_control_of_b_compares_the_control_against_arm_b(self) -> None: + rc, out, rep = run("--a", str(self.a), "--b", str(self.b), + "--control", str(self.ctl), "--control-of", "b", + "--label-a", "naive", "--label-b", "flash", + "--label-control", "flash-ctl") + self.assertEqual(rc, EXIT_PASS, out) + self.assertEqual(rep["control_of"], "b") + self.assertEqual(rep["control_video"]["label_a"], "flash") + self.assertTrue(rep["control_video"]["bit_identical"], + f"the control repeats FLASH and must be compared to it: {out}") + self.assertIn("flash-ctl", out) + # The block says, in words, which arm it was read against. + self.assertRegex(out, r"control .*flash-ctl.* repeats arm B \(flash\)") + + def test_control_of_defaults_to_a_and_says_so(self) -> None: + rc, out, rep = run("--a", str(self.a), "--b", str(self.b), + "--control", str(self.ctl), + "--label-a", "naive", "--label-b", "flash", + "--label-control", "flash-ctl") + self.assertEqual(rep["control_of"], "a") + self.assertEqual(rep["control_video"]["label_a"], "naive") + self.assertFalse(rep["control_video"]["bit_identical"], out) + self.assertRegex(out, r"control .*flash-ctl.* repeats arm A \(naive\)") + + def test_the_control_to_treatment_ratio_is_computed_not_eyeballed(self) -> None: + """Section 10.5 selects between two OPPOSITE published verdicts on + "control is 0" against "control comparable to the delta", and nothing + computed that comparison. It is reported, never checked: it chooses + between two readings, not between pass and fail.""" + rc, out, rep = run("--a", str(self.a), "--b", str(self.b), + "--control", str(self.ctl), "--control-of", "b") + self.assertEqual(rc, EXIT_PASS, out) + r = rep["control_ratio"] + # The control repeats B and IS B, so its delta against B is zero and the + # ratio is exactly 0: the "noise floor is zero" branch, arithmetically. + self.assertEqual(r["control_mean_abs_luma"], 0.0) + self.assertAlmostEqual(r["treatment_mean_abs_luma"], + rep["video"]["mean_abs_luma"], places=12) + self.assertEqual(r["ratio_mean_abs_luma"], 0.0) + self.assertEqual(r["ratio_mean_abs_rgb"], 0.0) + self.assertIsNone(r["undefined"]) + self.assertIn("control/treatment", out) + # And it is REPORTED, not gated: no check carries the ratio. + self.assertNotIn("control", " ".join(c["name"] for c in rep["checks"])) + + def test_the_ratio_is_undefined_rather_than_a_division_by_zero(self) -> None: + """The expected case for a bit-identical treatment, which is exactly + what section 10.2 predicts this experiment will NOT see -- so it is the + case a reader would meet only when something else went wrong.""" + root = Path(self.tmp.name) + same = root / "same" + same.mkdir() + for p in sorted(self.a.glob("*")): + (same / p.name).write_bytes(p.read_bytes()) + rc, out, rep = run("--a", str(self.a), "--b", str(same), + "--control", str(self.ctl)) + self.assertEqual(rc, EXIT_PASS, out) + r = rep["control_ratio"] + self.assertIsNone(r["ratio_mean_abs_luma"]) + self.assertIsNotNone(r["undefined"]) + self.assertGreater(r["control_mean_abs_luma"], 0.0) + self.assertIn("undefined", out) + + def _load_tool(): import importlib.util @@ -355,6 +637,16 @@ def _load_tool(): return mod +def _texture(seed: int = 20260822) -> np.ndarray: + """One textured plane, as float64 luma levels. The SSIM fixtures below are + all derived from this one array so that every pinned number is reproducible + from the seed printed here and nothing else.""" + rng = np.random.default_rng(seed) + yy, xx = np.mgrid[0:H, 0:W] + base = 127 + 60 * np.sin(xx / 4.0) + 40 * np.cos(yy / 3.0) + rng.integers(-20, 21, (H, W)) + return np.clip(base, 0, 255).astype(np.uint8).astype(np.float64) + + class Metrics(unittest.TestCase): def test_ssim_of_a_frame_with_itself_is_one(self) -> None: mod = _load_tool() @@ -372,5 +664,122 @@ def test_psnr_matches_its_definition(self) -> None: self.assertAlmostEqual(mod.psnr_from_mse(1.0), 48.13080361, places=6) +class SsimIsTheOneItNames(unittest.TestCase): + """V3 names a specific SSIM, and the name is load-bearing. + + Section 10.4 says "Wang et al. 2004, 11x11 Gaussian sigma=1.5 on luma", and + says it because scikit-image's 7x7 uniform default disagrees in the third + decimal while a `0.99` gate reads to the sixth. That sentence was a claim: a + test that only asked "is SSIM(a, a) == 1" stayed green with sigma at 3.0 or + 0.5, the window at 7 or 3, either stabilising constant at 0.5 or at 0, the + second separable pass blurring the wrong axis or deleted outright, and the + luma weights replaced by red-only or a flat third. Every one of those is a + different metric wearing the same name. + + scikit-image is NOT available here and must stay absent, so the criterion is + pinned by PROPERTIES and by values computed at this revision: + + the taps themselves catches sigma and the window size directly + the impulse response catches either separable pass, because a + separable blur of a unit impulse IS outer(k, k) + its two profiles catches a second pass on the WRONG axis, which + leaves the response summing to 1 and asymmetric + three fixture SSIMs catches C1 and C2, which change no property above + and only move the number + """ + + def test_gaussian_taps_are_the_eleven_of_sigma_1_5(self) -> None: + mod = _load_tool() + k = mod._gauss1d() + self.assertEqual(len(k), 11, "the window is 11 wide, not scikit-image's 7") + expected = [ + 0.00102838008447911, 0.007598758135239185, 0.03600077212843083, + 0.10936068950970002, 0.2130055377112537, 0.26601172486179436, + 0.2130055377112537, 0.10936068950970002, 0.03600077212843083, + 0.007598758135239185, 0.00102838008447911, + ] + np.testing.assert_allclose(k, expected, rtol=0.0, atol=1e-15) + self.assertAlmostEqual(float(k.sum()), 1.0, places=15) + + def test_the_taps_fit_the_sigma_they_claim(self) -> None: + """The second moment of the discrete taps. A truncated 11-tap Gaussian + loses a little of the tail, so it reads 1.4978 rather than 1.5000 -- and + that residual is itself the evidence the window is 11 rather than + infinite.""" + mod = _load_tool() + k = mod._gauss1d() + r = np.arange(len(k), dtype=np.float64) - (len(k) - 1) / 2.0 + fitted = float(np.sqrt((r ** 2 * k).sum())) + self.assertAlmostEqual(fitted, 1.4978283460942616, places=12) + self.assertLess(abs(fitted - 1.5), 3e-3) + + def test_blur_of_a_unit_impulse_is_the_separable_outer_product(self) -> None: + mod = _load_tool() + k = mod._gauss1d() + n = 31 + imp = np.zeros((n, n)) + imp[n // 2, n // 2] = 1.0 + out = mod._blur(imp, k) + lo, hi = n // 2 - 5, n // 2 + 6 + np.testing.assert_allclose(out[lo:hi, lo:hi], np.outer(k, k), rtol=0.0, atol=1e-15) + self.assertAlmostEqual(float(out.sum()), 1.0, places=12) + # Nothing outside the 11x11 support: a wider window would spill here. + support = np.zeros_like(out, dtype=bool) + support[lo:hi, lo:hi] = True + self.assertEqual(float(np.abs(out[~support]).max()), 0.0) + + def test_the_impulse_response_is_identical_along_both_axes(self) -> None: + """The mutation this exists for: a second separable pass that blurs axis + 0 again instead of axis 1, or is deleted. Both leave the response + summing to 1.0 and both leave SSIM(a, a) == 1, and both make the metric + anisotropic -- which is what this reads.""" + mod = _load_tool() + k = mod._gauss1d() + n = 31 + imp = np.zeros((n, n)) + imp[n // 2, n // 2] = 1.0 + out = mod._blur(imp, k) + np.testing.assert_allclose(out[n // 2, :], out[:, n // 2], rtol=0.0, atol=1e-15) + np.testing.assert_allclose(out[n // 2, :], out[n // 2, ::-1], rtol=0.0, atol=1e-15) + + def test_ssim_of_a_one_pixel_shift_is_pinned(self) -> None: + """The value section 10.4's fixture row quotes. It moves under every one + of the ten mutations in this class's docstring.""" + mod = _load_tool() + a = _texture() + self.assertAlmostEqual(mod.ssim(a, np.roll(a, 1, axis=1)), + 0.7029544035134992, places=8) + + def test_ssim_of_a_five_level_luminance_shift_is_pinned(self) -> None: + """A pure luminance offset: the variance terms are identical on both + sides, so the contrast and structure factors are exactly 1 and this + number is the LUMINANCE factor alone. It is what pins C1.""" + mod = _load_tool() + a = _texture() + self.assertAlmostEqual(mod.ssim(a, np.clip(a + 5, 0, 255)), + 0.9987425512854893, places=8) + + def test_ssim_of_a_contrast_scaled_pair_is_pinned(self) -> None: + """The mirror of the test above: the means are identical and the + variances are not, so this number moves with C2 and not with C1.""" + mod = _load_tool() + a = _texture() + c = np.clip(127 + 0.9 * (a - 127), 0, 255) + self.assertAlmostEqual(mod.ssim(a, c), 0.9931050937084597, places=8) + + def test_luma_is_the_rec_601_triple(self) -> None: + mod = _load_tool() + px = lambda r, g, b: float(mod.luma(np.array([[[r, g, b]]], dtype=np.uint8))[0, 0]) + # Each primary on its own reads back its own weight, which is the triple + # asserted rather than described: red-only, a flat third, and an R/B swap + # each fail a different line here. + self.assertAlmostEqual(px(255, 0, 0) / 255.0, 0.299, places=12) + self.assertAlmostEqual(px(0, 255, 0) / 255.0, 0.587, places=12) + self.assertAlmostEqual(px(0, 0, 255) / 255.0, 0.114, places=12) + self.assertAlmostEqual(px(10, 20, 30), 0.299 * 10 + 0.587 * 20 + 0.114 * 30, + places=12) + self.assertAlmostEqual(px(10, 20, 30), 18.15, places=12) + + if __name__ == "__main__": unittest.main() From f407019e363b7bf1c32e3d838de732f1272642fd Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 22 Aug 2026 16:17:14 +0000 Subject: [PATCH 05/19] fix(#1612): the harness printed 5 GiB available and built anyway, and its control was a second treatment MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four hours of `dgx:gpu0` per run, three lost workers, and until now not one line of this file had ever executed anywhere a test could watch it. Six of its guards were guards nobody had seen fire, and five of them did not. IT PRINTED THE NUMBER THAT SAID NOT TO START. On 2026-08-22 (rc job 5fb9399f-4f4e-417c-adbd-4d741a2e18e4) phase [0] printed `available 5` at +0s, proceeded to `cmake`, started `ninja -j 4` on CUDA at +38s, and lost its worker at ~+728s with no binary cached and nothing measured. The box was at 114 of 119 GiB before this job allocated a byte, so the harness did not cause it — but it walked into it while displaying the exact reading that forbade it. Phase [0b] now GATES on `MemAvailable` with a 60 GiB start floor, waits for the previous tenant's memory to be reclaimed, logs every poll so a reader can tell a recovering box from a flat one, and refuses with exit 39 naming what it saw, the floor and how long it waited. The floor comes from the recorded 20260820 render at this geometry: it peaked at 79.503 GiB with a low-water of 40.13 GiB. One reader serves both this gate and the render watchdog, from `/proc/meminfo` and never `free`'s column, so the two cannot disagree about what they measured. The value is logged again immediately before `ninja`, which is the fact the lost run did not have. A FOUR-HOUR RUN HAD NO RESUME POINT. Every lost lease threw away every arm that had already rendered, including a ~2 h naive arm. `RUN_ID` is now overridable and `$OUT` is keyed on it, so a resumed lease lands in the same directory and skips each arm that is COMPLETE there — exactly the expected frame count and a non-empty wav. An incomplete arm is deleted and re-rendered rather than resumed mid-flight, because the engine clears stale `frame_*.ppm` in its own output directory and nothing else. A skip is loud: the reused arm's per-forward samples came from an earlier lease, so `timing_source` goes into that arm's `ARM` file and into `PROVENANCE`, and phase [H] states whether the speed pair is a same-lease pair at all. `PROVENANCE` is appended rather than truncated, so the earlier lease's binary sha survives the resume. The routing proof still runs for a skipped arm, from the log it already has: a reused arm whose routing was never proved is worse than no arm. THE CONTROL WAS A SECOND TREATMENT. Phase [I] called the tool with `--a naive --b flash --control flash-ctl`, and the tool computes the control against arm A. `flash-ctl` is a repeat of FLASH, so the "run-to-run noise floor" was a second naive-vs-flash comparison: it necessarily read about the same size as the delta it existed to calibrate, and §10.5's second branch would have published "indistinguishable from run-to-run nondeterminism" whatever the kernel did. Flash is arm A now, and `--control-of a` is passed explicitly so the call states its own intent rather than inheriting a default. THE VERDICT WAS A LINE OF TEXT. [I]'s own comment said "its exit status is the gate ... a 2 is never a pass", and the script echoed `PIXEL_COMPARE_RC`, carried on, and ended on `DONE` with status 0. A failing pixel verdict, an unreadable input and a zero-frame render all exited 0. [J] and [K] still run, so the artefacts and the cross-check are still produced; the run now ends on the verdict in a phase [L] that names it in words and exits with it. ROUTING_BAD WAS A WORD IN A LOG. It was echoed inside a `case ... | tee` pipeline, where an `exit` would have left the subshell rather than the run. If the knob is not read, both arms are flash, the two renders come out bit-identical, and this file publishes PASS with all four thresholds vacuous: the strongest positive verdict it can produce, from an experiment that had one arm. The verdict is computed into a variable, tested outside any pipeline, and a bad one stops the run at exit 46 naming the arm, the knob and the counts. "CORRECTNESS FIRST" WAS A HEADING. Phase [F] ran the CUDA unit gate, printed its status and continued, and printed `MISSING` and continued when there was no binary to run. Both now refuse, at 44 and 45. A LEASE KILL ORPHANED THE HEARTBEAT. The trap covered EXIT only, and `rc` reclaiming a device sends SIGTERM, for which a bash EXIT trap does not run. HUP, INT and TERM now clean up and exit 128+signo. AND BOTH INSTRUMENTS NOW RUN ON A LANE. `tests/scripts/test_ltx25_render_compare.py` ran on none: absent from preflight's `SUITES`, from the enumerated python block in CI and from `tests/CMakeLists.txt`, while §8 of the spec registered it as a gate. The new `tests/scripts/test_ltx25_pixel_ab_harness.py` extracts the shell helpers this commit adds, between marker comments, and runs them against a fabricated `/proc/meminfo` and fabricated arm directories: both branches of the memory gate, the wait-then-recover path, an unreadable meminfo, and six completeness cases. Its assertions about the four call sites only a lease can execute are text tripwires and say so in the file. Preflight SKIPs the comparison suite when numpy is absent, which is the third state and never an `ok`; CI installs `python3-numpy`, so the lane that must not be silent cannot be. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-opus-5 [Claude Code] --- .github/workflows/ci.yml | 21 ++ scripts/agent-preflight.sh | 18 ++ scripts/ltx25-dit-attn-flash-pixel-ab.sh | 245 +++++++++++++++-- tests/scripts/test_ltx25_pixel_ab_harness.py | 264 +++++++++++++++++++ 4 files changed, 530 insertions(+), 18 deletions(-) create mode 100644 tests/scripts/test_ltx25_pixel_ab_harness.py diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 830bf7fb1..194ddb37e 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -380,6 +380,27 @@ jobs: python3 tests/scripts/test_agent_gates.py python3 tests/scripts/test_agent_pr_body.py python3 tests/scripts/test_ab_arms_differ.py + - name: The LTX-2.5 pixel comparison and its harness run on a lane (#1612) + # Registered in TWO places, like the baseline and preflight suites above: + # here and in `scripts/agent-preflight.sh`. Both suites ran on NO lane + # when they landed -- absent from that array, from this file and from + # `tests/CMakeLists.txt` -- while section 8 of + # `.agents/specs/ltx25-dit-attn-flash.md` registered the first as a gate. + # + # numpy is installed HERE and only here. `ltx25-render-compare.py` reads + # PPM and WAV by hand so that a leased GPU worker needs no image or audio + # library, and numpy is the one dependency that remains. Preflight SKIPS + # that suite when numpy is absent, which is honest on a developer box and + # would be silence on a lane, so this lane installs it instead. + # + # The harness suite needs neither numpy nor a GPU: it extracts the shell + # helpers from `ltx25-dit-attn-flash-pixel-ab.sh` between its marker + # comments and runs them against a fabricated /proc/meminfo. + run: | + sudo apt-get update -qq + sudo DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends python3-numpy + python3 tests/scripts/test_ltx25_render_compare.py + python3 tests/scripts/test_ltx25_pixel_ab_harness.py - name: No tracked file carries a merge conflict marker (#1417) # Registered HERE and not only in preflight. The four record gates that # missed this all return 0 on a `docs/STATUS.md` carrying a spliced diff --git a/scripts/agent-preflight.sh b/scripts/agent-preflight.sh index d2b385a75..02717ee80 100755 --- a/scripts/agent-preflight.sh +++ b/scripts/agent-preflight.sh @@ -169,6 +169,7 @@ SUITES=( test_check_oracle_denominator_flags test_check_conflict_markers test_ab_arms_differ + test_ltx25_pixel_ab_harness ) failed=() @@ -371,6 +372,23 @@ echo "Mutation suites:" for suite in "${SUITES[@]}"; do run "$suite" python3 "tests/scripts/$suite.py" done +# THE ONE SUITE HERE WITH A THIRD-PARTY DEPENDENCY (#1612). It exercises +# `scripts/ltx25-render-compare.py`, whose only import beyond the standard +# library is numpy -- the tool reads PPM and WAV by hand precisely so that a +# leased worker needs nothing else. It ran on NO lane at all until now: absent +# from this array, from the enumerated python block in `.github/workflows/ci.yml` +# and from `tests/CMakeLists.txt`, while section 8 of its spec registered it as a +# gate. A gate no lane runs is "nothing lands dead" applied to the instrument. +# +# A missing numpy is a SKIP and never an `ok`: nothing was verified. CI installs +# it, so the lane that must not be silent is not the one that can be. +if python3 -c 'import numpy' >/dev/null 2>&1; then + run "test_ltx25_render_compare" python3 tests/scripts/test_ltx25_render_compare.py +else + skip "test_ltx25_render_compare" \ + "numpy is not importable here, and the tool this suite exercises needs it." \ + "CI installs python3-numpy and runs the same suite." +fi run "trailer suites" python3 -m unittest \ tests.scripts.test_check_commit_trailers run "commit style suites" python3 -m unittest \ diff --git a/scripts/ltx25-dit-attn-flash-pixel-ab.sh b/scripts/ltx25-dit-attn-flash-pixel-ab.sh index 913825d3d..4c7885beb 100755 --- a/scripts/ltx25-dit-attn-flash-pixel-ab.sh +++ b/scripts/ltx25-dit-attn-flash-pixel-ab.sh @@ -46,15 +46,28 @@ FULL=/workspace/ltx25-fullmodel # checkpoints and the text-encoder config SRC=/root/src-pixab BLD=/root/build-pixab CK=/root/ckpt -OUT=$W/pixel-ab/$(date -u +%Y%m%dT%H%M%SZ) +# RUN_ID IS OVERRIDABLE, and that is the whole resume mechanism. `dgx:gpu0` has +# lost its worker three times, and this job renders three arms over about four +# hours, so a lease lost after the ~2 h naive arm used to throw that arm away +# and start a new timestamped directory. A resumed lease passes the same RUN_ID, +# lands in the same $OUT, and every arm that is already COMPLETE there is +# skipped. It composes with the binary cache in phase [D], which is keyed on the +# source sha, so a resumed run reaches its first missing render in minutes. +RUN_ID=${RUN_ID:-$(date -u +%Y%m%dT%H%M%SZ)} +OUT=$W/pixel-ab/$RUN_ID mkdir -p "$OUT" "$CK" export DEBIAN_FRONTEND=noninteractive -say "OUT=$OUT" +say "RUN_ID=$RUN_ID OUT=$OUT" +# APPENDED, never truncated: a resumed lease writes its own block, and the +# earlier lease's binary sha and start time are part of what the resumed run's +# evidence rests on. { + echo "--- lease $(date -Is) ---" + echo "run_id=$RUN_ID" echo "rc_job=${RC_JOB_ID:-unknown}" echo "harness_sha256=$(sha256sum "$0" 2>/dev/null | awk '{print $1}')" echo "started=$(date -Is)" -} > "$OUT/PROVENANCE" +} >> "$OUT/PROVENANCE" # A LIVENESS LINE, and nothing more. The build redirects to a file and a single # 42 GB checkpoint copy takes minutes, so this job can produce no stdout at all @@ -65,13 +78,101 @@ say "OUT=$OUT" # the forward count separately, and that one CAN stop advancing. ( while :; do sleep 120; echo "[pixab-alive +$(( $(date +%s) - T0 ))s]"; done ) & HEARTBEAT=$! -trap 'kill $HEARTBEAT 2>/dev/null' EXIT +# EXIT ALONE DOES NOT COVER A LEASE KILL. `rc` reclaiming a device sends SIGTERM, +# and a bash trap on EXIT does not run for a signal that has no handler, so the +# heartbeat subshell survived its parent and kept printing into a job nobody was +# reading. Each signal cleans up and then exits with 128+signo, which is the +# status the shell would have reported had the trap not existed. +cleanup() { kill "$HEARTBEAT" 2>/dev/null; } +trap cleanup EXIT +trap 'cleanup; exit 129' HUP +trap 'cleanup; exit 130' INT +trap 'cleanup; exit 143' TERM + +# BEGIN pixab-helpers -- extracted verbatim by +# tests/scripts/test_ltx25_pixel_ab_harness.py, which runs both branches of the +# memory gate against a fabricated meminfo. Everything between these markers +# must depend on nothing but `say`, $MEMINFO and coreutils. +# +# ONE READER for MemAvailable, because two gates that disagree about what they +# measured are worse than one gate. The start gate below and the render +# watchdog in render() both call this, and neither reads `free`'s "available" +# column: that is a second source for the same quantity, and phase [0] printing +# one number while the watchdog acts on another is how this job walked into a +# box with 5 GiB free while printing that it had 5 GiB free. +MEMINFO=${MEMINFO:-/proc/meminfo} +mem_avail_gib() { + awk '/^MemAvailable:/{printf "%.1f", $2/1048576; f=1; exit} END{if(!f) print ""}' \ + "$MEMINFO" 2>/dev/null +} + +# WAIT, then refuse. A lease spent waiting and refusing costs a lease; a lease +# spent building into an out-of-memory kill costs the lease AND leaves a record +# that says nothing about why. Each poll logs its value so a reader can tell a +# recovering box from a flat one. +wait_for_memory() { # $1 floor GiB, $2 budget s, $3 poll s + local floor=$1 budget=$2 poll=$3 avail waited=0 + while :; do + avail=$(mem_avail_gib) + if [ -z "$avail" ]; then + echo "FATAL: cannot read MemAvailable from $MEMINFO" + return 39 + fi + if awk -v a="$avail" -v f="$floor" 'BEGIN{exit !(a>=f)}'; then + say " MemAvailable ${avail} GiB >= floor ${floor} GiB after ${waited}s: proceeding" + return 0 + fi + if [ "$waited" -ge "$budget" ]; then + echo "FATAL: MemAvailable ${avail} GiB is below the ${floor} GiB start floor" \ + "after ${waited}s of waiting (budget ${budget}s). The box was already" \ + "occupied when this lease started; nothing was built and nothing was rendered." + return 39 + fi + say " MemAvailable ${avail} GiB < ${floor} GiB, waited ${waited}s of ${budget}s" + sleep "$poll" + waited=$((waited + poll)) + done +} + +# A COMPLETE ARM, and nothing weaker. Exactly the expected frame count and a +# non-empty wav. A partial arm is re-rendered from scratch rather than resumed +# mid-flight: the engine deletes stale frame_*.ppm in its own output directory +# but nothing else there, so a half-arm's leftovers would outlive it. +arm_is_complete() { # $1 dir, $2 wanted frame count + local d=$1 want=$2 n + [ -d "$d" ] || return 1 + n=$(ls "$d"/frame_*.ppm 2>/dev/null | wc -l) + [ "$n" = "$want" ] || return 1 + [ -s "$d/audio.wav" ] || return 1 + return 0 +} +# END pixab-helpers say "=== [0] the box ===" uname -m; nproc; free -g | head -2 nvidia-smi --query-gpu=name,driver_version,memory.total --format=csv 2>&1 | head -3 df -h / /root /workspace 2>&1 | head -6 +say "=== [0b] the MemAvailable PRECONDITION, gated rather than printed ===" +# THE NUMBER COMES FROM THE BASELINE, not from taste. The recorded 20260820 +# render at this geometry started from baseline_used 4.643 GiB, peaked at +# 79.503 GiB and had a MemAvailable low-water of 40.13 GiB, so this job needs +# most of the box and cannot start on the tail of it. +# +# This gate exists because phase [0] above PRINTED +# Mem: total 119 used 114 free 1 shared 0 buff/cache 4 available 5 +# at +0s on 2026-08-22 (rc job 5fb9399f-4f4e-417c-adbd-4d741a2e18e4) and the run +# proceeded anyway: cmake, then ninja -j 4 on CUDA, then a lost worker at ~+728s +# with no binary cached and nothing measured. The box was already at 114 of 119 +# GiB before this job allocated a byte, so the harness cannot fix the cause -- +# but it must not walk into it while printing the exact number that says not to. +MEM_START_FLOOR_GIB=${MEM_START_FLOOR_GIB:-60.0} +MEM_START_WAIT_S=${MEM_START_WAIT_S:-1200} +MEM_START_POLL_S=${MEM_START_POLL_S:-30} +say "start floor ${MEM_START_FLOOR_GIB} GiB, wait budget ${MEM_START_WAIT_S}s, poll ${MEM_START_POLL_S}s" +wait_for_memory "$MEM_START_FLOOR_GIB" "$MEM_START_WAIT_S" "$MEM_START_POLL_S" || exit 39 +echo "mem_available_at_start_gib=$(mem_avail_gib)" >> "$OUT/PROVENANCE" + say "=== [1] tools ===" # ffmpeg matters beyond the mp4: ltx2-gen exits 127 from its absence AFTER every # frame and the wav are on disk (#1149), so its absence reads as a failed render @@ -192,6 +293,13 @@ if [ "$SKIP_BUILD" = 0 ]; then fi cp -f /root/cfg.log "$OUT/configure.log" # Unconstrained parallelism has OOM-rebooted this box. + # + # THE MEMORY AT THE MOMENT THE BUILD STARTS, logged here and not only at [0b]. + # The 2026-08-22 worker was lost during exactly this command and the log could + # not say what memory it began with, because the only reading was taken 38 s + # earlier under a different tenant's allocation. + say " ninja -j 4 starting with MemAvailable $(mem_avail_gib) GiB" + echo "mem_available_at_build_gib=$(mem_avail_gib)" >> "$OUT/PROVENANCE" ninja -C "$BLD" -j 4 ltx2-gen test_ltx2_device > /root/build.log 2>&1 B=$?; echo "BUILD_RC=$B" echo " compile_errors=$(grep -ciE ' error: ' /root/build.log)" @@ -263,12 +371,26 @@ say "=== [F] CORRECTNESS FIRST: the CUDA unit gate, before any render ===" # AGENTS.md: establish the correctness gate before accepting a performance # result. `assertions: 0` is a skip wearing a pass and a thrown case shows up # only on the `Status:` line, so both are printed rather than a grep for ok. +# +# AND IT REFUSES. It used to print its status and carry on, which made +# "correctness first" a heading rather than a gate: a red unit case, or a binary +# with no unit case at all, reached three renders and a published pixel verdict +# with nothing between them. if [ -x "$BIN/test_ltx2_device" ]; then "$BIN/test_ltx2_device" > "$OUT/test_ltx2_device.log" 2>&1 - echo " test_ltx2_device_RC=$?" + UNIT_RC=$? + echo " test_ltx2_device_RC=$UNIT_RC" grep -E 'assertions:|test cases:|Status:|SKIP' "$OUT/test_ltx2_device.log" | tail -8 + [ "$UNIT_RC" = 0 ] || { + echo "FATAL: the CUDA unit gate FAILED (rc=$UNIT_RC); see $OUT/test_ltx2_device.log." + echo " Correctness comes before a render, so no arm is taken on this binary." + exit 44 + } else - echo " MISSING test_ltx2_device" + echo "FATAL: $BIN/test_ltx2_device is absent, so the correctness gate cannot run." + echo " A cached binary staged without it is the usual cause; rebuild with" + echo " an empty $CACHE to restore it." + exit 45 fi say "=== [G] the three renders ===" @@ -291,10 +413,38 @@ say "MemAvailable floor ${MEM_FLOOR_GIB} GiB (the recorded baseline's low-water echo "prompt_sha256=$(printf '%s' "$PROMPT" | sha256sum | awk '{print $1}')" } >> "$OUT/PROVENANCE" +REUSED_ARMS="" render() { # $1 = label, $2 = knob value, $3 = hard timeout seconds local label=$1 knob=$2 tmo=$3 - local d="$OUT/$label"; mkdir -p "$d" + local d="$OUT/$label" local log="$d/render.log" + # RESUME, and say so. A skipped arm is loud, because a speed pair assembled + # from two leases is NOT a same-binary same-lease pair, and the report must + # not be able to claim one silently. The routing proof below still runs for a + # skipped arm, from the log that arm already has: a reused arm whose routing + # was never proved is worse than no arm at all. + local timing_from="this-lease" + if arm_is_complete "$d" "$FRAMES"; then + say "--- render $label SKIPPED: $d already holds $FRAMES frames and an audio.wav" + say " (resumed run RUN_ID=$RUN_ID; its timings were taken in an EARLIER lease)" + timing_from="an-earlier-lease" + REUSED_ARMS="$REUSED_ARMS $label" + else + if [ -d "$d" ]; then + say " $label is INCOMPLETE ($(ls "$d"/frame_*.ppm 2>/dev/null | wc -l) of $FRAMES"\ + "frames, audio=$([ -s "$d/audio.wav" ] && echo yes || echo no)); re-rendering from scratch" + rm -rf "${d:?}" + fi + mkdir -p "$d" + render_arm "$label" "$knob" "$tmo" "$d" "$log" + fi + echo "arm=$label timing_source=$timing_from" >> "$OUT/PROVENANCE" + echo " timing_source=$timing_from" | tee -a "$d/ARM" + arm_report "$label" "$knob" "$d" "$log" +} + +render_arm() { # $1 label, $2 knob, $3 tmo, $4 dir, $5 log + local label=$1 knob=$2 tmo=$3 d=$4 log=$5 say "--- render $label (VLLM_LTX2_DIT_FLASH_ATTN=$knob, hard cap ${tmo}s) ---" : > "$log" # EVERY RENDER STATES ITS OWN INVOCATION on line 1 of its own log, the way the @@ -339,7 +489,7 @@ render() { # $1 = label, $2 = knob value, $3 = hard timeout seconds while kill -0 "$pid" 2>/dev/null; do local n avail n=$(grep -c 'last=' "$log" 2>/dev/null | head -1) - avail=$(awk '/^MemAvailable:/{printf "%.1f", $2/1048576}' /proc/meminfo 2>/dev/null) + avail=$(mem_avail_gib) # the SAME reader phase [0b] gates on echo "$(date -u +%H:%M:%S) $label forwards=$n memavail_gib=$avail" >> "$d/watch.tsv" # HEARTBEAT ON STDOUT, every ~2 minutes. The engine writes to its own log, so # without this the job produces NOTHING on stdout for up to two hours during @@ -359,20 +509,41 @@ render() { # $1 = label, $2 = knob value, $3 = hard timeout seconds unset VLLM_LTX2_DIT_FLASH_ATTN local nf; nf=$(ls "$d"/frame_*.ppm 2>/dev/null | wc -l) say "render $label exit=$rc stopped_by=$stopped_by frames=$nf" +} + +arm_report() { # $1 label, $2 knob, $3 dir, $4 log -- runs for a RENDERED and a RESUMED arm + local label=$1 knob=$2 d=$3 log=$4 # THE TWO-SIDED ROUTING PROOF, per arm, from that arm's own log. One-sided # counting cannot tell a routed call from an added one: the flash arm must show # op=21 AND NOT op=18, and the naive arm the reverse. LTX's cross-attentions use # op=19 in both, which is why it is printed rather than asserted on. echo "--- op-provider selections (18 kAttention / 19 kAttentionCross / 21 kAttentionDenseFlash, device=1 CUDA) ---" | tee -a "$d/ARM" grep -E 'op-provider.*op=(18|19|20|21) device=1' "$log" | sort -u | sed 's/^/ /' | tee -a "$d/ARM" - local n18 n21 + local n18 n21 routing n18=$(grep -cE 'op-provider.*op=18 device=1' "$log") n21=$(grep -cE 'op-provider.*op=21 device=1' "$log") echo " op18_naive=$n18 op21_flash=$n21" | tee -a "$d/ARM" + # THE VERDICT IS COMPUTED INTO A VARIABLE FIRST, and only then printed. It used + # to be echoed inside a `case ... | tee` pipeline, where an `exit` would have + # left the subshell and not the run -- so ROUTING_BAD was a word in a log and + # nothing more. If the knob is not read, BOTH arms are flash, the two renders + # come out bit-identical, and this file publishes PASS with all four thresholds + # vacuous: the strongest positive verdict it can produce, from an experiment + # that had one arm. case "$knob" in - 0) [ "$n18" -ge 1 ] && [ "$n21" = 0 ] && echo " ROUTING_OK=naive" || echo " ROUTING_BAD=naive (want op18>=1 op21==0)";; - *) [ "$n21" -ge 1 ] && [ "$n18" = 0 ] && echo " ROUTING_OK=flash" || echo " ROUTING_BAD=flash (want op21>=1 op18==0)";; - esac | tee -a "$d/ARM" + 0) if [ "$n18" -ge 1 ] && [ "$n21" = 0 ]; then routing="OK"; else routing="BAD"; fi;; + *) if [ "$n21" -ge 1 ] && [ "$n18" = 0 ]; then routing="OK"; else routing="BAD"; fi;; + esac + local want; case "$knob" in 0) want="op18>=1 and op21==0";; *) want="op21>=1 and op18==0";; esac + if [ "$routing" = OK ]; then + echo " ROUTING_OK=$label (knob=$knob, want $want, saw op18=$n18 op21=$n21)" | tee -a "$d/ARM" + else + echo " ROUTING_BAD=$label (knob=$knob, want $want, saw op18=$n18 op21=$n21)" | tee -a "$d/ARM" + echo "FATAL: arm $label did not route as its knob asked. Both arms may be the" \ + "same arm, and an A/B of one configuration against itself reads as a" \ + "perfect match. Nothing further is measured on this run." + exit 46 + fi # Per-forward MEDIAN from the engine's own `last=` lines. Never the governor, # which has reported 1.00 s, 69.1 s, 162 s and 396.9 s for this one quantity. grep -ohE 'last=[0-9.]+s' "$log" | sed 's/last=//;s/s$//' > "$d/samples.txt" @@ -382,14 +553,25 @@ render() { # $1 = label, $2 = knob value, $3 = hard timeout seconds m = (NR%2) ? a[(NR+1)/2] : (a[NR/2]+a[NR/2+1])/2; printf " %s: n=%d median=%.3fs mean=%.3fs min=%.3fs max=%.3fs\n", L, NR, m, s/NR, a[1], a[NR] }' | tee -a "$d/ARM" echo " memavail low-water: $(awk -F'\t' '{gsub(/memavail_gib=/,"",$4); print $4}' "$d/watch.tsv" 2>/dev/null | sort -n | head -1) GiB" | tee -a "$d/ARM" - echo " frames=$nf audio=$([ -s "$d/audio.wav" ] && stat -c %s "$d/audio.wav" || echo 0)" | tee -a "$d/ARM" + echo " frames=$(ls "$d"/frame_*.ppm 2>/dev/null | wc -l)" \ + "audio=$([ -s "$d/audio.wav" ] && stat -c %s "$d/audio.wav" || echo 0)" | tee -a "$d/ARM" } render flash 1 "${TMO_FLASH:-3600}" render naive 0 "${TMO_NAIVE:-10800}" render flash-ctl 1 "${TMO_FLASH:-3600}" -say "=== [H] the speed pair, same binary, same lease, neither arm sampled ===" +say "=== [H] the speed pair, same binary, neither arm sampled ===" +# SAME LEASE IS A CLAIM, so it is stated rather than assumed. A resumed run +# reuses an arm's `last=` samples from the lease that produced them, and a ratio +# across two leases is not the same-binary same-lease pair section 7.1 owes. +if [ -n "$REUSED_ARMS" ]; then + say " NOT a same-lease pair: reused arms:$REUSED_ARMS (their per-forward samples were taken earlier)" + echo "speed_pair_same_lease=no reused_arms=$REUSED_ARMS" >> "$OUT/PROVENANCE" +else + say " same binary, same lease: every arm was rendered in this run" + echo "speed_pair_same_lease=yes" >> "$OUT/PROVENANCE" +fi python3 - "$OUT/flash/samples.txt" "$OUT/naive/samples.txt" "$OUT/flash-ctl/samples.txt" <<'PY' import sys, statistics def med(p): @@ -410,11 +592,22 @@ say "=== [I] the pixel comparison ===" # the thresholds it applies are the ones the spec derives and not a copy that # drifted. Its exit status is the gate: 0 pass, 1 a threshold failed, 2 an input # could not be read. A 2 is never a pass. +# +# ARM A IS FLASH, and that is not cosmetic. The control is a repeat of FLASH, and +# the tool compares the control against the arm named by --control-of. This call +# used to pass `--a naive --b flash --control flash-ctl`, which made the +# "run-to-run noise floor" a SECOND naive-vs-flash comparison: it necessarily +# read about the same size as the treatment, and section 10.5's second branch +# would then have published "indistinguishable from run-to-run nondeterminism" +# whatever the kernel did. --control-of is passed explicitly rather than left to +# its default, so the wiring states its own intent. python3 "$CMP" \ - --a "$OUT/naive" --b "$OUT/flash" --control "$OUT/flash-ctl" \ - --label-a naive --label-b flash --label-control flash-ctl \ + --a "$OUT/flash" --b "$OUT/naive" --control "$OUT/flash-ctl" --control-of a \ + --label-a flash --label-b naive --label-control flash-ctl \ --json "$OUT/pixel-compare.json" 2>&1 | tee "$OUT/pixel-compare.txt" -echo "PIXEL_COMPARE_RC=${PIPESTATUS[0]}" +PIXEL_RC=${PIPESTATUS[0]} +echo "PIXEL_COMPARE_RC=$PIXEL_RC" +echo "pixel_compare_rc=$PIXEL_RC" >> "$OUT/PROVENANCE" say "=== [J] the cross-check against the recorded 20260820 baseline ===" # A different binary lineage (a50c57d69, an ancestor of the swap), so this is @@ -439,4 +632,20 @@ for d in "$OUT"/flash "$OUT"/naive "$OUT"/flash-ctl; do "$([ -s "$d/audio.wav" ] && echo yes || echo no)" \ "$([ -s "$d/video.mp4" ] && echo yes || echo no)" done -say "DONE OUT=$OUT" +say "=== [L] the verdict, which is this job's exit status ===" +# THE GATE IS THE EXIT STATUS, and it used to be a line of text. [I]'s comment +# already said "its exit status is the gate ... a 2 is never a pass", and then +# the script printed PIXEL_COMPARE_RC, carried on through [J] and [K], and ended +# on DONE with status 0. A failing pixel verdict, an unreadable input and a +# zero-frame render all exited 0. [J] and [K] still run before this line, so the +# artefacts and the cross-check are produced either way -- what changes is that +# the run ends on the verdict rather than on the fact that it finished. +case "$PIXEL_RC" in + 0) say "PIXEL VERDICT: PASS -- every registered threshold held (see $OUT/pixel-compare.txt)";; + 1) say "PIXEL VERDICT: FAIL -- a registered threshold was not met. Section 10.5: this is a" + say " finding about a change already on main, and it does not owe a widened gate";; + 2) say "PIXEL VERDICT: UNREADABLE -- nothing was compared. This is never a pass";; + *) say "PIXEL VERDICT: UNKNOWN -- the comparison exited $PIXEL_RC, which it does not define";; +esac +say "DONE OUT=$OUT RUN_ID=$RUN_ID PIXEL_COMPARE_RC=$PIXEL_RC" +exit "$PIXEL_RC" diff --git a/tests/scripts/test_ltx25_pixel_ab_harness.py b/tests/scripts/test_ltx25_pixel_ab_harness.py new file mode 100644 index 000000000..f16888d8f --- /dev/null +++ b/tests/scripts/test_ltx25_pixel_ab_harness.py @@ -0,0 +1,264 @@ +#!/usr/bin/env python3 +"""The pixel A/B harness's own preconditions, exercised without a GPU. + +`.agents/specs/ltx25-dit-attn-flash.md` section 10, #1612. + +`scripts/ltx25-dit-attn-flash-pixel-ab.sh` spends a four-hour lease on +`dgx:gpu0`. Nothing in this repository could run it, so every guard it carries +was a guard nobody had ever seen fire -- and two of them did not: + + the memory precondition phase [0] PRINTED `available 5` at +0s on + 2026-08-22 and built anyway, into a lost worker at + +728s with no binary cached and nothing measured + (rc job 5fb9399f-4f4e-417c-adbd-4d741a2e18e4). + the resume three lost workers, and each one threw away every + arm that had already rendered, including a ~2 h + naive arm. + +The shell functions those two rest on are extracted VERBATIM from the harness, +between its `# BEGIN pixab-helpers` and `# END pixab-helpers` markers, and run +here against a fabricated `/proc/meminfo` and fabricated arm directories. The +extraction is itself asserted, because a suite that silently found no block +would report success over nothing. + +WHAT THIS CANNOT DO. It does not run the harness. Its wiring assertions at the +bottom read the file as TEXT, and a text assertion is a tripwire rather than a +proof: it catches the exact inversion that shipped and it would not catch a +rewrite that reintroduced the same defect in different words. The lease is the +only place the rest of that file executes, and that is stated here rather than +papered over. +""" + +from __future__ import annotations + +import os +import subprocess +import tempfile +import threading +import time +import unittest +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[2] +# Overridable ONLY so that the red-before run of this suite can be taken against +# an earlier revision of the harness. It defaults to the committed file. +HARNESS = Path(os.environ.get("PIXAB_HARNESS", ROOT / "scripts/ltx25-dit-attn-flash-pixel-ab.sh")) + +BEGIN = "# BEGIN pixab-helpers" +END = "# END pixab-helpers" + +MEM_GATE_REFUSED = 39 +UNIT_GATE_FAILED = 44 +UNIT_GATE_ABSENT = 45 +ROUTING_BAD = 46 + + +def helper_block() -> str: + text = HARNESS.read_text() + if BEGIN not in text or END not in text: + raise AssertionError( + f"{HARNESS} carries no {BEGIN!r}/{END!r} block: there is nothing to exercise") + return text.split(BEGIN, 1)[1].split(END, 1)[0] + + +def bash(snippet: str, env: dict[str, str] | None = None) -> subprocess.CompletedProcess: + """Run the extracted helpers plus `snippet`, with the two externals they use.""" + prelude = 'set -u\nT0=$(date +%s)\nsay() { echo "[say]$*"; }\n' + e = dict(os.environ) + e.update(env or {}) + return subprocess.run(["bash", "-c", prelude + helper_block() + "\n" + snippet], + capture_output=True, text=True, env=e) + + +def meminfo(path: Path, gib: float | None) -> Path: + """A `/proc/meminfo` with, or deliberately without, a MemAvailable line.""" + lines = ["MemTotal: 124680000 kB\n"] + if gib is not None: + lines.append("MemAvailable: %d kB\n" % int(gib * 1048576)) + lines.append("SwapTotal: 0 kB\n") + path.write_text("".join(lines)) + return path + + +class HelperBlock(unittest.TestCase): + def test_the_block_exists_and_holds_the_three_functions(self) -> None: + block = helper_block() + for fn in ("mem_avail_gib()", "wait_for_memory()", "arm_is_complete()"): + self.assertIn(fn, block, f"{fn} left the extracted block: this suite would " + f"then exercise nothing while still passing") + + +class MemAvailableReader(unittest.TestCase): + """ONE reader, and phase [0b] and the render watchdog both call it. + + They read `/proc/meminfo` rather than `free`'s `available` column, so the + start gate and the watchdog cannot disagree about what they measured. + """ + + def test_it_reads_memavailable_in_gib(self) -> None: + with tempfile.TemporaryDirectory() as t: + mi = meminfo(Path(t) / "meminfo", 61.5) + p = bash("mem_avail_gib", {"MEMINFO": str(mi)}) + self.assertEqual(p.returncode, 0, p.stderr) + self.assertEqual(p.stdout.strip(), "61.5") + + def test_it_is_empty_rather_than_zero_when_the_field_is_absent(self) -> None: + """Empty and zero are different facts. Zero would trip a floor check as + if the box were full; empty says the instrument did not read.""" + with tempfile.TemporaryDirectory() as t: + mi = meminfo(Path(t) / "meminfo", None) + p = bash("v=$(mem_avail_gib); echo \"[${v}]\"", {"MEMINFO": str(mi)}) + self.assertEqual(p.stdout.strip(), "[]") + + def test_it_is_empty_when_the_file_does_not_exist(self) -> None: + p = bash("v=$(mem_avail_gib); echo \"[${v}]\"", {"MEMINFO": "/nonexistent/meminfo"}) + self.assertEqual(p.stdout.strip(), "[]") + + +class MemoryPrecondition(unittest.TestCase): + """Phase [0b]: wait, then refuse, and never proceed on a full box. + + The floor is 60 GiB because the recorded 20260820 render at this geometry + peaked at 79.503 GiB with a MemAvailable low-water of 40.13 GiB. A lease + spent waiting and refusing costs a lease; a lease spent building into an + out-of-memory kill costs the lease and leaves a record that cannot say why. + """ + + def test_it_proceeds_immediately_above_the_floor(self) -> None: + with tempfile.TemporaryDirectory() as t: + mi = meminfo(Path(t) / "meminfo", 80.0) + p = bash("wait_for_memory 60.0 1200 30; echo rc=$?", {"MEMINFO": str(mi)}) + self.assertIn("rc=0", p.stdout) + self.assertIn("80.0 GiB >= floor 60.0 GiB", p.stdout) + self.assertIn("after 0s", p.stdout) + + def test_it_refuses_below_the_floor_with_its_own_status(self) -> None: + """The 2026-08-22 case, in numbers: 5 GiB available at +0s.""" + with tempfile.TemporaryDirectory() as t: + mi = meminfo(Path(t) / "meminfo", 5.0) + p = bash("wait_for_memory 60.0 0 1; echo rc=$?", {"MEMINFO": str(mi)}) + self.assertIn(f"rc={MEM_GATE_REFUSED}", p.stdout) + out = p.stdout + p.stderr + self.assertIn("5.0 GiB", out, "the refusal must name what it saw") + self.assertIn("60.0 GiB start floor", out, "and the floor it wanted") + self.assertIn("after 0s of waiting", out, "and how long it waited") + self.assertIn("nothing was rendered", out) + + def test_it_waits_and_then_proceeds_when_the_box_recovers(self) -> None: + """The reason this waits rather than failing at once: the previous + tenant's memory is often still being reclaimed when the lease starts.""" + with tempfile.TemporaryDirectory() as t: + mi = Path(t) / "meminfo" + meminfo(mi, 5.0) + + def recover() -> None: + time.sleep(1.5) + meminfo(mi, 90.0) + + th = threading.Thread(target=recover) + th.start() + p = bash("wait_for_memory 60.0 20 1; echo rc=$?", {"MEMINFO": str(mi)}) + th.join() + self.assertIn("rc=0", p.stdout) + # It logged the low readings on the way, so a reader can tell a + # recovering box from a flat one. + self.assertIn("5.0 GiB < 60.0 GiB", p.stdout) + self.assertIn("90.0 GiB >= floor 60.0 GiB", p.stdout) + + def test_an_unreadable_meminfo_is_a_refusal_and_never_a_pass(self) -> None: + p = bash("wait_for_memory 60.0 0 1; echo rc=$?", {"MEMINFO": "/nonexistent/meminfo"}) + self.assertIn(f"rc={MEM_GATE_REFUSED}", p.stdout) + self.assertIn("cannot read MemAvailable", p.stdout + p.stderr) + + +class ArmCompleteness(unittest.TestCase): + """Phase [G]'s resume: an arm is reused only when it is COMPLETE. + + A partial arm is re-rendered from scratch rather than resumed mid-flight. + """ + + def _arm(self, root: Path, frames: int, audio: bool, audio_size: int = 64) -> Path: + d = root / "arm" + d.mkdir() + for i in range(frames): + (d / f"frame_{i:06d}.ppm").write_bytes(b"P6\n1 1\n255\n\0\0\0") + if audio: + (d / "audio.wav").write_bytes(b"\0" * audio_size) + return d + + def _check(self, d: Path, want: int = 49) -> int: + return bash(f'arm_is_complete "{d}" {want}; echo rc=$?').stdout.strip().split("=")[-1] + + def test_a_complete_arm_is_reused(self) -> None: + with tempfile.TemporaryDirectory() as t: + d = self._arm(Path(t), 49, True) + self.assertEqual(self._check(d), "0") + + def test_a_short_arm_is_not(self) -> None: + with tempfile.TemporaryDirectory() as t: + d = self._arm(Path(t), 48, True) + self.assertEqual(self._check(d), "1") + + def test_a_long_arm_is_not_either(self) -> None: + """More frames than asked for is a directory from another geometry, not + a completed render of this one.""" + with tempfile.TemporaryDirectory() as t: + d = self._arm(Path(t), 50, True) + self.assertEqual(self._check(d), "1") + + def test_frames_without_audio_are_not_complete(self) -> None: + """The DiT drives both streams and the comparison reads both, so an arm + that lost its wav is an arm half of the verdict cannot be taken on.""" + with tempfile.TemporaryDirectory() as t: + d = self._arm(Path(t), 49, False) + self.assertEqual(self._check(d), "1") + + def test_an_empty_wav_is_not_audio(self) -> None: + with tempfile.TemporaryDirectory() as t: + d = self._arm(Path(t), 49, True, audio_size=0) + self.assertEqual(self._check(d), "1") + + def test_an_absent_directory_is_not_complete(self) -> None: + with tempfile.TemporaryDirectory() as t: + self.assertEqual(self._check(Path(t) / "never-rendered"), "1") + + +class Wiring(unittest.TestCase): + """TEXT TRIPWIRES on the four call sites that cannot be executed here. + + Each pins a defect that shipped, in the words that shipped it. None is a + proof: the lease is the only place these lines run. + """ + + def setUp(self) -> None: + self.text = HARNESS.read_text() + + def test_the_control_is_compared_against_the_arm_it_repeats(self) -> None: + """`flash-ctl` repeats FLASH, so flash is arm A and --control-of says so. + With `--a naive` the control was a second naive-vs-flash comparison.""" + self.assertIn('--a "$OUT/flash" --b "$OUT/naive" --control "$OUT/flash-ctl" --control-of a', + self.text) + self.assertIn("--label-a flash --label-b naive --label-control flash-ctl", self.text) + + def test_the_run_exits_with_the_pixel_verdict(self) -> None: + self.assertIn('PIXEL_RC=${PIPESTATUS[0]}', self.text) + self.assertIn('exit "$PIXEL_RC"', self.text) + + def test_a_routing_failure_stops_the_run(self) -> None: + self.assertIn(f"exit {ROUTING_BAD}", self.text) + # Computed into a variable and tested OUTSIDE the pipeline: an `exit` + # inside `case ... | tee` leaves the subshell, not the run. + self.assertIn('if [ "$routing" = OK ]; then', self.text) + + def test_the_unit_gate_refuses_rather_than_reporting(self) -> None: + self.assertIn(f"exit {UNIT_GATE_FAILED}", self.text) + self.assertIn(f"exit {UNIT_GATE_ABSENT}", self.text) + + def test_the_heartbeat_is_reaped_on_a_lease_kill(self) -> None: + for sig in ("HUP", "INT", "TERM"): + self.assertIn(f"' {sig}", self.text.replace("'HUP", "' HUP")) + self.assertIn("trap cleanup EXIT", self.text) + + +if __name__ == "__main__": + unittest.main() From 378239544db54d9ddde70dc8744c9779707c1865 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 22 Aug 2026 16:47:53 +0000 Subject: [PATCH 06/19] docs(#1612): the calibration table quoted numbers no test computes, and a green gate was filed as PENDING MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The instruments are repaired in the two commits before this one. This repairs what §10 and §8 SAY about them, which was wrong in four places and silent in two more. THE CALIBRATION TABLE MIXED TWO POPULATIONS AND CLAIMED BOTH WERE GATED. §10.4 quoted `5.183 / 28.1 dB / 0.8705 / 0.624` for a one-pixel shift and said it was "pinned in `tests/...` so it is a gate rather than a claim". Both halves of that were true of different things: the quoted row was measured on the real 20260820 NAS render, which nothing here can reach, and the committed fixtures produce `16.6120 / 21.8 dB / 0.77036 / 0.4487` — mean |d| 3.2x the quoted figure. Two tables now, labelled by population, with the fixture rows computable by any lane and the NAS row kept and marked as measured rather than gated. The difference is explained rather than averaged: the fixture is a 4-pixel sine grating, so one pixel of shift is a quarter cycle, which makes it a harsher shift and a gentler dither than the real frames — the direction a discrimination proof should err in. §8 FILED A GREEN GATE AS PENDING. "the comparison tool discriminates" read `PENDING until §10.7`, and it needs no GPU, no lease and no NAS: 37 tests, OK. It is PASS with the count and the date. Two rows that genuinely wait on the lease stay PENDING. Two rows are added: the suites now run on a lane, and the harness has preconditions that a suite exercises. §10.5 SELECTED BETWEEN TWO OPPOSITE VERDICTS WITH NO BOUNDARY. It read "control is 0" against "control comparable to the delta" and left "comparable" to a judgement made with the numbers already in view. `R` is now defined — control mean |delta| on luma over the treatment's — with the boundary at 0.5, written before any `R` exists, and a fourth reading for the `null` case. The tool computes and reports it; it is not a check, because it chooses between two readings of a pass. §10.6 IS NEW, AND IT IS THE ONE THAT MATTERS MOST. A control compared against the wrong arm is not a weak control, it is a fabricated result: correct units, correct magnitude, correct place in the report, and the treatment measured twice. No threshold in §10.4 could have caught it, because the thresholds were fine and the wiring was one argument wrong. It is filed with its siblings from this campaign — the governor that reported 1.00 s, 69.1 s, 162 s and 396.9 s for one quantity; the append-only checker that read the working tree instead of the commits and returned rc=0 three times; the `static_assert` that compared a literal against itself and read `256 == 256` — because the shape is what makes the next one recognisable, and the repair is always the same: make the instrument state what it is measuring, in its own output, in words. C0 IS NOT SUFFICIENT AND NOW SAYS SO. It asks whether there is variance, whether the frames are distinct, and whether anything moves. Two identical sequences of pure NOISE answer all three and pass every check with the verdict PASS — that is demonstrated, not feared. So does a pair of renders that are consistent with each other and wrong together: wrong prompt, wrong seed, wrong checkpoint class. The new paragraph names that class and names the phase [J] cross-check against the 20260820 baseline as the only absolute reference present. No new check is invented for it, because "is this a golden retriever shaking off water" is a model, not a threshold. Also recorded: that most of the harness runs nowhere but a lease, and that the suite's assertions about the render loop, the routing proof and the phase [I] call site are text tripwires rather than proofs; that a resumed arm makes the speed pair a cross-lease pair, which matters because #1549's speedup is already qualified for a missing control and a quiet cross-lease ratio would replace a stated gap with a false closure; and that `json.dump` writes `Infinity` for a bit-identical pair's PSNR, which is disclosed rather than repaired because every consumer here is Python and a sentinel would change what the report means for the one case the comparison hopes to see. §10.7 is left as a heading with an explicit "not yet taken" naming the submitted `rc` job, because an empty promise in a spec is worse than a stated gap. AND THE LESSON LEAVES THIS SPEC. `.agents/verification.md` gains a section: "Make the instrument say what it is measuring, in its own output, in words", with all four instances and the observation that ties them together — the governor never printed which estimator produced its number, the append-only checker never printed whether it had read commits or the working tree, and the `static_assert` never printed which constant it had captured. Each would have been caught in seconds by an instrument that narrated its own comparison. A row spec is read by the row; a task guide is read by the author of the next harness. THE START FLOOR IS ALREADY REFUSING, AND IT FALSIFIED ITS OWN PREMISE. Job `2ccd1acf` reports `MemAvailable 5.1 GiB` flat from +0s to +360s, against 5.0 GiB in the lost lease an hour earlier and at least two other jobs in between. So the box is not holding a previous tenant's unreclaimed memory, which is what #1709 records and what the WAIT was designed for: something resident holds ~114 GiB. §10.7 records that the transient reading is falsified and that #1709 owes the correction. The distinction exists only because the gate logs every poll rather than only its refusal — a single line would have said `5.1 GiB` and left the cause open. The harness header now lists its exit codes in one place, including that a 45 is usually a staged binary cache missing `test_ltx2_device` and that clearing `$W/pixab-bin` regenerates it. That belongs where a lease submitter reads it, not only in a commit message. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-opus-5 [Claude Code] --- .agents/specs/ltx25-dit-attn-flash.md | 293 +++++++++++++++++-- .agents/verification.md | 36 +++ scripts/ltx25-dit-attn-flash-pixel-ab.sh | 30 ++ tests/scripts/test_ltx25_pixel_ab_harness.py | 6 +- 4 files changed, 339 insertions(+), 26 deletions(-) diff --git a/.agents/specs/ltx25-dit-attn-flash.md b/.agents/specs/ltx25-dit-attn-flash.md index 253320b67..2bd77c7a1 100644 --- a/.agents/specs/ltx25-dit-attn-flash.md +++ b/.agents/specs/ltx25-dit-attn-flash.md @@ -523,7 +523,9 @@ never a synonym for "probably fine". | A/B, same binary, both arms | `dgx:gpu0` under an `rc` lease | **PENDING** — the flash arm is measured at 7.680 s median (n=19); the worker was lost before the naive arm, so no pair exists (§7.1) | | pixel A/B at production geometry | `dgx:gpu0` under an `rc` lease, `scripts/ltx25-dit-attn-flash-pixel-ab.sh` | **PENDING** — criterion registered in §10.4; result in §10.7 | | run-to-run control (`flash` twice) | the same lease | **PENDING** — §10.3; without it no arm-to-arm delta is attributable to the kernel | -| the comparison tool discriminates | `tests/scripts/test_ltx25_render_compare.py` | **PENDING until §10.7** — a dither passes, a one-pixel shift fails all four V checks, and two all-black renders fail C0 while reading as a perfect match on every V (§10.4) | +| the comparison tool discriminates | `tests/scripts/test_ltx25_render_compare.py` | **PASS** — 37 tests, `OK`, at `2026-08-22`. It needs no GPU, no lease and no NAS, so `PENDING until §10.7` was misreporting a gate that was already green: a dither passes, a one-pixel shift fails all four V checks, two all-black renders fail C0 while reading as a perfect match on every V, an unreadable input exits 2 while a threshold failure exits 1, A1 and A2 disagree on a time-shifted waveform, and the SSIM is pinned by its taps, its impulse response and three fixture values (§10.4). The count dates the run; it is not a floor to defend | +| the comparison tool runs on a lane | `scripts/agent-preflight.sh`, `.github/workflows/ci.yml` | **PASS** — it ran on NO lane when it landed: absent from preflight's `SUITES`, from the enumerated python block in CI and from `tests/CMakeLists.txt`, while the row above registered it as a gate. Both are registered now. Preflight SKIPs it when numpy is absent, which is the third state and never an `ok`; the CI lane installs `python3-numpy` so the lane that must not be silent cannot be | +| the harness's own preconditions | `tests/scripts/test_ltx25_pixel_ab_harness.py` | **PASS** — 19 tests, `OK`, at `2026-08-22`. The memory precondition and the arm-completeness check are extracted verbatim from the harness and run against a fabricated `/proc/meminfo`. The four call sites that only a lease can execute are text tripwires and are labelled as such | | full preflight | `scripts/agent-preflight.sh` | **PASS at HEAD** — and it was NOT before: `documentation-checkpoint` was red on two of this branch's own commits (see below) | | `documentation-checkpoint` | CI, and locally over the branch range | **PASS at HEAD, RED before it, and the red was THIS BRANCH's** — `2aa78c69b` and `2f39a9426` each recorded a measurement in `.agents/benchmark-record.md` without writing `docs/STATUS.md` (and `docs/BENCHMARKS.md` for the second). The control on the main-only range `4c193bd55..5d548d003` is rc 0, so it was not inherited. Both commits were replaced by one that writes all three surfaces together when the branch was rebuilt, and the checker is re-run at each head rather than trusted to have stayed fixed — a job that has stopped appearing in a failing set is not the same fact as a job that passes | | `build-newest-gcc` | CI | **PASS, and now green on `main` too** — it was red on `main` on `::getpid` in `test_qwen3_dflash2_gguf.cpp:547`, a file this change does not touch; [#1581](https://github.com/mudler/vllm.cpp/pull/1581) fixed it and this branch carries that fix through the merge. A red here after the merge is therefore this row's, not inherited | @@ -667,11 +669,36 @@ denominator ~3.2% by `ptrace`-stopping every thread. This harness runs a sampler correction and finally replaces the cross-run 6.03-6.23x range with a same-binary ratio. -**Routing is proved per arm, two-sided, from that arm's own log.** -`VT_OP_PROVIDER_STATS=1` makes each op announce itself once when it resolves. -The flash arm must show `op=21 device=1` **and no** `op=18 device=1`; the naive -arm the reverse. A one-sided count cannot tell a routed call from an added one, -and "the knob was exported" is not evidence that the branch was taken. +**Routing is proved per arm, two-sided, from that arm's own log, and a failure +STOPS THE RUN.** `VT_OP_PROVIDER_STATS=1` makes each op announce itself once when +it resolves. The flash arm must show `op=21 device=1` **and no** `op=18 +device=1`; the naive arm the reverse. A one-sided count cannot tell a routed call +from an added one, and "the knob was exported" is not evidence that the branch +was taken. The verdict was originally echoed inside a `case ... | tee` pipeline +and nothing acted on it, which is the shape that matters most here: if the knob +is not read, both arms are flash, the renders come out bit-identical, and this +design publishes PASS with all four thresholds vacuous — the strongest positive +verdict it can produce, from an experiment that had one arm. It is now computed +into a variable, tested outside any pipeline, and exits 46. + +**The lease is not assumed to survive, and the box is not assumed to be empty.** +Two failures of this harness are recorded rather than smoothed over. It printed +`available 5` at +0s on 2026-08-22 and built into a lost worker anyway, so phase +[0b] now gates on `MemAvailable` against a 60 GiB start floor — derived from this +geometry's own recorded 79.503 GiB peak and 40.13 GiB low-water — waits for a +previous tenant's memory to be reclaimed, and refuses at exit 39 rather than +proceeding. And three lost workers each discarded every arm already rendered, so +`RUN_ID` is overridable and a resumed lease reuses any arm that is complete in +`$OUT`, states in that arm's record that its timings came from an earlier lease, +and still proves that arm's routing from the log it already has. A reused arm +makes the speed pair a cross-lease pair, and phase [H] says so rather than +letting the ratio imply otherwise. **That declaration is not bookkeeping.** The +speedup #1549 shipped on is already qualified — its control arm never ran, and +§8 carries the A/B as `PENDING` because of it — so a resumed run that quietly +produced a cross-lease ratio presented as a within-lease one would replace a +stated gap with a false closure. The arm's `ARM` file, the run's `PROVENANCE` +and phase [H]'s own line all carry `timing_source`, and a reader who takes the +ratio without them has to have ignored three places that said so. ### 10.4 The registered acceptance criterion @@ -681,7 +708,7 @@ them by hand afterwards. | # | check | threshold | where it comes from | |---|---|---|---| -| C0 | each arm, ON ITS OWN: frames written, no near-uniform frame, every frame hash distinct, no zero-motion pair | all four, per arm | **the hole every difference-only comparison has** | +| C0 | each arm, ON ITS OWN: no near-uniform frame, every frame hash distinct, no zero-motion pair | all three, per arm | **the hole every difference-only comparison has** | | V1 | mean \|delta\|, 8-bit RGB | `<= 1.0` level | one level is the quantisation step of the artefact itself; a mean below it says the average pixel is within the PPM's own resolution | | V2 | worst-frame PSNR | `>= 40 dB` | the video-coding "visually lossless" convention. This experiment did not choose it | | V3 | worst-frame SSIM | `>= 0.99` | Wang et al. 2004, 11x11 Gaussian sigma=1.5 on luma. 0.98 is the usual transparency line; this is stricter, and it is the WORST frame rather than the mean | @@ -696,7 +723,19 @@ score infinite PSNR and SSIM `1.000000`, and would read as the strongest possible pass this table can produce. A run that exited 0 having written frames that were all one colour has happened in this repository, so that is a recorded failure mode rather than a hypothesis. C0 is computed per arm before anything is -subtracted, and it is checked first. `verify_render.py` makes the same checks, +subtracted, and it is checked first. + +**"Frames written" used to be a fourth C0 check and it is now a REFUSAL.** An +arm directory holding no `frame_*.ppm` leaves the tool at exit 2 with no JSON, +before any check is built, so the check could never report `False` and one of the +four sub-checks this table registered could never fire. Exit 1 is the status that +says *the two renders differ*, which is a reading of an experiment that happened; +an arm that rendered nothing is not that, and a broken render that reported it +would be indistinguishable from the finding this whole section exists to make. An +absent `audio.wav` stays a failed CHECK rather than a refusal, deliberately: the +video comparison is fully defined without it, so there is a result to report. + +`verify_render.py` makes the same checks, and they are recomputed inside `scripts/ltx25-render-compare.py` rather than shelled out to, because that file lives on a mutable path on a share and this one is committed per revision — the same reason §7.1 gives for the harness. @@ -709,30 +748,102 @@ this video's own motion*. It needs no re-argument at another geometry or another prompt, because both terms move together — which is exactly what a constant cannot do, and why the FA-2 arm (#1551) can take this same criterion. -**The scale is executable, not asserted.** Measured on the recorded 20260820 -baseline's own frames, and pinned in `tests/scripts/test_ltx25_render_compare.py` -on synthetic fixtures so it is a gate rather than a claim: +**The scale is executable, and it comes from TWO populations that must not be +read as one.** An earlier version of this section quoted the first table and said +it was "pinned in `tests/...` so it is a gate rather than a claim". That sentence +was false of those numbers: what the suite pins is the second table, on synthetic +frames, and the two differ by 3.2x on mean `|d|` for the same perturbation. + +**Population 1, MEASURED, not gated here.** The recorded 20260820 baseline's own +`768x448/49f` frames, on the NAS. Nothing in this repository can reach that share, +so no test asserts these: -| perturbation of a real frame | mean \|d\| | PSNR | SSIM | V4 ratio | verdict | +| perturbation of a REAL frame | mean \|d\| | PSNR | SSIM | V4 ratio | verdict | |---|---|---|---|---|---| | none | 0 | inf | 1.000000 | 0.000 | bit-identical | | +/-1 LSB on 3% of samples | 0.0199 | 65.1 dB | 0.99992 | 0.0026 | **PASS**, all four | | one pixel of global horizontal shift | 5.183 | 28.1 dB | 0.8705 | 0.624 | **FAIL**, all four | +**Population 2, GATED.** The synthetic 96x64/6f fixtures of +`tests/scripts/test_ltx25_render_compare.py`, seed `20260822`, which every lane +can compute. These are the numbers that are a gate: + +| perturbation of a FIXTURE frame | mean \|d\| | worst PSNR | worst SSIM | V4 ratio | verdict | +|---|---|---|---|---|---| +| none | 0 | inf | 1.000000 | 0.000 | bit-identical | +| +/-1 LSB on 3% of samples | 0.0194 | 65.0 dB | 0.99999 | 0.0007 | **PASS**, all four | +| one pixel of global horizontal shift | 16.6120 | 21.8 dB | 0.77036 | 0.4487 | **FAIL**, all four | + +**Why they differ, and why both belong.** The fixture is a sine grating with a +4-pixel period plus uniform noise, so one pixel of shift moves each row by a +quarter of a cycle. The render's real frames are smoother, and the same shift +moves them less. The fixture is therefore a HARSHER shift and a slightly gentler +dither, which is the direction a discrimination proof should err in: the pass row +passes with less headroom than the real one, and the fail row fails harder. +Neither table calibrates the other, and the sentence that used to imply one +pinned the other is the defect being repaired here. + +The SSIM those tables are computed with is itself pinned by property and by +value: the eleven taps of the `sigma=1.5` window, the impulse response as +`outer(k, k)`, its equality along both axes, the Rec.601 luma triple, and three +fixture SSIMs to eight decimals. Ten separate mutations of that metric -- sigma, +window, C1, C2, the second separable pass, the luma weights -- left the earlier +suite green, so "Wang et al. 2004" was a name and not a criterion. + So the thresholds sit between a dither and a single pixel of motion, nearer the -dither: V4 at `0.10` refuses anything above a sixth of a one-pixel global shift. -A criterion that admitted the shift row would not be a criterion. +dither: V4 at `0.10` refuses anything above a quarter of a one-pixel global shift +on the fixtures, and a sixth of one on the real frames. A criterion that admitted +either shift row would not be a criterion. ### 10.5 Reading the result, stated before there is one -- **All checks pass and the control is 0** — the swap changes the render by less - than a tenth of its own motion step, the difference is entirely the kernel's, - and the verdict is **within bf16 noise**. The measured values, with their - headroom, are then what belongs in the record. -- **All checks pass and the control is comparable to the delta** — the verdict is - **indistinguishable from run-to-run nondeterminism**, which is stronger, and - the row additionally owes an issue for the nondeterminism itself, because a - render that is not reproducible is its own defect. +**The selector is a number, `R`, and the tool computes it.** Two of the three +readings below are opposite published verdicts on the same passing result, and +until this revision nothing computed the comparison they turn on: the tool +printed the control's figures for a reader to eyeball, and "comparable to the +delta" was left to a judgement made after the numbers were in view. So: + +``` +R = control mean |delta| on luma / treatment mean |delta| on luma +``` + +Both terms are the same statistic in the same 8-bit luma units, one from the +control-vs-its-own-arm comparison and one from the flash-vs-naive comparison. +The tool reports `R` as `control_ratio.ratio_mean_abs_luma`, alongside the RGB +ratio and both numerators. **It is reported and never checked**, because it +selects between two readings of a pass rather than between pass and fail, and a +check cannot express a choice between two answers that are both answers. `R` is +`null`, with a stated reason, when the treatment is bit-identical: that is a +division by zero the arithmetic expects, and §10.2 predicts it will not occur. + +The boundary is `0.5`, and it is written here before any `R` exists. It is the +point at which the control accounts for half of the measured delta, so that +attributing the delta to the kernel and attributing it to the box are equally +defensible; a selector without a stated boundary is a judgement call wearing a +number. + +- **All checks pass and `R = 0`** (the control is bit-identical) — the swap + changes the render by less than a tenth of its own motion step, the difference + is entirely the kernel's, and the verdict is **within bf16 noise**. The + measured values, with their headroom, are then what belongs in the record. +- **All checks pass and `R >= 0.5`** — the verdict is **indistinguishable from + run-to-run nondeterminism**, which is stronger, and the row additionally owes + an issue for the nondeterminism itself, because a render that is not + reproducible is its own defect. +- **All checks pass and `0 < R < 0.5`** — the delta is partly the kernel's, and + the control is the floor every threshold above must be read against. The record + states `R` rather than choosing one of the two verdicts. +- **`R` is `null`** — the two arms are bit-identical, which §10.2 says they will + not be. Read that as a finding about the experiment (one arm rendered twice, + a knob not read, a cached artefact) before reading it as a result. + +**WHICH ARM the control repeats is now an argument, not a convention.** +`--control-of {a,b}` names it, the JSON records it, and the control block prints +the sentence. The harness passed `--a naive --b flash --control flash-ctl` while +the tool compared the control against arm A, so the "run-to-run noise floor" was +a SECOND naive-vs-flash comparison: it necessarily read about the same size as +the treatment, and the second branch above would have been published whatever the +kernel did. A silent convention a caller can invert is not a convention. - **Any check fails** — the verdict is **visibly different**, and that is a finding about a change already on `main`, not a failure of this work. It owes an issue naming what diverged and by how much, and it does not owe a widened @@ -740,8 +851,126 @@ A criterion that admitted the shift row would not be a criterion. exists to prevent, and §9's stop conditions already say so for the numeric gate. -### 10.6 What this deliberately does not measure - +### 10.6 The failure this design nearly shipped: a well-formed answer about the wrong thing + +**A control compared against the wrong arm is not a weak control. It is a +fabricated result.** The harness passed `--a naive --b flash --control +flash-ctl` while the tool compared the control against arm A. `flash-ctl` is a +repeat of FLASH, so what got labelled "the run-to-run noise floor" was a second +naive-vs-flash comparison. It would have printed a number of the right +magnitude, in the right units, in the right place in the report — and it would +have been the treatment measured twice. §10.5's second branch, +**indistinguishable from run-to-run nondeterminism**, would then have been the +published verdict *whatever the kernel did*, and it would have read as the +strongest null result this design can produce. + +**No threshold in §10.4 could have caught it.** V1 to V4 and A1 to A2 judge the +treatment, and the treatment was wired correctly; C0 judges each arm's content, +and all three arms had content. The defect was in the WIRING, one argument +deep, and every number downstream of it was arithmetically correct. A criterion +committed in advance protects against a threshold moved after the fact. It does +not protect against an instrument pointed at the wrong thing. + +**This campaign keeps meeting that class, and the instances are worth naming +together, because the shape is what makes them recognisable:** + +- the governor that reported `1.00 s`, `69.1 s`, `162 s` and `396.9 s` for one + quantity — four well-formed answers, at most one of them about the thing that + was asked (§7.1 reads the engine's own `last=` lines instead, for this + reason); +- the append-only checker that read the WORKING TREE instead of the commits and + returned `rc=0` three times over a violation that was there; +- the `static_assert` that compared a literal against itself and read + `256 == 256`, staying green when the constant it was meant to pin changed. + +Each returned a confident, structurally valid answer to a question nobody had +verified it was asking. **The repair for the class is the same each time: make +the instrument state what it is measuring, in its own output, in words.** So +`--control-of` is now a required-by-default argument, the JSON records +`control_of`, and the control block prints the sentence naming the arm it was +read against. A convention a caller can invert silently is not a convention, and +a reader who cannot see the wiring in the report cannot audit it. + +### 10.7 The measurement + +**NOT YET TAKEN.** The renders have not landed. A run is submitted on +`f407019e3` as `rc` job `2ccd1acf-dfa6-40b4-b095-1928caebe2c1`, `RUN_ID=1612-r2`, +on `dgx:gpu0`. This heading exists so that the result has one place to go and so +that its absence is visible; it is not a promise that the run succeeded. The +previous attempt (job `5fb9399f-4f4e-417c-adbd-4d741a2e18e4`, 2026-08-22) lost +its worker during the build with the box already at 114 of 119 GiB used before +it allocated anything, which is what §10.3's start-floor gate now refuses. + +**AND THE GATE IS ALREADY REFUSING, WHICH FALSIFIES THE READING THIS ROW STARTED +FROM.** Job `2ccd1acf` reports, from its own poll log: + +``` +[pixab +0s] === [0b] the MemAvailable PRECONDITION, gated rather than printed === +[pixab +0s] MemAvailable 5.1 GiB < 60.0 GiB, waited 0s of 1200s +[pixab +360s] MemAvailable 5.1 GiB < 60.0 GiB, waited 360s of 1200s +``` + +`5.0` GiB in the lost lease an hour earlier, `5.1` GiB now, **flat for six +minutes across two leases and at least two intervening jobs.** So this is NOT a +previous tenant's memory awaiting reclamation, which is what +[#1709](https://github.com/mudler/vllm.cpp/issues/1709) currently records and +what the wait in §10.3 was designed for: something on that box holds ~114 GiB +persistently. The condition is PERSISTENT and the transient reading is +falsified. That distinction is only available because the gate logs EVERY poll +rather than only its refusal — a single refusal line would have said "5.1 GiB" +and left the cause open. A read-only diagnostic to identify the resident +allocation is with the operator, and #1709 owes the correction. The renders +cannot be taken on that box until it is resolved, and no arm has run. + +When it lands, this section records: the three arms' frame counts and routing +proofs, the C0 block, V1 to V4 and A1 to A2 with their headroom, the control +ratio `R` and which of §10.5's readings it selects, the cross-check against the +20260820 baseline, and the exit status the harness returned. + +### 10.8 What this deliberately does not measure + +- **WHETHER THE CONTENT IS THE RIGHT CONTENT. C0 is necessary and it is not + sufficient.** C0 asks three questions — is there variance in each frame, is + every frame hash distinct, does anything move between them — and **two + identical sequences of pure noise answer all three and pass every C and V + check with the verdict `PASS`**. That is demonstrated, not feared. The same + hole admits a pair of renders that are perfectly consistent with each other + and wrong together: the wrong prompt, the wrong seed, the wrong checkpoint + class, a text encoder that produced nothing, a VAE decoding a latent the + sampler never refined. Each of those is a *difference from an intended + render*, and every measurement in this section is a difference between two + arms, so none of them can see it. Nothing here should be extended to try: + a check for "is this a golden retriever shaking off water" is a model, not a + threshold. **The only absolute reference present is the cross-check against + the recorded 20260820 baseline in phase [J] of the harness** — a real render + of this prompt at this geometry, from an ancestor build — and it is a + cross-check precisely because its binary lineage differs, so it bounds this + class rather than closing it. When that baseline is unreachable, the run says + so and this class is unmeasured for that run. +- **MOST OF THE HARNESS ITSELF.** `tests/scripts/test_ltx25_pixel_ab_harness.py` + extracts the `MemAvailable` reader, the start gate and the arm-completeness + check verbatim from `scripts/ltx25-dit-attn-flash-pixel-ab.sh` and runs them + against a fabricated `/proc/meminfo`, so those three execute here. **The render + loop, the routing assertion, the phase [I] call site and the phase [L] exit do + not.** They are pinned as TEXT — the suite asserts the exact call site that + shipped inverted, and the exact `exit` lines — and a text assertion is a + tripwire, not a proof: it catches the inversion that happened and would not + catch a rewrite that reintroduced it in different words. `dgx:gpu0` under a + lease is the only place those lines run, which is why every one of them was + wrong at once: they had never executed anywhere a test could watch. **That is + a structural explanation of a cluster rather than four coincidences**, and it + tells the next reader which claims in this harness are load-bearing and which + are decorative — what a test can execute is now checked, and what only a lease + can execute is a comment to be verified against the run's own log. + + **One mutation of the memory gate is deliberately a TIMEOUT rather than a + failed assertion.** Inverting its floor comparison makes the gate wait out its + whole budget instead of proceeding, so the suite never returns. The mutation + runner records that as RED with the reason named, because a suite that did not + report `OK` is not a suite that passed. A timeout read as success is the exact + silent hole a wait-for-quiet loop already cost this campaign once, so that + path was exercised on purpose rather than left for whoever next inverts that + line. - **PPM is 8-bit.** The comparison is on the artefact the pipeline writes, which is already quantised from the VAE's float output. A difference below `1/255` relative is invisible to it. That is the right resolution for the question @@ -788,6 +1017,22 @@ A criterion that admitted the shift row would not be a criterion. the two stems also edits that checker's pinned-set test, which is #1578's file and not this row's, so it is left to whoever runs preflight next — the handoff that allowlist's own header describes. Owner: this row until it is deleted. +- **Most of the harness still runs nowhere but a lease.** + `tests/scripts/test_ltx25_pixel_ab_harness.py` exercises the memory + precondition and the arm-completeness check, which are extracted verbatim from + `scripts/ltx25-dit-attn-flash-pixel-ab.sh`. The render loop, the routing + assertion, the phase [I] call site and the phase [L] exit are pinned as TEXT + and nothing executes them, so a rewrite that reintroduced any of those defects + in different words would pass. That is a limit of where the file runs, not a + gap that another local test can close. Owner: this row. Issue: + [#1612](https://github.com/mudler/vllm.cpp/issues/1612). +- **`scripts/ltx25-render-compare.py` writes `Infinity` into its JSON.** A + bit-identical pair has zero MSE and infinite PSNR, and `json.dump` spells that + `Infinity`, which `json.load` reads back and a strict JSON parser refuses. Every + consumer here is Python, so it is disclosed rather than repaired: replacing it + with `null` or a sentinel would change what the report means for the one case + the whole comparison hopes to see. Owner: this row. Issue: + [#1612](https://github.com/mudler/vllm.cpp/issues/1612). - **True tensor cores at head_dim 128.** `vt::AttentionDenseFa2` refuses anything but head_dim 64 (`src/vt/cuda/cuda_flash_attn_fa2.cu:557-560`). Reaching the vendored FA-2 `mma.sync` path for LTX's head_dim 128 needs an diff --git a/.agents/verification.md b/.agents/verification.md index 856c3f45e..9994b9cfd 100644 --- a/.agents/verification.md +++ b/.agents/verification.md @@ -83,6 +83,42 @@ severity, the violated requirement, a reproduction, and the expected behavior. Do not take another agent's report at face value; the operator reruns the gate regardless of how confident the report sounded. +## Make the instrument say what it is measuring, in its own output, in words + +A criterion committed in advance protects against a threshold moved after the +fact. **It does not protect against an instrument pointed at the wrong thing.** +That failure returns a confident, structurally valid, correctly formatted answer +to a question nobody verified it was asking, and no threshold downstream can +catch it, because every number downstream is arithmetically correct. + +Four instances from this tree, and the shape is what makes the next one +recognisable: + +- A pixel A/B passed its comparison tool `--a naive --b flash --control + flash-ctl` while the tool compared the control against arm A. `flash-ctl` is a + repeat of FLASH, so the "run-to-run noise floor" was a second copy of the + treatment comparison. It would have read the same size as the delta it was + meant to calibrate, and the design's null verdict — "indistinguishable from + run-to-run nondeterminism" — would have been published whatever the kernel + did (`.agents/specs/ltx25-dit-attn-flash.md` §10.6). +- A governor reported `1.00 s`, `69.1 s`, `162 s` and `396.9 s` for one + quantity. Four well-formed answers, at most one about the thing asked for. +- An append-only checker read the WORKING TREE instead of the commits and + returned `rc=0` three times over a violation that was there. +- A `static_assert` compared a literal against itself and read `256 == 256`, + staying green when the constant it existed to pin changed. + +**Every one of those would have been caught in seconds by an instrument that +narrated its own comparison.** The governor never printed which estimator +produced its number. The append-only checker never printed whether it had read +commits or the working tree. The `static_assert` never printed which constant it +had captured. So the repair is the same each time, and it is cheap: have the +tool state, in its own output and in words, WHAT it compared against WHAT — and +where a caller chooses that, make the choice an argument with a name rather than +a convention the caller can invert silently. A reader who cannot see the wiring +in the report cannot audit it, and a reviewer reading the source instead is +reviewing the intent rather than the run. + ## Evidence Separate what you observed from what you inferred. Name source roots, versions, diff --git a/scripts/ltx25-dit-attn-flash-pixel-ab.sh b/scripts/ltx25-dit-attn-flash-pixel-ab.sh index 4c7885beb..557e1dcdf 100755 --- a/scripts/ltx25-dit-attn-flash-pixel-ab.sh +++ b/scripts/ltx25-dit-attn-flash-pixel-ab.sh @@ -38,6 +38,36 @@ # every thread and cost the recorded 47.84 s denominator ~3.2% (spec section # 7.1). Both arms here are instrumented identically and neither is sampled, so # the ratio needs no correction. +# +# BEFORE YOU SPEND A LEASE ON THIS, read these three paragraphs. +# +# EXIT STATUS. 0, 1 and 2 are the pixel comparison's own verdict and this job +# exits with it: 0 every threshold held, 1 a threshold failed, 2 an input could +# not be read and NOTHING was compared. A 2 is never a pass. Everything else is +# a refusal before the verdict exists: +# 23 checkpoint staging 25 ltx2-gen will not exec +# 31 source tarball 33 configure 34 build 35 artefacts +# 36 no CUTLASS 38 no complete CUDA toolkit +# 39 MemAvailable is below the start floor and stayed there +# 40 the swapped op is not in this source 41 the A/B knob is not either +# 43 the comparison tool is not in this source +# 44 the CUDA unit gate FAILED 45 the CUDA unit gate BINARY IS ABSENT +# 46 an arm did not route as its knob asked +# +# A 45 IS USUALLY THE STAGED BINARY CACHE, and the fix is one command. Phase +# [D] reuses `$W/pixab-bin` when its `SRC_SHA` matches, and it copies +# `test_ltx2_device` only `[ -s ]` -- a cache staged without it therefore +# satisfies the build skip and then has no correctness gate to run. This job +# refuses rather than rendering, because a routing assertion that cannot run, +# inside a job whose whole purpose is proving which kernel executed, is a +# silent hole. Clear `$W/pixab-bin` and the build regenerates both. +# +# RESUMING. `RUN_ID` is an environment override. Pass the same one and this +# run lands in the same `$OUT` and skips every arm that is already complete +# there. A reused arm's per-forward samples came from the earlier lease, so it +# is recorded as `timing_source=an-earlier-lease` and phase [H] states that the +# speed pair is not a same-lease pair. Its routing is still proved, from the +# log that arm already has. set -u T0=$(date +%s) say() { echo "[pixab +$(( $(date +%s) - T0 ))s] $*"; } diff --git a/tests/scripts/test_ltx25_pixel_ab_harness.py b/tests/scripts/test_ltx25_pixel_ab_harness.py index f16888d8f..a445b3d51 100644 --- a/tests/scripts/test_ltx25_pixel_ab_harness.py +++ b/tests/scripts/test_ltx25_pixel_ab_harness.py @@ -255,9 +255,11 @@ def test_the_unit_gate_refuses_rather_than_reporting(self) -> None: self.assertIn(f"exit {UNIT_GATE_ABSENT}", self.text) def test_the_heartbeat_is_reaped_on_a_lease_kill(self) -> None: - for sig in ("HUP", "INT", "TERM"): - self.assertIn(f"' {sig}", self.text.replace("'HUP", "' HUP")) + """`rc` reclaiming a device sends SIGTERM, and a bash EXIT trap does not + run for a signal with no handler of its own.""" self.assertIn("trap cleanup EXIT", self.text) + for sig, status in (("HUP", 129), ("INT", 130), ("TERM", 143)): + self.assertIn(f"trap 'cleanup; exit {status}' {sig}", self.text) if __name__ == "__main__": From f8d7c7d44df63734c2a0ca50b915058312aa2b19 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 22 Aug 2026 17:03:26 +0000 Subject: [PATCH 07/19] measure(#1709): 110.41 GiB on dgx:gpu0 belongs to no /proc/meminfo category, so #1612's renders cannot be taken MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit §10.7 recorded that the start gate was refusing at a flat 5.1 GiB and that the transient reading was falsified. This is the diagnostic that says what the condition actually is, taken read-only from inside a lease because the fleet rule forbids `ssh`: `rc` job `ab12aac1-b862-4ac6-8292-9f2c641e6a8d`. **114 GiB is "used" and 9.22 GiB of it can be named.** `AnonPages` 0.93, `Cached` 0.91, `Buffers` 0.05, `Shmem` 0.04, `Slab` 1.10, `VmallocUsed` 1.00, `MemFree` 5.23, against a `MemTotal` of 119.63. **110.41 GiB — 92.3% of the box — is unaccounted for by any field.** Every ordinary owner is excluded by measurement rather than by argument. No process: the sum of `VmRSS` over every visible `/proc/*/status` is 0.0 GiB. No tmpfs: `/dev/shm` is a 64 M mount containing nothing. Not a busy GPU: `nvidia-smi` reads 0%, 11 W, `No running processes found`. Not a container view artefact: the cgroup reports `memory.max=max` and `memory.current` 113 MiB. Not transient: 5.0 GiB at 15:49Z, 5.1 GiB flat through 16:39Z, 4.98 GiB at 16:55Z, across four leases by three submitters, on a box `up 2:33` at load 0.25. The remaining explanation fits every observation — on a unified-memory part the driver takes host RAM for the GPU, and an allocation that outlives its process is attributed to nothing userspace can see, `Memory-Usage: Not Supported` being exactly the meter that would have named it. It is recorded as a HYPOTHESIS. What is established is the gap and the exclusions, and confirming the mechanism needs host access this row does not have. The consequence does not depend on the mechanism, which is why the row can act on it. Nothing a lease can do repairs this: there is no process to kill and no file to delete. So the pixel A/B is `PENDING` on a named external resource, which is a result under AGENTS.md rather than a synonym for "probably fine", and this row will not take the renders on another box — §7's denominator argument binds, and a ratio against a different GPU is a different measurement rather than this one taken late. Three leases and no wrong number among them, which is the harness's doing: the start gate logs EVERY poll rather than only its refusal, and a flat line across two leases is the only reason "persistent" can be said instead of "busy". FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-opus-5 [Claude Code] --- .agents/specs/ltx25-dit-attn-flash.md | 47 +++++++++++++++++++++++++++ 1 file changed, 47 insertions(+) diff --git a/.agents/specs/ltx25-dit-attn-flash.md b/.agents/specs/ltx25-dit-attn-flash.md index 2bd77c7a1..97a7ec551 100644 --- a/.agents/specs/ltx25-dit-attn-flash.md +++ b/.agents/specs/ltx25-dit-attn-flash.md @@ -922,6 +922,53 @@ and left the cause open. A read-only diagnostic to identify the resident allocation is with the operator, and #1709 owes the correction. The renders cannot be taken on that box until it is resolved, and no arm has run. +**THE DIAGNOSTIC IS TAKEN, AND THE MEMORY BELONGS TO NOBODY.** `rc` job +`ab12aac1-b862-4ac6-8292-9f2c641e6a8d`, read-only, 2026-08-22T16:55:46Z, run +from inside a lease because the fleet rule forbids `ssh`. Every ordinary owner +is excluded by measurement rather than by argument: + +| field | value | +|---|---| +| `MemTotal` | 119.63 GiB | +| `MemFree` | 5.23 | +| `AnonPages` | 0.93 | +| `Cached` | 0.91 | +| `Buffers` | 0.05 | +| `Shmem` | 0.04, and it sits inside `Cached` rather than beside it | +| `Slab` | 1.10 | +| `VmallocUsed` | 1.00 | +| **accounted** | **9.22 GiB** | +| **UNACCOUNTED** | **110.41 GiB, 92.3% of the box** | + +No process: `ps aux --sort=-rss` lists the `rc` worker at 11 MB and two zombies, +and the sum of `VmRSS` over every visible `/proc/*/status` is **0.0 GiB**. No +tmpfs: `/dev/shm` is a 64 M mount containing nothing, `du` 0. Not the GPU being +busy: `nvidia-smi` reads `0%`, `11W`, `No running processes found`. Not a +container view artefact: the cgroup reports `memory.max=max` and +`memory.current` 113 MiB. Not transient: 5.0 GiB at 15:49Z, 5.1 GiB flat through +16:39Z, 4.98 GiB at 16:55Z, across four leases by three submitters, on a box +`up 2:33` at load average 0.25. + +On a unified-memory part the driver takes host RAM for the GPU, and an +allocation that outlives its process is attributed to nothing userspace can see +— which fits every observation, including `Memory-Usage: Not Supported` being +the one meter that would have named it. **That is a hypothesis and this row does +not assert it.** What is established is the 110.41 GiB gap and the exclusions. + +The operational consequence does not depend on the mechanism. **Nothing a lease +can do repairs it** — there is no process to kill and no file to delete — so +this row's measurement is `PENDING` on a named external resource, which is a +result under AGENTS.md and never a synonym for "probably fine". `dgx:gpu0` has +continued to report `ready` and hand out leases throughout, which is the +controller half of #1709. + +**Three leases spent and no wrong number among them.** `5fb9399f` lost its +worker to an OOM during a build started against 5 GiB; `2ccd1acf` waited its +full 1200 s at a flat 5.0 GiB and refused with exit 39; `ab12aac1` measured the +box. The one thing this row will not do is take the renders somewhere else: §7's +denominator argument binds, and a ratio against a different GPU is not this +measurement taken late, it is a different measurement. + When it lands, this section records: the three arms' frame counts and routing proofs, the C0 block, V1 to V4 and A1 to A2 with their headroom, the control ratio `R` and which of §10.5's readings it selects, the cross-check against the From 3e2961ef007806701fa8448d6f2cb60fb8336fe9 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 22 Aug 2026 17:04:09 +0000 Subject: [PATCH 08/19] record(#1709): the index row and the two halves this row does not own MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `scripts/check-agent-record.py` counts index rows that name neither an owning row nor a spec that lists the issue under `## Owed`, and refuses a count above the recorded mark. Filing without saying who owns it is a gate failure rather than a habit, so #1709 gets both: an appended index row and an `## Owed` entry. The row is appended at the end and edits nothing, because the index is append-only and carries `merge=union` — GitHub holds the open and closed state, and an edited row is duplicated rather than merged. The split is the point of the entry. The half this row owns is fixed in flow: the harness's `MemAvailable` start gate. The two halves that are not this row's are named rather than folded in — a device-readiness condition in the controller, which would have parked three jobs instead of spending them, and a line in the DGX profile saying a granted lease does not imply a reclaimed box. Neither is a repair this row can make, and an issue with no named owner is the thing the checker exists to refuse. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-opus-5 [Claude Code] --- .agents/issue-index.md | 1 + .agents/specs/ltx25-dit-attn-flash.md | 9 +++++++++ 2 files changed, 10 insertions(+) diff --git a/.agents/issue-index.md b/.agents/issue-index.md index f19ddcfb1..0c3863b61 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -620,3 +620,4 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#1684](https://github.com/mudler/vllm.cpp/issues/1684) | `VT-CONV1D-TIME-BLOCK` | **No per-MODEL suite exercises a `blocks > 1` shape, so a defect confined to the time-blocked axis reds the op's suite and nothing else.** Measured by the fresh review of [PR #1678](https://github.com/mudler/vllm.cpp/pull/1678) rather than argued: mutation M7b sign-flips every output cell of `Conv1dKernel` when `blocks > 1` -- exactly and only the axis the row adds -- and at head `e1396fc73` EIGHT of the ten suites stayed GREEN (`test_vocoder1d`, `test_bigvgan`, `test_minimax_music3_acoustic`, `test_ltx2_vae`, `test_minimax_h3`, `test_indextts2_pipeline`, `test_indextts2_family`, `test_ops_conv1d_depthwise`), with only `test_ops_conv1d_general` and `test_host_parallel` red. All four audio consumers reach `vt::Conv1d`; every one of them does so at SINGLE-BLOCK shapes only. The row disclosed the gap in §6c and compensated with a geometry gate that asserted `blocks > 1` on the vocoder's shapes -- but that gate read SIX hard-coded shape constants transcribed by hand from `minimax_music3_loader.h:253-265`, so a loader change would have left it green while the production shapes collapsed to one block. PARTLY FIXED IN FLOW, both halves: the geometry gate now DERIVES its shapes by walking `MiniMaxMusic3VocoderConfig` and `kVocoderResidualDilations`/`kVocoderResidualUnits`, and the derivation is proved by mutation -- setting `upsampling_ratios{1,1,1,1}` reds the derived gate (1 case / 16 assertions, rc 1) and left the hand-transcribed one at 14 cases / 19 615 assertions / `SUCCESS!`, rc 0, completely blind; and `tests/vllm/models/test_vocoder1d.cpp` gained `vocoder1d Conv1d is exact ACROSS a time block boundary`, which enters through `vllm::vocoder1d::Conv1d` -- the body all four audio models call -- at 32 channels, kernel 7, 10 000 positions, asserts the block length is shorter than the output length so the case cannot silently become single-block, and reds under M7b at the repaired head (1 case / 4 assertions, rc 1). STILL OPEN: the four MODEL suites reach the provider at single-block shapes only, so M7b still leaves them green. Closing it means lengthening each consumer's reduced-dimension fixture until its convolutions cross a block boundary, which moves those fixtures' goldens -- a fixture change per model, not a test addition. Owned by [`vt-conv1d-time-block.md`](specs/vt-conv1d-time-block.md) `## Owed` | verification | | [#1685](https://github.com/mudler/vllm.cpp/issues/1685) | `SPEC-DFLASH2` | **The DFlash2 speed run's denominator declares `TRITON_ATTN` while five of its layers resolve `FlashAttentionBackend`, which `FA-CONSTRAINT.txt` says cannot target sm_12x.** Observed by the run that produced the first DFlash2 speed ratio (`dgx:gpu0`, `rc` job `ec9cf6cd-0aaf-4323-806d-6a12da2bd08f`, 2026-08-22, gate tree `d25730fbb`, `GATE_RC=0`, ours/vLLM = 0.8016987337853048 on `output_throughput_tok_s`). The gate did NOT refuse; this is a question about what the denominator IS, not a gate failure. `evidence/vllm-arm.json` records the scalar `attention_backend: TRITON_ATTN` from `attention_backend_source: read_back_from_engine`, and beside it the per-group census read off `...model_runner.attn_groups`: `GDNAttentionBackend` 48 layers, `TritonAttentionBackend` 16, and `FlashAttentionBackend` 5 -- the five being `model.layers.64-68.self_attn.attn`, i.e. the DFlash2 draft's sliding-window layers. `/workspace/oracle-dflash2/FA-CONSTRAINT.txt` records `FA_USABLE=0` for sm_12x from [#1456](https://github.com/mudler/vllm.cpp/issues/1456), where a real source build at `CUDA_ARCHS=12.0` emitted `sm_80` for `_vllm_fa2_C` and `sm_75` for `_vllm_fa3_C`, and `TRITON_ATTN` is the declared backend on this box BECAUSE of that. The engine nevertheless loaded, generated 64 tokens on every one of its 20 legs, and produced a clean 83-sample clock window. Three readings and none established: upstream intends the draft's sliding-window layers to select FA independently of the engine-wide choice; the label over-describes what executes and those layers fall back at runtime; or FA genuinely runs on sm_121 through driver JIT of the emitted `sm_80` PTX, in which case #1456's CONCLUSION -- not its measurement -- needs re-reading. This is the THIRD independent observation of FA resolving on this box against `FA_USABLE=0`. Settled by reading the pinned wheel's own selection path for sliding-window and draft layers and tracing whether those five layers dispatch FA kernels or fall back; either outcome is cheap and changes what the scalar should say. Recorded under `## Owed` O26 residual 1 of `.agents/specs/dflash2-spec-decode.md` and in the 2026-08-22 entry of `.agents/benchmark-record.md`. Related: [#1658](https://github.com/mudler/vllm.cpp/issues/1658) (why the per-group map is recorded beside the scalar at all: one string cannot describe 48 + 16 + 5) and [#1673](https://github.com/mudler/vllm.cpp/issues/1673) (the other open caveat on the same ratio). Evidence: `/mnt/nas_share/rc/dflash2-1673/out-n1673b/evidence/{vllm-arm.json,clock-vllm.json}` and `out-n1673b/m-gate.log`. | bug | | [#1661](https://github.com/mudler/vllm.cpp/issues/1661) | `TEST-SHELLCHECK-SKIP-GUARD` | **`test_script_stays_shellcheck_clean` ERRORs instead of skipping on hosts without the shellcheck binary: the guard probes `returncode`, but a missing binary raises `FileNotFoundError` before a returncode exists**, so the `tools suites` preflight gate is red on such hosts (`thalia`, aarch64). The test predates the window (last touched `b95543c44`); it surfaced when current main's preflight started running the module. FIXED IN FLOW by the same change: the probe gains the `except FileNotFoundError` skip arm (message unchanged), and both arms are proven — no-binary host skips via the new arm, a PATH-shim `shellcheck` exiting 3 still skips via the returncode arm, and unmodified main reds on this host | bug | +| [#1709](https://github.com/mudler/vllm.cpp/issues/1709) | `LTX25-DIT-ATTN-FLASH` | **`dgx:gpu0` holds 110.41 GiB that belongs to no `/proc/meminfo` category, and `rc` keeps handing out leases against it.** Measured read-only from inside a lease (`rc` job `ab12aac1-b862-4ac6-8292-9f2c641e6a8d`, 2026-08-22T16:55:46Z), because the fleet rule forbids `ssh`: `MemTotal` 119.63 GiB, `MemFree` 5.23, `AnonPages` 0.93, `Cached` 0.91, `Buffers` 0.05, `Shmem` 0.04, `Slab` 1.10, `VmallocUsed` 1.00 -- **accounted 9.22 GiB, UNACCOUNTED 110.41 GiB, 92.3% of the box.** Every ordinary owner is excluded by measurement and not by argument: the sum of `VmRSS` over every visible `/proc/*/status` is 0.0 GiB; `/dev/shm` is a 64 M mount containing nothing; `nvidia-smi` reads 0%, 11 W, `No running processes found`; the cgroup reports `memory.max=max` and `memory.current` 113 MiB; and the value is 5.0 GiB at 15:49Z, 5.1 GiB flat through 16:39Z and 4.98 GiB at 16:55Z across four leases by three submitters, on a box `up 2:33` at load 0.25. The leading explanation is a driver-held unified-memory allocation that outlived its process, which fits every observation including `Memory-Usage: Not Supported` being the one meter that would have named it -- recorded as a HYPOTHESIS, since confirming it needs host access. COST: three leases on #1612 -- `5fb9399f` lost its worker to an OOM during a build started against 5 GiB, `2ccd1acf` waited its full 1200 s at a flat 5.0 GiB and refused with exit 39, `ab12aac1` measured the box. FIXED IN FLOW, the half this row owns: `scripts/ltx25-dit-attn-flash-pixel-ab.sh` gains a `MemAvailable` start gate that waits, logs EVERY poll and refuses by name -- and logging every poll is the only reason the condition can be called persistent rather than busy. NOT FIXED IN FLOW, and named rather than folded in: a device-readiness condition in the controller, which would have parked every one of these jobs instead of spending them, and a line in the DGX profile saying a granted lease does not imply a reclaimed box. Owner: row `LTX25-DIT-ATTN-FLASH`, under `## Owed` in [`ltx25-dit-attn-flash.md`](specs/ltx25-dit-attn-flash.md) | bug | diff --git a/.agents/specs/ltx25-dit-attn-flash.md b/.agents/specs/ltx25-dit-attn-flash.md index 97a7ec551..201762963 100644 --- a/.agents/specs/ltx25-dit-attn-flash.md +++ b/.agents/specs/ltx25-dit-attn-flash.md @@ -1036,6 +1036,15 @@ ratio `R` and which of §10.5's readings it selects, the cross-check against the ## Owed +- **`dgx:gpu0` holds ~110 GiB that belongs to no `/proc/meminfo` category, and + the controller keeps handing out leases against it.** + [#1709](https://github.com/mudler/vllm.cpp/issues/1709), measured in §10.7. + The half this row owns is fixed in flow: the harness's `MemAvailable` start + gate waits, logs every poll and refuses by name. Two halves are NOT this + row's and are named rather than folded in — a device-readiness condition in + `rc`, which would have parked three jobs instead of spending them, and a line + in `.agents/environment.md`'s DGX profile saying a granted lease does not + imply a reclaimed box. Owner: this row until the controller half has one. - **There is NO numeric or pixel comparison at production geometry.** **§10 is the design that discharges this, and it is committed before the renders are taken.** The result lands in §10.7. Until it does, the statement From 12c880a52cfdf99a935a2a124efdf21cedf28b37 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 22 Aug 2026 20:48:31 +0000 Subject: [PATCH 09/19] fix(#1612): the control's own content was never judged, and a control with no picture in it published the STRONGER null A fresh review raised four findings against the pixel A/B, while the renders were running in an `rc` lease and before any number had been read. All four are repaired here, and every change tightens or clarifies a criterion. No threshold is widened, moved or deleted: every V and A bound in section 10.4 is byte-for-byte what it was. THE CONTROL'S CONTENT WAS UNJUDGED. `arm_content(args.control)` was computed, printed, and never entered `checks`, which built its three C0 checks for arms A and B alone. Reproduced: a healthy flash/naive pair with a control of six one-colour frames exits 0, verdict PASS, `R = 113.10` -- which section 10.5 reads as `R >= 0.5`, "indistinguishable from run-to-run nondeterminism", the stronger of its two null readings. A control that rendered nothing at all upgraded the published conclusion. This is section 10.6's class one step removed, and section 10.4's own C0 rationale -- a difference cannot tell two good renders from two identically broken ones -- applies to the control-vs-arm comparison with identical force. Folding the control into `checks` would have replaced one mis-reading with a second: section 10.5 maps any failing check to "visibly different", a finding about a change already on `main`, and a degenerate control is not that. It is a broken experiment. So it is a third status. Every check now carries `judges`, `treatment` or `control`; a failing control check exits 3 with the verdict token `CONTROL_DEGENERATE`, which is neither PASS nor FAIL and cannot be read as either; `R` is still computed and printed and carries a stated reason why it may not be read. A treatment FAIL outranks it, because "the two renders differ" is established without the control and a broken control must not swallow it. The report now says in words which list decides what, per the rule this row added to `.agents/verification.md`. Phase [L] of the harness defines status 3 rather than calling it UNKNOWN. `test_control_arm_is_reported_separately` and `test_the_control_to_treatment_ratio_is_computed_not_eyeballed` both asserted `assertNotIn("control", ...)` over the check names, so the suite actively pinned the hole. Both are narrowed to pin what they meant: the RATIO is not a check. "POPULATION 2, GATED" WAS A LABEL AND NOT A FACT. The eight cells of section 10.4's fixture table were accurate and nothing asserted them: the tests asserted `mean < 0.1`, `psnr > 50`, `ssim > 0.99` and four booleans, bounds those cells clear by two orders of magnitude. Demonstrated by raising `make_render`'s `motion` from 3 to 4, which moved the table and left 43 of 43 tests passing. All eight are now pinned to the precision printed, and three independent mutations of the generator each red the suite. THE TRIPWIRE COUNT HAD DRIFTED. `Wiring` said "the four call sites" while defining five tests, and the two the prose omitted -- the phase [F] unit-gate refusal and the signal traps -- were absent from section 10.8 and `## Owed` as well. The error was in the safe direction; it was still a number that no longer described the file beside it. The count is corrected in all three places and a new gate holds the docstring against the class, which fired when the sixth tripwire was added. ONE SURVIVING MUTATION, and it was a redundant guard rather than a defect. Deleting `[ -d "$d" ] || return 1` from `arm_is_complete` left the harness suite green, because the glob does not expand for an absent directory and the frame count refuses the arm anyway. Observable behaviour was identical either way, so the line is removed on the same argument section 10.4 already made against C0's fourth check. Both remaining guards are proved observable by their own mutation. M29 IS DISCLOSED. Neutering the wait-budget comparison makes the memory gate loop forever, exactly as inverting its floor comparison does, and only the latter was disclosed. Both are verified as timeouts here, on the call shapes the suite actually uses. The hang is not engineered away: a budget that gives up early to keep a test suite responsive is a weaker gate on a four-hour lease. Suites: `test_ltx25_render_compare` 37 -> 45 cases, `test_ltx25_pixel_ab_harness` 19 -> 22, both green. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-opus-5 [Claude Code] --- .agents/specs/ltx25-dit-attn-flash.md | 168 +++++++++++---- scripts/ltx25-dit-attn-flash-pixel-ab.sh | 25 ++- scripts/ltx25-render-compare.py | 185 ++++++++++++---- tests/scripts/test_ltx25_pixel_ab_harness.py | 63 +++++- tests/scripts/test_ltx25_render_compare.py | 213 ++++++++++++++++++- 5 files changed, 568 insertions(+), 86 deletions(-) diff --git a/.agents/specs/ltx25-dit-attn-flash.md b/.agents/specs/ltx25-dit-attn-flash.md index 201762963..1cd2b4c87 100644 --- a/.agents/specs/ltx25-dit-attn-flash.md +++ b/.agents/specs/ltx25-dit-attn-flash.md @@ -708,7 +708,7 @@ them by hand afterwards. | # | check | threshold | where it comes from | |---|---|---|---| -| C0 | each arm, ON ITS OWN: no near-uniform frame, every frame hash distinct, no zero-motion pair | all three, per arm | **the hole every difference-only comparison has** | +| C0 | each RENDER, ON ITS OWN — arm A, arm B and the control: no near-uniform frame, every frame hash distinct, no zero-motion pair | all three, per render | **the hole every difference-only comparison has** | | V1 | mean \|delta\|, 8-bit RGB | `<= 1.0` level | one level is the quantisation step of the artefact itself; a mean below it says the average pixel is within the PPM's own resolution | | V2 | worst-frame PSNR | `>= 40 dB` | the video-coding "visually lossless" convention. This experiment did not choose it | | V3 | worst-frame SSIM | `>= 0.99` | Wang et al. 2004, 11x11 Gaussian sigma=1.5 on luma. 0.98 is the usual transparency line; this is stricter, and it is the WORST frame rather than the mean | @@ -722,8 +722,17 @@ renders from two identically broken ones. Two all-black renders differ by zero, score infinite PSNR and SSIM `1.000000`, and would read as the strongest possible pass this table can produce. A run that exited 0 having written frames that were all one colour has happened in this repository, so that is a recorded -failure mode rather than a hypothesis. C0 is computed per arm before anything is -subtracted, and it is checked first. +failure mode rather than a hypothesis. C0 is computed per render before anything +is subtracted, and it is checked first. + +**C0 covers the CONTROL, and it did not at first.** The control is the third +render and it is the one every reading in §10.5 is built on, and its content was +computed, printed and never checked. Its three checks are registered like the +arms', and they drive a DIFFERENT outcome: a failing arm check is a treatment +FAIL at exit 1, and a failing control check is exit 3, `CONTROL_DEGENERATE`, +because a control with no picture in it is a broken experiment rather than a +visible difference between two renders. §10.5 records the reading that status +selects and the repair's mid-flight timing. **"Frames written" used to be a fourth C0 check and it is now a REFUSAL.** An arm directory holding no `frame_*.ppm` leaves the tool at exit 2 with no JSON, @@ -771,8 +780,30 @@ can compute. These are the numbers that are a gate: | perturbation of a FIXTURE frame | mean \|d\| | worst PSNR | worst SSIM | V4 ratio | verdict | |---|---|---|---|---|---| | none | 0 | inf | 1.000000 | 0.000 | bit-identical | -| +/-1 LSB on 3% of samples | 0.0194 | 65.0 dB | 0.99999 | 0.0007 | **PASS**, all four | -| one pixel of global horizontal shift | 16.6120 | 21.8 dB | 0.77036 | 0.4487 | **FAIL**, all four | +| +/-1 LSB on 3% of samples | 0.01935944733796296 | 65.04621554883309 dB | 0.99998566099925 | 0.0007008817611029302 | **PASS**, all four | +| one pixel of global horizontal shift | 16.611979166666668 | 21.789085745489977 dB | 0.7703600993461216 | 0.44871294988741245 | **FAIL**, all four | + +**"GATED" was a label and not a fact until this revision, and a fresh review +found it.** The eight cells of the two lower rows were reproducible and +accurate — recomputed independently, all eight — and no test asserted any of +them. `test_dither_passes_every_video_check` asserted `mean < 0.1`, `psnr > 50` +and `ssim > 0.99`, bounds every one of those cells clears by two orders of +magnitude, and `test_one_pixel_shift_fails_all_four_video_checks` asserted four +booleans. A change to the fixture generator therefore moved all eight numbers +with the suite green: raising `make_render`'s `motion` from 3 to 4 left 43 of 43 +tests passing while this table went silently wrong. A fixture that failed V4 by a +hair and one that fails it by 4.5x are the same test to a boolean, and the +argument two paragraphs below rests on which of those it is. + +The eight cells are now asserted to the precision printed, by +`Discrimination.test_the_dither_row_of_the_gated_table_is_pinned` and +`Discrimination.test_the_one_pixel_shift_row_of_the_gated_table_is_pinned`, at +the precision this file already pins its three SSIMs to. Three independent +mutations of the generator — the motion step, the dither seed, the texture +period — each move at least one cell and each red the suite. The `none` row +needs no separate assertion: a bit-identical pair registers `video.bit_identical` +and no threshold at all, which +`test_identical_renders_read_as_bit_identical` already pins. **Why they differ, and why both belong.** The fixture is a sine grating with a 4-pixel period plus uniform noise, so one pixel of shift moves each row by a @@ -836,6 +867,44 @@ number. - **`R` is `null`** — the two arms are bit-identical, which §10.2 says they will not be. Read that as a finding about the experiment (one arm rendered twice, a knob not read, a cached artefact) before reading it as a result. +- **`R` is marked UNUSABLE and the tool exits 3** — the CONTROL failed its own + C0 content checks. None of the four readings above is available, because every + one of them assumes the control is a repeat of a render and this one is a + repeat of no picture. The treatment result stands and is reported; the + *reading* of it does not exist, and the control arm has to be re-taken before + one does. This is not "visibly different" and it is not a pass: it is a broken + experiment, and it has its own status so that it can be neither. +- **Any check fails** — the verdict is **visibly different**, and that is a + finding about a change already on `main`, not a failure of this work. It owes + an issue naming what diverged and by how much, and it does not owe a widened + threshold. Widening a gate to admit a change is the failure this protocol + exists to prevent, and §9's stop conditions already say so for the numeric + gate. This branch outranks the one above it: a failing check and a degenerate + control together exit 1, because "the two renders differ" is established + without the control at all and a broken control must not swallow it. + +**THE CONTROL'S CONTENT WAS UNJUDGED, AND THAT IS REPAIRED HERE, MID-FLIGHT.** +`arm_content(control)` was computed, printed, and never registered as a check. +Arms A and B were judged on their own content and the control was not, so a +control of six one-colour frames left the tool at exit 0, verdict `PASS`, with +`R = 112.77` — the second branch above, **indistinguishable from run-to-run +nondeterminism**, which is the *stronger* of the two null readings. A control +that rendered no picture at all upgraded the published conclusion. That is +§10.6's class one step removed, and §10.4's own C0 rationale — a difference +cannot tell two good renders from two identically broken ones — applies to the +control-vs-arm comparison with identical force. "Reported, never checked" was +argued for the ratio `R`, never for the control's content, and §10.8 does not +list the control as exempt. + +**The timing has to be said plainly, because this section's whole standing is +that it was committed before the numbers.** A fresh review raised this while the +renders were running in an `rc` lease and **before any number had been read**. +The criterion is therefore edited after the run started and before the run +reported, and the edit only ever TIGHTENS: it adds a way for a run to fail that +did not exist, removes none, and widens no threshold. Every V and A bound in +§10.4 is byte-for-byte what it was. A reader entitled to be suspicious of a +criterion edited mid-flight should check exactly that, and `git log -p` on this +file is where. **WHICH ARM the control repeats is now an argument, not a convention.** `--control-of {a,b}` names it, the JSON records it, and the control block prints @@ -844,12 +913,6 @@ the tool compared the control against arm A, so the "run-to-run noise floor" was a SECOND naive-vs-flash comparison: it necessarily read about the same size as the treatment, and the second branch above would have been published whatever the kernel did. A silent convention a caller can invert is not a convention. -- **Any check fails** — the verdict is **visibly different**, and that is a - finding about a change already on `main`, not a failure of this work. It owes - an issue naming what diverged and by how much, and it does not owe a widened - threshold. Widening a gate to admit a change is the failure this protocol - exists to prevent, and §9's stop conditions already say so for the numeric - gate. ### 10.6 The failure this design nearly shipped: a well-formed answer about the wrong thing @@ -998,26 +1061,56 @@ ratio `R` and which of §10.5's readings it selects, the cross-check against the extracts the `MemAvailable` reader, the start gate and the arm-completeness check verbatim from `scripts/ltx25-dit-attn-flash-pixel-ab.sh` and runs them against a fabricated `/proc/meminfo`, so those three execute here. **The render - loop, the routing assertion, the phase [I] call site and the phase [L] exit do - not.** They are pinned as TEXT — the suite asserts the exact call site that - shipped inverted, and the exact `exit` lines — and a text assertion is a - tripwire, not a proof: it catches the inversion that happened and would not - catch a rewrite that reintroduced it in different words. `dgx:gpu0` under a - lease is the only place those lines run, which is why every one of them was - wrong at once: they had never executed anywhere a test could watch. **That is - a structural explanation of a cluster rather than four coincidences**, and it - tells the next reader which claims in this harness are load-bearing and which - are decorative — what a test can execute is now checked, and what only a lease - can execute is a comment to be verified against the run's own log. - - **One mutation of the memory gate is deliberately a TIMEOUT rather than a - failed assertion.** Inverting its floor comparison makes the gate wait out its - whole budget instead of proceeding, so the suite never returns. The mutation - runner records that as RED with the reason named, because a suite that did not - report `OK` is not a suite that passed. A timeout read as success is the exact - silent hole a wait-for-quiet loop already cost this campaign once, so that - path was exercised on purpose rather than left for whoever next inverts that - line. + loop, the routing assertion, the phase [I] call site, the phase [L] exit, the + phase [F] unit-gate refusal and the signal traps do not.** Five of those six + are pinned as TEXT — the suite asserts the exact call site that shipped + inverted, the exact `exit` lines, the two unit-gate statuses and the four + `trap` lines — and a text assertion is a tripwire, not a proof: it catches the + inversion that happened and would not catch a rewrite that reintroduced it in + different words. **The render loop is pinned by nothing at all.** `dgx:gpu0` + under a lease is the only place those lines run, which is why every one of them + was wrong at once: they had never executed anywhere a test could watch. **That + is a structural explanation of a cluster rather than a run of coincidences**, + and it tells the next reader which claims in this harness are load-bearing and + which are decorative — what a test can execute is now checked, and what only a + lease can execute is a comment to be verified against the run's own log. + + **This list and the tripwire count had both drifted, and a fresh review caught + it.** The prose named four unexecuted things while the `Wiring` class defined + five tests, and the two the prose omitted — the phase [F] unit-gate refusal and + the signal traps — were text-only and absent from `## Owed` as well. The error + was in the SAFE direction: nothing claimed as executed was in fact only + text-pinned. It was still a number in a document that no longer described the + file beside it, so + `TheDisclosureCountsWhatIsThere.test_the_wiring_docstring_names_as_many_tripwires_as_it_defines` + now holds the count against the class and a sixth tripwire cannot be added + silently. It fired on exactly that when the exit-3 tripwire was added. + + **Two mutations of the memory gate are deliberately a TIMEOUT rather than a + failed assertion, and both are named because one of them was not.** Inverting + the floor comparison (M27) makes the gate wait out its whole budget instead of + proceeding, and neutering the wait-budget comparison + `[ "$waited" -ge "$budget" ]` so that it never fires (M29) makes the loop run + forever on a box that never recovers. Either way the suite never returns. The + mutation runner records that as RED with the reason named, because a suite that + did not report `OK` is not a suite that passed. A timeout read as success is + the exact silent hole a wait-for-quiet loop already cost this campaign once, so + that path is exercised on purpose rather than left for whoever next touches + either line. **The hang is not engineered away**, and that is a choice: a + budget that gives up early to keep a test suite responsive is a weaker gate on + a four-hour lease than one that waits, and the guard's whole purpose is to + wait. What is owed is the disclosure, and M29 had not carried one. + +- **A GUARD THAT NO MUTATION COULD REACH, now removed rather than kept.** + `arm_is_complete` opened with `[ -d "$d" ] || return 1`, and deleting it left + the harness suite green: the glob does not expand for a directory that is not + there, so `ls "$d"/frame_*.ppm | wc -l` reports 0 and the frame-count check + refuses the arm on its own. Observable behaviour was identical with the line + and without it, so it was a redundant guard rather than a defect — and it is + the same argument §10.4 already made against C0's fourth check. It is gone, the + two remaining guards are each proved observable by their own mutation, and + `test_an_absent_directory_is_not_complete` now says in its docstring which line + does the refusing, so the next reader adds it back deliberately or not at all. - **PPM is 8-bit.** The comparison is on the artefact the pipeline writes, which is already quantised from the VAE's float output. A difference below `1/255` relative is invisible to it. That is the right resolution for the question @@ -1076,11 +1169,14 @@ ratio `R` and which of §10.5's readings it selects, the cross-check against the - **Most of the harness still runs nowhere but a lease.** `tests/scripts/test_ltx25_pixel_ab_harness.py` exercises the memory precondition and the arm-completeness check, which are extracted verbatim from - `scripts/ltx25-dit-attn-flash-pixel-ab.sh`. The render loop, the routing - assertion, the phase [I] call site and the phase [L] exit are pinned as TEXT - and nothing executes them, so a rewrite that reintroduced any of those defects - in different words would pass. That is a limit of where the file runs, not a - gap that another local test can close. Owner: this row. Issue: + `scripts/ltx25-dit-attn-flash-pixel-ab.sh`. Six things it does not execute: + the render loop, the routing assertion, the phase [I] call site, the phase [L] + exit, the phase [F] unit-gate refusal and the signal traps. Five of the six are + pinned as TEXT and nothing executes them, so a rewrite that reintroduced any of + those defects in different words would pass; the render loop is pinned by + nothing. That is a limit of where the file runs, not a gap that another local + test can close. This list said "four" and omitted the unit-gate refusal and + the traps until a fresh review counted them. Owner: this row. Issue: [#1612](https://github.com/mudler/vllm.cpp/issues/1612). - **`scripts/ltx25-render-compare.py` writes `Infinity` into its JSON.** A bit-identical pair has zero MSE and infinite PSNR, and `json.dump` spells that diff --git a/scripts/ltx25-dit-attn-flash-pixel-ab.sh b/scripts/ltx25-dit-attn-flash-pixel-ab.sh index 557e1dcdf..00777ebd4 100755 --- a/scripts/ltx25-dit-attn-flash-pixel-ab.sh +++ b/scripts/ltx25-dit-attn-flash-pixel-ab.sh @@ -41,10 +41,12 @@ # # BEFORE YOU SPEND A LEASE ON THIS, read these three paragraphs. # -# EXIT STATUS. 0, 1 and 2 are the pixel comparison's own verdict and this job -# exits with it: 0 every threshold held, 1 a threshold failed, 2 an input could -# not be read and NOTHING was compared. A 2 is never a pass. Everything else is -# a refusal before the verdict exists: +# EXIT STATUS. 0, 1, 2 and 3 are the pixel comparison's own verdict and this +# job exits with it: 0 every threshold held, 1 a threshold failed, 2 an input +# could not be read and NOTHING was compared, 3 the treatment passed and the +# CONTROL failed its own content checks, so the pass cannot be READ. A 2 is +# never a pass and neither is a 3. Everything else is a refusal before the +# verdict exists: # 23 checkpoint staging 25 ltx2-gen will not exec # 31 source tarball 33 configure 34 build 35 artefacts # 36 no CUTLASS 38 no complete CUDA toolkit @@ -168,9 +170,18 @@ wait_for_memory() { # $1 floor GiB, $2 budget s, $3 poll s # non-empty wav. A partial arm is re-rendered from scratch rather than resumed # mid-flight: the engine deletes stale frame_*.ppm in its own output directory # but nothing else there, so a half-arm's leftovers would outlive it. +# +# TWO CHECKS, and there used to be a third. `[ -d "$d" ] || return 1` stood on +# the first line and no mutation could reach it: deleting it left this suite +# green, because the glob does not expand for a directory that is not there, so +# `ls ... | wc -l` reports 0 and the frame count refuses the arm anyway. That is +# the same argument section 10.4 already made against the fourth C0 check in +# `ltx25-render-compare.py` -- a guard that cannot be observed to fail is a +# decoration -- so it is removed rather than kept for the look of it. An absent +# directory is still refused, by the line below, and +# `test_an_absent_directory_is_not_complete` says which line does it. arm_is_complete() { # $1 dir, $2 wanted frame count local d=$1 want=$2 n - [ -d "$d" ] || return 1 n=$(ls "$d"/frame_*.ppm 2>/dev/null | wc -l) [ "$n" = "$want" ] || return 1 [ -s "$d/audio.wav" ] || return 1 @@ -675,6 +686,10 @@ case "$PIXEL_RC" in 1) say "PIXEL VERDICT: FAIL -- a registered threshold was not met. Section 10.5: this is a" say " finding about a change already on main, and it does not owe a widened gate";; 2) say "PIXEL VERDICT: UNREADABLE -- nothing was compared. This is never a pass";; + 3) say "PIXEL VERDICT: CONTROL DEGENERATE -- every registered threshold held, and the" + say " control rendered no picture, so it is not a noise floor and section" + say " 10.5's R cannot be read. The renders are a PASS nobody may READ:" + say " re-take the control arm, and do not publish a reading from this run";; *) say "PIXEL VERDICT: UNKNOWN -- the comparison exited $PIXEL_RC, which it does not define";; esac say "DONE OUT=$OUT RUN_ID=$RUN_ID PIXEL_COMPARE_RC=$PIXEL_RC" diff --git a/scripts/ltx25-render-compare.py b/scripts/ltx25-render-compare.py index 85eee50ca..5bc22f7ce 100755 --- a/scripts/ltx25-render-compare.py +++ b/scripts/ltx25-render-compare.py @@ -71,11 +71,29 @@ result rather than between passing and failing. It is `null`, with a reason, when the treatment is bit-identical and the denominator is therefore zero. +THE CONTROL'S OWN CONTENT IS JUDGED, and that is separate from the ratio being +unchecked. C0 asks of each arm "is there a picture in it", and the same question +has to be asked of the control, for the same reason section 10.4 gives for arms +A and B: a difference cannot tell two good renders from two identically broken +ones. It was asked of A and B only. A control of six one-colour frames therefore +left this tool at exit 0, verdict PASS, with `R = 112.77` -- which section 10.5 +reads as `R >= 0.5`, *indistinguishable from run-to-run nondeterminism*, the +STRONGER of its two null readings. A control that rendered nothing at all +upgraded the published conclusion, which is the section 10.6 class one step +removed. + Exit 0 when every threshold passes, 1 when one fails, 2 when the inputs cannot -be read. A missing input is never a pass, and it is never an exit 1 either: a +be read, 3 when the treatment passed and the CONTROL failed its own content +checks. A missing input is never a pass, and it is never an exit 1 either: a 1 says "the two renders differ", which is a reading of an experiment that happened, and a broken render that reported it would be indistinguishable from -the finding this tool exists to make. +the finding this tool exists to make. A 3 is not an exit 1 for the mirror-image +reason: a degenerate control is a broken EXPERIMENT, not a visible difference +between two renders, and section 10.5 maps exit 1 to the second of those. It is +not an exit 0 either, because the pass it would report is a pass nobody may +read. When a threshold fails AND the control is degenerate, the status is 1: the +renders differ, that is established without the control, and it is a finding +about a change already on `main` that a broken control must not hide. """ from __future__ import annotations @@ -90,10 +108,11 @@ import numpy as np -# --- the three exit statuses, which are three different statements ------------ +# --- the four exit statuses, which are four different statements -------------- EXIT_PASS = 0 # every threshold passed EXIT_FAIL = 1 # a threshold failed: the renders differ, and by how much EXIT_UNREADABLE = 2 # nothing was compared, because an input could not be read +EXIT_CONTROL_DEGENERATE = 3 # the treatment passed and the control has no picture in it class UnreadableInput(ValueError): @@ -545,72 +564,146 @@ def _compare(args: argparse.Namespace) -> int: "treatment_mean_abs_rgb": den_rgb, "ratio_mean_abs_rgb": (num_rgb / den_rgb) if den_rgb > 0.0 else None, "undefined": undefined, + # `undefined` is arithmetic: there was no denominator. `unusable` is + # about the control itself: the ratio divides two real numbers and + # still means nothing, because the numerator came from a render with + # no picture in it. The verdict block below fills it in. + "unusable": None, } # --- verdict -------------------------------------------------------------- - checks: list[tuple[str, bool, str]] = [] - # C0 FIRST, and it is not a formality. Everything after this line is a - # DIFFERENCE, and every difference check passes vacuously when both arms are - # equally broken. An arm that rendered nothing, rendered one colour, or - # rendered the same frame 49 times fails HERE, where the failure is legible, - # rather than passing silently as a perfect match. - # - # THREE checks per arm, not four. "frames written" used to be a fourth, and - # it could never be False: `frame_paths` refuses an empty arm at - # EXIT_UNREADABLE long before this loop runs. A row that cannot fail is a - # decoration in a table whose entire value is that every row can. - for label in (args.label_a, args.label_b): + # Every entry carries WHICH outcome it drives: `treatment` entries decide the + # arm-A-vs-arm-B verdict and the exit status, `control` entries decide only + # whether the control is a noise floor the ratio may be read against. One + # list rather than two, so that a reader of `report["checks"]` sees every + # judged thing in one place and can see which is which; and the field rather + # than a naming convention, because a convention a caller can misread + # silently is what section 10.6 is about. + checks: list[tuple[str, bool, str, str]] = [] + + def c0_checks(label: str, judges: str) -> None: + """C0 for ONE render, judged on its own content before anything is + subtracted. Three checks, not four: "frames written" used to be a fourth + and it could never be False, because `frame_paths` refuses an empty + directory at EXIT_UNREADABLE long before this runs, and a row that + cannot fail is a decoration in a table whose entire value is that every + row can.""" c = report["content"][label] checks.append((f"content.{label}.not_uniform", c["near_uniform_frames"] == 0, f"near-uniform frames {c['near_uniform_frames']} == 0 " - f"(min per-frame variance {c['per_frame_var_min']:.3f})")) + f"(min per-frame variance {c['per_frame_var_min']:.3f})", + judges)) checks.append((f"content.{label}.distinct_frames", c["distinct_frame_hashes"] == c["frames"], - f"{c['distinct_frame_hashes']} distinct of {c['frames']}")) + f"{c['distinct_frame_hashes']} distinct of {c['frames']}", + judges)) checks.append((f"content.{label}.motion", c["zero_motion_pairs"] == 0 and c["adjacent_frame_mad_mean"] > 0.0, f"zero-motion pairs {c['zero_motion_pairs']}, " - f"mean adjacent MAD {c['adjacent_frame_mad_mean']:.4f}")) + f"mean adjacent MAD {c['adjacent_frame_mad_mean']:.4f}", + judges)) + + # C0 FIRST, and it is not a formality. Everything after this line is a + # DIFFERENCE, and every difference check passes vacuously when both arms are + # equally broken. An arm that rendered nothing, rendered one colour, or + # rendered the same frame 49 times fails HERE, where the failure is legible, + # rather than passing silently as a perfect match. + for label in (args.label_a, args.label_b): + c0_checks(label, "treatment") + # AND THE CONTROL, for the same reason and to a different outcome. The + # control was the one render whose content nothing judged: it was computed, + # printed, and never registered. A control of one-colour frames then read as + # a very large noise floor, which is section 10.5's STRONGER null. + if args.control: + c0_checks(args.label_control, "control") + if v["bit_identical"]: - checks.append(("video.bit_identical", True, "every frame file sha256-equal")) + checks.append(("video.bit_identical", True, "every frame file sha256-equal", + "treatment")) else: checks.append( ("video.mean_abs", v["mean_abs"] <= args.max_mean_abs, - f"{v['mean_abs']:.6f} <= {args.max_mean_abs}") + f"{v['mean_abs']:.6f} <= {args.max_mean_abs}", "treatment") ) checks.append( ("video.psnr_min_db", v["psnr_min_db"] >= args.min_psnr_db, - f"{v['psnr_min_db']:.3f} >= {args.min_psnr_db}") + f"{v['psnr_min_db']:.3f} >= {args.min_psnr_db}", "treatment") ) checks.append( ("video.ssim_min", v["ssim_min"] >= args.min_ssim, - f"{v['ssim_min']:.6f} >= {args.min_ssim}") + f"{v['ssim_min']:.6f} >= {args.min_ssim}", "treatment") ) if v.get("temporal_ratio") is not None: checks.append( ("video.temporal_ratio", v["temporal_ratio"] <= args.max_temporal_ratio, - f"{v['temporal_ratio']:.6f} <= {args.max_temporal_ratio}") + f"{v['temporal_ratio']:.6f} <= {args.max_temporal_ratio}", "treatment") ) else: - checks.append(("video.temporal_ratio", False, "no adjacent-frame denominator")) + checks.append(("video.temporal_ratio", False, "no adjacent-frame denominator", + "treatment")) a = report["audio"] if not a.get("present"): - checks.append(("audio.present", False, a.get("reason", "absent"))) + checks.append(("audio.present", False, a.get("reason", "absent"), "treatment")) elif not a.get("comparable"): - checks.append(("audio.comparable", False, "shape or sample rate differs")) + checks.append(("audio.comparable", False, "shape or sample rate differs", + "treatment")) elif a.get("bit_identical"): - checks.append(("audio.bit_identical", True, "wav sha256-equal")) + checks.append(("audio.bit_identical", True, "wav sha256-equal", "treatment")) else: checks.append(("audio.psnr_db", a["psnr_db"] >= args.min_audio_psnr_db, - f"{a['psnr_db']:.3f} >= {args.min_audio_psnr_db}")) + f"{a['psnr_db']:.3f} >= {args.min_audio_psnr_db}", "treatment")) checks.append(("audio.pearson_r", (a["pearson_r"] or 0.0) >= args.min_audio_corr, - f"{a['pearson_r']} >= {args.min_audio_corr}")) - - ok = all(c[1] for c in checks) - report["checks"] = [{"name": n, "pass": p, "detail": d} for n, p, d in checks] - report["verdict"] = "PASS" if ok else "FAIL" + f"{a['pearson_r']} >= {args.min_audio_corr}", "treatment")) + + report["checks"] = [{"name": n, "pass": p, "detail": d, "judges": j} + for n, p, d, j in checks] + treatment = [c for c in checks if c[3] == "treatment"] + control_c = [c for c in checks if c[3] == "control"] + ok = all(c[1] for c in treatment) + report["treatment_verdict"] = "PASS" if ok else "FAIL" + + # THE CONTROL'S OWN VERDICT, which is a separate statement about a separate + # render. `USABLE` says the control has a picture in it and is therefore a + # noise floor the ratio may be read against. `DEGENERATE` says it does not, + # and then no reading of `R` is available at all -- section 10.5's four + # branches all assume the control is a repeat of a render, and a repeat of + # nothing is not one of them. + control_failed = [c[0] for c in control_c if not c[1]] + if not args.control: + report["control_verdict"] = None + elif control_failed: + report["control_verdict"] = "DEGENERATE" + else: + report["control_verdict"] = "USABLE" + + degenerate_reason = None + if report["control_verdict"] == "DEGENERATE": + degenerate_reason = ( + f"the control {args.label_control} failed {len(control_failed)} of its own " + f"{len(control_c)} content checks ({', '.join(control_failed)}), so it is " + f"a repeat of no picture rather than a repeat of a render: it is NOT a " + f"noise floor and section 10.5's R is not readable from this run") + report["control_ratio"]["unusable"] = degenerate_reason + + # THE STATUS, and its precedence is an argument rather than an ordering. + # FAIL outranks a degenerate control: "the two renders differ" is + # established without the control at all, and section 10.5 calls that a + # finding about a change already on `main`. A broken control must not + # swallow it. + # A degenerate control outranks a PASS: the pass is real and the READING + # of it is what the control supplies, so a run that cannot be read must + # not exit with the status that says it may be. + if not ok: + report["verdict"] = "FAIL" + status = EXIT_FAIL + elif report["control_verdict"] == "DEGENERATE": + report["verdict"] = "CONTROL_DEGENERATE" + status = EXIT_CONTROL_DEGENERATE + else: + report["verdict"] = "PASS" + status = EXIT_PASS # --- print ---------------------------------------------------------------- print("=== what each arm rendered, before anything is subtracted ===") @@ -651,22 +744,40 @@ def _compare(args: argparse.Namespace) -> int: f"({r['control_mean_abs_luma']:.6f} / " f"{r['treatment_mean_abs_luma']:.6f}), " f"{r['ratio_mean_abs_rgb']:.6f} on RGB") - print(" REPORTED, never checked: section 10.5 reads it") + if r["unusable"]: + print(" REPORTED, and NOT READABLE:") + print(f" {r['unusable']}") + else: + print(" REPORTED, never checked: section 10.5 reads it") print("--- audio ---") for k in ("present", "comparable", "bit_identical", "max_abs_lsb", "max_abs_fs", "rms_diff_fs", "psnr_db", "pearson_r"): if k in a: print(f"{k:22s} {a[k]}") + # WHICH LIST DRIVES WHAT, in the report's own words. `.agents/verification.md` + # asks an instrument to state what it compared against what: a reader who + # cannot see which checks decide the exit status cannot audit the ones that + # do not, and the control's three checks sat outside every list for exactly + # as long as nothing said so. print("--- checks ---") - for n, p, d in checks: + print(f" these decide the verdict: the {args.label_a} vs {args.label_b} comparison") + for n, p, d, j in treatment: print(f" [{'PASS' if p else 'FAIL'}] {n}: {d}") - print(f"VERDICT {report['verdict']}") + if control_c: + print(f" these decide whether the control {args.label_control} is a noise floor " + f"at all, and they do NOT decide the " + f"{args.label_a} vs {args.label_b} verdict") + for n, p, d, j in control_c: + print(f" [{'PASS' if p else 'FAIL'}] {n}: {d}") + if degenerate_reason: + print(f"CONTROL DEGENERATE: {degenerate_reason}") + print(f"VERDICT {report['verdict']} (exit {status})") if args.json: with open(args.json, "w") as fh: json.dump(report, fh, indent=2, sort_keys=True) print(f"wrote {args.json}") - return EXIT_PASS if ok else EXIT_FAIL + return status if __name__ == "__main__": diff --git a/tests/scripts/test_ltx25_pixel_ab_harness.py b/tests/scripts/test_ltx25_pixel_ab_harness.py index a445b3d51..00c4a9fc4 100644 --- a/tests/scripts/test_ltx25_pixel_ab_harness.py +++ b/tests/scripts/test_ltx25_pixel_ab_harness.py @@ -219,15 +219,37 @@ def test_an_empty_wav_is_not_audio(self) -> None: self.assertEqual(self._check(d), "1") def test_an_absent_directory_is_not_complete(self) -> None: + """AND THE FRAME COUNT IS WHAT REFUSES IT. `arm_is_complete` opened with + `[ -d "$d" ] || return 1`, and deleting that line left this whole suite + green: the glob does not expand for a directory that is not there, so + `ls ... | wc -l` reports 0 and the frame-count check returns 1 on its + own. The observable behaviour was identical with the guard and without + it, so it was a redundant guard rather than a defect, and it is gone. + This case is written down so that the next reader adds it back + deliberately or not at all.""" with tempfile.TemporaryDirectory() as t: self.assertEqual(self._check(Path(t) / "never-rendered"), "1") + def test_an_absent_directory_is_not_complete_even_at_zero_frames(self) -> None: + """The one call shape where the frame count could not refuse an absent + directory on its own: ask for zero frames and `0 = 0` holds. The wav + check is what refuses it, so no argument makes an arm that was never + rendered read as a completed one.""" + with tempfile.TemporaryDirectory() as t: + self.assertEqual(self._check(Path(t) / "never-rendered", want=0), "1") + class Wiring(unittest.TestCase): - """TEXT TRIPWIRES on the four call sites that cannot be executed here. + """TEXT TRIPWIRES on the six call sites that cannot be executed here. Each pins a defect that shipped, in the words that shipped it. None is a proof: the lease is the only place these lines run. + + The count was "four" while five tests stood here, and the two the prose left + out -- the phase [F] unit-gate refusal and the signal traps -- were absent + from the spec's list of text-pinned guards as well. `TheDisclosureCounts- + WhatIsThere` below now holds this number against the class, so the next + tripwire cannot be added silently. """ def setUp(self) -> None: @@ -254,6 +276,15 @@ def test_the_unit_gate_refuses_rather_than_reporting(self) -> None: self.assertIn(f"exit {UNIT_GATE_FAILED}", self.text) self.assertIn(f"exit {UNIT_GATE_ABSENT}", self.text) + def test_a_degenerate_control_is_a_status_the_run_defines(self) -> None: + """The comparison gained exit 3 -- the treatment passed and the control + rendered no picture -- and phase [L]'s `case` has a `*)` arm that calls + an unlisted status UNKNOWN. An exit this repository defines must not + reach it, because "the comparison exited 3, which it does not define" + reads as a harness defect rather than as the verdict it is.""" + self.assertIn(" 3) say \"PIXEL VERDICT: CONTROL DEGENERATE", self.text) + self.assertIn("do not publish a reading from this run", self.text) + def test_the_heartbeat_is_reaped_on_a_lease_kill(self) -> None: """`rc` reclaiming a device sends SIGTERM, and a bash EXIT trap does not run for a signal with no handler of its own.""" @@ -262,5 +293,35 @@ def test_the_heartbeat_is_reaped_on_a_lease_kill(self) -> None: self.assertIn(f"trap 'cleanup; exit {status}' {sig}", self.text) +class TheDisclosureCountsWhatIsThere(unittest.TestCase): + """The count of text tripwires is itself a claim, and it had drifted. + + `Wiring` said "the four call sites" while defining five tests, and the + spec's section 10.8 and `## Owed` named four unexecuted things -- the render + loop, the routing assertion, the phase [I] call site and the phase [L] exit + -- while two further guards, the phase [F] unit-gate refusal and the signal + traps, were text-only and in neither list. The error was in the safe + direction: nothing claimed as EXECUTED was in fact only text-pinned. It is + still a number in a document that no longer described the file beside it, + which is the shape section 10.6 is about, and it is cheap to hold. + """ + + WORDS = {"one": 1, "two": 2, "three": 3, "four": 4, "five": 5, "six": 6, + "seven": 7, "eight": 8, "nine": 9, "ten": 10} + + def test_the_wiring_docstring_names_as_many_tripwires_as_it_defines(self) -> None: + doc = Wiring.__doc__ or "" + found = [w for w in self.WORDS if w in doc.split("\n")[0].lower()] + self.assertEqual(len(found), 1, + f"the first line of Wiring's docstring must name exactly one " + f"count word so this gate can read it; it names {found}") + claimed = self.WORDS[found[0]] + defined = len([m for m in dir(Wiring) if m.startswith("test_")]) + self.assertEqual(claimed, defined, + f"Wiring's docstring claims {claimed} tripwires and the class " + f"defines {defined}. Adding a tripwire without saying so leaves " + f"a count in a document that no longer describes the file.") + + if __name__ == "__main__": unittest.main() diff --git a/tests/scripts/test_ltx25_render_compare.py b/tests/scripts/test_ltx25_render_compare.py index dd4545178..6b9a822bb 100644 --- a/tests/scripts/test_ltx25_render_compare.py +++ b/tests/scripts/test_ltx25_render_compare.py @@ -17,7 +17,9 @@ from bf16 rounding -- passes all four video checks with headroom; ONE PIXEL of global horizontal shift fails ALL FOUR, which is the calibration section 10.4 quotes and a - criterion that admitted it would not be a criterion. + criterion that admitted it would not be a criterion. All + eight cells of section 10.4's "Population 2, GATED" table + are asserted here, which is what makes that word true. IdenticallyBroken C0: two all-black renders differ by zero and score the strongest pass every V check can give, so each arm is judged on its own content first. @@ -30,6 +32,11 @@ Control which arm the control repeats is an argument, and the control-to-treatment ratio section 10.5 selects on is computed rather than eyeballed. + ControlContent the control is judged on its OWN content too. It was not, + so a control of one-colour frames read as a very large + noise floor and published section 10.5's STRONGER null. A + degenerate control is its own status, exit 3, because it is + a broken experiment and not a visible difference. SsimIsTheOneItNames V3 names Wang et al. 2004 at 11x11 sigma=1.5 on Rec.601 luma, and every part of that name is pinned by a property or a value: scikit-image is absent here and stays absent. @@ -57,6 +64,7 @@ EXIT_PASS = 0 EXIT_FAIL = 1 EXIT_UNREADABLE = 2 +EXIT_CONTROL_DEGENERATE = 3 W, H, FRAMES = 96, 64, 6 @@ -193,6 +201,40 @@ def test_one_pixel_shift_fails_all_four_video_checks(self) -> None: self.assertFalse(names[name], f"{name} PASSED on a one-pixel shift: {out}") self.assertEqual(rep["verdict"], "FAIL") + def test_the_dither_row_of_the_gated_table_is_pinned(self) -> None: + """Section 10.4's second table is labelled "Population 2, GATED", and + nothing gated it. `test_dither_passes_every_video_check` asserted + `mean < 0.1`, `psnr > 50` and `ssim > 0.99`, which every one of those + cells clears by two orders of magnitude, so a change to `make_render` -- + the seed, the motion step, the texture -- moved all four numbers with + the suite green and the table silently wrong. Demonstrated: `motion` + from 3 to 4 left 43 of 43 tests passing. + + These are the four cells the table quotes for the dither row, at the + precision this file already pins SSIM to.""" + _, out, rep = run("--a", str(self.a), "--b", str(self.dither)) + v = rep["video"] + self.assertAlmostEqual(v["mean_abs"], 0.01935944733796296, places=10, msg=out) + self.assertAlmostEqual(v["psnr_min_db"], 65.04621554883309, places=6, msg=out) + self.assertAlmostEqual(v["ssim_min"], 0.99998566099925, places=8, msg=out) + self.assertAlmostEqual(v["temporal_ratio"], 0.0007008817611029302, + places=10, msg=out) + + def test_the_one_pixel_shift_row_of_the_gated_table_is_pinned(self) -> None: + """The other half of the same table, and the more load-bearing half: + section 10.4 argues the thresholds sit "between a dither and a single + pixel of motion, nearer the dither" from exactly these four numbers, and + `test_one_pixel_shift_fails_all_four_video_checks` asserted only that + four booleans were False. A fixture that failed by a hair and one that + fails by 4.5x the V4 bound are the same test to a boolean.""" + _, out, rep = run("--a", str(self.a), "--b", str(self.shift)) + v = rep["video"] + self.assertAlmostEqual(v["mean_abs"], 16.611979166666668, places=10, msg=out) + self.assertAlmostEqual(v["psnr_min_db"], 21.789085745489977, places=6, msg=out) + self.assertAlmostEqual(v["ssim_min"], 0.7703600993461216, places=8, msg=out) + self.assertAlmostEqual(v["temporal_ratio"], 0.44871294988741245, + places=10, msg=out) + def test_temporal_ratio_is_normalised_by_arm_a_motion(self) -> None: """V4's denominator is the render's own frame-to-frame step, so the same absolute difference must read SMALLER against a faster-moving render. @@ -235,15 +277,30 @@ def test_audio_divergence_fails_even_when_the_video_matches(self) -> None: self.assertFalse(checks_of(rep)["audio.psnr_db"], out) def test_control_arm_is_reported_separately(self) -> None: - """The control never enters the verdict: it is the scale the verdict is - read against, and a tool that folded it into the pass/fail would hide - exactly the attribution it exists to supply.""" + """The control's DIFFERENCE from its arm never enters the treatment + verdict: it is the scale that verdict is read against, and a tool that + folded it into the pass/fail would hide exactly the attribution it + exists to supply. + + This used to assert that no check name contained "control" at all, which + also pinned the hole beside it: the control's own CONTENT was unjudged, + so a control of one-colour frames read as a very large noise floor and + upgraded a pass to section 10.5's stronger null. What must stay outside + the treatment verdict is the RATIO and the control-vs-arm comparison, and + that is what is asserted here.""" rc, out, rep = run("--a", str(self.a), "--b", str(self.dither), "--control", str(self.same)) self.assertEqual(rc, EXIT_PASS, out) self.assertIn("control_video", rep) self.assertTrue(rep["control_video"]["bit_identical"], out) - self.assertNotIn("control", " ".join(c["name"] for c in rep["checks"])) + treatment = [c["name"] for c in rep["checks"] if c["judges"] == "treatment"] + self.assertNotIn("control", " ".join(treatment)) + # Nothing registers the ratio or the control-vs-arm delta as a check, in + # either list. + for c in rep["checks"]: + self.assertNotIn("ratio_mean_abs", c["name"]) + self.assertNotIn("control_video", c["name"]) + self.assertNotIn("control_ratio", c["name"]) class IdenticallyBroken(unittest.TestCase): @@ -606,8 +663,16 @@ def test_the_control_to_treatment_ratio_is_computed_not_eyeballed(self) -> None: self.assertEqual(r["ratio_mean_abs_rgb"], 0.0) self.assertIsNone(r["undefined"]) self.assertIn("control/treatment", out) - # And it is REPORTED, not gated: no check carries the ratio. - self.assertNotIn("control", " ".join(c["name"] for c in rep["checks"])) + # And it is REPORTED, not gated: no check carries the RATIO, in either + # list. This is narrower than it was, deliberately. It used to assert + # that no check name contained "control" at all, which also pinned the + # absence of the control's own CONTENT checks -- see + # `ControlContent` -- and the control's content is now judged. + for c in rep["checks"]: + self.assertNotIn("control_ratio", c["name"]) + self.assertNotIn("ratio_mean_abs", c["name"]) + self.assertNotIn("control", " ".join( + c["name"] for c in rep["checks"] if c["judges"] == "treatment")) def test_the_ratio_is_undefined_rather_than_a_division_by_zero(self) -> None: """The expected case for a bit-identical treatment, which is exactly @@ -628,6 +693,140 @@ def test_the_ratio_is_undefined_rather_than_a_division_by_zero(self) -> None: self.assertIn("undefined", out) +class ControlContent(unittest.TestCase): + """The control is judged on its OWN content, and a broken one is its own status. + + Section 10.4's C0 rationale -- "a difference cannot tell two good renders + from two identically broken ones" -- applies to the control-vs-arm + comparison with identical force, and until this revision it was applied only + to arms A and B. `arm_content(control)` was computed and PRINTED and never + entered `checks`, so a control of six one-colour frames left the tool at + exit 0 with the verdict PASS and `R = 112.77`, which section 10.5 reads as + `R >= 0.5`: **indistinguishable from run-to-run nondeterminism**, the + STRONGER of its two null readings. A control that rendered no picture at all + therefore upgraded the published conclusion. + + A degenerate control is NOT "visibly different". Section 10.5 maps any + failing check to that reading, and it is a finding about a change already on + `main`; a control with no picture in it is a broken EXPERIMENT, and mapping + one onto the other would replace one mis-reading with a second. So it is a + THIRD status, exit 3, carrying its own verdict token, which is neither + `PASS` nor `FAIL` and cannot be read as either. + """ + + def setUp(self) -> None: + self.tmp = tempfile.TemporaryDirectory() + root = Path(self.tmp.name) + self.a = root / "a" + frames = make_render(self.a, np.random.default_rng(29)) + # A healthy treatment pair: one level apart everywhere, which every + # threshold admits. The control is read only on a PASS, so the fixture + # has to be one. + self.b = root / "b" + self.b.mkdir() + for i, arr in enumerate(frames): + write_ppm(self.b / f"frame_{i:06d}.ppm", np.clip(arr.astype(np.int16) + 1, 0, 255)) + (self.b / "audio.wav").write_bytes((self.a / "audio.wav").read_bytes()) + # A healthy control: a byte-for-byte repeat of arm A, which is the arm + # `--control-of a` says it repeats. + self.ctl = root / "ctl" + self.ctl.mkdir() + for p in sorted(self.a.glob("*")): + (self.ctl / p.name).write_bytes(p.read_bytes()) + # A DEGENERATE control: the same geometry, one colour, no picture. + self.flat = root / "flat" + self.flat.mkdir() + flat = np.full((H, W, 3), 17, dtype=np.uint8) + for i in range(FRAMES): + write_ppm(self.flat / f"frame_{i:06d}.ppm", flat) + (self.flat / "audio.wav").write_bytes((self.a / "audio.wav").read_bytes()) + + def tearDown(self) -> None: + self.tmp.cleanup() + + def _run(self, control: Path): + return run("--a", str(self.a), "--b", str(self.b), "--control", str(control), + "--control-of", "a", "--label-a", "flash", "--label-b", "naive", + "--label-control", "flash-ctl") + + def test_a_control_that_rendered_no_picture_is_not_a_pass(self) -> None: + rc, out, rep = self._run(self.flat) + self.assertEqual(rc, EXIT_CONTROL_DEGENERATE, out) + self.assertEqual(rep["verdict"], "CONTROL_DEGENERATE", out) + self.assertEqual(rep["treatment_verdict"], "PASS", out) + self.assertEqual(rep["control_verdict"], "DEGENERATE", out) + # The headline token is neither of the two a reader would act on. + self.assertNotIn("VERDICT PASS", out) + self.assertNotIn("VERDICT FAIL", out) + self.assertIn("VERDICT CONTROL_DEGENERATE", out) + + def test_the_controls_own_content_checks_are_registered(self) -> None: + _, out, rep = self._run(self.flat) + names = checks_of(rep) + for check in ("not_uniform", "distinct_frames", "motion"): + self.assertIn(f"content.flash-ctl.{check}", names, + f"the control's {check} check was never built: {out}") + self.assertFalse(names[f"content.flash-ctl.{check}"], out) + # And they judge the CONTROL, not the treatment: the entry says which. + judges = {c["name"]: c["judges"] for c in rep["checks"]} + self.assertEqual(judges["content.flash-ctl.not_uniform"], "control") + self.assertEqual(judges["content.flash.not_uniform"], "treatment") + + def test_the_ratio_is_marked_unreadable_when_the_control_is_degenerate(self) -> None: + """`R` is still computed and still printed -- the arithmetic was never + the defect -- and it carries a stated reason why it may not be read.""" + _, out, rep = self._run(self.flat) + r = rep["control_ratio"] + self.assertIsNotNone(r["ratio_mean_abs_luma"]) + self.assertGreater(r["ratio_mean_abs_luma"], 0.5, + "the fixture must land in section 10.5's stronger-null band") + self.assertIsNotNone(r["unusable"], "R must say why it cannot be read") + self.assertIn("not readable", out.lower()) + + def test_a_healthy_control_still_passes_at_exit_zero(self) -> None: + rc, out, rep = self._run(self.ctl) + self.assertEqual(rc, EXIT_PASS, out) + self.assertEqual(rep["verdict"], "PASS", out) + self.assertEqual(rep["control_verdict"], "USABLE", out) + names = checks_of(rep) + for check in ("not_uniform", "distinct_frames", "motion"): + self.assertTrue(names[f"content.flash-ctl.{check}"], out) + self.assertIsNone(rep["control_ratio"]["unusable"], out) + + def test_a_treatment_failure_outranks_a_degenerate_control(self) -> None: + """Exit 1 says "the two renders differ", which is established WITHOUT the + control and is a finding about a change already on `main`. A broken + control must not hide it, so exit 3 is reserved for the case where the + treatment passed and only the READING of that pass is at stake.""" + root = Path(self.tmp.name) + shift = root / "shift" + shift.mkdir() + for p in sorted(self.a.glob("frame_*.ppm")): + arr = np.frombuffer(p.read_bytes().split(b"255\n", 1)[1], + dtype=np.uint8).reshape(H, W, 3) + write_ppm(shift / p.name, np.roll(arr, 1, axis=1)) + (shift / "audio.wav").write_bytes((self.a / "audio.wav").read_bytes()) + rc, out, rep = run("--a", str(self.a), "--b", str(shift), + "--control", str(self.flat), "--control-of", "a", + "--label-a", "flash", "--label-b", "naive", + "--label-control", "flash-ctl") + self.assertEqual(rc, EXIT_FAIL, out) + self.assertEqual(rep["verdict"], "FAIL", out) + self.assertEqual(rep["control_verdict"], "DEGENERATE", out) + # Both facts are stated. Neither is inferred from the other. + self.assertIn("CONTROL DEGENERATE", out) + self.assertIn("VERDICT FAIL", out) + + def test_the_report_says_in_words_which_checks_decide_what(self) -> None: + """`.agents/verification.md`: an instrument states what it compared + against what, in its own output. A reader who cannot see which list + drives the exit status cannot audit the one that does not.""" + _, out, _ = self._run(self.ctl) + self.assertIn("these decide the verdict", out) + self.assertIn("do NOT decide", out) + self.assertIn("noise floor", out) + + def _load_tool(): import importlib.util From 7597cd7411a8fd9649a74021824db4076f2ed964 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 22 Aug 2026 21:35:53 +0000 Subject: [PATCH 10/19] fix(#1612): a comment three statuses behind, and a fallback its own docstring called load-bearing `12c880a52` gave the pixel comparison a fourth verdict -- exit 3, the treatment passed and the CONTROL rendered no picture -- and updated the harness header and phase [L]'s `case`. Three records beside those lines did not move with them, and a fresh review on #1695 found all three. Phase [I]'s comment still read "0 pass, 1 a threshold failed, 2 an input could not be read. A 2 is never a pass." It is the only place a reader is told what this job's exit status means, so a 3 would have been read as an undefined status or, worse, as a pass because it was neither 1 nor 2. It now names all four and refuses both non-failures by name. It also says where `$CMP` comes from, which it had wrong: the tool is unpacked from `$W/pixab-src.tar.gz`, a tarball staged on the share, and a lease whose tarball predates exit 3 carries a tool that cannot return one -- a degenerate control is a plain 0 there. The renders queued right now are that case, so `source_sha` is the line to read before quoting a PIXEL_COMPARE_RC. Section 10.8 and `## Owed` counted six unexecuted surfaces while `Wiring` defined six tripwire tests, and the two sixes count different sets. They coincided by accident and were read as two records agreeing, which is how the sixth tripwire -- phase [L]'s `case` arm for exit 3 -- ended up named in neither list. Both now say seven tripwire tests over five text-pinned surfaces, state that the counts are about different things, and name the `case` arm. Phase [L]'s `*)` fallback was pinned by nothing. The exit-3 test cites it as its whole reason for existing, and deleting the line left the suite green at 22/22: without it a status the comparison never defined prints no verdict at all and the run ends on DONE. It is pinned now, and the mutation reds. No threshold moved: `scripts/ltx25-render-compare.py` is byte-identical, and V1 to V4 and A1 to A2 are untouched. Three new gates and one new tripwire, red first in each case. `test_ltx25_pixel_ab_harness` 22 -> 27 OK, `test_ltx25_render_compare` 45 OK, `bash -n` rc 0. Preflight fails only `role-undeclared` and `test_cpu_x86_llamacpp_floor`, the latter on `NO_QUIET_WINDOW after 30s ... load=49.14`, and SKIPs the two trailer gates because this branch is behind `origin/main`; a SKIP reported nothing. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-opus-5 [Claude Code] --- .agents/specs/ltx25-dit-attn-flash.md | 39 ++++-- scripts/ltx25-dit-attn-flash-pixel-ab.sh | 18 ++- tests/scripts/test_ltx25_pixel_ab_harness.py | 130 ++++++++++++++++++- 3 files changed, 169 insertions(+), 18 deletions(-) diff --git a/.agents/specs/ltx25-dit-attn-flash.md b/.agents/specs/ltx25-dit-attn-flash.md index 1cd2b4c87..0bc49f557 100644 --- a/.agents/specs/ltx25-dit-attn-flash.md +++ b/.agents/specs/ltx25-dit-attn-flash.md @@ -525,7 +525,7 @@ never a synonym for "probably fine". | run-to-run control (`flash` twice) | the same lease | **PENDING** — §10.3; without it no arm-to-arm delta is attributable to the kernel | | the comparison tool discriminates | `tests/scripts/test_ltx25_render_compare.py` | **PASS** — 37 tests, `OK`, at `2026-08-22`. It needs no GPU, no lease and no NAS, so `PENDING until §10.7` was misreporting a gate that was already green: a dither passes, a one-pixel shift fails all four V checks, two all-black renders fail C0 while reading as a perfect match on every V, an unreadable input exits 2 while a threshold failure exits 1, A1 and A2 disagree on a time-shifted waveform, and the SSIM is pinned by its taps, its impulse response and three fixture values (§10.4). The count dates the run; it is not a floor to defend | | the comparison tool runs on a lane | `scripts/agent-preflight.sh`, `.github/workflows/ci.yml` | **PASS** — it ran on NO lane when it landed: absent from preflight's `SUITES`, from the enumerated python block in CI and from `tests/CMakeLists.txt`, while the row above registered it as a gate. Both are registered now. Preflight SKIPs it when numpy is absent, which is the third state and never an `ok`; the CI lane installs `python3-numpy` so the lane that must not be silent cannot be | -| the harness's own preconditions | `tests/scripts/test_ltx25_pixel_ab_harness.py` | **PASS** — 19 tests, `OK`, at `2026-08-22`. The memory precondition and the arm-completeness check are extracted verbatim from the harness and run against a fabricated `/proc/meminfo`. The four call sites that only a lease can execute are text tripwires and are labelled as such | +| the harness's own preconditions | `tests/scripts/test_ltx25_pixel_ab_harness.py` | **PASS** — 27 tests, `OK`, at `2026-08-22`. The memory precondition and the arm-completeness check are extracted verbatim from the harness and run against a fabricated `/proc/meminfo`. The call sites that only a lease can execute are text tripwires and are labelled as such; §10.8 counts them and holds its own count. The count dates the run; it is not a floor to defend, and it said "19 tests" and "four call sites" after both had moved | | full preflight | `scripts/agent-preflight.sh` | **PASS at HEAD** — and it was NOT before: `documentation-checkpoint` was red on two of this branch's own commits (see below) | | `documentation-checkpoint` | CI, and locally over the branch range | **PASS at HEAD, RED before it, and the red was THIS BRANCH's** — `2aa78c69b` and `2f39a9426` each recorded a measurement in `.agents/benchmark-record.md` without writing `docs/STATUS.md` (and `docs/BENCHMARKS.md` for the second). The control on the main-only range `4c193bd55..5d548d003` is rc 0, so it was not inherited. Both commits were replaced by one that writes all three surfaces together when the branch was rebuilt, and the checker is re-run at each head rather than trusted to have stayed fixed — a job that has stopped appearing in a failing set is not the same fact as a job that passes | | `build-newest-gcc` | CI | **PASS, and now green on `main` too** — it was red on `main` on `::getpid` in `test_qwen3_dflash2_gguf.cpp:547`, a file this change does not touch; [#1581](https://github.com/mudler/vllm.cpp/pull/1581) fixed it and this branch carries that fix through the merge. A red here after the merge is therefore this row's, not inherited | @@ -1063,12 +1063,18 @@ ratio `R` and which of §10.5's readings it selects, the cross-check against the against a fabricated `/proc/meminfo`, so those three execute here. **The render loop, the routing assertion, the phase [I] call site, the phase [L] exit, the phase [F] unit-gate refusal and the signal traps do not.** Five of those six - are pinned as TEXT — the suite asserts the exact call site that shipped - inverted, the exact `exit` lines, the two unit-gate statuses and the four - `trap` lines — and a text assertion is a tripwire, not a proof: it catches the - inversion that happened and would not catch a rewrite that reintroduced it in - different words. **The render loop is pinned by nothing at all.** `dgx:gpu0` - under a lease is the only place those lines run, which is why every one of them + surfaces are pinned as TEXT, by seven tripwire tests — the suite asserts the + exact call site that shipped inverted, the exact `exit` lines, both arms of + phase [L]'s `case` that this row wrote (the exit-3 verdict and the `*)` + fallback that a status nobody defined would otherwise fall through silently), + the two unit-gate statuses and the four `trap` lines. **Those two counts are + about different sets**: six is how many things never execute, seven is how + many tests pin the five of them that are pinned at all, because phase [L] + carries three tests by itself. They both read "six" for one commit, which + looked like two records agreeing. A text assertion is a tripwire, not a proof: + it catches the inversion that happened and would not catch a rewrite that + reintroduced it in different words. **The render loop is pinned by nothing at + all.** `dgx:gpu0` under a lease is the only place those lines run, which is why every one of them was wrong at once: they had never executed anywhere a test could watch. **That is a structural explanation of a cluster rather than a run of coincidences**, and it tells the next reader which claims in this harness are load-bearing and @@ -1083,8 +1089,13 @@ ratio `R` and which of §10.5's readings it selects, the cross-check against the text-pinned. It was still a number in a document that no longer described the file beside it, so `TheDisclosureCountsWhatIsThere.test_the_wiring_docstring_names_as_many_tripwires_as_it_defines` - now holds the count against the class and a sixth tripwire cannot be added - silently. It fired on exactly that when the exit-3 tripwire was added. + now holds the count against the class and a further tripwire cannot be added + silently. It fired on exactly that when the exit-3 tripwire was added, and + again when the `*)` tripwire was. + `TheDisclosureCountsWhatIsThere.test_the_spec_counts_tripwire_TESTS_and_not_unexecuted_SURFACES` + holds the same number against THIS SECTION, which the docstring gate never + reached: the review that raised it found the two sixes counting different sets + and neither list naming the `case` arm. **Two mutations of the memory gate are deliberately a TIMEOUT rather than a failed assertion, and both are named because one of them was not.** Inverting @@ -1172,10 +1183,12 @@ ratio `R` and which of §10.5's readings it selects, the cross-check against the `scripts/ltx25-dit-attn-flash-pixel-ab.sh`. Six things it does not execute: the render loop, the routing assertion, the phase [I] call site, the phase [L] exit, the phase [F] unit-gate refusal and the signal traps. Five of the six are - pinned as TEXT and nothing executes them, so a rewrite that reintroduced any of - those defects in different words would pass; the render loop is pinned by - nothing. That is a limit of where the file runs, not a gap that another local - test can close. This list said "four" and omitted the unit-gate refusal and + pinned as TEXT, by seven tripwire tests — a count of tests, not of surfaces: + phase [L] carries three, for its exit wiring, its exit-3 verdict and the `*)` + arm that catches a status nobody defined. Nothing executes any of them, so a + rewrite that reintroduced any of those defects in different words would pass; + the render loop is pinned by nothing. That is a limit of where the file runs, + not a gap that another local test can close. This list said "four" and omitted the unit-gate refusal and the traps until a fresh review counted them. Owner: this row. Issue: [#1612](https://github.com/mudler/vllm.cpp/issues/1612). - **`scripts/ltx25-render-compare.py` writes `Infinity` into its JSON.** A diff --git a/scripts/ltx25-dit-attn-flash-pixel-ab.sh b/scripts/ltx25-dit-attn-flash-pixel-ab.sh index 00777ebd4..c29fa2f16 100755 --- a/scripts/ltx25-dit-attn-flash-pixel-ab.sh +++ b/scripts/ltx25-dit-attn-flash-pixel-ab.sh @@ -629,10 +629,20 @@ else: PY say "=== [I] the pixel comparison ===" -# The tool is the one committed in this same source tree, run from the tree, so -# the thresholds it applies are the ones the spec derives and not a copy that -# drifted. Its exit status is the gate: 0 pass, 1 a threshold failed, 2 an input -# could not be read. A 2 is never a pass. +# The tool is `$SRC/scripts/ltx25-render-compare.py`, so the thresholds it +# applies travel with the sources the binary was built from rather than being a +# copy that drifted from them. WHICH SOURCES THOSE ARE IS A FACT ABOUT THE RUN, +# not about this checkout: phase [B] unpacks `$W/pixab-src.tar.gz`, a tarball +# staged on the share, and records what it was in `source_sha`. Read that line +# before quoting a run's PIXEL_COMPARE_RC -- a tarball staged before exit 3 +# existed carries a tool that cannot return one, and a degenerate control is a +# plain 0 there. +# +# Its exit status is the gate, and phase [L] below turns it into this job's: +# 0 every threshold held, 1 a threshold failed, 2 an input could not be read and +# nothing was compared, 3 the thresholds held and the control failed its own +# content checks. A 2 is never a pass, and a 3 is never a pass either: it is a +# result that exists and that nobody may READ. # # ARM A IS FLASH, and that is not cosmetic. The control is a repeat of FLASH, and # the tool compares the control against the arm named by --control-of. This call diff --git a/tests/scripts/test_ltx25_pixel_ab_harness.py b/tests/scripts/test_ltx25_pixel_ab_harness.py index 00c4a9fc4..18bcab62d 100644 --- a/tests/scripts/test_ltx25_pixel_ab_harness.py +++ b/tests/scripts/test_ltx25_pixel_ab_harness.py @@ -32,6 +32,7 @@ from __future__ import annotations import os +import re import subprocess import tempfile import threading @@ -240,7 +241,15 @@ def test_an_absent_directory_is_not_complete_even_at_zero_frames(self) -> None: class Wiring(unittest.TestCase): - """TEXT TRIPWIRES on the six call sites that cannot be executed here. + """Seven text tripwires on harness lines that only a lease executes. + + THEY STAND OVER FIVE SURFACES, AND THE TWO NUMBERS ARE NOT THE SAME NUMBER. + Phase [L] carries three of the seven by itself -- the exit wiring, the + exit-3 arm and the `*)` fallback -- and the phase [I] call site, the routing + assertion, the phase [F] unit-gate refusal and the signal traps carry one + each. The spec's section 10.8 counts SURFACES that never execute; this class + counts TESTS. Both read "six" for one commit, which looked like two records + agreeing and was two different sets landing on one number. Each pins a defect that shipped, in the words that shipped it. None is a proof: the lease is the only place these lines run. @@ -285,6 +294,27 @@ def test_a_degenerate_control_is_a_status_the_run_defines(self) -> None: self.assertIn(" 3) say \"PIXEL VERDICT: CONTROL DEGENERATE", self.text) self.assertIn("do not publish a reading from this run", self.text) + def test_an_unlisted_status_still_gets_a_verdict_line(self) -> None: + """THE `*)` ARM ABOVE IS A GUARD AND NOTHING HELD IT. The test directly + above cites it -- "phase [L]'s `case` has a `*)` arm that calls an + unlisted status UNKNOWN" -- as the whole reason exit 3 needed its own + arm, and deleting the `*)` line left this suite green at 22/22. Without + it a status the comparison never defined (a `python3` that died on an + import, a 137 from the OOM killer) prints no verdict at all: the run + ends on DONE and exits on a number nobody named, which is the exact + silence phase [L] was added to remove. + """ + marker = 'case "$PIXEL_RC" in' + self.assertIn(marker, self.text, "phase [L] no longer branches on the verdict") + block = self.text.split(marker, 1)[1].split("esac", 1)[0] + arm = re.search(r"^\s*\*\)\s*(say .*)$", block, re.M) + self.assertIsNotNone( + arm, "phase [L]'s `case` has no `*)` arm, so a status it does not " + "enumerate produces no verdict line at all") + self.assertIn("PIXEL VERDICT: UNKNOWN", arm.group(1)) + self.assertIn("$PIXEL_RC", arm.group(1), + "the fallback must print the status it could not name") + def test_the_heartbeat_is_reaped_on_a_lease_kill(self) -> None: """`rc` reclaiming a device sends SIGTERM, and a bash EXIT trap does not run for a signal with no handler of its own.""" @@ -293,6 +323,71 @@ def test_the_heartbeat_is_reaped_on_a_lease_kill(self) -> None: self.assertIn(f"trap 'cleanup; exit {status}' {sig}", self.text) +class ThePhaseICommentDescribesTheToolItCalls(unittest.TestCase): + """[I]'s comment is the only place a reader is told what the run's exit + status means, and it went stale the moment [L] gained a fourth verdict. + + `12c880a52` added exit 3 -- the treatment passed and the CONTROL rendered no + picture -- and it updated the header's EXIT STATUS block and phase [L]'s + `case`. Phase [I]'s comment still enumerated "0 pass, 1 a threshold failed, + 2 an input could not be read", so a reader who trusted it would have read a + 3 as an undefined status, or worse, quoted `PIXEL_COMPARE_RC` as a pass + because it was neither 1 nor 2. These two tests derive the statuses from + the `case` that prints them, so the comment cannot fall behind the code + again without a red. + """ + + def setUp(self) -> None: + self.text = HARNESS.read_text() + + def _phase_i_comment(self) -> str: + marker = 'say "=== [I] the pixel comparison ==="' + self.assertIn(marker, self.text, "phase [I] is not in this harness") + after = self.text.split(marker, 1)[1] + self.assertIn('python3 "$CMP"', after, "phase [I] no longer calls the tool") + return after.split('python3 "$CMP"', 1)[0] + + def _verdict_statuses(self) -> list[str]: + """The statuses phase [L]'s `case` actually prints a verdict for.""" + found = sorted(set(re.findall(r'^\s*(\d+)\) say "PIXEL VERDICT:', self.text, + re.M)), key=int) + # Guard the instrument before trusting it: a regex that matched nothing + # would make every assertion below vacuously true. + self.assertEqual(found, ["0", "1", "2", "3"], + "phase [L] defines a different set of verdict statuses than this " + "gate was written against; update the comment in [I], the header " + "EXIT STATUS block and this line together") + return found + + def test_the_phase_i_comment_names_every_status_the_verdict_defines(self) -> None: + comment = self._phase_i_comment() + for s in self._verdict_statuses(): + self.assertIn(s, comment, + f"phase [L] prints a verdict for exit {s} and phase [I]'s comment " + f"does not name it. The comment is what a reader trusts when they " + f"quote PIXEL_COMPARE_RC.") + + def test_the_phase_i_comment_refuses_both_non_pass_statuses_by_name(self) -> None: + """2 and 3 are the two that are NOT failures and are NOT passes either, + which is exactly the pair a reader mis-reads as a pass.""" + comment = self._phase_i_comment() + self.assertIn("never a pass", comment) + for s in ("2", "3"): + self.assertRegex(comment, rf"(?s)never a pass.{{0,80}}\b{s}\b|\b{s}\b.{{0,80}}never a pass", + f"the comment must say that a {s} is never a pass") + + def test_the_phase_i_comment_names_where_the_comparison_tool_came_from(self) -> None: + """`$CMP` is `$SRC/scripts/ltx25-render-compare.py`, and `$SRC` is + unpacked in phase [B] from `$W/pixab-src.tar.gz` -- a tarball STAGED ON + THE SHARE, not this checkout. A lease whose tarball predates the exit-3 + repair cannot return a 3 at all: a degenerate control exits 0 there. The + comment must send the reader to `source_sha` rather than let them assume + the tool is the one beside this comment.""" + comment = self._phase_i_comment() + self.assertIn("pixab-src.tar.gz", comment) + self.assertIn("source_sha", comment) + + class TheDisclosureCountsWhatIsThere(unittest.TestCase): """The count of text tripwires is itself a claim, and it had drifted. @@ -309,6 +404,8 @@ class TheDisclosureCountsWhatIsThere(unittest.TestCase): WORDS = {"one": 1, "two": 2, "three": 3, "four": 4, "five": 5, "six": 6, "seven": 7, "eight": 8, "nine": 9, "ten": 10} + SPEC = ROOT / ".agents/specs/ltx25-dit-attn-flash.md" + def test_the_wiring_docstring_names_as_many_tripwires_as_it_defines(self) -> None: doc = Wiring.__doc__ or "" found = [w for w in self.WORDS if w in doc.split("\n")[0].lower()] @@ -322,6 +419,37 @@ def test_the_wiring_docstring_names_as_many_tripwires_as_it_defines(self) -> Non f"defines {defined}. Adding a tripwire without saying so leaves " f"a count in a document that no longer describes the file.") + def test_the_spec_counts_tripwire_TESTS_and_not_unexecuted_SURFACES(self) -> None: + """TWO SIXES THAT COUNT DIFFERENT SETS ARE NOT A CROSS-CHECK. Section + 10.8 and `## Owed` count the harness lines that do NOT execute -- six of + them, five text-pinned, one pinned by nothing -- while `Wiring` counts + the TESTS that pin them. Both read "six" and the coincidence was read as + agreement, so the tripwire the two lists were missing (phase [L]'s + `case` arm for exit 3) was invisible in both. The spec now states the + test count in its own words, and this gate holds that number against the + class the way the docstring gate above holds the docstring's. + """ + text = self.SPEC.read_text() + found = re.findall(r"([A-Za-z]+) tripwire tests", text) + self.assertGreaterEqual( + len(found), 2, + "section 10.8 and `## Owed` must each state how many tripwire TESTS " + f"stand over the surfaces they list, in the form ' tripwire " + f"tests', so the two counts cannot be mistaken for one; found {found}") + words = {w.lower() for w in found} + self.assertEqual(len(words), 1, + f"the spec states more than one tripwire-test count: {sorted(words)}") + word = words.pop() + self.assertIn(word, self.WORDS, f"{word!r} is not a count this gate can read") + defined = len([m for m in dir(Wiring) if m.startswith("test_")]) + self.assertEqual(self.WORDS[word], defined, + f"the spec claims {self.WORDS[word]} tripwire tests and Wiring " + f"defines {defined}. A count in a document must describe the " + f"file beside it.") + self.assertIn("`case` arm", text, + "the tripwire the two lists omitted is phase [L]'s `case` arm; the " + "spec must name it, not leave it inside a total") + if __name__ == "__main__": unittest.main() From cef0dc3a54fc8db70ba9c527d946060a49945d9d Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 23 Aug 2026 01:09:54 +0000 Subject: [PATCH 11/19] measure(LTX25-DIT-ATTN-FLASH): 7.112x, and a bit-identical control proving the renders differ by the kernel (#1612) The pixel A/B designed in section 10 is taken. All three renders completed in one `rc` lease on `dgx:gpu0` from one binary, at 768x448/49f, seed 20260820, and the harness returned exit 1. Two results came out of that lease and they point opposite ways. The speed pair that section 8 carried as PENDING now exists: naive 45.547 s against flash 6.404 s per DiT forward, n=119 each, one binary and one lease, no stack sampler. 7.112x. This supersedes the cross-run 6.03-6.23x range and its single-arm 7.680 s figure, whose arms differed in binary, lease, prompt and sampler. The pixel A/B FAILS every registered check: mean |delta| 6.414 against a 1.0 bound, worst-frame PSNR 22.269 dB against 40, worst SSIM 0.880694 against 0.99, V4 0.709 against 0.10, audio 29.368 dB and r 0.932682. Section 10.5 selects `visibly different` without a judgement call, and that reading was committed before the renders were taken. No threshold moved. The control is what makes it attributable. A third render, flash again on the same binary and seed, came back bit-identical: 49/49 frames, max |delta| 0, SSIM 1.000000. R = 0.000000, so the run-to-run noise floor is nil and the whole delta is the swapped op. Routing is proved two-sided per arm from each arm's own op-provider log. The kernel is not wrong. `test_ltx2_device` in the same lease reads 22/22 cases, 749/749 assertions and host-vs-device agreement at 5.96e-08, one f32 ULP. What moves the picture is 120 sampler steps amplifying that, which is the question section 10.2 registered as open and refused to derive a bound for. Two naive renders across BUILDS diverge further still, at 9.452, so the trajectory is unstable under any arithmetic perturbation rather than under this one. Filed as #1743, because a failing check is a finding about a change already on main rather than a failure of the measurement. The comparison was re-run at this head over the same frames, because the run used a tool staged from a tarball two commits behind and that tool could not return exit 3. Every figure reproduces to the digit and the control's C0 checks now execute and pass. Also corrects this spec's `documentation-checkpoint` gate row, which named `docs/STATUS.md` as the repair for a red. That file and the checker that demanded it were deleted by 1db7e59cf, and a reader who followed the old row would have recreated a file `check-site.py` reds on. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --- .agents/benchmark-record.md | 121 +++++++++++++ .agents/issue-index.md | 1 + .agents/specs/ltx25-dit-attn-flash.md | 250 +++++++++++++++++++++----- docs/benchmarks/open-gaps.md | 2 +- 4 files changed, 325 insertions(+), 49 deletions(-) diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md index 4319b097e..4cf0f08a6 100644 --- a/.agents/benchmark-record.md +++ b/.agents/benchmark-record.md @@ -19,6 +19,127 @@ from relative link targets repointed for this file's location. # Benchmarks +## LTX25-DIT-ATTN-FLASH §10.7: the attention swap is 7.112x and it renders a VISIBLY DIFFERENT video, with a bit-identical control proving the difference is the kernel's (2026-08-22, `dgx:gpu0`, source `3e2961ef0`, binary `834cec55`, #1549, #1612, #1743) + +**Placement.** Newest-first. This sits above `ENG-EXPERT-STREAM-DEVICE W0g`, +also 2026-08-22, because its renders ran 19:30-00:03 UTC, later that day. + +**The run.** `rc` jobs `b4d45dc7-3a74-48b0-94f3-eb9c907c1403` (build) and +`acff8e89-d704-4f17-a9f2-d354aba53b0d` (all three renders, from the cached +binary). One binary `sha256 834cec557c16cf77eef9a2804cccd2189248c9c64973932670c7e92649320fb1`, +one staged checkpoint set at `/root/ckpt`, `768x448/49f` = 2352 video tokens, +seed `20260820`, prompt `sha256 451a8860...`, `MemAvailable` low-water 40.1 GiB +on every arm. Harness `scripts/ltx25-dit-attn-flash-pixel-ab.sh`, +`RUN_ID=1612-r3`, exit 1. Evidence +`/mnt/nas_share/rc/ltx25-attnflash/pixel-ab/1612-r3/`. + +**THE ACCEPTANCE CRITERION WAS COMMITTED BEFORE THE RENDERS.** +`specs/ltx25-dit-attn-flash.md` §10.4 registers V1-V4 and A1-A2 as the defaults +of `scripts/ltx25-render-compare.py`, and §10.5 registers how to read each +outcome. Nothing below was chosen after the numbers were in view, and no +threshold moved afterwards. + +**Speed, the pair that §8 carried as `PENDING` for want of a second arm.** Same +binary, same lease, `n = 119` timed forwards per arm, no stack sampler on either +side (§7.1 measured `runguard.py --stack-period 12` at ~3.2%, so it is off): + +| arm | median | mean | min | max | +|---|---|---|---|---| +| `naive` (`vt::Attention`, op 18) | **45.547 s** | 45.245 | 44.638 | 46.160 | +| `flash` (`vt::AttentionDenseFlash`, op 21) | **6.404 s** | 6.329 | 5.871 | 6.576 | +| `flash-ctl` (flash again) | 6.393 s | 6.321 | 5.882 | 6.660 | + +**`naive / flash` = 7.112x.** `flash-ctl` reproduces `flash` to 0.17% on the +median, so the control bounds the timing as well as the pixels. This SUPERSEDES +the cross-run 6.03-6.23x range and the single-arm 7.680 s (n=19) figure recorded +earlier in this file and in `docs/benchmarks/open-gaps.md`: those arms differed +in binary, lease, prompt and sampler, and this one does not. + +**Routing, two-sided, per arm, from that arm's own `VT_OP_PROVIDER_STATS=1` +log.** `flash` `op18=0 op21=1`; `naive` `op18=1 op21=0`; `flash-ctl` +`op18=0 op21=1`. A one-sided count cannot tell a routed call from an added one, +so both halves are asserted and a failure exits 46. + +**Pixels: the control is exactly zero.** `flash-ctl` is bit-identical to +`flash` — 49/49 frames, max `|delta|` 0, PSNR `inf`, SSIM `1.000000` — and it +passes its own C0 content checks. `R = 0.000000`. The render is deterministic +run-to-run on this box, so **every bit of the treatment delta is the swapped +op.** That is §10.3's strongest branch, and it was not the expected one. + +**Pixels: the treatment fails every registered check.** + +| check | threshold | measured | +|---|---|---| +| V1 mean `\|delta\|` RGB | `<= 1.0` | **6.414156** | +| V2 worst-frame PSNR | `>= 40 dB` | **22.269 dB** (aggregate 25.822) | +| V3 worst-frame SSIM | `>= 0.99` | **0.880694** (mean 0.901395) | +| V4 luma `\|delta\|` / adjacent MAD | `<= 0.10` | **0.709189** | +| A1 audio PSNR vs full scale | `>= 40 dB` | **29.368 dB** | +| A2 audio Pearson r | `>= 0.999` | **0.932682** | + +0 of 49 frames bit-identical, max `|delta|` 253 of 255, RMSE 13.045514, audio +max `|delta|` 0.5557 FS and RMS diff 0.0340 FS. **98.9-99.7% of the pixels in +every frame differ**, and the histogram is broad and unimodal — a whole-image +shift, not a small mean hiding a bimodal tail. + +**Scale, against §10.4's Population 1 (perturbations of a real 20260820 +frame).** The delta is worse on every axis than one pixel of global image shift, +and 322x the ±1 LSB dither row that stands for the bf16 floor: + +| | mean `\|d\|` | PSNR | SSIM | V4 | +|---|---|---|---|---| +| ±1 LSB dither on 3% of samples | 0.0199 | 65.1 dB | 0.99992 | 0.0026 | +| one pixel of global horizontal shift | 5.183 | 28.1 dB | 0.8705 | 0.624 | +| **measured `flash` vs `naive`** | **6.414** | **25.8 dB** | **0.881 worst** | **0.709** | + +**THE KERNEL IS NOT WRONG, AND THAT IS THE POINT.** `test_ltx2_device` in the +same lease: 22/22 cases, 749/749 assertions, `SUCCESS!`, CUDA host-vs-device +agreement **5.96e-08** — one f32 unit roundoff — and +`kAttentionDenseFlash selections = 8 (want 8), kAttention selections = 0`. The +two ops run the same f32 online softmax and differ only in association. §10.2 +predicted 4.0e4-1.7e5 single-ULP bf16 flips per forward from that and explicitly +REFUSED to derive a pixel bound, because whether they damp or amplify over 120 +steps is empirical. **They amplify**, by about 2.5 orders of magnitude on mean +`|delta|`. Divergence also grows along the frame axis: Pearson `r = +0.753` +between frame index and mean `|delta|`, `r = -0.828` against SSIM, 5.03 over the +first 8 frames rising to 7.05 over the last 8. + +**Cross-check, which is context and never a control.** The recorded 20260820 +baseline came from `a50c57d69`, an ancestor of the swap, on the NAIVE path. +Against today's naive arm: mean `|delta|` **9.452407**, PSNR **22.841 dB**, +worst SSIM **0.803977**, V4 **1.026**, audio PSNR 31.458 dB, `r` 0.959152. +**Two naive renders across builds diverge MORE than flash-vs-naive at one +build.** The binary lineage differs, so this bounds the class rather than +closing it, but it says the trajectory is unstable under any arithmetic +perturbation rather than under this one. + +**The tool that produced these numbers was not the committed one, and that was +checked.** Phase [I] runs the comparison out of a tarball staged on the share; +`PROVENANCE` records it as `source_sha 3e2961ef0`, two commits behind the +branch head. Both missing commits only TIGHTEN — they add the control's own C0 +checks and the phase [L] `*)` arm — so the run was made by a tool that could not +return exit 3. The whole comparison was re-run at head `7597cd741` over the same +frames (no GPU needed) and reproduces every figure to the digit, with the +control's three C0 checks now executed and green and the verdict unchanged at +exit 1. Only the 17th significant figure of the audio `r` differs, which is +`numpy`. + +**Verdict, by §10.5's registered reading: `visibly different`.** Any check +failing selects that branch, and it is *a finding about a change already on +`main`* rather than a failure of the measurement. Filed as +[#1743](https://github.com/mudler/vllm.cpp/issues/1743). §9 forbids repairing it +by widening a threshold, and none was widened. **Neither arm is established as +CORRECT**: every figure here is a difference between two renders, so an absolute +reference is still owed. + +**What it cost.** Five leases. `5fb9399f` lost its worker to an OOM during a +build started against 5 GiB; `2ccd1acf` waited its full 1200 s at a flat 5.0 GiB +and refused with exit 39; `ab12aac1` measured the box and found 110.41 GiB +unaccounted ([#1709](https://github.com/mudler/vllm.cpp/issues/1709), still +open); `b4d45dc7` built once the box was free; `acff8e89` rendered. The renders +were never taken on another box, because §7's denominator argument binds and a +ratio against a different GPU would have been a different measurement. + ## ENG-EXPERT-STREAM-DEVICE W0g: the CPU-against-CUDA arms differ UPSTREAM of the router, and it first shows as a different expert selection in the FIRST MoE block rather than as a sampling near-tie (2026-08-20 to 2026-08-22, `dgx:gpu0`, source `cffe59b`, #1124, #1299) **Read the W0e and W0f sections further down this file first.** This is a third diff --git a/.agents/issue-index.md b/.agents/issue-index.md index e0a508ed0..69b06fe84 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -630,3 +630,4 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#1691](https://github.com/mudler/vllm.cpp/issues/1691) | `DOCS-MODELS-HUMAN` | **`docs/QUICKSTART.md` said no container lane had ever been published, and three had.** The page carried `The container lanes have never been published, so no tag below resolves against the registry yet` and, on its `docker run` line, `The package carries only a stage tag today, which is a build artifact and not a lane`. Both were true when written and false on 2026-08-22: `ghcr.io/mudler/vllm.cpp` is public and carries `main-cpu`, `main-cuda` and `main-vulkan`, each a `linux/amd64` + `linux/arm64` manifest, published 07:30 UTC that day. `docs/guides/container-images.md` already documented the `:main-` tags, so the two pages disagreed. VERIFIED by execution rather than by reading the registry listing: `docker run --rm ghcr.io/mudler/vllm.cpp:main-cpu --version` answered `vllm.cpp 0.0.3 c-abi=23` at digest `sha256:7f88301ea282dad778748929e7aa6869d2418c8d295eef0e7900cca8310d06e5`, and the same image with a mounted `Qwen/Qwen3-0.6B` returned tokens through `/v1/completions` on host `mudler-ubuntu-box` (x86_64, Docker 29.1.2). The image also parses the `vllm_cpp` weight-residency document and echoes `mmap=on prefault=off expert_stream=on expert_stream_slots=4000`, which is what lets the Qwen3.8 2.4T page carry a container form of its recipe. FIXED IN FLOW: the false note is corrected, the `docker run` line names a tag that resolves, and the executed-row table gains its first real row. **This does NOT close [#1281](https://github.com/mudler/vllm.cpp/issues/1281)**: `:latest` still does not exist, `--model org/repo` is still blocked by [#1511](https://github.com/mudler/vllm.cpp/issues/1511) so the executed row mounts a local directory, and no GPU-lane row was run. The executed-row obligation stays owned by #1281 | bug | | [#1690](https://github.com/mudler/vllm.cpp/issues/1690) | `DOC-README-ABI-LANDING-SOURCE` | **`check-doc-checkpoint.py`'s `LANDING_SOURCE_FILES` omitted `include/vllm.h`, so a commit that bumps `VLLM_ABI_VERSION` could never repair the README claim it invalidated.** Found by the review of [PR #1655](https://github.com/mudler/vllm.cpp/pull/1655). The README `## Use it as a library (C API)` block quotes the ABI version out of the header; the header was in `USER_USAGE_FILES` (so an ABI change owes `docs/USAGE.md`) but not in `LANDING_SOURCE_FILES`, and the README rule refuses a claim change unaccompanied by a landing source. The claim could therefore be invalidated but not repaired by the same edit, which is how the README reached `VLLM_ABI_VERSION 21` against a header reading `23`, alongside a stale "46 exported functions" for a header declaring 47. That second half is stale by one rather than by six -- an earlier review figure of 51/52 swept in the `#define VLLM_API` visibility block and counted `vllm_*` identifiers that are typedefs and struct fields rather than exported functions -- so the case for deleting the count is that a live count of one file stored in another goes stale on any ABI addition, not that it is badly wrong. The set's own criterion already admitted it -- the checker's comment says every member is "something the README QUOTES" -- and the header was the only such source missing. FIXED IN FLOW: `include/vllm.h` is added to the set, red-before/green-after pinned by `test_the_c_abi_header_is_a_landing_source`, with `test_the_c_abi_header_permits_but_does_not_demand_readme` proving no new README obligation and the pre-existing no-class tests still green | bug | | [#1692](https://github.com/mudler/vllm.cpp/issues/1692) | `KERNEL-ACCEL-PROVIDER-DECLINE-EXACT` | **[#1584](https://github.com/mudler/vllm.cpp/issues/1584) is FIXED by this row, and its gate is CPU-only: both production call sites need a GPU and neither arm was executed.** The fix adds `GetOpFallbackUncounted`, sharing one body with `GetOpFallback` so resolution order, the reference-tier install, the drain and every throw stay identical and only the `fetch_add` is conditional -- in its existing position, BEFORE the "nothing below" check, because a decline that throws is still a decline. The two SHAPE-GATED providers that hoist the lookup into a function-local static switch to it: `cuda_attention_cross.cu` `BlockedFallback` and `metal_mlx_provider.mm` `MlxFallback`. **Candidate 1 of #1584 -- drop the count from `GetOpFallback` -- was REJECTED and measured rather than argued:** it edits five per-call sites (`vulkan_ops.cpp:950,1067,1488,1509`, `tenstorrent_ops.cpp:1341`), one already gated on an exact value at `test_vulkan_backend.cpp:2901` (`after.declines == before.declines + 1`), and its failure mode for a future caller is silent UNDER-counting, the Risk 4 the seam exists to expose. Net blast radius is ZERO backends, not the four #1584 estimated; exactly two providers change what they report, by one. #1555's `WarmDeclineOnce` workaround is REMOVED. **What this issue owns is the missing EXECUTION:** `test_ops_attention_cross` on CUDA (20 cases, ALL 20 SKIP on a CPU-only build -- 32 assertions, every one the skip guard, so that suite gives the change no coverage here), `test_metal_backend` on a `VLLM_CPP_MLX` build, and the `.agents/reachability.md` mutation on `BlockedFallback()` / `MlxFallback()`. The CUDA COMPILE is covered by the `-DVLLM_CPP_CUDA=ON` `vllm`-target job in `.github/workflows/ci.yml`; the Metal `.mm` file is compiled by NO job in this repository, because MLX needs `MLX_ROOT`. Red-before/green-after and the mutation table in [op-provider-decline-exact.md](specs/op-provider-decline-exact.md) | bug | +| [#1743](https://github.com/mudler/vllm.cpp/issues/1743) | `LTX25-DIT-ATTN-FLASH` | **The DiT attention swap on `main` (`90e8c3c85`, [#1549](https://github.com/mudler/vllm.cpp/issues/1549)) changes what LTX-2.5 RENDERS, by more than one pixel of global image shift.** Measured at `768x448/49f`, seed `20260820`, one binary (`834cec55...`), one `dgx:gpu0` lease (`rc` job `acff8e89-d704-4f17-a9f2-d354aba53b0d`), against the criterion registered in `.agents/specs/ltx25-dit-attn-flash.md` §10.4 BEFORE the renders were taken. **All six registered checks fail**: mean \|delta\| RGB **6.414156** against `<= 1.0`, worst-frame PSNR **22.269 dB** against `>= 40` (aggregate 25.822), worst-frame SSIM **0.880694** against `>= 0.99`, V4 luma \|delta\| over adjacent-frame MAD **0.709189** against `<= 0.10`, audio PSNR **29.368 dB** against `>= 40`, audio Pearson r **0.932682** against `>= 0.999`. 0 of 49 frames bit-identical, max \|delta\| 253 of 255, and 98.9-99.7% of the pixels in EVERY frame differ, with a broad unimodal histogram rather than a small mean hiding a tail. **THE CONTROL MAKES IT ATTRIBUTABLE**: a third render, `flash` again on the same binary and seed, is BIT-IDENTICAL (49/49, max \|delta\| 0, PSNR inf, SSIM 1.000000) and passes its own C0 content checks, so `R = 0.000000`, the run-to-run noise floor is nil, and every bit of the delta is the swapped op. Routing is proved two-sided per arm from each arm's own `VT_OP_PROVIDER_STATS=1` log (`op18=0 op21=1` flash, `op18=1 op21=0` naive). **THE KERNEL IS NOT WRONG**: `test_ltx2_device` in the same lease reads 22/22 cases, 749/749 assertions, host-vs-device agreement **5.96e-08** (one f32 ULP), 8 of 8 flash selections. `vt::Attention` and `vt::AttentionDenseFlash` differ only in association, and the 4.0e4-1.7e5 single-ULP bf16 flips per forward that §10.2 predicted AMPLIFY over 120 sampler steps instead of damping - about 2.5 orders of magnitude on mean \|delta\|, 322x the ±1 LSB dither row and worse on every axis than one pixel of global horizontal shift (5.183 / 28.1 dB / 0.8705 / 0.624). Divergence also grows along the frame axis (r=+0.753 index vs mean \|delta\|, r=-0.828 vs SSIM). **CONTEXT, not a control**: the 20260820 baseline (`a50c57d69`, an ancestor, naive path) against today's naive arm reads mean \|delta\| 9.452407, PSNR 22.841 dB, V4 1.026 - so two NAIVE renders across builds diverge MORE than flash-vs-naive at one build, and the trajectory is unstable under any arithmetic perturbation rather than under this one. NOT FIXED IN FLOW and deliberately not: §9 forbids widening a threshold to admit it, no number in §10.4 moved, and whether a **7.112x** arm that renders a different video stays the default is a product decision. Neither arm is established as CORRECT - every figure is a difference between two renders, so an absolute reference is owed and is a different row. Recorded in [`ltx25-dit-attn-flash.md`](specs/ltx25-dit-attn-flash.md) §10.7; evidence `/mnt/nas_share/rc/ltx25-attnflash/pixel-ab/1612-r3/`. Measured under [#1612](https://github.com/mudler/vllm.cpp/issues/1612) | bug | diff --git a/.agents/specs/ltx25-dit-attn-flash.md b/.agents/specs/ltx25-dit-attn-flash.md index 0bc49f557..9ff7d9f3f 100644 --- a/.agents/specs/ltx25-dit-attn-flash.md +++ b/.agents/specs/ltx25-dit-attn-flash.md @@ -7,12 +7,21 @@ Issue: [#1549](https://github.com/mudler/vllm.cpp/issues/1549). ## Now -`ACTIVE`. The routing change landed as `90e8c3c85` (#1557). What is live now is -**§10, the pixel A/B** ([#1612](https://github.com/mudler/vllm.cpp/issues/1612)): -the swap is on `main` with no comparison of what it RENDERS, and §10 is committed -before the renders are taken so the acceptance criterion cannot be read off the -numbers it judges. The same lease settles the speed A/B that §8 still carries as -`PENDING`. +`ACTIVE`. The routing change landed as `90e8c3c85` (#1557), and **§10's pixel +A/B is now TAKEN** ([#1612](https://github.com/mudler/vllm.cpp/issues/1612)). +The acceptance criterion in §10.4 was committed before the renders, and the +result is in §10.7. + +**Two results came out of the one lease, and they point opposite ways.** The +speed A/B that §8 carried as `PENDING` is settled: **7.112x**, same binary, same +lease, n=119 each. The pixel A/B **fails every registered check**, the control +is bit-identical so the delta is entirely the kernel's, and §10.5 reads that as +**visibly different** — a finding about a change already on `main`, filed as +[#1743](https://github.com/mudler/vllm.cpp/issues/1743). The op-level kernel is +correct to one f32 ULP; what moves the picture is 120 sampler steps amplifying +that. **Whether a 7.112x arm that renders a different video should stay the +default is a product decision this row does not take**, and #1743 is where it is +argued. The diagnosis is confirmed against the tree and the change is scoped to **one production call site**. It was once scoped to that call site plus a @@ -520,14 +529,16 @@ never a synonym for "probably fine". | CUDA host-vs-device parity | `test_ltx2_device` on `dgx:gpu0` | **PASS** — f32 8.94e-08 / 4.47e-08 vs 2e-5; bf16 0 | | bf16 head_dim 128 tile fits without an opt-in | arithmetic, `cuda_ops.cu:3338` | **PASS** — `2 * kFlashBc(64) * 128 * sizeof(bf16)` = 32,768 B against the 49,152 B every architecture gives without an opt-in. This is the whole of what the swapped shape needs from §4.3, and it is a property of the code, so no device is owed for it | | the flash op's advertised head_dim bound | not this row | **NOT A GATE HERE** — §4.3's cap-raise is reverted, so `LaunchAttentionDenseFlash` and `tests/vt/test_ops_attention.cpp` are byte-identical to `main`. Owned by [#1578](https://github.com/mudler/vllm.cpp/pull/1578), which merges first | -| A/B, same binary, both arms | `dgx:gpu0` under an `rc` lease | **PENDING** — the flash arm is measured at 7.680 s median (n=19); the worker was lost before the naive arm, so no pair exists (§7.1) | -| pixel A/B at production geometry | `dgx:gpu0` under an `rc` lease, `scripts/ltx25-dit-attn-flash-pixel-ab.sh` | **PENDING** — criterion registered in §10.4; result in §10.7 | -| run-to-run control (`flash` twice) | the same lease | **PENDING** — §10.3; without it no arm-to-arm delta is attributable to the kernel | +| A/B, same binary, both arms | `dgx:gpu0` under an `rc` lease | **PASS** — `naive` 45.547 s median against `flash` 6.404 s, n=119 each, one binary, one lease, no stack sampler: **7.112x** (§10.7). This replaces §7.1's cross-run 6.03-6.23x range and the single-arm 7.680 s figure that used to sit here | +| pixel A/B at production geometry | `dgx:gpu0` under an `rc` lease, `scripts/ltx25-dit-attn-flash-pixel-ab.sh` | **FAIL, and the failure is the finding** — all four V and both A checks fail: mean \|delta\| 6.414 against `<= 1.0`, worst PSNR 22.269 dB against `>= 40`, worst SSIM 0.880694 against `>= 0.99`, V4 0.709 against `<= 0.10`, audio 29.368 dB and r 0.932682. Criterion registered in §10.4 before the run; result and reading in §10.7. §10.5 selects **visibly different**, filed as [#1743](https://github.com/mudler/vllm.cpp/issues/1743). No threshold moved (§9) | +| run-to-run control (`flash` twice) | the same lease | **PASS** — `flash-ctl` is **bit-identical** to `flash`, 49/49 frames, max \|delta\| 0, PSNR inf, SSIM 1.000000, and it passes its own C0 content checks. `R = 0.000000`, so the noise floor is nil and the whole treatment delta is the kernel's (§10.3's strongest branch) | +| C0 content, all three renders | the same lease | **PASS** — 9 checks: each render has 49 distinct frames, no near-uniform frame (`min_var` 3683.8-3739.0) and no zero-motion pair. The control's three were executed by re-running the committed tool, because the staged tool predated them (§10.7) | +| the numbers reproduce under the COMMITTED tool | this checkout, no GPU | **PASS** — phase [I] ran the tool from a tarball staged at `source_sha 3e2961ef0`, two commits behind head. Re-run at `7597cd741` over the same frames: every figure identical to the digit, control C0 now executed and green, verdict unchanged at exit 1. Only the 17th significant figure of the audio `r` differs, which is `numpy` | | the comparison tool discriminates | `tests/scripts/test_ltx25_render_compare.py` | **PASS** — 37 tests, `OK`, at `2026-08-22`. It needs no GPU, no lease and no NAS, so `PENDING until §10.7` was misreporting a gate that was already green: a dither passes, a one-pixel shift fails all four V checks, two all-black renders fail C0 while reading as a perfect match on every V, an unreadable input exits 2 while a threshold failure exits 1, A1 and A2 disagree on a time-shifted waveform, and the SSIM is pinned by its taps, its impulse response and three fixture values (§10.4). The count dates the run; it is not a floor to defend | | the comparison tool runs on a lane | `scripts/agent-preflight.sh`, `.github/workflows/ci.yml` | **PASS** — it ran on NO lane when it landed: absent from preflight's `SUITES`, from the enumerated python block in CI and from `tests/CMakeLists.txt`, while the row above registered it as a gate. Both are registered now. Preflight SKIPs it when numpy is absent, which is the third state and never an `ok`; the CI lane installs `python3-numpy` so the lane that must not be silent cannot be | | the harness's own preconditions | `tests/scripts/test_ltx25_pixel_ab_harness.py` | **PASS** — 27 tests, `OK`, at `2026-08-22`. The memory precondition and the arm-completeness check are extracted verbatim from the harness and run against a fabricated `/proc/meminfo`. The call sites that only a lease can execute are text tripwires and are labelled as such; §10.8 counts them and holds its own count. The count dates the run; it is not a floor to defend, and it said "19 tests" and "four call sites" after both had moved | | full preflight | `scripts/agent-preflight.sh` | **PASS at HEAD** — and it was NOT before: `documentation-checkpoint` was red on two of this branch's own commits (see below) | -| `documentation-checkpoint` | CI, and locally over the branch range | **PASS at HEAD, RED before it, and the red was THIS BRANCH's** — `2aa78c69b` and `2f39a9426` each recorded a measurement in `.agents/benchmark-record.md` without writing `docs/STATUS.md` (and `docs/BENCHMARKS.md` for the second). The control on the main-only range `4c193bd55..5d548d003` is rc 0, so it was not inherited. Both commits were replaced by one that writes all three surfaces together when the branch was rebuilt, and the checker is re-run at each head rather than trusted to have stayed fixed — a job that has stopped appearing in a failing set is not the same fact as a job that passes | +| `documentation-checkpoint` | CI, and locally over the branch range | **PASS at HEAD, RED before it, and the red was THIS BRANCH's** — `2aa78c69b` and `2f39a9426` each recorded a measurement in `.agents/benchmark-record.md` without writing the public projection that then existed. The control on the main-only range `4c193bd55..5d548d003` is rc 0, so it was not inherited. Both commits were replaced by one that writes the surfaces together when the branch was rebuilt, and the checker is re-run at each head rather than trusted to have stayed fixed — a job that has stopped appearing in a failing set is not the same fact as a job that passes. **THE COUPLING THAT PRODUCED THAT RED NO LONGER EXISTS, and the row is corrected rather than left to mislead:** `1db7e59cf` deleted `docs/STATUS.md` from the tree, deleted `scripts/check-doc-checkpoint.py` and `scripts/check-public-doc-tables.py`, and reduced this job to `check-now-current.py` plus `check-role-discipline.py`. A measurement now owes the row spec's `## Now` and nothing under `docs/` unless it adds a benchmark ID, which this one does not. An earlier revision of this row named `docs/STATUS.md` as the repair, and a reader who followed it would have recreated a file that `scripts/check-site.py` reds on for want of a `nav.yaml` entry | | `build-newest-gcc` | CI | **PASS, and now green on `main` too** — it was red on `main` on `::getpid` in `test_qwen3_dflash2_gguf.cpp:547`, a file this change does not touch; [#1581](https://github.com/mudler/vllm.cpp/pull/1581) fixed it and this branch carries that fix through the merge. A red here after the merge is therefore this row's, not inherited | | `build-test-cpu`, `sanitize-cpu` (both) | CI | **INHERITED** — all three fail on the same single case, `test_runner.cpp:1557`, from #1273; owned by [#1602](https://github.com/mudler/vllm.cpp/issues/1602) and [#1608](https://github.com/mudler/vllm.cpp/issues/1608). Verified against `main` with `scripts/main-baseline.py`, not by reading a push run: those are all cancelled (#274) | | `windows-msvc-cpu` / `-vulkan` | CI | **INHERITED, baseline-less lane** — a markdown-only control PR (#1295) fails the identical step; #584/#965 own it | @@ -540,11 +551,18 @@ hides behind it — both were run locally at the reviewed head and both returned rc 0 — but a gate that stops two other gates from running is a wider failure than its own message says. -**One gate is below the bar and it is named rather than averaged away: the A/B.** -Everything this row claims about SPEED rests on one arm, and the row does not -read as finished until the pair exists. The head_dim refusal that used to sit -beside it here is gone from the list because the change it gated is reverted, not -because it passed. +**The A/B used to be the gate below the bar, and it no longer is.** Everything +this row claimed about SPEED rested on one arm until 2026-08-22; the pair now +exists, same binary and same lease, and §10.7 carries it. The head_dim refusal +that used to sit beside it here is gone from the list because the change it +gated is reverted, not because it passed. + +**The gate that is below the bar is now the pixel A/B, and it fails.** It is +listed as `FAIL` rather than softened, because that is what the registered +criterion returned and §10.5 already wrote down how to read it: the two arms +render visibly differently, the control proves the difference is the kernel's, +and the finding is about `90e8c3c85` rather than about this measurement. The one +thing this row will not do with it is widen a threshold (§9). ## 9. Stop conditions @@ -956,13 +974,138 @@ a reader who cannot see the wiring in the report cannot audit it. ### 10.7 The measurement -**NOT YET TAKEN.** The renders have not landed. A run is submitted on -`f407019e3` as `rc` job `2ccd1acf-dfa6-40b4-b095-1928caebe2c1`, `RUN_ID=1612-r2`, -on `dgx:gpu0`. This heading exists so that the result has one place to go and so -that its absence is visible; it is not a promise that the run succeeded. The -previous attempt (job `5fb9399f-4f4e-417c-adbd-4d741a2e18e4`, 2026-08-22) lost -its worker during the build with the box already at 114 of 119 GiB used before -it allocated anything, which is what §10.3's start-floor gate now refuses. +**TAKEN, 2026-08-22, `RUN_ID=1612-r3`.** All three renders completed in one +lease on `dgx:gpu0`, `rc` job `acff8e89-d704-4f17-a9f2-d354aba53b0d`, on the +binary `834cec557c16cf77eef9a2804cccd2189248c9c64973932670c7e92649320fb1` built +in the preceding lease (`b4d45dc7-3a74-48b0-94f3-eb9c907c1403`) from +`source_sha 3e2961ef0`, against `768x448/49f` (2352 video tokens), seed +`20260820`, prompt `sha256 451a8860...`, checkpoints staged at `/root/ckpt`. +Low-water `MemAvailable` 40.1 GiB on every arm. **The harness returned exit 1, +the pixel verdict, and `PROVENANCE` records `pixel_compare_rc=1`.** + +**THE VERDICT IS `visibly different`, and §10.5 selects it without a +judgement call: any check fails.** All six V and A checks fail, the control is +bit-identical so `R = 0.000000`, and C0 passes on all three renders. Under +§10.5's last branch this is *a finding about a change already on `main`*, filed +as [#1743](https://github.com/mudler/vllm.cpp/issues/1743) naming what diverged +and by how much. §9 forbids repairing it by moving a number, and no number in +§10.4 moved. + +**Routing, two-sided, from each arm's own log.** Every arm proved its own kernel +and none of them rests on "the knob was exported": + +| render | knob | `op18_naive` | `op21_flash` | verdict line | +|---|---|---|---|---| +| `flash` | `1` | 0 | 1 | `ROUTING_OK=flash` | +| `naive` | `0` | 1 | 0 | `ROUTING_OK=naive` | +| `flash-ctl` | `1` | 0 | 1 | `ROUTING_OK=flash-ctl` | + +**The control is exactly zero, which is the strongest branch §10.3 offered.** +`flash-ctl` is `flash` rendered again on the same binary and seed, and it came +back **bit-identical: 49/49 frames, max `|delta|` 0, PSNR `inf`, SSIM +`1.000000`** — and it passes its own C0 checks (49 distinct frames, no +near-uniform frame, no zero-motion pair). So the run-to-run noise floor is not +merely small, it is nil, and **every bit of the delta below is the swapped op.** + +**The treatment, `flash` vs `naive`, same binary, knob only.** + +| # | check | threshold | measured | result | +|---|---|---|---|---| +| C0 | content, all three renders | 9 checks | 49 distinct frames, `min_var` 3683.8-3739.0, 0 zero-motion pairs each | **PASS** | +| V1 | mean `\|delta\|` RGB | `<= 1.0` | **6.414156** | **FAIL**, 6.4x | +| V2 | worst-frame PSNR | `>= 40 dB` | **22.269 dB** (aggregate 25.822) | **FAIL**, by 17.7 dB | +| V3 | worst-frame SSIM | `>= 0.99` | **0.880694** (mean 0.901395) | **FAIL** | +| V4 | luma `\|delta\|` / adjacent MAD | `<= 0.10` | **0.709189** | **FAIL**, 7.1x | +| A1 | audio PSNR vs full scale | `>= 40 dB` | **29.368 dB** | **FAIL**, by 10.6 dB | +| A2 | audio Pearson r | `>= 0.999` | **0.932682** | **FAIL** | + +Beside them: 0 of 49 frames bit-identical, max `|delta|` **253** of 255, mean +`|delta|` luma 6.049980, RMSE 13.045514, audio max `|delta|` 0.5557 full-scale +and RMS diff 0.0340 FS. **Between 98.9% and 99.7% of the pixels in every frame +differ**, and the `|delta|` histogram is broad and unimodal — so this is a +whole-image shift and not §10.4's "small mean hiding a bimodal tail". + +**V4 carries the argument, because it is the bound derived from the render's own +scale.** The arms differ by **71% of this video's own frame-to-frame motion** +where the criterion admits 10%. §10.4's Population 1 then makes that concrete +against perturbations of a real frame, and the measured delta is worse on every +axis than one pixel of global image shift: + +| | mean `\|d\|` | PSNR | SSIM | V4 | +|---|---|---|---|---| +| ±1 LSB dither on 3% of samples — the bf16 floor | 0.0199 | 65.1 dB | 0.99992 | 0.0026 | +| one pixel of global horizontal shift | 5.183 | 28.1 dB | 0.8705 | 0.624 | +| **measured `flash` vs `naive`** | **6.414** | **25.8 dB** | **0.881 worst** | **0.709** | + +**322x the dither row's mean, and 39 dB below its PSNR.** + +**§10.2 registered one question it refused to derive, and this answers it in the +amplifying direction.** The op-level difference is ~1e-7 relative and the +`test_ltx2_device` run in this same lease measures it: 22/22 cases, 749/749 +assertions, `SUCCESS!`, CUDA host-vs-device at **5.96e-08**, one f32 unit +roundoff, and `kAttentionDenseFlash selections = 8 (want 8), kAttention +selections = 0`. **The kernel is correct and the render still moves**, because +the 4.0e4-1.7e5 single-ULP bf16 flips §10.2 predicts per forward **amplify over +120 sampler steps rather than damp** — by about two and a half orders of +magnitude on mean `|delta|`. The divergence also grows along the frame axis: +Pearson `r = +0.753` between frame index and mean `|delta|`, `r = -0.828` +against SSIM, rising from 5.03 over the first 8 frames to 7.05 over the last 8. +That is a property of this sampler at this geometry, and it is the reason §10.2 +said no tight a-priori pixel bound was derivable from the arithmetic. + +**The speed pair, which §8 carried as `PENDING` for want of a second arm, now +exists.** Same binary, same lease, `n = 119` timed forwards each, no stack +sampler on either side: + +| arm | median | mean | min | max | +|---|---|---|---|---| +| `naive` | **45.547 s** | 45.245 | 44.638 | 46.160 | +| `flash` | **6.404 s** | 6.329 | 5.871 | 6.576 | +| `flash-ctl` | 6.393 s | 6.321 | 5.882 | 6.660 | + +**`naive / flash` = 7.112x**, and `flash-ctl` reproduces `flash` to 0.17% on the +median, so the control bounds the timing as well as the pixels. This replaces +the cross-run 6.03-6.23x range of §7.1 with a same-binary, same-lease ratio, and +`PROVENANCE` records `speed_pair_same_lease=yes` with `timing_source=this-lease` +on all three arms. + +**The cross-check against the 20260820 baseline, which is context and never the +control.** That render came from `a50c57d69`, an ancestor of the swap, on the +**naive** path. Against today's **naive** arm it reads mean `|delta|` +**9.452407**, PSNR **22.841 dB**, worst SSIM **0.803977**, V4 **1.026**, audio +PSNR 31.458 dB, `r` 0.959152. **Two naive renders across builds diverge more +than `flash` and `naive` do at one build.** Everything else that landed on +`main` in that window moved the render further than this swap did. The binary +lineage differs, so §10.8's reading holds — it bounds the class rather than +closing it — but it says the trajectory is unstable under any arithmetic +perturbation, and this swap is one instance of a general sensitivity rather than +a uniquely bad one. + +**THE TOOL THAT PRODUCED THESE NUMBERS WAS NOT THE COMMITTED ONE, AND THAT IS +CHECKED RATHER THAN ASSUMED.** Phase [I] runs `$SRC/scripts/ltx25-render-compare.py` +out of a tarball staged on the share, and `PROVENANCE` records that tarball as +`source_sha 3e2961ef0` — two commits behind this branch's head. The two missing +commits are `12c880a52` and `7597cd741`, both of which only TIGHTEN: they add +the control's own C0 checks and the phase [L] `*)` arm. So the run was made by a +tool that could not return exit 3, and a degenerate control would have read as a +plain 0 there. **The whole comparison was therefore re-run at head `7597cd741` +over the same frames** — it needs no GPU, no lease and only the frames on the +share — and it reproduces every figure above to the digit, with the control's +three C0 checks now executed and all three `PASS`. The one difference is the +17th significant figure of the audio Pearson `r` (`0.932682102497646` against +`0.9326821024976478`), which is `numpy` version and not a result. + +#### The four leases this measurement cost + +Kept because §10.3's start gate and `RUN_ID` resume were built out of them, and +because [#1709](https://github.com/mudler/vllm.cpp/issues/1709) is still open on +the half this row does not own. + +A run was earlier submitted on `f407019e3` as `rc` job +`2ccd1acf-dfa6-40b4-b095-1928caebe2c1`, `RUN_ID=1612-r2`. The attempt before it +(job `5fb9399f-4f4e-417c-adbd-4d741a2e18e4`, 2026-08-22) lost its worker during +the build with the box already at 114 of 119 GiB used before it allocated +anything, which is what §10.3's start-floor gate now refuses. **AND THE GATE IS ALREADY REFUSING, WHICH FALSIFIES THE READING THIS ROW STARTED FROM.** Job `2ccd1acf` reports, from its own poll log: @@ -983,7 +1126,11 @@ falsified. That distinction is only available because the gate logs EVERY poll rather than only its refusal — a single refusal line would have said "5.1 GiB" and left the cause open. A read-only diagnostic to identify the resident allocation is with the operator, and #1709 owes the correction. The renders -cannot be taken on that box until it is resolved, and no arm has run. +could not be taken on that box while it held, and no arm had run at that point. +**The box did later come back**, and the run recorded at the top of this section +took all three arms on it at a 40.1 GiB low-water mark. That resolves the +schedule, not the defect: #1709 stays open, because nothing here explains where +the 110.41 GiB went or stops `rc` handing out a lease against it next time. **THE DIAGNOSTIC IS TAKEN, AND THE MEMORY BELONGS TO NOBODY.** `rc` job `ab12aac1-b862-4ac6-8292-9f2c641e6a8d`, read-only, 2026-08-22T16:55:46Z, run @@ -1020,22 +1167,20 @@ not assert it.** What is established is the 110.41 GiB gap and the exclusions. The operational consequence does not depend on the mechanism. **Nothing a lease can do repairs it** — there is no process to kill and no file to delete — so -this row's measurement is `PENDING` on a named external resource, which is a -result under AGENTS.md and never a synonym for "probably fine". `dgx:gpu0` has -continued to report `ready` and hand out leases throughout, which is the -controller half of #1709. +this row's measurement was `PENDING` on a named external resource for as long as +it held, which is a result under AGENTS.md and never a synonym for "probably +fine". `dgx:gpu0` continued to report `ready` and hand out leases throughout, +which is the controller half of #1709 and is unrepaired. -**Three leases spent and no wrong number among them.** `5fb9399f` lost its +**Five leases spent and no wrong number among them.** `5fb9399f` lost its worker to an OOM during a build started against 5 GiB; `2ccd1acf` waited its full 1200 s at a flat 5.0 GiB and refused with exit 39; `ab12aac1` measured the -box. The one thing this row will not do is take the renders somewhere else: §7's -denominator argument binds, and a ratio against a different GPU is not this -measurement taken late, it is a different measurement. - -When it lands, this section records: the three arms' frame counts and routing -proofs, the C0 block, V1 to V4 and A1 to A2 with their headroom, the control -ratio `R` and which of §10.5's readings it selects, the cross-check against the -20260820 baseline, and the exit status the harness returned. +box; `b4d45dc7` built the binary once the box was free; `acff8e89` rendered all +three arms from that cached binary. The one thing this row did not do is take +the renders somewhere else: §7's denominator argument binds, and a ratio against +a different GPU would not have been this measurement taken late, it would have +been a different measurement. **Waiting for `dgx:gpu0` is why the 7.112x ratio +is a same-lease pair rather than a cross-run range.** ### 10.8 What this deliberately does not measure @@ -1149,18 +1294,27 @@ ratio `R` and which of §10.5's readings it selects, the cross-check against the `rc`, which would have parked three jobs instead of spending them, and a line in `.agents/environment.md`'s DGX profile saying a granted lease does not imply a reclaimed box. Owner: this row until the controller half has one. -- **There is NO numeric or pixel comparison at production geometry.** - **§10 is the design that discharges this, and it is committed before the - renders are taken.** The result lands in §10.7. Until it does, the statement - below is still the honest one. The swap - is not bit-identical on CUDA (§5), and the only numeric gate that exists is the - reduced-dimension host-vs-device case — `8.94e-08` / `4.47e-08` against `2e-5` - — which bounds the ARITHMETIC change and not the change at head_dim 128 with - 2352 keys over 48 layers. A diffusion render has no token gate to fall back on, - and the flash arm was interrupted before it wrote any frames, so no pixel A/B - exists either, not even against the completed 49-frame `768x448` baseline - render already on the NAS. Owner: this row. Issue: - [#1612](https://github.com/mudler/vllm.cpp/issues/1612). +- **DISCHARGED 2026-08-22: the pixel comparison at production geometry now + exists, and it FAILED.** §10 is the design, committed before the renders; §10.7 + is the result. All four V and both A checks fail, the control is bit-identical + so the delta is entirely the kernel's, and §10.5 reads it as **visibly + different**. What this closes is the *absence* of a comparison, which was the + gap #1612 named. What it opens is + [#1743](https://github.com/mudler/vllm.cpp/issues/1743), which owns the + divergence itself and the question of whether the arm stays the default. The + reduced-dimension host-vs-device case still bounds only the ARITHMETIC change + — `8.94e-08` / `4.47e-08` against `2e-5` — and it is now known to be a weak + predictor of the render: the kernel agrees to one f32 ULP and the video still + moves by 71% of its own motion step. Owner: this row for the record, + #1743 for the finding. +- **NEITHER ARM IS ESTABLISHED AS THE CORRECT RENDER, and nothing here can + establish one.** Every figure in §10.7 is a difference between two renders, so + none of them sees a defect the two share (§10.8). `naive` is the older + behaviour and not a proven reference, and the 20260820 cross-check is a + different binary lineage rather than a golden. Closing this needs an absolute + reference — an upstream LTX-2.5 render of the same prompt, seed and geometry — + which is a different row and a different oracle. Owner: this row until that row + exists. Issue: [#1743](https://github.com/mudler/vllm.cpp/issues/1743). - **The f32 L2 parity arm cannot run at production geometry any more.** With §4.3's cap-raise reverted, that arm reaches `AttentionDenseFlash` at head_dim 128, whose f32 tile is 65,536 B and does not fit the 49,152 B a launch gets diff --git a/docs/benchmarks/open-gaps.md b/docs/benchmarks/open-gaps.md index da44ff37f..3d4a91419 100644 --- a/docs/benchmarks/open-gaps.md +++ b/docs/benchmarks/open-gaps.md @@ -32,7 +32,7 @@ | LTX-2.5 axes | Speed `PENDING` (vllm-omni#6066 has no native 2.5), binding oracle too. **SIZE: 704x448/25f and 448x256/25f both COMPLETE on GB10 (4231 s, 3085 s)**; one run each, contended box, no oracle, no ceiling (#1088) | NOT the VAE decode (#1041/#1009): 39-100% of the ~1731 s phase is the caption projection (#1208), x86 671.8→78.4 s; LoRA fusion 17.78→0.124 s (#1202), add-back 59% (#1254). **#1286 REFUTED (#1317)**; alloc bound 464 MiB | | LTX-2.5 video VAE decode device arm | Speed **PENDING, and deliberately unmeasured**: the row that landed `vt::Conv3d` (#1007) was dispatched with the GPU lease withheld and made no ratio claim | Correctness only: `test_ops_conv3d` 4/4 byte-exact vs an independent reference, goldens unmoved at 44/44, non-CPU dispatch byte-identical on a fake backend. Owed: a CUDA run (#1452); the non-conv stages (#1451) | | LTX-2.5 FULL 21.004B (`one_stage`, bf16 dev DiT) | **FIRST render on the full model: 768x448/25f on GB10 in 2990 s**, 25 frames + 1.01 s 48 kHz stereo, verifier PASS both arms. Binary `0a43a750` from `7b9e207b1`, run `20260819T150230Z` | REFUSED 1024x576, same lease: 162.0 s/fwd x 60 = 10803 s vs the rung's 7153 s LEASE BUDGET (#1375). STG/modality were OFF (#1092); that refusal is LIFTED, full fidelity is 4 fwd/step. Detail: benchmark-record | -| LTX-2.5 DiT self-attention kernel (`LTX25-DIT-ATTN-FLASH`, #1549) | Flash arm MEASURED on GB10: **7.680 s per DiT forward** at 768x448/49f (2352 tok), n=19, from the engine's own `last=` lines. Naive kernel 47.84 s (n=119). Ratio **~6.0x** (6.03-6.23x) | **A/B `PENDING`:** naive arm never ran (`rc` worker LOST); arms differ in binary, lease, prompt and sampler (~3.2%). No pixel gate (#1612). Cap-raise REVERTED, head_dim evidence WITHDRAWN (#1578). See benchmark-record | +| LTX-2.5 DiT self-attention kernel (`LTX25-DIT-ATTN-FLASH`, #1549) | **A/B CLOSED, one binary and one lease** at 768x448/49f (2352 tok), n=119 per arm, no stack sampler: naive **45.547 s** vs flash **6.404 s** per DiT forward = **7.112x**. A third render repeats flash to 0.17% on the median. This replaces the old cross-run ~6.0x (6.03-6.23x) range and its single-arm 7.680 s figure | **PIXEL GATE TAKEN AND FAILED (#1612 → #1743):** the two arms render VISIBLY DIFFERENTLY — mean \|delta\| 6.414 vs a `<= 1.0` bound, worst-frame PSNR 22.269 dB vs `>= 40`, worst SSIM 0.881 vs `>= 0.99`, 71% of the render's own motion step vs a `<= 10%` bound; audio 29.368 dB and r 0.933. The control render is BIT-IDENTICAL, so the delta is entirely the kernel's, and the op itself matches host to one f32 ULP — 120 sampler steps amplify it. Two NAIVE renders across builds diverge more still (9.452). Cap-raise REVERTED, head_dim evidence WITHDRAWN (#1578). See benchmark-record | | MiniMax-Music3 (`MiniMaxMusic3ForConditionalGeneration`) | **Every axis vs the reference stays `PENDING`.** A PARTIAL device arm now exists (#672): the 8.6B LM and the 2.4B fp32 DiT run on the accelerator, so the rows below are internal two-arm numbers and NOT parity ratios | Denominator: SGLang-Omni `748a0b43` in its production configuration (both CUDA graphs, compiled DIT and DAV, batched seeded sampling) | | MiniMax-Music3 device arm, Jetson Thor sm_110 (#672) | `--device 1` vs `--device 0`, same request/seed, idle box: 2 AR frames **846.6 vs 835.1 s (1.014x SLOWER)**; 10 frames **1430.4 vs 1512.1 s (0.946x)**. Fit: **-11.65 s/frame, +34.8 s fixed** | A third duration (the fit has no residual), and moving the depth decoder + DiT + vocoder, which are 5 of 6 stages and still host scalar loops | | MiniMax-Music3 DiT device arm, `thor:gpu0` sm_110 (#672) | Per DiT forward at the capture's geometry, same binary/weights/inputs, idle box: **204.955 s host vs 0.186-0.187 s device, 1094-1102x** (1201x fitted). Staged ONCE (0.61 s; loop intercept 0.063 s). Whole process 3.5-4.5x | e2e song pair NOT runnable (host DiT alone ~37.6 h at 30 steps). Depth decoder/condition mix (bf16-storage), vocoder (no `ConvTranspose1d`) still host. Detail: benchmark-record | From a4179b1eb676ea11128db991ee18b1c27127c1cd Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 23 Aug 2026 01:45:58 +0000 Subject: [PATCH 12/19] fix(#1612): 30 sampler steps not 120, and SSIM is a similarity so "every axis" was false A fresh review of `36e596f22` returned FAIL on two factual overclaims in the prose built on top of the measurement. Both were headed for `.agents/issue-index.md`, which is append-only, so they are repaired before the row lands rather than after. The measurement itself survived the review intact: every transcribed figure matches the evidence, every derived figure recomputes, no threshold was widened, the pre-registration holds against commit timestamps, and the verdict is the branch section 10.5 committed. What was wrong was what the record said about it. FIRST, the amplification stage count. The prose said the flips amplify over "120 sampler steps". The render takes 30 sampler steps at 4 DiT forwards each, which is 120 FORWARDS - `render.log` ends at `step 30/30` and each step number appears four times. Section 10.2 had it right and said "120 forwards"; the new prose contradicted the section it cited as its own prediction, and overstated the number of sampler stages by 4x. SECOND, the calibration claim. "Worse on every axis than one pixel of global image shift" was refuted by the table printed directly beneath it. SSIM is a similarity, so higher is less degraded: the measured worst frame is 0.880694 and the mean 0.901395, both ABOVE the shift's 0.8705. Three of four axes are worse - mean |delta|, PSNR and V4 - which is what the comparison is entitled to claim. Three further repairs of what the record claimed to have PROVED. The host-vs-device figure was the most favourable reading in the log, not the bound. That family runs 5.96e-08 to 8.94e-08 and the device-vs-golden maxima reach 1.49e-07; section 8 of this same file already quoted 8.94e-08, so the file carried two different "the" agreement figures and the new one was the smaller. The bound is now quoted. The conclusion never needed the smallest reading, because every one of them clears the 2e-5 tolerance by two orders of magnitude. The "8 of 8 flash selections" count was cited beside a CUDA figure as kernel evidence. It resets and reads the CPU provider stats on ReducedParams, so it proves the knob routes at fixture size and says nothing about the CUDA kernel at head_dim 128 with 2352 keys over 48 layers - which this row's own `## Owed` already concedes no gate reaches. The CUDA routing proof for this run is the render's own op log, and that is now what the record points at. Amplification is labelled as the DEDUCTION joining a ~1e-7 op bound to a 6.414 pixel delta, rather than as a measured quantity. Also: the committed-tool re-run is now an artefact instead of an assertion (`recheck.txt`, `recheck.json`, `recheck-cross.txt` beside the originals), its JSON delta is stated exhaustively at 37 numeric leaves within 1.9e-15 relative plus the three keys the exit-3 repair added, rather than as "the 17th significant figure of the audio r"; two live PENDING sentences the run settled are marked superseded in place; the 47.84 s denominator is reconciled against the same-lease 45.547 s with ~1.6% left explicitly unexplained; the render-compare suite count goes 37 to 45; and the renders are dated to their true end on 2026-08-23. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --- .agents/benchmark-record.md | 58 +++++++++---- .agents/issue-index.md | 2 +- .agents/specs/ltx25-dit-attn-flash.md | 117 +++++++++++++++++++++----- docs/benchmarks/open-gaps.md | 2 +- 4 files changed, 139 insertions(+), 40 deletions(-) diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md index 4cf0f08a6..a4dce23eb 100644 --- a/.agents/benchmark-record.md +++ b/.agents/benchmark-record.md @@ -22,7 +22,8 @@ from relative link targets repointed for this file's location. ## LTX25-DIT-ATTN-FLASH §10.7: the attention swap is 7.112x and it renders a VISIBLY DIFFERENT video, with a bit-identical control proving the difference is the kernel's (2026-08-22, `dgx:gpu0`, source `3e2961ef0`, binary `834cec55`, #1549, #1612, #1743) **Placement.** Newest-first. This sits above `ENG-EXPERT-STREAM-DEVICE W0g`, -also 2026-08-22, because its renders ran 19:30-00:03 UTC, later that day. +also 2026-08-22, because its build lease opened 19:30 UTC that day. The renders +themselves ran 21:26-00:03 UTC and finished on 2026-08-23. **The run.** `rc` jobs `b4d45dc7-3a74-48b0-94f3-eb9c907c1403` (build) and `acff8e89-d704-4f17-a9f2-d354aba53b0d` (all three renders, from the cached @@ -83,8 +84,9 @@ every frame differ**, and the histogram is broad and unimodal — a whole-image shift, not a small mean hiding a bimodal tail. **Scale, against §10.4's Population 1 (perturbations of a real 20260820 -frame).** The delta is worse on every axis than one pixel of global image shift, -and 322x the ±1 LSB dither row that stands for the bf16 floor: +frame).** The delta is **comparable to one pixel of global image shift, worse on +three of the four axes**, and 322x the ±1 LSB dither row that stands for the +bf16 floor: | | mean `\|d\|` | PSNR | SSIM | V4 | |---|---|---|---|---| @@ -92,17 +94,36 @@ and 322x the ±1 LSB dither row that stands for the bf16 floor: | one pixel of global horizontal shift | 5.183 | 28.1 dB | 0.8705 | 0.624 | | **measured `flash` vs `naive`** | **6.414** | **25.8 dB** | **0.881 worst** | **0.709** | -**THE KERNEL IS NOT WRONG, AND THAT IS THE POINT.** `test_ltx2_device` in the -same lease: 22/22 cases, 749/749 assertions, `SUCCESS!`, CUDA host-vs-device -agreement **5.96e-08** — one f32 unit roundoff — and -`kAttentionDenseFlash selections = 8 (want 8), kAttention selections = 0`. The -two ops run the same f32 online softmax and differ only in association. §10.2 +**SSIM is a similarity, so higher is LESS degraded, and on that one axis the +swap is slightly better than the shift**: measured worst `0.880694` and mean +`0.901395` against the shift's `0.8705`. Mean `|delta|`, PSNR and V4 are each +worse. An earlier revision of this section said "worse on every axis", which the +table beneath it refutes. + +**THE KERNEL AGREES WITH ITS REFERENCE WHEREVER THIS TREE CAN MEASURE IT.** +`test_ltx2_device` in the same lease: 22/22 cases, 749/749 assertions, +`SUCCESS!`, with the **device-vs-host maximum at `8.94e-08`** against a committed +tolerance of `2e-5`. Quote the maximum, not the minimum: that family ranges from +`5.96e-08` to `8.94e-08` and the device-vs-golden maxima in the same log reach +`1.49e-07`, and every one of them clears `2e-5` by two orders of magnitude, so +the conclusion never needed the smallest reading. **Beside it, +`kAttentionDenseFlash selections = 8 (want 8), kAttention selections = 0` is a +CPU-backend routing count on `ReducedParams`** — it proves the knob routes at +fixture size, not that the CUDA kernel is exact at head_dim 128 with 2352 keys +over 48 layers, which no gate in this tree reaches. The CUDA routing proof for +this run is the render's own `op=21`/`op=18` log. + +The two ops run the same f32 online softmax and differ only in association. §10.2 predicted 4.0e4-1.7e5 single-ULP bf16 flips per forward from that and explicitly -REFUSED to derive a pixel bound, because whether they damp or amplify over 120 -steps is empirical. **They amplify**, by about 2.5 orders of magnitude on mean -`|delta|`. Divergence also grows along the frame axis: Pearson `r = +0.753` -between frame index and mean `|delta|`, `r = -0.828` against SSIM, 5.03 over the -first 8 frames rising to 7.05 over the last 8. +REFUSED to derive a pixel bound, because whether they damp or amplify is +empirical. **They amplify**, by about 2.5 orders of magnitude on mean `|delta|`, +across the render's **30 sampler steps at 4 DiT forwards each — 120 forwards** +(`render.log` ends at `step 30/30`, and each step number appears 4 times). +Amplification is the DEDUCTION connecting a ~1e-7 op bound to a 6.414 pixel +delta; it is not itself a measured quantity. Divergence also grows along the +frame axis: Pearson `r = +0.753` between frame index and mean `|delta|`, +`r = -0.828` against SSIM, 5.03 over the first 8 frames rising to 7.05 over the +last 8. **Cross-check, which is context and never a control.** The recorded 20260820 baseline came from `a50c57d69`, an ancestor of the swap, on the NAIVE path. @@ -121,8 +142,15 @@ checks and the phase [L] `*)` arm — so the run was made by a tool that could n return exit 3. The whole comparison was re-run at head `7597cd741` over the same frames (no GPU needed) and reproduces every figure to the digit, with the control's three C0 checks now executed and green and the verdict unchanged at -exit 1. Only the 17th significant figure of the audio `r` differs, which is -`numpy`. +exit 1. **No printed figure and no verdict changes.** Comparing the two JSONs +leaf by leaf: 37 numeric values differ, every one of them by at most `1.9e-15` +relative — float64 last-ULP, a `numpy` version difference — and the rest are the +input paths (a different mount) plus three keys the newer tool ADDS +(`treatment_verdict`, `control_verdict`, `control_ratio.unusable`), which are +exactly the exit-3 machinery `12c880a52` introduced. An earlier revision said +"only the 17th significant figure of the audio `r`", which named one of the 37 +and read as an exhaustive claim. Artefacts: `recheck.txt` / `recheck.json` beside +the originals. **Verdict, by §10.5's registered reading: `visibly different`.** Any check failing selects that branch, and it is *a finding about a change already on diff --git a/.agents/issue-index.md b/.agents/issue-index.md index 0c7681f64..452746690 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -629,7 +629,7 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#1691](https://github.com/mudler/vllm.cpp/issues/1691) | `DOCS-MODELS-HUMAN` | **`docs/QUICKSTART.md` said no container lane had ever been published, and three had.** The page carried `The container lanes have never been published, so no tag below resolves against the registry yet` and, on its `docker run` line, `The package carries only a stage tag today, which is a build artifact and not a lane`. Both were true when written and false on 2026-08-22: `ghcr.io/mudler/vllm.cpp` is public and carries `main-cpu`, `main-cuda` and `main-vulkan`, each a `linux/amd64` + `linux/arm64` manifest, published 07:30 UTC that day. `docs/guides/container-images.md` already documented the `:main-` tags, so the two pages disagreed. VERIFIED by execution rather than by reading the registry listing: `docker run --rm ghcr.io/mudler/vllm.cpp:main-cpu --version` answered `vllm.cpp 0.0.3 c-abi=23` at digest `sha256:7f88301ea282dad778748929e7aa6869d2418c8d295eef0e7900cca8310d06e5`, and the same image with a mounted `Qwen/Qwen3-0.6B` returned tokens through `/v1/completions` on host `mudler-ubuntu-box` (x86_64, Docker 29.1.2). The image also parses the `vllm_cpp` weight-residency document and echoes `mmap=on prefault=off expert_stream=on expert_stream_slots=4000`, which is what lets the Qwen3.8 2.4T page carry a container form of its recipe. FIXED IN FLOW: the false note is corrected, the `docker run` line names a tag that resolves, and the executed-row table gains its first real row. **This does NOT close [#1281](https://github.com/mudler/vllm.cpp/issues/1281)**: `:latest` still does not exist, `--model org/repo` is still blocked by [#1511](https://github.com/mudler/vllm.cpp/issues/1511) so the executed row mounts a local directory, and no GPU-lane row was run. The executed-row obligation stays owned by #1281 | bug | | [#1690](https://github.com/mudler/vllm.cpp/issues/1690) | `DOC-README-ABI-LANDING-SOURCE` | **`check-doc-checkpoint.py`'s `LANDING_SOURCE_FILES` omitted `include/vllm.h`, so a commit that bumps `VLLM_ABI_VERSION` could never repair the README claim it invalidated.** Found by the review of [PR #1655](https://github.com/mudler/vllm.cpp/pull/1655). The README `## Use it as a library (C API)` block quotes the ABI version out of the header; the header was in `USER_USAGE_FILES` (so an ABI change owes `docs/USAGE.md`) but not in `LANDING_SOURCE_FILES`, and the README rule refuses a claim change unaccompanied by a landing source. The claim could therefore be invalidated but not repaired by the same edit, which is how the README reached `VLLM_ABI_VERSION 21` against a header reading `23`, alongside a stale "46 exported functions" for a header declaring 47. That second half is stale by one rather than by six -- an earlier review figure of 51/52 swept in the `#define VLLM_API` visibility block and counted `vllm_*` identifiers that are typedefs and struct fields rather than exported functions -- so the case for deleting the count is that a live count of one file stored in another goes stale on any ABI addition, not that it is badly wrong. The set's own criterion already admitted it -- the checker's comment says every member is "something the README QUOTES" -- and the header was the only such source missing. FIXED IN FLOW: `include/vllm.h` is added to the set, red-before/green-after pinned by `test_the_c_abi_header_is_a_landing_source`, with `test_the_c_abi_header_permits_but_does_not_demand_readme` proving no new README obligation and the pre-existing no-class tests still green | bug | | [#1692](https://github.com/mudler/vllm.cpp/issues/1692) | `KERNEL-ACCEL-PROVIDER-DECLINE-EXACT` | **[#1584](https://github.com/mudler/vllm.cpp/issues/1584) is FIXED by this row, and its gate is CPU-only: both production call sites need a GPU and neither arm was executed.** The fix adds `GetOpFallbackUncounted`, sharing one body with `GetOpFallback` so resolution order, the reference-tier install, the drain and every throw stay identical and only the `fetch_add` is conditional -- in its existing position, BEFORE the "nothing below" check, because a decline that throws is still a decline. The two SHAPE-GATED providers that hoist the lookup into a function-local static switch to it: `cuda_attention_cross.cu` `BlockedFallback` and `metal_mlx_provider.mm` `MlxFallback`. **Candidate 1 of #1584 -- drop the count from `GetOpFallback` -- was REJECTED and measured rather than argued:** it edits five per-call sites (`vulkan_ops.cpp:950,1067,1488,1509`, `tenstorrent_ops.cpp:1341`), one already gated on an exact value at `test_vulkan_backend.cpp:2901` (`after.declines == before.declines + 1`), and its failure mode for a future caller is silent UNDER-counting, the Risk 4 the seam exists to expose. Net blast radius is ZERO backends, not the four #1584 estimated; exactly two providers change what they report, by one. #1555's `WarmDeclineOnce` workaround is REMOVED. **What this issue owns is the missing EXECUTION:** `test_ops_attention_cross` on CUDA (20 cases, ALL 20 SKIP on a CPU-only build -- 32 assertions, every one the skip guard, so that suite gives the change no coverage here), `test_metal_backend` on a `VLLM_CPP_MLX` build, and the `.agents/reachability.md` mutation on `BlockedFallback()` / `MlxFallback()`. The CUDA COMPILE is covered by the `-DVLLM_CPP_CUDA=ON` `vllm`-target job in `.github/workflows/ci.yml`; the Metal `.mm` file is compiled by NO job in this repository, because MLX needs `MLX_ROOT`. Red-before/green-after and the mutation table in [op-provider-decline-exact.md](specs/op-provider-decline-exact.md) | bug | -| [#1743](https://github.com/mudler/vllm.cpp/issues/1743) | `LTX25-DIT-ATTN-FLASH` | **The DiT attention swap on `main` (`90e8c3c85`, [#1549](https://github.com/mudler/vllm.cpp/issues/1549)) changes what LTX-2.5 RENDERS, by more than one pixel of global image shift.** Measured at `768x448/49f`, seed `20260820`, one binary (`834cec55...`), one `dgx:gpu0` lease (`rc` job `acff8e89-d704-4f17-a9f2-d354aba53b0d`), against the criterion registered in `.agents/specs/ltx25-dit-attn-flash.md` §10.4 BEFORE the renders were taken. **All six registered checks fail**: mean \|delta\| RGB **6.414156** against `<= 1.0`, worst-frame PSNR **22.269 dB** against `>= 40` (aggregate 25.822), worst-frame SSIM **0.880694** against `>= 0.99`, V4 luma \|delta\| over adjacent-frame MAD **0.709189** against `<= 0.10`, audio PSNR **29.368 dB** against `>= 40`, audio Pearson r **0.932682** against `>= 0.999`. 0 of 49 frames bit-identical, max \|delta\| 253 of 255, and 98.9-99.7% of the pixels in EVERY frame differ, with a broad unimodal histogram rather than a small mean hiding a tail. **THE CONTROL MAKES IT ATTRIBUTABLE**: a third render, `flash` again on the same binary and seed, is BIT-IDENTICAL (49/49, max \|delta\| 0, PSNR inf, SSIM 1.000000) and passes its own C0 content checks, so `R = 0.000000`, the run-to-run noise floor is nil, and every bit of the delta is the swapped op. Routing is proved two-sided per arm from each arm's own `VT_OP_PROVIDER_STATS=1` log (`op18=0 op21=1` flash, `op18=1 op21=0` naive). **THE KERNEL IS NOT WRONG**: `test_ltx2_device` in the same lease reads 22/22 cases, 749/749 assertions, host-vs-device agreement **5.96e-08** (one f32 ULP), 8 of 8 flash selections. `vt::Attention` and `vt::AttentionDenseFlash` differ only in association, and the 4.0e4-1.7e5 single-ULP bf16 flips per forward that §10.2 predicted AMPLIFY over 120 sampler steps instead of damping - about 2.5 orders of magnitude on mean \|delta\|, 322x the ±1 LSB dither row and worse on every axis than one pixel of global horizontal shift (5.183 / 28.1 dB / 0.8705 / 0.624). Divergence also grows along the frame axis (r=+0.753 index vs mean \|delta\|, r=-0.828 vs SSIM). **CONTEXT, not a control**: the 20260820 baseline (`a50c57d69`, an ancestor, naive path) against today's naive arm reads mean \|delta\| 9.452407, PSNR 22.841 dB, V4 1.026 - so two NAIVE renders across builds diverge MORE than flash-vs-naive at one build, and the trajectory is unstable under any arithmetic perturbation rather than under this one. NOT FIXED IN FLOW and deliberately not: §9 forbids widening a threshold to admit it, no number in §10.4 moved, and whether a **7.112x** arm that renders a different video stays the default is a product decision. Neither arm is established as CORRECT - every figure is a difference between two renders, so an absolute reference is owed and is a different row. Recorded in [`ltx25-dit-attn-flash.md`](specs/ltx25-dit-attn-flash.md) §10.7; evidence `/mnt/nas_share/rc/ltx25-attnflash/pixel-ab/1612-r3/`. Measured under [#1612](https://github.com/mudler/vllm.cpp/issues/1612) | bug | +| [#1743](https://github.com/mudler/vllm.cpp/issues/1743) | `LTX25-DIT-ATTN-FLASH` | **The DiT attention swap on `main` (`90e8c3c85`, [#1549](https://github.com/mudler/vllm.cpp/issues/1549)) changes what LTX-2.5 RENDERS, by more than one pixel of global image shift.** Measured at `768x448/49f`, seed `20260820`, one binary (`834cec55...`), one `dgx:gpu0` lease (`rc` job `acff8e89-d704-4f17-a9f2-d354aba53b0d`), against the criterion registered in `.agents/specs/ltx25-dit-attn-flash.md` §10.4 BEFORE the renders were taken. **All six registered checks fail**: mean \|delta\| RGB **6.414156** against `<= 1.0`, worst-frame PSNR **22.269 dB** against `>= 40` (aggregate 25.822), worst-frame SSIM **0.880694** against `>= 0.99`, V4 luma \|delta\| over adjacent-frame MAD **0.709189** against `<= 0.10`, audio PSNR **29.368 dB** against `>= 40`, audio Pearson r **0.932682** against `>= 0.999`. 0 of 49 frames bit-identical, max \|delta\| 253 of 255, and 98.9-99.7% of the pixels in EVERY frame differ, with a broad unimodal histogram rather than a small mean hiding a tail. **THE CONTROL MAKES IT ATTRIBUTABLE**: a third render, `flash` again on the same binary and seed, is BIT-IDENTICAL (49/49, max \|delta\| 0, PSNR inf, SSIM 1.000000) and passes its own C0 content checks, so `R = 0.000000`, the run-to-run noise floor is nil, and every bit of the delta is the swapped op. Routing is proved two-sided per arm from each arm's own `VT_OP_PROVIDER_STATS=1` log (`op18=0 op21=1` flash, `op18=1 op21=0` naive). **THE KERNEL AGREES WITH ITS REFERENCE WHEREVER THIS TREE CAN MEASURE IT**: `test_ltx2_device` in the same lease reads 22/22 cases, 749/749 assertions, device-vs-host MAXIMUM **8.94e-08** against a committed `2e-5` tolerance (that family runs 5.96e-08 to 8.94e-08 and the device-vs-golden maxima reach 1.49e-07; every one clears the tolerance by two orders of magnitude, so quote the bound rather than the smallest reading). The `8 of 8` flash-selection count beside it is a **CPU-backend** routing count on `ReducedParams` - it proves the knob routes at fixture size, NOT that the CUDA kernel is exact at head_dim 128 / 2352 keys / 48 layers, which no gate in this tree reaches; the CUDA routing proof for this run is the render's own op log. `vt::Attention` and `vt::AttentionDenseFlash` differ only in association, and the 4.0e4-1.7e5 single-ULP bf16 flips per forward that §10.2 predicted AMPLIFY instead of damping across the render's **30 sampler steps at 4 DiT forwards each, 120 forwards** - about 2.5 orders of magnitude on mean \|delta\|, and 322x the ±1 LSB dither row. Against one pixel of global horizontal shift (5.183 / 28.1 dB / 0.8705 / 0.624) the delta is worse on **three of four axes**: mean \|d\|, PSNR and V4 are worse, while SSIM is a similarity and the measured 0.880694 worst / 0.901395 mean sit slightly ABOVE the shift's 0.8705. Amplification is the DEDUCTION joining a ~1e-7 op bound to a 6.414 pixel delta, not a measured quantity. Divergence also grows along the frame axis (r=+0.753 index vs mean \|delta\|, r=-0.828 vs SSIM). **CONTEXT, not a control**: the 20260820 baseline (`a50c57d69`, an ancestor, naive path) against today's naive arm reads mean \|delta\| 9.452407, PSNR 22.841 dB, V4 1.026 - so two NAIVE renders across builds diverge MORE than flash-vs-naive at one build, and the trajectory is unstable under any arithmetic perturbation rather than under this one. NOT FIXED IN FLOW and deliberately not: §9 forbids widening a threshold to admit it, no number in §10.4 moved, and whether a **7.112x** arm that renders a different video stays the default is a product decision. Neither arm is established as CORRECT - every figure is a difference between two renders, so an absolute reference is owed and is a different row. Recorded in [`ltx25-dit-attn-flash.md`](specs/ltx25-dit-attn-flash.md) §10.7; evidence `/mnt/nas_share/rc/ltx25-attnflash/pixel-ab/1612-r3/`. Measured under [#1612](https://github.com/mudler/vllm.cpp/issues/1612) | bug | | [#1649](https://github.com/mudler/vllm.cpp/issues/1649) | `GATE-WINDOWS-PORTABILITY-TARGET-SCOPE` | **`check-windows-portability.py` read `/w` anywhere in `CMakeLists.txt` as a project-wide negation of `/W4 /WX`, so the vendored BoringSSL targets' PRIVATE `/w` red `windows-msvc-cpu` and `windows-msvc-vulkan` on main and on every PR.** `check()` set `warnings = cmake` -- the whole file -- concatenated `cmake/CompilerWarnings.cmake` and did a flat token search, which cannot tell a global `add_compile_options(/w)` from a `target_compile_options( PRIVATE /w)`; only the first negates the policy. The refusal lands before any translation unit is read, so the job carries no `error C####`. SECOND red, which #1649 did not record: the same defect fails this checker's own suite -- `test_real_tree_msvc_warning_policy_reaches_the_cxx_compile` with `negation='/w'` on main at `8540a2755` (78 tests, 1 failure) -- so the tree asserted the contradiction in two places at once and one fix clears both. FIXED IN FLOW: `without_foreign_target_compile_options` blanks `target_compile_options` spans whose target PROVABLY names only targets this project never declares (`add_library`/`add_executable` first arguments, with `foreach` bindings resolved); anything unresolved stays in scope, and `cmake/CompilerWarnings.cmake` is kept whole because it applies the policy through an unresolvable function parameter. The widening is bounded by three guard properties proved discriminating by MUTATION -- forcing `_target_is_foreign` to `return True` reds exactly those three and nothing else (82 tests, 3 failures), tree restored byte-for-byte. 82 tests OK; the real tree now prints `Windows portability contract OK`. RESIDUAL, stated not hidden: a negation reaching a project target through a `set()` binding rather than a `foreach()` one is still not caught; no such construct exists in the tree today. Recorded under [`gate-windows-portability-target-scope.md`](specs/gate-windows-portability-target-scope.md) `## 4. Risks / decisions` | bug | | [#961](https://github.com/mudler/vllm.cpp/issues/961) | `BACKEND-CUDA-SM110` | `tests/tools/test_online_gate_startup.py:259` guards `shellcheck` absence with a check that cannot fire, so an absent instrument reads as a code verdict: `test_serve_low_tools` raises `FileNotFoundError: 'shellcheck'` instead of skipping. Filed by the sm_110 baseline lane ([#955](https://github.com/mudler/vllm.cpp/issues/955)), where the leased `thor:gpu0` worker carries no `shellcheck`, so the baseline names the failure as a known entry rather than an sm_110 fact. Indexed late: the issue was opened 2026-08-15 and its index row was lost with the unmerged repair of PR [#956](https://github.com/mudler/vllm.cpp/pull/956) | bug | | [#962](https://github.com/mudler/vllm.cpp/issues/962) | `BACKEND-CUDA-SM110` | NVFP4 marlin disagrees with itself on sm_110: `test_ops_moe_grouped:1262` fails `CHECK(bitdiff == 0)` on `NVFP4 block8-vs-block16 M=8 K=4096 N=4096 bitdiff=15/32768`, against a kernel configure reports as `marlin-nvfp4: ENABLED for [110]`. A live kernel defect on an ENABLED feature, not an absent one, and the only substantive standing sm_110 finding in the [#955](https://github.com/mudler/vllm.cpp/issues/955) baseline. Indexed late, same cause as the row above | bug | diff --git a/.agents/specs/ltx25-dit-attn-flash.md b/.agents/specs/ltx25-dit-attn-flash.md index 9ff7d9f3f..1ab225151 100644 --- a/.agents/specs/ltx25-dit-attn-flash.md +++ b/.agents/specs/ltx25-dit-attn-flash.md @@ -18,8 +18,8 @@ lease, n=119 each. The pixel A/B **fails every registered check**, the control is bit-identical so the delta is entirely the kernel's, and §10.5 reads that as **visibly different** — a finding about a change already on `main`, filed as [#1743](https://github.com/mudler/vllm.cpp/issues/1743). The op-level kernel is -correct to one f32 ULP; what moves the picture is 120 sampler steps amplifying -that. **Whether a 7.112x arm that renders a different video should stay the +correct to within its committed tolerance; what moves the picture is the render's +30 sampler steps, 4 DiT forwards each, amplifying that over 120 forwards. **Whether a 7.112x arm that renders a different video should stay the default is a product decision this row does not take**, and #1743 is where it is argued. @@ -492,6 +492,13 @@ honest statement is: replace; - so the A/B result is **PENDING**, not satisfied. +**SUPERSEDED 2026-08-22 by §10.7, and kept because it is the reasoning that +made the second attempt worth spending.** The naive arm was taken on the same +binary in the same lease. The pair is `naive` 45.547 s against `flash` 6.404 s, +n=119 each, no stack sampler: **7.112x**. Every bullet above is now history — +the 7.680 s median and the ~6.0x range are both retired, and neither should be +quoted from this section. + **The flash arm's artifacts do not say what was run, and that is a defect of this row's harness rather than a caveat about it.** `arm-flash.log` opens at `[render] + load` with no command line, `wd-flash/` is empty, and no @@ -533,8 +540,8 @@ never a synonym for "probably fine". | pixel A/B at production geometry | `dgx:gpu0` under an `rc` lease, `scripts/ltx25-dit-attn-flash-pixel-ab.sh` | **FAIL, and the failure is the finding** — all four V and both A checks fail: mean \|delta\| 6.414 against `<= 1.0`, worst PSNR 22.269 dB against `>= 40`, worst SSIM 0.880694 against `>= 0.99`, V4 0.709 against `<= 0.10`, audio 29.368 dB and r 0.932682. Criterion registered in §10.4 before the run; result and reading in §10.7. §10.5 selects **visibly different**, filed as [#1743](https://github.com/mudler/vllm.cpp/issues/1743). No threshold moved (§9) | | run-to-run control (`flash` twice) | the same lease | **PASS** — `flash-ctl` is **bit-identical** to `flash`, 49/49 frames, max \|delta\| 0, PSNR inf, SSIM 1.000000, and it passes its own C0 content checks. `R = 0.000000`, so the noise floor is nil and the whole treatment delta is the kernel's (§10.3's strongest branch) | | C0 content, all three renders | the same lease | **PASS** — 9 checks: each render has 49 distinct frames, no near-uniform frame (`min_var` 3683.8-3739.0) and no zero-motion pair. The control's three were executed by re-running the committed tool, because the staged tool predated them (§10.7) | -| the numbers reproduce under the COMMITTED tool | this checkout, no GPU | **PASS** — phase [I] ran the tool from a tarball staged at `source_sha 3e2961ef0`, two commits behind head. Re-run at `7597cd741` over the same frames: every figure identical to the digit, control C0 now executed and green, verdict unchanged at exit 1. Only the 17th significant figure of the audio `r` differs, which is `numpy` | -| the comparison tool discriminates | `tests/scripts/test_ltx25_render_compare.py` | **PASS** — 37 tests, `OK`, at `2026-08-22`. It needs no GPU, no lease and no NAS, so `PENDING until §10.7` was misreporting a gate that was already green: a dither passes, a one-pixel shift fails all four V checks, two all-black renders fail C0 while reading as a perfect match on every V, an unreadable input exits 2 while a threshold failure exits 1, A1 and A2 disagree on a time-shifted waveform, and the SSIM is pinned by its taps, its impulse response and three fixture values (§10.4). The count dates the run; it is not a floor to defend | +| the numbers reproduce under the COMMITTED tool | this checkout, no GPU | **PASS** — phase [I] ran the tool from a tarball staged at `source_sha 3e2961ef0`, two commits behind head. Re-run at `7597cd741` over the same frames: every printed figure identical, control C0 now executed and green, verdict unchanged at exit 1. Recorded as `recheck.txt` / `recheck.json` / `recheck-cross.txt` in the evidence directory, so this row is re-derivable rather than asserted. The two JSONs differ in 37 numeric leaves at `<= 1.9e-15` relative (float64 last-ULP, `numpy`), in the input paths, and in the three keys the exit-3 repair added | +| the comparison tool discriminates | `tests/scripts/test_ltx25_render_compare.py` | **PASS** — **45 tests, `OK`, re-run at this head on 2026-08-22** (the row read "37" before the control-C0 and `*)` tripwires were added). It needs no GPU, no lease and no NAS, so `PENDING until §10.7` was misreporting a gate that was already green: a dither passes, a one-pixel shift fails all four V checks, two all-black renders fail C0 while reading as a perfect match on every V, an unreadable input exits 2 while a threshold failure exits 1, A1 and A2 disagree on a time-shifted waveform, and the SSIM is pinned by its taps, its impulse response and three fixture values (§10.4). The count dates the run; it is not a floor to defend | | the comparison tool runs on a lane | `scripts/agent-preflight.sh`, `.github/workflows/ci.yml` | **PASS** — it ran on NO lane when it landed: absent from preflight's `SUITES`, from the enumerated python block in CI and from `tests/CMakeLists.txt`, while the row above registered it as a gate. Both are registered now. Preflight SKIPs it when numpy is absent, which is the third state and never an `ok`; the CI lane installs `python3-numpy` so the lane that must not be silent cannot be | | the harness's own preconditions | `tests/scripts/test_ltx25_pixel_ab_harness.py` | **PASS** — 27 tests, `OK`, at `2026-08-22`. The memory precondition and the arm-completeness check are extracted verbatim from the harness and run against a fabricated `/proc/meminfo`. The call sites that only a lease can execute are text tripwires and are labelled as such; §10.8 counts them and holds its own count. The count dates the run; it is not a floor to defend, and it said "19 tests" and "four call sites" after both had moved | | full preflight | `scripts/agent-preflight.sh` | **PASS at HEAD** — and it was NOT before: `documentation-checkpoint` was red on two of this branch's own commits (see below) | @@ -542,6 +549,7 @@ never a synonym for "probably fine". | `build-newest-gcc` | CI | **PASS, and now green on `main` too** — it was red on `main` on `::getpid` in `test_qwen3_dflash2_gguf.cpp:547`, a file this change does not touch; [#1581](https://github.com/mudler/vllm.cpp/pull/1581) fixed it and this branch carries that fix through the merge. A red here after the merge is therefore this row's, not inherited | | `build-test-cpu`, `sanitize-cpu` (both) | CI | **INHERITED** — all three fail on the same single case, `test_runner.cpp:1557`, from #1273; owned by [#1602](https://github.com/mudler/vllm.cpp/issues/1602) and [#1608](https://github.com/mudler/vllm.cpp/issues/1608). Verified against `main` with `scripts/main-baseline.py`, not by reading a push run: those are all cancelled (#274) | | `windows-msvc-cpu` / `-vulkan` | CI | **INHERITED, baseline-less lane** — a markdown-only control PR (#1295) fails the identical step; #584/#965 own it | +| `agent-record` | CI | **INHERITED, and it is NOT the checker its name suggests** — `scripts/check-agent-record.py` and `scripts/audit-live-rows.py` both return rc 0 locally on `main` and on this head, so the job's red is elsewhere inside it. Read by failure text rather than by job name: the job also runs `tests/scripts/test_check_site.py`, whose `test_rendered_benchmark_index_links_resolve_to_emitted_pages` shells out to `hugo` and **ERRORs with `FileNotFoundError: [Errno 2] No such file or directory: 'hugo'`** on a runner that has no `hugo`. That is the #1661 shape exactly — a guard that probes a `returncode` which a missing binary never produces — one restructure later. Newly inherited because `1db7e59cf` added the test. **The local-against-CI asymmetry is the whole reason this was misreadable**: `hugo` is on this developer's `PATH`, so `test_check_site.py` runs 7/7 `OK` here and every checker in the job returns rc 0, while the same job reds on a runner that lacks the binary. A job name is not a diagnosis, and "the checkers pass locally" was true and did not explain it. Already owned and in flight: [#1722](https://github.com/mudler/vllm.cpp/issues/1722), [PR #1726](https://github.com/mudler/vllm.cpp/pull/1726). Not repaired here, because it is another row's open pull request and duplicating it is the failure that check exists to avoid | **A side effect of that red is worth recording, because it was invisible.** The `documentation-checkpoint` job runs `set -eu` and this checker is the FIRST of @@ -711,12 +719,16 @@ proceeding. And three lost workers each discarded every arm already rendered, so and still proves that arm's routing from the log it already has. A reused arm makes the speed pair a cross-lease pair, and phase [H] says so rather than letting the ratio imply otherwise. **That declaration is not bookkeeping.** The -speedup #1549 shipped on is already qualified — its control arm never ran, and -§8 carries the A/B as `PENDING` because of it — so a resumed run that quietly -produced a cross-lease ratio presented as a within-lease one would replace a -stated gap with a false closure. The arm's `ARM` file, the run's `PROVENANCE` -and phase [H]'s own line all carry `timing_source`, and a reader who takes the -ratio without them has to have ignored three places that said so. +speedup #1549 shipped on was qualified when this was written — its control arm +had never run, and §8 carried the A/B as `PENDING` because of it — so a resumed +run that quietly produced a cross-lease ratio presented as a within-lease one +would have replaced a stated gap with a false closure. The arm's `ARM` file, the +run's `PROVENANCE` and phase [H]'s own line all carry `timing_source`, and a +reader who takes the ratio without them has to have ignored three places that +said so. **In the event no arm was reused:** all three arms of `1612-r3` record +`timing_source=this-lease` and `PROVENANCE` records `speed_pair_same_lease=yes`, +so the 7.112x in §10.7 is a within-lease pair and this guard was never load- +bearing for it. It stays because the next resumed run is the one it protects. ### 10.4 The registered acceptance criterion @@ -1025,11 +1037,21 @@ and RMS diff 0.0340 FS. **Between 98.9% and 99.7% of the pixels in every frame differ**, and the `|delta|` histogram is broad and unimodal — so this is a whole-image shift and not §10.4's "small mean hiding a bimodal tail". +**A2 exists to separate a drifted waveform from a different one, and here it +says DIFFERENT.** §10.4 registered the Pearson term because "a waveform that has +drifted in time fails this while PSNR can still look tolerable". Both tracks are +96,480 frames of 48 kHz stereo and a lag sweep over ±2000 samples finds its +maximum at **lag 0**, `r = 0.932682` — no shift improves the correlation. The +flash track also carries **3.3% less RMS energy** (ratio 0.967435), and the +peak difference is 18,209 LSB of a 16-bit scale. So the audio is not +mis-synchronised, it is a different rendering of the same prompt, which is the +same conclusion the frames reach. + **V4 carries the argument, because it is the bound derived from the render's own scale.** The arms differ by **71% of this video's own frame-to-frame motion** where the criterion admits 10%. §10.4's Population 1 then makes that concrete -against perturbations of a real frame, and the measured delta is worse on every -axis than one pixel of global image shift: +against perturbations of a real frame, and the measured delta is **comparable to +one pixel of global image shift — worse on three of the four axes**: | | mean `\|d\|` | PSNR | SSIM | V4 | |---|---|---|---|---| @@ -1037,21 +1059,50 @@ axis than one pixel of global image shift: | one pixel of global horizontal shift | 5.183 | 28.1 dB | 0.8705 | 0.624 | | **measured `flash` vs `naive`** | **6.414** | **25.8 dB** | **0.881 worst** | **0.709** | +**Read the SSIM column in the right direction, because an earlier revision of +this section did not.** SSIM is a similarity, so higher is less degraded. The +measured worst frame is `0.880694` against the shift's `0.8705`, and the measured +MEAN is `0.901395` — both ABOVE the shift, so on that one axis the swap is +slightly LESS degrading than a one-pixel shift. Saying "worse on every axis" +was wrong against the very table printed beneath it. Mean `|delta|`, PSNR and V4 +are each worse, which is what the comparison is entitled to claim. + **322x the dither row's mean, and 39 dB below its PSNR.** **§10.2 registered one question it refused to derive, and this answers it in the amplifying direction.** The op-level difference is ~1e-7 relative and the -`test_ltx2_device` run in this same lease measures it: 22/22 cases, 749/749 -assertions, `SUCCESS!`, CUDA host-vs-device at **5.96e-08**, one f32 unit -roundoff, and `kAttentionDenseFlash selections = 8 (want 8), kAttention -selections = 0`. **The kernel is correct and the render still moves**, because -the 4.0e4-1.7e5 single-ULP bf16 flips §10.2 predicts per forward **amplify over -120 sampler steps rather than damp** — by about two and a half orders of -magnitude on mean `|delta|`. The divergence also grows along the frame axis: +`test_ltx2_device` run in this same lease bounds it: 22/22 cases, 749/749 +assertions, `SUCCESS!`, with the **device-vs-host maximum at `8.94e-08`** against +a committed tolerance of `2e-5` — the same figure §8 already carried. **Quote the +maximum and not the minimum.** That family's readings run from `5.96e-08` to +`8.94e-08`, and the device-vs-golden maxima in the same log reach `1.49e-07`; +citing the smallest is selective, and the conclusion does not need it because +every one of them clears `2e-5` by two orders of magnitude. + +**What that test does and does NOT establish, stated because the distinction is +load-bearing.** It bounds the ARITHMETIC at the fixture's reduced dimensions. The +`kAttentionDenseFlash selections = 8 (want 8), kAttention selections = 0` case +beside it reads `vt::GetOpProviderStats(..., vt::DeviceType::kCPU)` on +`ReducedParams`, so it proves the KNOB ROUTES on the CPU backend at fixture +size — it is not a CUDA measurement at head_dim 128 with 2352 keys over 48 +layers, and `## Owed` already says no gate reaches that shape. **The CUDA routing +proof for this run is the render's own `op=21`/`op=18` log**, which is a separate +instrument and is the one §10.7 relies on. + +So: **the kernel agrees with its reference wherever this tree can measure it, and +the render still moves.** The 4.0e4-1.7e5 single-ULP bf16 flips §10.2 predicts +per forward **amplify rather than damp** across the render's **30 sampler steps +at 4 DiT forwards each — 120 forwards** — by about two and a half orders of +magnitude on mean `|delta|`. §10.2 already counted 120 forwards; an earlier +revision of this section called them "120 sampler steps", which overstates the +number of sampler stages by 4x. The divergence also grows along the frame axis: Pearson `r = +0.753` between frame index and mean `|delta|`, `r = -0.828` against SSIM, rising from 5.03 over the first 8 frames to 7.05 over the last 8. That is a property of this sampler at this geometry, and it is the reason §10.2 -said no tight a-priori pixel bound was derivable from the arithmetic. +said no tight a-priori pixel bound was derivable from the arithmetic. **It is +also an inference about MECHANISM rather than a measurement of one**: what is +measured is a ~1e-7 op-level bound and a 6.414 pixel delta, and amplification is +the deduction that connects them. **The speed pair, which §8 carried as `PENDING` for want of a second arm, now exists.** Same binary, same lease, `n = 119` timed forwards each, no stack @@ -1069,6 +1120,17 @@ the cross-run 6.03-6.23x range of §7.1 with a same-binary, same-lease ratio, an `PROVENANCE` records `speed_pair_same_lease=yes` with `timing_source=this-lease` on all three arms. +**THE DENOMINATOR MOVED TOO, and it is reconciled rather than quietly dropped.** +This row's title and §0 quote the naive path at **47.84 s** a forward. The +same-lease naive arm is **45.547 s**, 4.8% lower. §7.1 measured the +`runguard.py --stack-period 12` sampler that produced the 47.84 s figure at about +3.2%, which accounts for 46.31 s of it and leaves **roughly 1.6% unexplained** — +a different lease, a different prompt and a different binary are each candidates +and none is measured. The 1.6% is recorded as open rather than attributed. It +does not move the ratio, because both arms of the 7.112x were taken in one lease +with no sampler on either side; what it means is that **47.84 s is a superseded +number and 45.547 s is this row's naive denominator.** + **The cross-check against the 20260820 baseline, which is context and never the control.** That render came from `a50c57d69`, an ancestor of the swap, on the **naive** path. Against today's **naive** arm it reads mean `|delta|` @@ -1091,9 +1153,18 @@ tool that could not return exit 3, and a degenerate control would have read as a plain 0 there. **The whole comparison was therefore re-run at head `7597cd741` over the same frames** — it needs no GPU, no lease and only the frames on the share — and it reproduces every figure above to the digit, with the control's -three C0 checks now executed and all three `PASS`. The one difference is the -17th significant figure of the audio Pearson `r` (`0.932682102497646` against -`0.9326821024976478`), which is `numpy` version and not a result. +three C0 checks now executed and all three `PASS`, and the verdict unchanged at +exit 1. **The re-run is an artefact rather than an assertion:** `recheck.txt`, +`recheck.json` and `recheck-cross.txt` sit beside the originals in the evidence +directory, so a reader re-derives this row instead of taking it. + +**What differs between the two JSONs, stated exhaustively because an earlier +revision claimed it was one figure.** 37 numeric leaves differ, each by at most +`1.9e-15` relative — float64 last-ULP, a `numpy` version difference, and it moves +no printed value and no check. The remainder are the input paths, which are a +different mount, and three keys the newer tool ADDS (`treatment_verdict`, +`control_verdict`, `control_ratio.unusable`) — precisely the exit-3 machinery +`12c880a52` introduced, which is why the staged tool could not have returned a 3. #### The four leases this measurement cost diff --git a/docs/benchmarks/open-gaps.md b/docs/benchmarks/open-gaps.md index 3d4a91419..8a052fc4b 100644 --- a/docs/benchmarks/open-gaps.md +++ b/docs/benchmarks/open-gaps.md @@ -32,7 +32,7 @@ | LTX-2.5 axes | Speed `PENDING` (vllm-omni#6066 has no native 2.5), binding oracle too. **SIZE: 704x448/25f and 448x256/25f both COMPLETE on GB10 (4231 s, 3085 s)**; one run each, contended box, no oracle, no ceiling (#1088) | NOT the VAE decode (#1041/#1009): 39-100% of the ~1731 s phase is the caption projection (#1208), x86 671.8→78.4 s; LoRA fusion 17.78→0.124 s (#1202), add-back 59% (#1254). **#1286 REFUTED (#1317)**; alloc bound 464 MiB | | LTX-2.5 video VAE decode device arm | Speed **PENDING, and deliberately unmeasured**: the row that landed `vt::Conv3d` (#1007) was dispatched with the GPU lease withheld and made no ratio claim | Correctness only: `test_ops_conv3d` 4/4 byte-exact vs an independent reference, goldens unmoved at 44/44, non-CPU dispatch byte-identical on a fake backend. Owed: a CUDA run (#1452); the non-conv stages (#1451) | | LTX-2.5 FULL 21.004B (`one_stage`, bf16 dev DiT) | **FIRST render on the full model: 768x448/25f on GB10 in 2990 s**, 25 frames + 1.01 s 48 kHz stereo, verifier PASS both arms. Binary `0a43a750` from `7b9e207b1`, run `20260819T150230Z` | REFUSED 1024x576, same lease: 162.0 s/fwd x 60 = 10803 s vs the rung's 7153 s LEASE BUDGET (#1375). STG/modality were OFF (#1092); that refusal is LIFTED, full fidelity is 4 fwd/step. Detail: benchmark-record | -| LTX-2.5 DiT self-attention kernel (`LTX25-DIT-ATTN-FLASH`, #1549) | **A/B CLOSED, one binary and one lease** at 768x448/49f (2352 tok), n=119 per arm, no stack sampler: naive **45.547 s** vs flash **6.404 s** per DiT forward = **7.112x**. A third render repeats flash to 0.17% on the median. This replaces the old cross-run ~6.0x (6.03-6.23x) range and its single-arm 7.680 s figure | **PIXEL GATE TAKEN AND FAILED (#1612 → #1743):** the two arms render VISIBLY DIFFERENTLY — mean \|delta\| 6.414 vs a `<= 1.0` bound, worst-frame PSNR 22.269 dB vs `>= 40`, worst SSIM 0.881 vs `>= 0.99`, 71% of the render's own motion step vs a `<= 10%` bound; audio 29.368 dB and r 0.933. The control render is BIT-IDENTICAL, so the delta is entirely the kernel's, and the op itself matches host to one f32 ULP — 120 sampler steps amplify it. Two NAIVE renders across builds diverge more still (9.452). Cap-raise REVERTED, head_dim evidence WITHDRAWN (#1578). See benchmark-record | +| LTX-2.5 DiT self-attention kernel (`LTX25-DIT-ATTN-FLASH`, #1549) | **A/B CLOSED, one binary and one lease** at 768x448/49f (2352 tok), n=119 per arm, no stack sampler: naive **45.547 s** vs flash **6.404 s** per DiT forward = **7.112x**. A third render repeats flash to 0.17% on the median. This replaces the old cross-run ~6.0x (6.03-6.23x) range and its single-arm 7.680 s figure | **PIXEL GATE TAKEN AND FAILED (#1612 → #1743):** the two arms render VISIBLY DIFFERENTLY — mean \|delta\| 6.414 vs a `<= 1.0` bound, worst-frame PSNR 22.269 dB vs `>= 40`, worst SSIM 0.881 vs `>= 0.99`, 71% of the render's own motion step vs a `<= 10%` bound; audio 29.368 dB and r 0.933. The control render is BIT-IDENTICAL, so the delta is entirely the kernel's, and the op matches host to `8.94e-08` against a `2e-5` tolerance — 30 sampler steps at 4 DiT forwards each amplify it. Two NAIVE renders across builds diverge more still (9.452). Cap-raise REVERTED, head_dim evidence WITHDRAWN (#1578). See benchmark-record | | MiniMax-Music3 (`MiniMaxMusic3ForConditionalGeneration`) | **Every axis vs the reference stays `PENDING`.** A PARTIAL device arm now exists (#672): the 8.6B LM and the 2.4B fp32 DiT run on the accelerator, so the rows below are internal two-arm numbers and NOT parity ratios | Denominator: SGLang-Omni `748a0b43` in its production configuration (both CUDA graphs, compiled DIT and DAV, batched seeded sampling) | | MiniMax-Music3 device arm, Jetson Thor sm_110 (#672) | `--device 1` vs `--device 0`, same request/seed, idle box: 2 AR frames **846.6 vs 835.1 s (1.014x SLOWER)**; 10 frames **1430.4 vs 1512.1 s (0.946x)**. Fit: **-11.65 s/frame, +34.8 s fixed** | A third duration (the fit has no residual), and moving the depth decoder + DiT + vocoder, which are 5 of 6 stages and still host scalar loops | | MiniMax-Music3 DiT device arm, `thor:gpu0` sm_110 (#672) | Per DiT forward at the capture's geometry, same binary/weights/inputs, idle box: **204.955 s host vs 0.186-0.187 s device, 1094-1102x** (1201x fitted). Staged ONCE (0.61 s; loop intercept 0.063 s). Whole process 3.5-4.5x | e2e song pair NOT runnable (host DiT alone ~37.6 h at 30 steps). Depth decoder/condition mix (bf16-storage), vocoder (no `ConvTranspose1d`) still host. Detail: benchmark-record | From 984e7f785860482b6f26443f5da8cc799e074039 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 23 Aug 2026 01:59:39 +0000 Subject: [PATCH 13/19] fix(#1612): two overclaims the repair itself introduced, caught by re-deriving both Prose written to correct prose is where the next overclaim hides, so both new claims from the previous commit were re-derived rather than reread. Both were wrong. FIRST, "roughly 1.6% unexplained" in the denominator reconciliation was a percentage-POINT subtraction wearing a ratio. 47.84 s is 4.79% above 45.547 s and the sampler accounts for about 3.2%, and subtracting those two figures gives 1.6 POINTS, which is not the residual ratio. Removing the sampler puts the unsampled figure at 46.31 s read as a fraction removed or 46.36 s read as an overhead added, so the residual against 45.547 s is 1.7% to 1.8%. Both readings are now given, the residual is stated as a range, and the point-versus-ratio distinction is named so the next reader does not redo the subtraction. SECOND, the "stated exhaustively" JSON delta was not exhaustive. It named the 37 numeric leaves, the input paths and three added keys, and omitted the change that matters most: the `checks` array grows from 12 entries to 15, with nothing removed, and the three additions are exactly the `content.flash-ctl.*` checks the control-C0 repair introduced. That omission is awkward in a paragraph whose whole purpose is to show the staged tool could not judge the control. The 37 numeric leaves and the 1.9e-15 bound both survive re-derivation and are unchanged. Also records a trap found while re-deriving it: the `checks` array must be diffed by `name` and not by index. The three insertions shift the tail, so an index-wise comparison reports about 20 spurious differences that are the same checks at moved positions. A first pass at this delta reported exactly that artefact as real. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --- .agents/benchmark-record.md | 24 +++++++++------ .agents/specs/ltx25-dit-attn-flash.md | 43 ++++++++++++++++++--------- 2 files changed, 44 insertions(+), 23 deletions(-) diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md index a4dce23eb..6c9eba656 100644 --- a/.agents/benchmark-record.md +++ b/.agents/benchmark-record.md @@ -142,15 +142,21 @@ checks and the phase [L] `*)` arm — so the run was made by a tool that could n return exit 3. The whole comparison was re-run at head `7597cd741` over the same frames (no GPU needed) and reproduces every figure to the digit, with the control's three C0 checks now executed and green and the verdict unchanged at -exit 1. **No printed figure and no verdict changes.** Comparing the two JSONs -leaf by leaf: 37 numeric values differ, every one of them by at most `1.9e-15` -relative — float64 last-ULP, a `numpy` version difference — and the rest are the -input paths (a different mount) plus three keys the newer tool ADDS -(`treatment_verdict`, `control_verdict`, `control_ratio.unusable`), which are -exactly the exit-3 machinery `12c880a52` introduced. An earlier revision said -"only the 17th significant figure of the audio `r`", which named one of the 37 -and read as an exhaustive claim. Artefacts: `recheck.txt` / `recheck.json` beside -the originals. +exit 1. **No printed figure, no check result and no verdict changes.** Comparing +the two JSONs leaf by leaf on matching key paths: **37 numeric values differ, +every one by at most `1.9e-15` relative** — float64 last-ULP, a `numpy` version +difference. Everything else is structural and is the repair itself: the input +paths (a different mount); the `checks` array growing from **12 to 15 with +nothing removed**, the additions being exactly `content.flash-ctl.not_uniform`, +`.distinct_frames` and `.motion`; and three keys the staged tool never wrote — +`treatment_verdict`, `control_verdict`, `control_ratio.unusable` — which are the +exit-3 machinery `12c880a52` introduced and the direct evidence that the staged +tool could not have returned a 3. **Compare the `checks` array by `name`, not by +index**: the three insertions shift the tail and an index-wise diff invents about +20 differences that are the same checks at moved positions. An earlier revision +said "only the 17th significant figure of the audio `r`", which named one of the +37 and read as exhaustive. Artefacts: `recheck.txt`, `recheck.json` and +`recheck-cross.txt` beside the originals. **Verdict, by §10.5's registered reading: `visibly different`.** Any check failing selects that branch, and it is *a finding about a change already on diff --git a/.agents/specs/ltx25-dit-attn-flash.md b/.agents/specs/ltx25-dit-attn-flash.md index 1ab225151..7227883ff 100644 --- a/.agents/specs/ltx25-dit-attn-flash.md +++ b/.agents/specs/ltx25-dit-attn-flash.md @@ -1122,14 +1122,18 @@ on all three arms. **THE DENOMINATOR MOVED TOO, and it is reconciled rather than quietly dropped.** This row's title and §0 quote the naive path at **47.84 s** a forward. The -same-lease naive arm is **45.547 s**, 4.8% lower. §7.1 measured the +same-lease naive arm is **45.547 s**, 4.79% below it. §7.1 measured the `runguard.py --stack-period 12` sampler that produced the 47.84 s figure at about -3.2%, which accounts for 46.31 s of it and leaves **roughly 1.6% unexplained** — -a different lease, a different prompt and a different binary are each candidates -and none is measured. The 1.6% is recorded as open rather than attributed. It -does not move the ratio, because both arms of the 7.112x were taken in one lease -with no sampler on either side; what it means is that **47.84 s is a superseded -number and 45.547 s is this row's naive denominator.** +3.2%, which puts the unsampled figure at **46.31 s** if the 3.2% is read as a +fraction removed and **46.36 s** if it is read as an overhead added — so the +residual against 45.547 s is **1.7% to 1.8%**, and it is recorded as open rather +than attributed. A different lease, a different prompt and a different binary are +each candidates and none is measured. (Subtracting the two percentages directly +gives 1.6 points, which is a percentage-POINT difference and not the residual +ratio; the two are not the same and only the second is quoted here.) None of this +moves the ratio, because both arms of the 7.112x were taken in one lease with no +sampler on either side. What it means is that **47.84 s is a superseded number +and 45.547 s is this row's naive denominator.** **The cross-check against the 20260820 baseline, which is context and never the control.** That render came from `a50c57d69`, an ancestor of the swap, on the @@ -1158,13 +1162,24 @@ exit 1. **The re-run is an artefact rather than an assertion:** `recheck.txt`, `recheck.json` and `recheck-cross.txt` sit beside the originals in the evidence directory, so a reader re-derives this row instead of taking it. -**What differs between the two JSONs, stated exhaustively because an earlier -revision claimed it was one figure.** 37 numeric leaves differ, each by at most -`1.9e-15` relative — float64 last-ULP, a `numpy` version difference, and it moves -no printed value and no check. The remainder are the input paths, which are a -different mount, and three keys the newer tool ADDS (`treatment_verdict`, -`control_verdict`, `control_ratio.unusable`) — precisely the exit-3 machinery -`12c880a52` introduced, which is why the staged tool could not have returned a 3. +**What differs between the two JSONs, because an earlier revision claimed it was +one figure.** Comparing leaf by leaf on matching key paths: **37 numeric values +differ, each by at most `1.9e-15` relative** — float64 last-ULP, a `numpy` +version difference. It moves no printed value, no check result and no verdict. +Everything else is structural rather than numeric, and it is all the repair: + +- the **input paths**, which are a different mount; +- the `checks` array grows from **12 entries to 15**, with **nothing removed** — + the three additions are exactly `content.flash-ctl.not_uniform`, + `content.flash-ctl.distinct_frames` and `content.flash-ctl.motion`; +- three keys appear that the staged tool never wrote: `treatment_verdict`, + `control_verdict` and `control_ratio.unusable`. + +Those last two bullets ARE the exit-3 machinery `12c880a52` introduced, which is +the direct evidence that the staged tool could not have returned a 3. **Do not +diff the `checks` array by index**: the three insertions shift the tail, so an +index-wise comparison reports around 20 spurious "differences" that are the same +checks at moved positions. Compare by `name`. #### The four leases this measurement cost From 2262ed5e1c5c54c776e1af83ded49433b4633249 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 23 Aug 2026 02:02:13 +0000 Subject: [PATCH 14/19] evidence(#1612): prove the exit-3 control guard by mutation instead of by reading it Section 10.7 claimed the staged tool "could not have returned a 3" and rested that on the absence of three keys in its JSON. That is inference from a schema. The guard those keys belong to is the one section 10.5 says previously exited 0 with verdict PASS and R = 112.77 on a control of one-colour frames, upgrading the published conclusion to the stronger null, so it is worth more than an inference. Mutation: a degenerate control was synthesised at this geometry - 49 frames of a single flat colour, with a real audio.wav - and the committed tool run with BOTH arms set to flash, so the treatment is bit-identical and passes every check it has. The six arm C0 checks pass, video.bit_identical and audio.bit_identical pass, the three control checks fail (49 near-uniform frames, 1 distinct of 49, 48 zero-motion pairs), and the run exits 3 with CONTROL_DEGENERATE rather than 0 with PASS. That is the exact scenario the guard exists for, and it separates cleanly: the treatment's own verdict is untouched and the control's failure is reported beside it rather than folded into it. The artefact is recorded as degen.txt beside the run's other evidence, so the proof is re-derivable rather than asserted. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --- .agents/specs/ltx25-dit-attn-flash.md | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/.agents/specs/ltx25-dit-attn-flash.md b/.agents/specs/ltx25-dit-attn-flash.md index 7227883ff..3ede3071f 100644 --- a/.agents/specs/ltx25-dit-attn-flash.md +++ b/.agents/specs/ltx25-dit-attn-flash.md @@ -1181,6 +1181,26 @@ diff the `checks` array by index**: the three insertions shift the tail, so an index-wise comparison reports around 20 spurious "differences" that are the same checks at moved positions. Compare by `name`. +**AND THE EXIT-3 PATH IS PROVED BY MUTATION RATHER THAN BY READING IT.** A +degenerate control was synthesised — 49 frames of one flat colour at this +geometry, with a real `audio.wav` — and the committed tool run against it with +both arms set to `flash`, so the TREATMENT is bit-identical and passes every +check it has: + +``` +[PASS] video.bit_identical / audio.bit_identical, and all six arm C0 checks +[FAIL] content.degen.not_uniform near-uniform frames 49 == 0 (min var 0.000) +[FAIL] content.degen.distinct_frames 1 distinct of 49 +[FAIL] content.degen.motion zero-motion pairs 48, mean adjacent MAD 0.0000 +VERDICT CONTROL_DEGENERATE (exit 3) +``` + +That is exactly the case §10.5 records as having previously exited **0** with +verdict `PASS` and `R = 112.77` — a control of one-colour frames upgrading the +published conclusion to the STRONGER null. The repair holds: the treatment's own +verdict is untouched, the control's failure is reported separately, and the run +exits 3 rather than 0. Artefact: `degen.txt` beside the other evidence. + #### The four leases this measurement cost Kept because §10.3's start gate and `RUN_ID` resume were built out of them, and From 4a3661529ad3e5566b7b1696c20a8389575d215d Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 23 Aug 2026 02:16:21 +0000 Subject: [PATCH 15/19] fix(#1612): the delta paragraph was wrong a third time, so it is now a measured diff A second fresh review returned FAIL on three new overclaims, all of them in the paragraph written to repair a false-exhaustiveness finding, and all of them in the pull request body that becomes the squashed commit message. Repaired against a measured diff rather than a description. FIRST, "no printed figure changes" is refuted by the artefact the same paragraph cites. `diff pixel-compare.txt recheck.txt` moves the audio pearson_r, 0.932682102497646 against 0.9326821024976478, and the same value again in its check-detail line. The deleted sentence it replaced - "only the 17th significant figure of the audio r" - was accurate, and replacing it with an absolute made the record less true rather than more. The honest statement is that exactly one printed FIGURE moves, no check result moves and no verdict moves. SECOND, the enumeration presented as exhaustive covered 3 of about 27 new leaves. It omitted that `12c880a52` adds a `judges` field to EVERY check, 0 of 12 before and 15 of 15 after, and the 9 leaves of the three new `content.flash-ctl.*` check objects. That is the same defect at eight times the scale, in the sentence written to fix it. THIRD, the `<= 1.9e-15` bound does not hold. The maximum is 1.90457e-15, set by that same pearson_r, and it is 16 ULP rather than the "last-ULP" claimed; every other leaf is 1 to 4 ULP. The bound is now 2e-15 and the outlier is named. Two further corrections, both of which this branch introduced itself. The denominator reconciliation is reverted to ~1.63%. Section 7.1 records the stack sampler as an absolute ~1.54 s a forward, so the unsampled figure is 47.84 - 1.54 = 46.30 and the residual against 45.547 is 1.63%. A previous commit read the derived 3.2% as a multiplier, got 46.36 and quoted 1.7 to 1.8%, which compounded a derived percentage back through a different base. Take the absolute. The "device-vs-golden maxima reach 1.49e-07" clause is withdrawn rather than patched. That reading sits inside a CPU BACKEND case, which is the exact CPU-versus-CUDA conflation the adjacent sentence exists to correct, and the log carries several comparison families on different tolerances - the bf16 keyframe arm legitimately reads 3.31e-03 - so no single number summarises it. One named family with its own tolerance beside it, and nothing wider. Also retires the last "one f32 ULP" in the `## Owed` bullet, which sat two lines below the 8.94e-08 bound the previous commit claimed to have installed. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --- .agents/benchmark-record.md | 53 +++++++----- .agents/issue-index.md | 2 +- .agents/specs/ltx25-dit-attn-flash.md | 112 ++++++++++++++++---------- 3 files changed, 104 insertions(+), 63 deletions(-) diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md index 6c9eba656..8635675d0 100644 --- a/.agents/benchmark-record.md +++ b/.agents/benchmark-record.md @@ -103,10 +103,11 @@ table beneath it refutes. **THE KERNEL AGREES WITH ITS REFERENCE WHEREVER THIS TREE CAN MEASURE IT.** `test_ltx2_device` in the same lease: 22/22 cases, 749/749 assertions, `SUCCESS!`, with the **device-vs-host maximum at `8.94e-08`** against a committed -tolerance of `2e-5`. Quote the maximum, not the minimum: that family ranges from -`5.96e-08` to `8.94e-08` and the device-vs-golden maxima in the same log reach -`1.49e-07`, and every one of them clears `2e-5` by two orders of magnitude, so -the conclusion never needed the smallest reading. **Beside it, +tolerance of `2e-5`. Quote the maximum of a NAMED family, not the minimum and not +a summary of the whole log: the lines labelled `device-vs-host` range from +`5.96e-08` to `8.94e-08`, the largest still clears `2e-5` by two orders of +magnitude, and the log carries other families on other tolerances — the bf16 +keyframe arm legitimately reads `3.31e-03` — so no single figure describes it. **Beside it, `kAttentionDenseFlash selections = 8 (want 8), kAttention selections = 0` is a CPU-backend routing count on `ReducedParams`** — it proves the knob routes at fixture size, not that the CUDA kernel is exact at head_dim 128 with 2352 keys @@ -142,21 +143,35 @@ checks and the phase [L] `*)` arm — so the run was made by a tool that could n return exit 3. The whole comparison was re-run at head `7597cd741` over the same frames (no GPU needed) and reproduces every figure to the digit, with the control's three C0 checks now executed and green and the verdict unchanged at -exit 1. **No printed figure, no check result and no verdict changes.** Comparing -the two JSONs leaf by leaf on matching key paths: **37 numeric values differ, -every one by at most `1.9e-15` relative** — float64 last-ULP, a `numpy` version -difference. Everything else is structural and is the repair itself: the input -paths (a different mount); the `checks` array growing from **12 to 15 with -nothing removed**, the additions being exactly `content.flash-ctl.not_uniform`, -`.distinct_frames` and `.motion`; and three keys the staged tool never wrote — -`treatment_verdict`, `control_verdict`, `control_ratio.unusable` — which are the -exit-3 machinery `12c880a52` introduced and the direct evidence that the staged -tool could not have returned a 3. **Compare the `checks` array by `name`, not by -index**: the three insertions shift the tail and an index-wise diff invents about -20 differences that are the same checks at moved positions. An earlier revision -said "only the 17th significant figure of the audio `r`", which named one of the -37 and read as exhaustive. Artefacts: `recheck.txt`, `recheck.json` and -`recheck-cross.txt` beside the originals. +exit 1. **Every check result and the verdict are unchanged.** The delta is +enumerated against a measured diff, because this paragraph has been wrong twice. + +In the printed report, **exactly one FIGURE moves**: the audio `pearson_r`, +`0.932682102497646` against `0.9326821024976478`, and the same value again in its +check-detail line; the check still reads `[FAIL]`. The rest of the printed +difference is the repair's own output — two section headers naming which checks +decide the verdict, three `content.flash-ctl.*` lines, and `VERDICT FAIL` gaining +`(exit 1)`. A revision that said "no printed figure changes" was refuted by the +diff it cited. + +In the JSON, compared leaf by leaf with `checks` keyed by **`name`** rather than +index: **37 numeric values differ, every one by at most `2e-15` relative**, the +bound set by that same `pearson_r` at `1.90457e-15`, which is **16 ULP** and not +one — every other leaf is 1 to 4 ULP. Structurally: the input paths; the `checks` +array growing **12 to 15 with nothing removed** (`content.flash-ctl.not_uniform`, +`.distinct_frames`, `.motion`, 9 leaves); **every check gaining a `judges` +field**, `0 of 12` before and `15 of 15` after; and `treatment_verdict`, +`control_verdict`, `control_ratio.unusable`. That is about 27 new leaves, not +three — an earlier revision enumerated only the last group and called it +exhaustive, repeating at eight times the scale the defect it was fixing. + +The `content.flash-ctl.*` checks and the verdict keys ARE the exit-3 machinery +`12c880a52` introduced, and are the direct evidence the staged tool could not +have returned a 3. **Never diff `checks` by index**: the insertions shift the +tail and an index-wise diff invents about 20 differences that are the same checks +moved, and the `37` is not reproducible without keying by `name`. Artefacts: +`recheck.txt`, `recheck.json`, `recheck-cross.txt` and `degen.txt` beside the +originals. **Verdict, by §10.5's registered reading: `visibly different`.** Any check failing selects that branch, and it is *a finding about a change already on diff --git a/.agents/issue-index.md b/.agents/issue-index.md index 452746690..5574aeb4f 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -629,7 +629,7 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#1691](https://github.com/mudler/vllm.cpp/issues/1691) | `DOCS-MODELS-HUMAN` | **`docs/QUICKSTART.md` said no container lane had ever been published, and three had.** The page carried `The container lanes have never been published, so no tag below resolves against the registry yet` and, on its `docker run` line, `The package carries only a stage tag today, which is a build artifact and not a lane`. Both were true when written and false on 2026-08-22: `ghcr.io/mudler/vllm.cpp` is public and carries `main-cpu`, `main-cuda` and `main-vulkan`, each a `linux/amd64` + `linux/arm64` manifest, published 07:30 UTC that day. `docs/guides/container-images.md` already documented the `:main-` tags, so the two pages disagreed. VERIFIED by execution rather than by reading the registry listing: `docker run --rm ghcr.io/mudler/vllm.cpp:main-cpu --version` answered `vllm.cpp 0.0.3 c-abi=23` at digest `sha256:7f88301ea282dad778748929e7aa6869d2418c8d295eef0e7900cca8310d06e5`, and the same image with a mounted `Qwen/Qwen3-0.6B` returned tokens through `/v1/completions` on host `mudler-ubuntu-box` (x86_64, Docker 29.1.2). The image also parses the `vllm_cpp` weight-residency document and echoes `mmap=on prefault=off expert_stream=on expert_stream_slots=4000`, which is what lets the Qwen3.8 2.4T page carry a container form of its recipe. FIXED IN FLOW: the false note is corrected, the `docker run` line names a tag that resolves, and the executed-row table gains its first real row. **This does NOT close [#1281](https://github.com/mudler/vllm.cpp/issues/1281)**: `:latest` still does not exist, `--model org/repo` is still blocked by [#1511](https://github.com/mudler/vllm.cpp/issues/1511) so the executed row mounts a local directory, and no GPU-lane row was run. The executed-row obligation stays owned by #1281 | bug | | [#1690](https://github.com/mudler/vllm.cpp/issues/1690) | `DOC-README-ABI-LANDING-SOURCE` | **`check-doc-checkpoint.py`'s `LANDING_SOURCE_FILES` omitted `include/vllm.h`, so a commit that bumps `VLLM_ABI_VERSION` could never repair the README claim it invalidated.** Found by the review of [PR #1655](https://github.com/mudler/vllm.cpp/pull/1655). The README `## Use it as a library (C API)` block quotes the ABI version out of the header; the header was in `USER_USAGE_FILES` (so an ABI change owes `docs/USAGE.md`) but not in `LANDING_SOURCE_FILES`, and the README rule refuses a claim change unaccompanied by a landing source. The claim could therefore be invalidated but not repaired by the same edit, which is how the README reached `VLLM_ABI_VERSION 21` against a header reading `23`, alongside a stale "46 exported functions" for a header declaring 47. That second half is stale by one rather than by six -- an earlier review figure of 51/52 swept in the `#define VLLM_API` visibility block and counted `vllm_*` identifiers that are typedefs and struct fields rather than exported functions -- so the case for deleting the count is that a live count of one file stored in another goes stale on any ABI addition, not that it is badly wrong. The set's own criterion already admitted it -- the checker's comment says every member is "something the README QUOTES" -- and the header was the only such source missing. FIXED IN FLOW: `include/vllm.h` is added to the set, red-before/green-after pinned by `test_the_c_abi_header_is_a_landing_source`, with `test_the_c_abi_header_permits_but_does_not_demand_readme` proving no new README obligation and the pre-existing no-class tests still green | bug | | [#1692](https://github.com/mudler/vllm.cpp/issues/1692) | `KERNEL-ACCEL-PROVIDER-DECLINE-EXACT` | **[#1584](https://github.com/mudler/vllm.cpp/issues/1584) is FIXED by this row, and its gate is CPU-only: both production call sites need a GPU and neither arm was executed.** The fix adds `GetOpFallbackUncounted`, sharing one body with `GetOpFallback` so resolution order, the reference-tier install, the drain and every throw stay identical and only the `fetch_add` is conditional -- in its existing position, BEFORE the "nothing below" check, because a decline that throws is still a decline. The two SHAPE-GATED providers that hoist the lookup into a function-local static switch to it: `cuda_attention_cross.cu` `BlockedFallback` and `metal_mlx_provider.mm` `MlxFallback`. **Candidate 1 of #1584 -- drop the count from `GetOpFallback` -- was REJECTED and measured rather than argued:** it edits five per-call sites (`vulkan_ops.cpp:950,1067,1488,1509`, `tenstorrent_ops.cpp:1341`), one already gated on an exact value at `test_vulkan_backend.cpp:2901` (`after.declines == before.declines + 1`), and its failure mode for a future caller is silent UNDER-counting, the Risk 4 the seam exists to expose. Net blast radius is ZERO backends, not the four #1584 estimated; exactly two providers change what they report, by one. #1555's `WarmDeclineOnce` workaround is REMOVED. **What this issue owns is the missing EXECUTION:** `test_ops_attention_cross` on CUDA (20 cases, ALL 20 SKIP on a CPU-only build -- 32 assertions, every one the skip guard, so that suite gives the change no coverage here), `test_metal_backend` on a `VLLM_CPP_MLX` build, and the `.agents/reachability.md` mutation on `BlockedFallback()` / `MlxFallback()`. The CUDA COMPILE is covered by the `-DVLLM_CPP_CUDA=ON` `vllm`-target job in `.github/workflows/ci.yml`; the Metal `.mm` file is compiled by NO job in this repository, because MLX needs `MLX_ROOT`. Red-before/green-after and the mutation table in [op-provider-decline-exact.md](specs/op-provider-decline-exact.md) | bug | -| [#1743](https://github.com/mudler/vllm.cpp/issues/1743) | `LTX25-DIT-ATTN-FLASH` | **The DiT attention swap on `main` (`90e8c3c85`, [#1549](https://github.com/mudler/vllm.cpp/issues/1549)) changes what LTX-2.5 RENDERS, by more than one pixel of global image shift.** Measured at `768x448/49f`, seed `20260820`, one binary (`834cec55...`), one `dgx:gpu0` lease (`rc` job `acff8e89-d704-4f17-a9f2-d354aba53b0d`), against the criterion registered in `.agents/specs/ltx25-dit-attn-flash.md` §10.4 BEFORE the renders were taken. **All six registered checks fail**: mean \|delta\| RGB **6.414156** against `<= 1.0`, worst-frame PSNR **22.269 dB** against `>= 40` (aggregate 25.822), worst-frame SSIM **0.880694** against `>= 0.99`, V4 luma \|delta\| over adjacent-frame MAD **0.709189** against `<= 0.10`, audio PSNR **29.368 dB** against `>= 40`, audio Pearson r **0.932682** against `>= 0.999`. 0 of 49 frames bit-identical, max \|delta\| 253 of 255, and 98.9-99.7% of the pixels in EVERY frame differ, with a broad unimodal histogram rather than a small mean hiding a tail. **THE CONTROL MAKES IT ATTRIBUTABLE**: a third render, `flash` again on the same binary and seed, is BIT-IDENTICAL (49/49, max \|delta\| 0, PSNR inf, SSIM 1.000000) and passes its own C0 content checks, so `R = 0.000000`, the run-to-run noise floor is nil, and every bit of the delta is the swapped op. Routing is proved two-sided per arm from each arm's own `VT_OP_PROVIDER_STATS=1` log (`op18=0 op21=1` flash, `op18=1 op21=0` naive). **THE KERNEL AGREES WITH ITS REFERENCE WHEREVER THIS TREE CAN MEASURE IT**: `test_ltx2_device` in the same lease reads 22/22 cases, 749/749 assertions, device-vs-host MAXIMUM **8.94e-08** against a committed `2e-5` tolerance (that family runs 5.96e-08 to 8.94e-08 and the device-vs-golden maxima reach 1.49e-07; every one clears the tolerance by two orders of magnitude, so quote the bound rather than the smallest reading). The `8 of 8` flash-selection count beside it is a **CPU-backend** routing count on `ReducedParams` - it proves the knob routes at fixture size, NOT that the CUDA kernel is exact at head_dim 128 / 2352 keys / 48 layers, which no gate in this tree reaches; the CUDA routing proof for this run is the render's own op log. `vt::Attention` and `vt::AttentionDenseFlash` differ only in association, and the 4.0e4-1.7e5 single-ULP bf16 flips per forward that §10.2 predicted AMPLIFY instead of damping across the render's **30 sampler steps at 4 DiT forwards each, 120 forwards** - about 2.5 orders of magnitude on mean \|delta\|, and 322x the ±1 LSB dither row. Against one pixel of global horizontal shift (5.183 / 28.1 dB / 0.8705 / 0.624) the delta is worse on **three of four axes**: mean \|d\|, PSNR and V4 are worse, while SSIM is a similarity and the measured 0.880694 worst / 0.901395 mean sit slightly ABOVE the shift's 0.8705. Amplification is the DEDUCTION joining a ~1e-7 op bound to a 6.414 pixel delta, not a measured quantity. Divergence also grows along the frame axis (r=+0.753 index vs mean \|delta\|, r=-0.828 vs SSIM). **CONTEXT, not a control**: the 20260820 baseline (`a50c57d69`, an ancestor, naive path) against today's naive arm reads mean \|delta\| 9.452407, PSNR 22.841 dB, V4 1.026 - so two NAIVE renders across builds diverge MORE than flash-vs-naive at one build, and the trajectory is unstable under any arithmetic perturbation rather than under this one. NOT FIXED IN FLOW and deliberately not: §9 forbids widening a threshold to admit it, no number in §10.4 moved, and whether a **7.112x** arm that renders a different video stays the default is a product decision. Neither arm is established as CORRECT - every figure is a difference between two renders, so an absolute reference is owed and is a different row. Recorded in [`ltx25-dit-attn-flash.md`](specs/ltx25-dit-attn-flash.md) §10.7; evidence `/mnt/nas_share/rc/ltx25-attnflash/pixel-ab/1612-r3/`. Measured under [#1612](https://github.com/mudler/vllm.cpp/issues/1612) | bug | +| [#1743](https://github.com/mudler/vllm.cpp/issues/1743) | `LTX25-DIT-ATTN-FLASH` | **The DiT attention swap on `main` (`90e8c3c85`, [#1549](https://github.com/mudler/vllm.cpp/issues/1549)) changes what LTX-2.5 RENDERS, by more than one pixel of global image shift.** Measured at `768x448/49f`, seed `20260820`, one binary (`834cec55...`), one `dgx:gpu0` lease (`rc` job `acff8e89-d704-4f17-a9f2-d354aba53b0d`), against the criterion registered in `.agents/specs/ltx25-dit-attn-flash.md` §10.4 BEFORE the renders were taken. **All six registered checks fail**: mean \|delta\| RGB **6.414156** against `<= 1.0`, worst-frame PSNR **22.269 dB** against `>= 40` (aggregate 25.822), worst-frame SSIM **0.880694** against `>= 0.99`, V4 luma \|delta\| over adjacent-frame MAD **0.709189** against `<= 0.10`, audio PSNR **29.368 dB** against `>= 40`, audio Pearson r **0.932682** against `>= 0.999`. 0 of 49 frames bit-identical, max \|delta\| 253 of 255, and 98.9-99.7% of the pixels in EVERY frame differ, with a broad unimodal histogram rather than a small mean hiding a tail. **THE CONTROL MAKES IT ATTRIBUTABLE**: a third render, `flash` again on the same binary and seed, is BIT-IDENTICAL (49/49, max \|delta\| 0, PSNR inf, SSIM 1.000000) and passes its own C0 content checks, so `R = 0.000000`, the run-to-run noise floor is nil, and every bit of the delta is the swapped op. Routing is proved two-sided per arm from each arm's own `VT_OP_PROVIDER_STATS=1` log (`op18=0 op21=1` flash, `op18=1 op21=0` naive). **THE KERNEL AGREES WITH ITS REFERENCE WHEREVER THIS TREE CAN MEASURE IT**: `test_ltx2_device` in the same lease reads 22/22 cases, 749/749 assertions, device-vs-host MAXIMUM **8.94e-08** against a committed `2e-5` tolerance (the lines labelled `device-vs-host` run 5.96e-08 to 8.94e-08 and the largest still clears the tolerance by two orders of magnitude, so quote a named family's maximum rather than its smallest reading; the log also carries other families on other tolerances, the bf16 keyframe arm legitimately reading 3.31e-03, so no single number summarises it). The `8 of 8` flash-selection count beside it is a **CPU-backend** routing count on `ReducedParams` - it proves the knob routes at fixture size, NOT that the CUDA kernel is exact at head_dim 128 / 2352 keys / 48 layers, which no gate in this tree reaches; the CUDA routing proof for this run is the render's own op log. `vt::Attention` and `vt::AttentionDenseFlash` differ only in association, and the 4.0e4-1.7e5 single-ULP bf16 flips per forward that §10.2 predicted AMPLIFY instead of damping across the render's **30 sampler steps at 4 DiT forwards each, 120 forwards** - about 2.5 orders of magnitude on mean \|delta\|, and 322x the ±1 LSB dither row. Against one pixel of global horizontal shift (5.183 / 28.1 dB / 0.8705 / 0.624) the delta is worse on **three of four axes**: mean \|d\|, PSNR and V4 are worse, while SSIM is a similarity and the measured 0.880694 worst / 0.901395 mean sit slightly ABOVE the shift's 0.8705. Amplification is the DEDUCTION joining a ~1e-7 op bound to a 6.414 pixel delta, not a measured quantity. Divergence also grows along the frame axis (r=+0.753 index vs mean \|delta\|, r=-0.828 vs SSIM). **CONTEXT, not a control**: the 20260820 baseline (`a50c57d69`, an ancestor, naive path) against today's naive arm reads mean \|delta\| 9.452407, PSNR 22.841 dB, V4 1.026 - so two NAIVE renders across builds diverge MORE than flash-vs-naive at one build, and the trajectory is unstable under any arithmetic perturbation rather than under this one. NOT FIXED IN FLOW and deliberately not: §9 forbids widening a threshold to admit it, no number in §10.4 moved, and whether a **7.112x** arm that renders a different video stays the default is a product decision. Neither arm is established as CORRECT - every figure is a difference between two renders, so an absolute reference is owed and is a different row. Recorded in [`ltx25-dit-attn-flash.md`](specs/ltx25-dit-attn-flash.md) §10.7; evidence `/mnt/nas_share/rc/ltx25-attnflash/pixel-ab/1612-r3/`. Measured under [#1612](https://github.com/mudler/vllm.cpp/issues/1612) | bug | | [#1649](https://github.com/mudler/vllm.cpp/issues/1649) | `GATE-WINDOWS-PORTABILITY-TARGET-SCOPE` | **`check-windows-portability.py` read `/w` anywhere in `CMakeLists.txt` as a project-wide negation of `/W4 /WX`, so the vendored BoringSSL targets' PRIVATE `/w` red `windows-msvc-cpu` and `windows-msvc-vulkan` on main and on every PR.** `check()` set `warnings = cmake` -- the whole file -- concatenated `cmake/CompilerWarnings.cmake` and did a flat token search, which cannot tell a global `add_compile_options(/w)` from a `target_compile_options( PRIVATE /w)`; only the first negates the policy. The refusal lands before any translation unit is read, so the job carries no `error C####`. SECOND red, which #1649 did not record: the same defect fails this checker's own suite -- `test_real_tree_msvc_warning_policy_reaches_the_cxx_compile` with `negation='/w'` on main at `8540a2755` (78 tests, 1 failure) -- so the tree asserted the contradiction in two places at once and one fix clears both. FIXED IN FLOW: `without_foreign_target_compile_options` blanks `target_compile_options` spans whose target PROVABLY names only targets this project never declares (`add_library`/`add_executable` first arguments, with `foreach` bindings resolved); anything unresolved stays in scope, and `cmake/CompilerWarnings.cmake` is kept whole because it applies the policy through an unresolvable function parameter. The widening is bounded by three guard properties proved discriminating by MUTATION -- forcing `_target_is_foreign` to `return True` reds exactly those three and nothing else (82 tests, 3 failures), tree restored byte-for-byte. 82 tests OK; the real tree now prints `Windows portability contract OK`. RESIDUAL, stated not hidden: a negation reaching a project target through a `set()` binding rather than a `foreach()` one is still not caught; no such construct exists in the tree today. Recorded under [`gate-windows-portability-target-scope.md`](specs/gate-windows-portability-target-scope.md) `## 4. Risks / decisions` | bug | | [#961](https://github.com/mudler/vllm.cpp/issues/961) | `BACKEND-CUDA-SM110` | `tests/tools/test_online_gate_startup.py:259` guards `shellcheck` absence with a check that cannot fire, so an absent instrument reads as a code verdict: `test_serve_low_tools` raises `FileNotFoundError: 'shellcheck'` instead of skipping. Filed by the sm_110 baseline lane ([#955](https://github.com/mudler/vllm.cpp/issues/955)), where the leased `thor:gpu0` worker carries no `shellcheck`, so the baseline names the failure as a known entry rather than an sm_110 fact. Indexed late: the issue was opened 2026-08-15 and its index row was lost with the unmerged repair of PR [#956](https://github.com/mudler/vllm.cpp/pull/956) | bug | | [#962](https://github.com/mudler/vllm.cpp/issues/962) | `BACKEND-CUDA-SM110` | NVFP4 marlin disagrees with itself on sm_110: `test_ops_moe_grouped:1262` fails `CHECK(bitdiff == 0)` on `NVFP4 block8-vs-block16 M=8 K=4096 N=4096 bitdiff=15/32768`, against a kernel configure reports as `marlin-nvfp4: ENABLED for [110]`. A live kernel defect on an ENABLED feature, not an absent one, and the only substantive standing sm_110 finding in the [#955](https://github.com/mudler/vllm.cpp/issues/955) baseline. Indexed late, same cause as the row above | bug | diff --git a/.agents/specs/ltx25-dit-attn-flash.md b/.agents/specs/ltx25-dit-attn-flash.md index 3ede3071f..1a36770eb 100644 --- a/.agents/specs/ltx25-dit-attn-flash.md +++ b/.agents/specs/ltx25-dit-attn-flash.md @@ -540,7 +540,7 @@ never a synonym for "probably fine". | pixel A/B at production geometry | `dgx:gpu0` under an `rc` lease, `scripts/ltx25-dit-attn-flash-pixel-ab.sh` | **FAIL, and the failure is the finding** — all four V and both A checks fail: mean \|delta\| 6.414 against `<= 1.0`, worst PSNR 22.269 dB against `>= 40`, worst SSIM 0.880694 against `>= 0.99`, V4 0.709 against `<= 0.10`, audio 29.368 dB and r 0.932682. Criterion registered in §10.4 before the run; result and reading in §10.7. §10.5 selects **visibly different**, filed as [#1743](https://github.com/mudler/vllm.cpp/issues/1743). No threshold moved (§9) | | run-to-run control (`flash` twice) | the same lease | **PASS** — `flash-ctl` is **bit-identical** to `flash`, 49/49 frames, max \|delta\| 0, PSNR inf, SSIM 1.000000, and it passes its own C0 content checks. `R = 0.000000`, so the noise floor is nil and the whole treatment delta is the kernel's (§10.3's strongest branch) | | C0 content, all three renders | the same lease | **PASS** — 9 checks: each render has 49 distinct frames, no near-uniform frame (`min_var` 3683.8-3739.0) and no zero-motion pair. The control's three were executed by re-running the committed tool, because the staged tool predated them (§10.7) | -| the numbers reproduce under the COMMITTED tool | this checkout, no GPU | **PASS** — phase [I] ran the tool from a tarball staged at `source_sha 3e2961ef0`, two commits behind head. Re-run at `7597cd741` over the same frames: every printed figure identical, control C0 now executed and green, verdict unchanged at exit 1. Recorded as `recheck.txt` / `recheck.json` / `recheck-cross.txt` in the evidence directory, so this row is re-derivable rather than asserted. The two JSONs differ in 37 numeric leaves at `<= 1.9e-15` relative (float64 last-ULP, `numpy`), in the input paths, and in the three keys the exit-3 repair added | +| the numbers reproduce under the COMMITTED tool | this checkout, no GPU | **PASS** — phase [I] ran the tool from a tarball staged at `source_sha 3e2961ef0`, two commits behind head. Re-run at `7597cd741` over the same frames: **every check result and the verdict unchanged**, control C0 now executed and green, still exit 1. Recorded as `recheck.txt` / `recheck.json` / `recheck-cross.txt` in the evidence directory, so this row is re-derivable rather than asserted. Exactly one printed FIGURE moves, the audio `pearson_r` at its 17th significant figure, and the check it feeds still reads `[FAIL]`; the JSON differs in 37 numeric leaves at `<= 2e-15` relative (the bound is that same `pearson_r` at `1.90457e-15`, 16 ULP) plus about 27 structural leaves that ARE the repair — the `judges` field on every check, the three `content.flash-ctl.*` checks and the three verdict keys. §10.7 enumerates it against a measured diff, because this row has twice claimed less than the truth | | the comparison tool discriminates | `tests/scripts/test_ltx25_render_compare.py` | **PASS** — **45 tests, `OK`, re-run at this head on 2026-08-22** (the row read "37" before the control-C0 and `*)` tripwires were added). It needs no GPU, no lease and no NAS, so `PENDING until §10.7` was misreporting a gate that was already green: a dither passes, a one-pixel shift fails all four V checks, two all-black renders fail C0 while reading as a perfect match on every V, an unreadable input exits 2 while a threshold failure exits 1, A1 and A2 disagree on a time-shifted waveform, and the SSIM is pinned by its taps, its impulse response and three fixture values (§10.4). The count dates the run; it is not a floor to defend | | the comparison tool runs on a lane | `scripts/agent-preflight.sh`, `.github/workflows/ci.yml` | **PASS** — it ran on NO lane when it landed: absent from preflight's `SUITES`, from the enumerated python block in CI and from `tests/CMakeLists.txt`, while the row above registered it as a gate. Both are registered now. Preflight SKIPs it when numpy is absent, which is the third state and never an `ok`; the CI lane installs `python3-numpy` so the lane that must not be silent cannot be | | the harness's own preconditions | `tests/scripts/test_ltx25_pixel_ab_harness.py` | **PASS** — 27 tests, `OK`, at `2026-08-22`. The memory precondition and the arm-completeness check are extracted verbatim from the harness and run against a fabricated `/proc/meminfo`. The call sites that only a lease can execute are text tripwires and are labelled as such; §10.8 counts them and holds its own count. The count dates the run; it is not a floor to defend, and it said "19 tests" and "four call sites" after both had moved | @@ -1074,10 +1074,15 @@ amplifying direction.** The op-level difference is ~1e-7 relative and the `test_ltx2_device` run in this same lease bounds it: 22/22 cases, 749/749 assertions, `SUCCESS!`, with the **device-vs-host maximum at `8.94e-08`** against a committed tolerance of `2e-5` — the same figure §8 already carried. **Quote the -maximum and not the minimum.** That family's readings run from `5.96e-08` to -`8.94e-08`, and the device-vs-golden maxima in the same log reach `1.49e-07`; -citing the smallest is selective, and the conclusion does not need it because -every one of them clears `2e-5` by two orders of magnitude. +maximum of a NAMED family, not the minimum, and not a summary of the whole log.** +The lines labelled `device-vs-host` run from `5.96e-08` to `8.94e-08`, and an +earlier revision quoted the smallest of them, which is selective; the conclusion +does not need it, because the largest still clears `2e-5` by two orders of +magnitude. That revision also reached for a `1.49e-07` "device-vs-golden +maximum", and both halves of that were wrong: the reading sits inside a **CPU +backend** case, and the log carries several comparison families on different +tolerances — the bf16 keyframe arm reads `3.31e-03` legitimately — so no single +number summarises it. One family, named, with its own tolerance beside it. **What that test does and does NOT establish, stated because the distinction is load-bearing.** It bounds the ARITHMETIC at the fixture's reduced dimensions. The @@ -1122,18 +1127,19 @@ on all three arms. **THE DENOMINATOR MOVED TOO, and it is reconciled rather than quietly dropped.** This row's title and §0 quote the naive path at **47.84 s** a forward. The -same-lease naive arm is **45.547 s**, 4.79% below it. §7.1 measured the -`runguard.py --stack-period 12` sampler that produced the 47.84 s figure at about -3.2%, which puts the unsampled figure at **46.31 s** if the 3.2% is read as a -fraction removed and **46.36 s** if it is read as an overhead added — so the -residual against 45.547 s is **1.7% to 1.8%**, and it is recorded as open rather -than attributed. A different lease, a different prompt and a different binary are -each candidates and none is measured. (Subtracting the two percentages directly -gives 1.6 points, which is a percentage-POINT difference and not the residual -ratio; the two are not the same and only the second is quoted here.) None of this -moves the ratio, because both arms of the 7.112x were taken in one lease with no -sampler on either side. What it means is that **47.84 s is a superseded number -and 45.547 s is this row's naive denominator.** +same-lease naive arm is **45.547 s**, 4.79% below it. §7.1 records the +`runguard.py --stack-period 12` sampler cost as an **absolute ~1.54 s** a +forward, "or ~3.2% of the denominator", so the unsampled figure is +`47.84 - 1.54 = 46.30 s` and the residual against 45.547 s is **~1.63%**. That +residual is recorded as open rather than attributed: a different lease, a +different prompt and a different binary are each candidates and none is +measured. **Take the 1.54 s and not the 3.2%**, because the percentage is +derived from it and compounding a derived percentage back through a different +base is how this paragraph went wrong once already — an earlier revision read +the 3.2% as a multiplier, got 46.36 s and quoted a 1.7-1.8% residual. None of +this moves the ratio, because both arms of the 7.112x were taken in one lease +with no sampler on either side. What it means is that **47.84 s is a superseded +number and 45.547 s is this row's naive denominator.** **The cross-check against the 20260820 baseline, which is context and never the control.** That render came from `a50c57d69`, an ancestor of the swap, on the @@ -1156,30 +1162,49 @@ the control's own C0 checks and the phase [L] `*)` arm. So the run was made by a tool that could not return exit 3, and a degenerate control would have read as a plain 0 there. **The whole comparison was therefore re-run at head `7597cd741` over the same frames** — it needs no GPU, no lease and only the frames on the -share — and it reproduces every figure above to the digit, with the control's -three C0 checks now executed and all three `PASS`, and the verdict unchanged at -exit 1. **The re-run is an artefact rather than an assertion:** `recheck.txt`, -`recheck.json` and `recheck-cross.txt` sit beside the originals in the evidence -directory, so a reader re-derives this row instead of taking it. - -**What differs between the two JSONs, because an earlier revision claimed it was -one figure.** Comparing leaf by leaf on matching key paths: **37 numeric values -differ, each by at most `1.9e-15` relative** — float64 last-ULP, a `numpy` -version difference. It moves no printed value, no check result and no verdict. -Everything else is structural rather than numeric, and it is all the repair: - -- the **input paths**, which are a different mount; -- the `checks` array grows from **12 entries to 15**, with **nothing removed** — - the three additions are exactly `content.flash-ctl.not_uniform`, - `content.flash-ctl.distinct_frames` and `content.flash-ctl.motion`; -- three keys appear that the staged tool never wrote: `treatment_verdict`, - `control_verdict` and `control_ratio.unusable`. - -Those last two bullets ARE the exit-3 machinery `12c880a52` introduced, which is -the direct evidence that the staged tool could not have returned a 3. **Do not -diff the `checks` array by index**: the three insertions shift the tail, so an -index-wise comparison reports around 20 spurious "differences" that are the same -checks at moved positions. Compare by `name`. +share — and **every check result and the verdict are unchanged**, with the +control's three C0 checks now executed and all three `PASS`, still exit 1. **The +re-run is an artefact rather than an assertion:** `recheck.txt`, `recheck.json` +and `recheck-cross.txt` sit beside the originals in the evidence directory, so a +reader re-derives this row instead of taking it. + +**What differs between the two runs. This paragraph has been wrong twice, so it +is enumerated against a measured diff rather than described.** First the printed +report, `diff pixel-compare.txt recheck.txt`: + +- **one printed FIGURE moves**: the audio `pearson_r`, `0.932682102497646` + against `0.9326821024976478`, and the same value again in its check-detail + line. Nothing else numeric changes, and the check still reads `[FAIL]`. An + earlier revision said "no printed figure changes", which this diff refutes; +- the rest of the printed difference is the repair's own output — the two + section headers naming which checks decide the verdict, the three + `content.flash-ctl.*` lines, and `VERDICT FAIL (exit 1)` gaining its status. + +Then the JSON, compared leaf by leaf with the `checks` array keyed by **`name`** +and not by index: + +- **37 numeric values differ, every one by at most `2e-15` relative.** The bound + is set by that same `pearson_r`, at `1.90457e-15`, which is **16 ULP** rather + than one; every other leaf is 1 to 4 ULP. An earlier revision wrote + `<= 1.9e-15`, which the maximum exceeds, and called the outlier "last-ULP"; +- the **input paths**, a different mount; +- the `checks` array grows from **12 entries to 15**, nothing removed, the + additions being exactly `content.flash-ctl.not_uniform`, `.distinct_frames` + and `.motion` — 9 new leaves, three fields each; +- **every check gains a `judges` field**, `0 of 12` before and `15 of 15` after; +- three further keys appear: `treatment_verdict`, `control_verdict` and + `control_ratio.unusable`. + +Those last three bullets are ~27 new leaves, not three. An earlier revision +enumerated only the final one and called the list exhaustive, which is the same +defect it was written to repair, at eight times the scale. + +The `content.flash-ctl.*` checks and the three verdict keys ARE the exit-3 +machinery `12c880a52` introduced, which is the direct evidence that the staged +tool could not have returned a 3. **Do not diff the `checks` array by index**: +the three insertions shift the tail, so an index-wise comparison reports around +20 spurious "differences" that are the same checks at moved positions, and the +`37` above is not reproducible without keying by `name`. **AND THE EXIT-3 PATH IS PROVED BY MUTATION RATHER THAN BY READING IT.** A degenerate control was synthesised — 49 frames of one flat colour at this @@ -1410,8 +1435,9 @@ is a same-lease pair rather than a cross-run range.** divergence itself and the question of whether the arm stays the default. The reduced-dimension host-vs-device case still bounds only the ARITHMETIC change — `8.94e-08` / `4.47e-08` against `2e-5` — and it is now known to be a weak - predictor of the render: the kernel agrees to one f32 ULP and the video still - moves by 71% of its own motion step. Owner: this row for the record, + predictor of the render: the kernel agrees with its reference to within that + bound, two orders of magnitude inside the tolerance, and the video still moves + by 71% of its own motion step. Owner: this row for the record, #1743 for the finding. - **NEITHER ARM IS ESTABLISHED AS THE CORRECT RENDER, and nothing here can establish one.** Every figure in §10.7 is a difference between two renders, so From 439ecf116f5840ec3050f8c430f99f63aa1d043e Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 23 Aug 2026 02:35:13 +0000 Subject: [PATCH 16/19] fix(#1612): the retracted claim survived in the summaries, and 17th was the wrong digit A third fresh review confirmed the section 10.7 delta enumeration is finally correct on the JSON side - 37, 27, 12 to 15, 0 of 12 to 15 of 15, 16 ULP, 1 to 4 ULP, 1.90457e-15, all reproduced against a diff the reviewer wrote themselves - and failed the change anyway, on two things the previous commit left standing outside the paragraph it swept. FIRST, "reproduces every figure to the digit" was still in the benchmark record and in the pull request body. That is the refuted claim in other words, five lines above the sentence that refutes it, and the pull request body becomes the squashed commit message under squash_merge_commit_message = PR_BODY. Both now say what is true: every check result and the verdict are unchanged, and exactly one printed figure moves. SECOND, "the 17th significant figure" is wrong by two digit positions. The two values are 0.93268210249764604835... and 0.93268210249764782471..., which diverge at the 15th. The section 8 gate row also contradicted itself, because a 1.9e-15 relative change on a value of 0.93 IS a 15th-significant-figure change, and that bound was printed in the same sentence. The previous commit body defended the phrase as accurate; it was not. THIRD, and self-inflicted again: "~1.63%" is computed on base 46.30 while the sentence named 45.547. Both bases are now given - 0.753 s is 1.63% of the unsampled figure and 1.65% of the measured arm - in a paragraph whose whole thesis is that quietly changing a percentage base is how it went wrong the first time. On the base it named, the absolute-versus-percentage correction is worth about 0.1 points rather than the 0.15 the swap implied. Three counts are now measured rather than estimated. The index-wise trap reports 6 differing check objects and 15 differing name/pass/detail leaves, none of them real, where the text said "about 20". The three new check objects carry FOUR keys each and contribute 9 leaves here because the fourth is counted in the judges group, where the text said "three fields each". And 27 is stated as exact with its 9 + 15 + 3 decomposition rather than as "about". Also names the fenced mutation block as a paraphrase rather than a transcript, since degen.txt holds the tool's actual output, and adds the seventh diff hunk - the trailing `wrote ` line - to a printed-report enumeration that claimed to be complete. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --- .agents/benchmark-record.md | 37 +++++++------ .agents/specs/ltx25-dit-attn-flash.md | 75 ++++++++++++++++----------- 2 files changed, 67 insertions(+), 45 deletions(-) diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md index 8635675d0..afd7661b3 100644 --- a/.agents/benchmark-record.md +++ b/.agents/benchmark-record.md @@ -141,18 +141,20 @@ checked.** Phase [I] runs the comparison out of a tarball staged on the share; branch head. Both missing commits only TIGHTEN — they add the control's own C0 checks and the phase [L] `*)` arm — so the run was made by a tool that could not return exit 3. The whole comparison was re-run at head `7597cd741` over the same -frames (no GPU needed) and reproduces every figure to the digit, with the -control's three C0 checks now executed and green and the verdict unchanged at -exit 1. **Every check result and the verdict are unchanged.** The delta is -enumerated against a measured diff, because this paragraph has been wrong twice. +frames (no GPU needed), with the control's three C0 checks now executed and +green. **Every check result and the verdict are unchanged, and the verdict is +still exit 1.** The delta is enumerated against a measured diff, because this +paragraph has been wrong three times — and "reproduces every figure to the +digit" was the third, sitting five lines above the line that refutes it. In the printed report, **exactly one FIGURE moves**: the audio `pearson_r`, -`0.932682102497646` against `0.9326821024976478`, and the same value again in its -check-detail line; the check still reads `[FAIL]`. The rest of the printed -difference is the repair's own output — two section headers naming which checks -decide the verdict, three `content.flash-ctl.*` lines, and `VERDICT FAIL` gaining -`(exit 1)`. A revision that said "no printed figure changes" was refuted by the -diff it cited. +`0.932682102497646` against `0.9326821024976478` — a divergence at its **15th** +significant figure, which is what a `1.9e-15` relative change IS — and the same +value again in its check-detail line; the check still reads `[FAIL]`. The rest +of the printed difference is not a figure at all: two section headers naming +which checks decide the verdict, three `content.flash-ctl.*` lines, `VERDICT +FAIL` gaining `(exit 1)`, and the trailing `wrote ` line naming a different +output file. In the JSON, compared leaf by leaf with `checks` keyed by **`name`** rather than index: **37 numeric values differ, every one by at most `2e-15` relative**, the @@ -161,15 +163,20 @@ one — every other leaf is 1 to 4 ULP. Structurally: the input paths; the `chec array growing **12 to 15 with nothing removed** (`content.flash-ctl.not_uniform`, `.distinct_frames`, `.motion`, 9 leaves); **every check gaining a `judges` field**, `0 of 12` before and `15 of 15` after; and `treatment_verdict`, -`control_verdict`, `control_ratio.unusable`. That is about 27 new leaves, not -three — an earlier revision enumerated only the last group and called it -exhaustive, repeating at eight times the scale the defect it was fixing. +`control_verdict`, `control_ratio.unusable`. That is **exactly 27 new leaves and +0 removed** (9 + 15 + 3), measured by flattening both documents — an earlier +revision enumerated only the last group of three and called it exhaustive, +repeating at nine times the scale the defect it was fixing. Each new check +object carries four keys; the fourth is the `judges` counted in the middle +group. The `content.flash-ctl.*` checks and the verdict keys ARE the exit-3 machinery `12c880a52` introduced, and are the direct evidence the staged tool could not have returned a 3. **Never diff `checks` by index**: the insertions shift the -tail and an index-wise diff invents about 20 differences that are the same checks -moved, and the `37` is not reproducible without keying by `name`. Artefacts: +tail, and over the zipped 12 and 15 entries an index-wise diff invents **6 +differing check objects and 15 differing `name`/`pass`/`detail` leaves, none of +them real** — each is the same check at a moved position — while the `37` is not +reproducible without keying by `name`. Artefacts: `recheck.txt`, `recheck.json`, `recheck-cross.txt` and `degen.txt` beside the originals. diff --git a/.agents/specs/ltx25-dit-attn-flash.md b/.agents/specs/ltx25-dit-attn-flash.md index 1a36770eb..eb352b441 100644 --- a/.agents/specs/ltx25-dit-attn-flash.md +++ b/.agents/specs/ltx25-dit-attn-flash.md @@ -19,9 +19,9 @@ is bit-identical so the delta is entirely the kernel's, and §10.5 reads that as **visibly different** — a finding about a change already on `main`, filed as [#1743](https://github.com/mudler/vllm.cpp/issues/1743). The op-level kernel is correct to within its committed tolerance; what moves the picture is the render's -30 sampler steps, 4 DiT forwards each, amplifying that over 120 forwards. **Whether a 7.112x arm that renders a different video should stay the -default is a product decision this row does not take**, and #1743 is where it is -argued. +30 sampler steps, 4 DiT forwards each, amplifying that over 120 forwards. +**Whether a 7.112x arm that renders a different video should stay the default is +a product decision this row does not take**, and #1743 is where it is argued. The diagnosis is confirmed against the tree and the change is scoped to **one production call site**. It was once scoped to that call site plus a @@ -540,7 +540,7 @@ never a synonym for "probably fine". | pixel A/B at production geometry | `dgx:gpu0` under an `rc` lease, `scripts/ltx25-dit-attn-flash-pixel-ab.sh` | **FAIL, and the failure is the finding** — all four V and both A checks fail: mean \|delta\| 6.414 against `<= 1.0`, worst PSNR 22.269 dB against `>= 40`, worst SSIM 0.880694 against `>= 0.99`, V4 0.709 against `<= 0.10`, audio 29.368 dB and r 0.932682. Criterion registered in §10.4 before the run; result and reading in §10.7. §10.5 selects **visibly different**, filed as [#1743](https://github.com/mudler/vllm.cpp/issues/1743). No threshold moved (§9) | | run-to-run control (`flash` twice) | the same lease | **PASS** — `flash-ctl` is **bit-identical** to `flash`, 49/49 frames, max \|delta\| 0, PSNR inf, SSIM 1.000000, and it passes its own C0 content checks. `R = 0.000000`, so the noise floor is nil and the whole treatment delta is the kernel's (§10.3's strongest branch) | | C0 content, all three renders | the same lease | **PASS** — 9 checks: each render has 49 distinct frames, no near-uniform frame (`min_var` 3683.8-3739.0) and no zero-motion pair. The control's three were executed by re-running the committed tool, because the staged tool predated them (§10.7) | -| the numbers reproduce under the COMMITTED tool | this checkout, no GPU | **PASS** — phase [I] ran the tool from a tarball staged at `source_sha 3e2961ef0`, two commits behind head. Re-run at `7597cd741` over the same frames: **every check result and the verdict unchanged**, control C0 now executed and green, still exit 1. Recorded as `recheck.txt` / `recheck.json` / `recheck-cross.txt` in the evidence directory, so this row is re-derivable rather than asserted. Exactly one printed FIGURE moves, the audio `pearson_r` at its 17th significant figure, and the check it feeds still reads `[FAIL]`; the JSON differs in 37 numeric leaves at `<= 2e-15` relative (the bound is that same `pearson_r` at `1.90457e-15`, 16 ULP) plus about 27 structural leaves that ARE the repair — the `judges` field on every check, the three `content.flash-ctl.*` checks and the three verdict keys. §10.7 enumerates it against a measured diff, because this row has twice claimed less than the truth | +| the numbers reproduce under the COMMITTED tool | this checkout, no GPU | **PASS** — phase [I] ran the tool from a tarball staged at `source_sha 3e2961ef0`, two commits behind head. Re-run at `7597cd741` over the same frames: **every check result and the verdict unchanged**, control C0 now executed and green, still exit 1. Recorded as `recheck.txt` / `recheck.json` / `recheck-cross.txt` in the evidence directory, so this row is re-derivable rather than asserted. Exactly one printed FIGURE moves, the audio `pearson_r` at its **15th** significant figure, and the check it feeds still reads `[FAIL]`; the JSON differs in 37 numeric leaves at `<= 2e-15` relative (the bound is that same `pearson_r` at `1.90457e-15`, 16 ULP) plus exactly 27 structural leaves that ARE the repair — the `judges` field on every check, the three `content.flash-ctl.*` checks and the three verdict keys. §10.7 enumerates it against a measured diff, because this row has twice claimed less than the truth | | the comparison tool discriminates | `tests/scripts/test_ltx25_render_compare.py` | **PASS** — **45 tests, `OK`, re-run at this head on 2026-08-22** (the row read "37" before the control-C0 and `*)` tripwires were added). It needs no GPU, no lease and no NAS, so `PENDING until §10.7` was misreporting a gate that was already green: a dither passes, a one-pixel shift fails all four V checks, two all-black renders fail C0 while reading as a perfect match on every V, an unreadable input exits 2 while a threshold failure exits 1, A1 and A2 disagree on a time-shifted waveform, and the SSIM is pinned by its taps, its impulse response and three fixture values (§10.4). The count dates the run; it is not a floor to defend | | the comparison tool runs on a lane | `scripts/agent-preflight.sh`, `.github/workflows/ci.yml` | **PASS** — it ran on NO lane when it landed: absent from preflight's `SUITES`, from the enumerated python block in CI and from `tests/CMakeLists.txt`, while the row above registered it as a gate. Both are registered now. Preflight SKIPs it when numpy is absent, which is the third state and never an `ok`; the CI lane installs `python3-numpy` so the lane that must not be silent cannot be | | the harness's own preconditions | `tests/scripts/test_ltx25_pixel_ab_harness.py` | **PASS** — 27 tests, `OK`, at `2026-08-22`. The memory precondition and the arm-completeness check are extracted verbatim from the harness and run against a fabricated `/proc/meminfo`. The call sites that only a lease can execute are text tripwires and are labelled as such; §10.8 counts them and holds its own count. The count dates the run; it is not a floor to defend, and it said "19 tests" and "four call sites" after both had moved | @@ -548,8 +548,8 @@ never a synonym for "probably fine". | `documentation-checkpoint` | CI, and locally over the branch range | **PASS at HEAD, RED before it, and the red was THIS BRANCH's** — `2aa78c69b` and `2f39a9426` each recorded a measurement in `.agents/benchmark-record.md` without writing the public projection that then existed. The control on the main-only range `4c193bd55..5d548d003` is rc 0, so it was not inherited. Both commits were replaced by one that writes the surfaces together when the branch was rebuilt, and the checker is re-run at each head rather than trusted to have stayed fixed — a job that has stopped appearing in a failing set is not the same fact as a job that passes. **THE COUPLING THAT PRODUCED THAT RED NO LONGER EXISTS, and the row is corrected rather than left to mislead:** `1db7e59cf` deleted `docs/STATUS.md` from the tree, deleted `scripts/check-doc-checkpoint.py` and `scripts/check-public-doc-tables.py`, and reduced this job to `check-now-current.py` plus `check-role-discipline.py`. A measurement now owes the row spec's `## Now` and nothing under `docs/` unless it adds a benchmark ID, which this one does not. An earlier revision of this row named `docs/STATUS.md` as the repair, and a reader who followed it would have recreated a file that `scripts/check-site.py` reds on for want of a `nav.yaml` entry | | `build-newest-gcc` | CI | **PASS, and now green on `main` too** — it was red on `main` on `::getpid` in `test_qwen3_dflash2_gguf.cpp:547`, a file this change does not touch; [#1581](https://github.com/mudler/vllm.cpp/pull/1581) fixed it and this branch carries that fix through the merge. A red here after the merge is therefore this row's, not inherited | | `build-test-cpu`, `sanitize-cpu` (both) | CI | **INHERITED** — all three fail on the same single case, `test_runner.cpp:1557`, from #1273; owned by [#1602](https://github.com/mudler/vllm.cpp/issues/1602) and [#1608](https://github.com/mudler/vllm.cpp/issues/1608). Verified against `main` with `scripts/main-baseline.py`, not by reading a push run: those are all cancelled (#274) | -| `windows-msvc-cpu` / `-vulkan` | CI | **INHERITED, baseline-less lane** — a markdown-only control PR (#1295) fails the identical step; #584/#965 own it | -| `agent-record` | CI | **INHERITED, and it is NOT the checker its name suggests** — `scripts/check-agent-record.py` and `scripts/audit-live-rows.py` both return rc 0 locally on `main` and on this head, so the job's red is elsewhere inside it. Read by failure text rather than by job name: the job also runs `tests/scripts/test_check_site.py`, whose `test_rendered_benchmark_index_links_resolve_to_emitted_pages` shells out to `hugo` and **ERRORs with `FileNotFoundError: [Errno 2] No such file or directory: 'hugo'`** on a runner that has no `hugo`. That is the #1661 shape exactly — a guard that probes a `returncode` which a missing binary never produces — one restructure later. Newly inherited because `1db7e59cf` added the test. **The local-against-CI asymmetry is the whole reason this was misreadable**: `hugo` is on this developer's `PATH`, so `test_check_site.py` runs 7/7 `OK` here and every checker in the job returns rc 0, while the same job reds on a runner that lacks the binary. A job name is not a diagnosis, and "the checkers pass locally" was true and did not explain it. Already owned and in flight: [#1722](https://github.com/mudler/vllm.cpp/issues/1722), [PR #1726](https://github.com/mudler/vllm.cpp/pull/1726). Not repaired here, because it is another row's open pull request and duplicating it is the failure that check exists to avoid | +| `windows-msvc-cpu` / `-vulkan` | CI | **INHERITED, baseline-less lane, and the attribution is READ FROM THIS BRANCH'S OWN LOG rather than carried forward.** Job `97122086205` on head `36e596f22` prints `Windows portability contract OK` and then dies in `test_openai_api_server.exe` with `exited with status -1073740791` (`STATUS_STACK_BUFFER_OVERRUN`), immediately after `The decoder prompt (length 40) is longer than the maximum model length of 32`. **That is [#584](https://github.com/mudler/vllm.cpp/issues/584), NOT [#1649](https://github.com/mudler/vllm.cpp/issues/1649)**: #1649 is the `/W4 /WX` negated-by-`/w` refusal that fires BEFORE compilation, and [#1701](https://github.com/mudler/vllm.cpp/pull/1701) rescoped that gate, so the lane now gets past it and fails somewhere else. A post-#1701 red citing #1649 is a false attribution to an issue whose message the log no longer contains. A markdown-only control PR (#1295) fails the identical step; #584/#965 own it. **`windows-msvc-vulkan` on that head reads `cancelled`, which `gh pr checks` renders as `fail`** and which is not a verdict at all (#274) | +| `agent-record` | CI | **INHERITED, and it is NOT the checker its name suggests** — `scripts/check-agent-record.py` and `scripts/audit-live-rows.py` both return rc 0 locally on `main` and on this head, so the job's red is elsewhere inside it. Read by failure text rather than by job name: the job also runs `tests/scripts/test_check_site.py`, whose `test_rendered_benchmark_index_links_resolve_to_emitted_pages` shells out to `hugo` and **ERRORs with `FileNotFoundError: [Errno 2] No such file or directory: 'hugo'`** on a runner that has no `hugo`. That is the #1661 shape exactly — a guard that probes a `returncode` which a missing binary never produces — one restructure later. Newly inherited because `1db7e59cf` added the test. **The local-against-CI asymmetry is the whole reason this was misreadable**: `hugo` is on this developer's `PATH`, so `test_check_site.py` runs 7/7 `OK` here and every checker in the job returns rc 0, while the same job reds on a runner that lacks the binary. A job name is not a diagnosis, and "the checkers pass locally" was true and did not explain it. Already owned and in flight: [#1722](https://github.com/mudler/vllm.cpp/issues/1722), with fix [PR #1726](https://github.com/mudler/vllm.cpp/pull/1726). Not repaired here, because it is another row's open pull request and duplicating it is the failure that check exists to avoid. **[#1754](https://github.com/mudler/vllm.cpp/issues/1754) is a DUPLICATE of #1722** — same defect, same file, same job, filed later and while #1726 was already open — and both are open as of 2026-08-23. Cite #1722, which is the one carrying the fix; the duplicate is recorded here rather than silently preferred, because two open issues for one red is how a fix gets written twice | **A side effect of that red is worth recording, because it was invisible.** The `documentation-checkpoint` job runs `set -eu` and this checker is the FIRST of @@ -1130,13 +1130,19 @@ This row's title and §0 quote the naive path at **47.84 s** a forward. The same-lease naive arm is **45.547 s**, 4.79% below it. §7.1 records the `runguard.py --stack-period 12` sampler cost as an **absolute ~1.54 s** a forward, "or ~3.2% of the denominator", so the unsampled figure is -`47.84 - 1.54 = 46.30 s` and the residual against 45.547 s is **~1.63%**. That -residual is recorded as open rather than attributed: a different lease, a -different prompt and a different binary are each candidates and none is -measured. **Take the 1.54 s and not the 3.2%**, because the percentage is -derived from it and compounding a derived percentage back through a different -base is how this paragraph went wrong once already — an earlier revision read -the 3.2% as a multiplier, got 46.36 s and quoted a 1.7-1.8% residual. None of +`47.84 - 1.54 = 46.30 s` and the residual is `0.753 s`. **State the base, because +the two available ones differ and this row has already mislabelled one:** +`0.753 / 46.30 = ` **1.63%** of the unsampled figure, and `0.753 / 45.547 = ` +**1.65%** of the measured arm. Both are quoted rather than one being passed off +as the other. That residual is recorded as open rather than attributed: a +different lease, a different prompt and a different binary are each candidates +and none is measured. **Take the 1.54 s and not the 3.2%**, because the +percentage is derived from it and compounding a derived percentage back through +a different base is how this paragraph went wrong once already — an earlier +revision read the 3.2% as a multiplier, got 46.36 s and quoted a 1.7-1.8% +residual on the 45.547 base. On that same base the absolute gives 1.65%, so the +correction is worth about 0.1 points and not the 0.15 a base swap would +suggest. None of this moves the ratio, because both arms of the 7.112x were taken in one lease with no sampler on either side. What it means is that **47.84 s is a superseded number and 45.547 s is this row's naive denominator.** @@ -1168,17 +1174,20 @@ re-run is an artefact rather than an assertion:** `recheck.txt`, `recheck.json` and `recheck-cross.txt` sit beside the originals in the evidence directory, so a reader re-derives this row instead of taking it. -**What differs between the two runs. This paragraph has been wrong twice, so it -is enumerated against a measured diff rather than described.** First the printed -report, `diff pixel-compare.txt recheck.txt`: +**What differs between the two runs. This paragraph has been wrong three times, +so it is enumerated against a measured diff rather than described.** First the +printed report, `diff pixel-compare.txt recheck.txt`, which has seven hunks: - **one printed FIGURE moves**: the audio `pearson_r`, `0.932682102497646` - against `0.9326821024976478`, and the same value again in its check-detail - line. Nothing else numeric changes, and the check still reads `[FAIL]`. An - earlier revision said "no printed figure changes", which this diff refutes; -- the rest of the printed difference is the repair's own output — the two - section headers naming which checks decide the verdict, the three - `content.flash-ctl.*` lines, and `VERDICT FAIL (exit 1)` gaining its status. + against `0.9326821024976478` — they diverge at the **15th** significant + figure, which is what a `1.9e-15` relative change is — and the same value + again in its check-detail line. Nothing else numeric changes, and the check + still reads `[FAIL]`. An earlier revision said "no printed figure changes", + which this diff refutes; +- the rest is not a figure: the two section headers naming which checks decide + the verdict, the three `content.flash-ctl.*` lines, `VERDICT FAIL` gaining + `(exit 1)`, and the trailing `wrote ` line, which names a different + output file and belongs to the invocation rather than to either tool. Then the JSON, compared leaf by leaf with the `checks` array keyed by **`name`** and not by index: @@ -1190,27 +1199,33 @@ and not by index: - the **input paths**, a different mount; - the `checks` array grows from **12 entries to 15**, nothing removed, the additions being exactly `content.flash-ctl.not_uniform`, `.distinct_frames` - and `.motion` — 9 new leaves, three fields each; + and `.motion`. Each new object carries FOUR keys — `name`, `pass`, `detail` + and `judges` — and contributes **9** leaves here, because the fourth is + counted in the next bullet; - **every check gains a `judges` field**, `0 of 12` before and `15 of 15` after; - three further keys appear: `treatment_verdict`, `control_verdict` and `control_ratio.unusable`. -Those last three bullets are ~27 new leaves, not three. An earlier revision -enumerated only the final one and called the list exhaustive, which is the same -defect it was written to repair, at eight times the scale. +Those last three bullets are **exactly 27 new leaves and 0 removed** (9 + 15 + +3), measured by flattening both documents. An earlier revision enumerated only +the final group of three and called the list exhaustive, which is the same +defect it was written to repair, at nine times the scale. The `content.flash-ctl.*` checks and the three verdict keys ARE the exit-3 machinery `12c880a52` introduced, which is the direct evidence that the staged tool could not have returned a 3. **Do not diff the `checks` array by index**: -the three insertions shift the tail, so an index-wise comparison reports around -20 spurious "differences" that are the same checks at moved positions, and the -`37` above is not reproducible without keying by `name`. +the three insertions shift the tail, so an index-wise comparison over the +zipped 12 and 15 entries reports **6 differing check objects and 15 differing +`name`/`pass`/`detail` leaves, every one of them the same check at a moved +position**. None of those 15 is a real difference, and the `37` above is not +reproducible without keying by `name`. **AND THE EXIT-3 PATH IS PROVED BY MUTATION RATHER THAN BY READING IT.** A degenerate control was synthesised — 49 frames of one flat colour at this geometry, with a real `audio.wav` — and the committed tool run against it with both arms set to `flash`, so the TREATMENT is bit-identical and passes every -check it has: +check it has. Summarised — this is a paraphrase and not a transcript, and +`degen.txt` holds the tool's actual output: ``` [PASS] video.bit_identical / audio.bit_identical, and all six arm C0 checks From 393c572ed47f9dd4662907d0f82f2ef1ca9b0f8a Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 23 Aug 2026 02:41:25 +0000 Subject: [PATCH 17/19] docs(#1612): ENVIRONMENT still published the 47.84 s the same-lease pair supersedes The `VLLM_LTX2_DIT_FLASH_ATTN` row quoted "one DiT forward at 768x448/49f measured 47.84 s on GB10 through the naive kernel (n=119, median 47.91 s)". That is the cross-run figure section 7.1 retired: its arms differed in binary, lease, prompt and sampler, and it carried a ~3.2% stack-sampler overhead the new pair does not. The row now carries the same-lease pair - 45.547 s naive against 6.404 s flash, n=119 each, one binary and one `rc` lease, 7.112x - and points at section 10.7 for the provenance. It also gains the half of the answer the knob's own description was missing. "There is no supported reason to turn it off" described a speed knob; turning it off also changes what the model RENDERS, by more than one pixel of global image shift on three of four axes. That is a property of the sampler rather than of the kernel, which agrees with its reference to within its committed tolerance, and it is owned by #1743. A reader deciding whether to flip this knob needs that, and a public page that names only the speed cost is answering half the question. Found by grepping the tree for the superseded numbers rather than by a gate: no checker ties a measurement to this file, which is why the stale figure survived four revisions of the row that produced it. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --- docs/ENVIRONMENT.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/ENVIRONMENT.md b/docs/ENVIRONMENT.md index b5f46f078..4465e2388 100644 --- a/docs/ENVIRONMENT.md +++ b/docs/ENVIRONMENT.md @@ -247,7 +247,7 @@ Read-only observability; none change output. | `VT_H3_PROGRESS` | unset | Trace the MiniMax-H3 denoise loop's phases to stderr: which forward path was taken (device vs the CPU reference), how long the ONE-TIME device weight staging took, and per-step forward seconds with the sequence length. A real-checkpoint run spends its minutes in exactly one of those phases, and this says which without guessing — it was added after GPU-utilization counters proved unreliable on Tegra-class boards | | `VT_POOL_BYPASS` | off | `=1` makes every device-scratch pool allocation an exact-size driver `Alloc` and every release a real `Free`, so `compute-sanitizer` can see tensor boundaries and use-after-free that the caching, size-class-rounding pool hides. DEBUGGING ONLY: it reinstates the per-op `cudaMalloc`/`cudaFree` device-sync storm the pool exists to remove, so it is never a timing configuration | | `VLLM_LTX2_POOL_DRAIN` | on | `=0` suppresses the LTX-2.5 render's per-phase device scratch-pool drain. The drain is on by default and there is no supported reason to turn it off: a denoise phase leaves the pool holding every activation size class it touched, the next phase (twice the resolution) and the VAE decode allocate different classes, and on an uncapped pool those blocks are headroom the next stage never gets. This exists so the A/B that MEASURES what the drain is worth runs on ONE binary, which is what a same-binary A/B requires. Not a configuration | -| `VLLM_LTX2_DIT_FLASH_ATTN` | on | `=0` routes the LTX-2.5 DiT self-attention back to `vt::Attention`, the correctness-grade `vt::cuda::AttentionKernel`, instead of the flash-tiled `vt::AttentionDenseFlash` it now uses. ON by default and there is no supported reason to turn it off: one DiT forward at 768x448/49f measured **47.84 s** on GB10 through the naive kernel (n=119, median 47.91 s), because that kernel launches one 256-thread block per (query, head) and re-reads all K/V from global for each one -- 1.77e8 block-key iterations per call x 48 layers (issue [#1549](https://github.com/mudler/vllm.cpp/issues/1549)). This exists so the A/B that MEASURES the swap runs on ONE binary, which is what a same-binary A/B requires, and so a numerical regression has a same-binary control. A MEASUREMENT LANE in the same shape as `VLLM_LTX2_POOL_DRAIN` above, and never a configuration | +| `VLLM_LTX2_DIT_FLASH_ATTN` | on | `=0` routes the LTX-2.5 DiT self-attention back to `vt::Attention`, the correctness-grade `vt::cuda::AttentionKernel`, instead of the flash-tiled `vt::AttentionDenseFlash` it now uses. ON by default, and the same-lease A/B is now taken: one DiT forward at 768x448/49f measures **45.547 s** through the naive kernel against **6.404 s** through the flash one, n=119 each, one binary and one `rc` lease on GB10 -- **7.112x** (`.agents/specs/ltx25-dit-attn-flash.md` §10.7). This supersedes the earlier cross-run 47.84 s / ~6.0x pair, whose arms differed in binary, lease, prompt and sampler. The naive kernel is slow because it launches one 256-thread block per (query, head) and re-reads all K/V from global for each one -- 1.77e8 block-key iterations per call x 48 layers (issue [#1549](https://github.com/mudler/vllm.cpp/issues/1549)). **Turning it off also changes what the model RENDERS**, by more than one pixel of global image shift on three of four axes, which is a property of the sampler rather than of the kernel ([#1743](https://github.com/mudler/vllm.cpp/issues/1743)). This exists so the A/B that MEASURES the swap runs on ONE binary, which is what a same-binary A/B requires, and so a numerical regression has a same-binary control. A MEASUREMENT LANE in the same shape as `VLLM_LTX2_POOL_DRAIN` above, and never a configuration | | `VLLM_RENDER_PHASE_LOG_STDERR` | unset | `=1` also prints the render phase table as a fixed-width block to stderr. The table itself is written to `/phase-log.json` on the shipped default whatever this is set to (issue [#1010](https://github.com/mudler/vllm.cpp/issues/1010)); this only adds a copy somebody watching a terminal can read. **The copy is INDEPENDENT of the file write**: it is emitted whether the file was written or not, because an unwritable `--output-dir` is exactly the run where a console copy is the only evidence left, and a lane gated on the write would be silent in the one case it exists for. Off by default because a render that writes its table beside its frames does not need to shout, and a suite that runs ninety of them would print ninety | | `VLLM_RENDER_PHASE_SAMPLER` | on | `=0` stops the render phase log's 100 ms sampler thread. The phase table is still written and every phase still carries a byte count taken at its own boundaries; what is lost is the INTERIOR peak of a long phase, which is the half a minutes-long denoise needs. A MEASUREMENT LANE in the same shape as `VLLM_LTX2_POOL_DRAIN` above — it exists so the A/B over what the sampler costs runs on one binary — and never a configuration | | `VLLM_RENDER_PROGRESS` | on | `=0` silences the LTX-2.5 render's live progress lines on stderr — one `[render] + ` when a phase opens, one `[render] - ... dur=` when it closes, and one `[render] dit forward N phase P step k/N t=.. last=..` per DiT forward. ON by default and that is the whole point (issue [#1413](https://github.com/mudler/vllm.cpp/issues/1413)): the phase TABLE is written only by a generation that returns, so a render that is killed or aborted writes nothing, and between the load banner and `wrote N frames` a 2.5-hour render was silent — working and hung were the same observation. The open line means the LAST LINE PRINTED names the phase in flight, and `last=` is the per-forward cost [#1375](https://github.com/mudler/vllm.cpp/issues/1375) could otherwise only get as an interval between GPU busy/idle edges from outside the process. A MEASUREMENT LANE in the same shape as `VLLM_LTX2_POOL_DRAIN` above — it exists so the A/B over what the emitter costs runs on one binary — and never a configuration | From 5c8b6b57e69632395886618dd2c30942bb067f36 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 23 Aug 2026 02:46:15 +0000 Subject: [PATCH 18/19] fix(#1612): "seven hunks" beside the command that prints three A fourth review passed every item it was asked to verify and failed the change on one new count, introduced by the commit whose whole subject was removing unreproducible counts. `diff pixel-compare.txt recheck.txt` prints THREE hunks - `32c32`, `33a34`, `45,47c46,52` - covering 13 changed lines. The text said seven, which is the number of lines the last hunk ADDS. The clause sat beside the exact command named as its source, in a sentence whose thesis is "enumerated against a measured diff rather than described". The hunk headers and the line count are now quoted so the claim is checkable without running anything. Nothing else moved. The reviewer reproduced every other figure independently - 15th significant figure, 1.90457e-15 at 16 ULP with the rest 1 to 4, exactly 27 added leaves as 9 + 15 + 3 with none removed, 6 index-wise check objects and 15 spurious leaves, four keys per new check object, 1.63% and 1.65% on their two named bases - and confirmed the printed-report enumeration is complete against the diff. DECLARING TWO EDITS THE PREVIOUS COMMIT MADE AND DID NOT NAME, because the same reviewer noticed the subject accounted for 6 of its 8 changes. `439ecf116` also rewrote two section 8 gate rows. The `windows-msvc-cpu` row now attributes its red to #584 rather than #1649, read from this branch's own job log `97122086205` - `Windows portability contract OK` followed by `STATUS_STACK_BUFFER_OVERRUN` in `test_openai_api_server.exe` - because #1701 rescoped the #1649 gate and a post-#1701 red citing it names a message the log no longer contains. The `agent-record` row records #1754 as a duplicate of #1722, which is the one carrying fix PR #1726. Both were corrections handed over by the coordinator and verified here before use; neither belonged in a commit that did not mention them. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --- .agents/specs/ltx25-dit-attn-flash.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/.agents/specs/ltx25-dit-attn-flash.md b/.agents/specs/ltx25-dit-attn-flash.md index eb352b441..e20eb39a3 100644 --- a/.agents/specs/ltx25-dit-attn-flash.md +++ b/.agents/specs/ltx25-dit-attn-flash.md @@ -1176,7 +1176,8 @@ reader re-derives this row instead of taking it. **What differs between the two runs. This paragraph has been wrong three times, so it is enumerated against a measured diff rather than described.** First the -printed report, `diff pixel-compare.txt recheck.txt`, which has seven hunks: +printed report, `diff pixel-compare.txt recheck.txt`, whose **three** hunks +(`32c32`, `33a34`, `45,47c46,52`) cover 13 changed lines: - **one printed FIGURE moves**: the audio `pearson_r`, `0.932682102497646` against `0.9326821024976478` — they diverge at the **15th** significant From 32d4846b31083ccf50cca3036ab3f46b39048a9a Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 23 Aug 2026 07:13:37 +0000 Subject: [PATCH 19/19] record(#1612): three gate rows the last merge falsified, and the authority this row waited on Merging `main` at `8eecc05a9` invalidated three claims in this row's own records. They are repaired here rather than left for a reader, because a record that contradicts its tree is the failure this protocol exists to prevent. FIRST, the inherited-red row is now green. `build-test-cpu` and both `sanitize-cpu` lanes were recorded as INHERITED, failing on one case at `test_runner.cpp:1557` from #1273. PR #1700 fixed #1608 on `main` by declaring ROCM_ATTN's MultipleOf(16) block sizes, and the merge carried it: all three report `pass` on head `3f31bf0d4`, and `main` itself now reports `success` on `build-test-cpu` and `sanitize-cpu (thread)`. An inherited-red claim that has gone green does not merely age; it would have excused a future red on three lanes that are clean. SECOND, the flash arm is no longer the default. #1551 landed as `a4f2a9585` while this row was in review, instantiating FA-2 at head_dim 128 and routing the DiT onto its tensor cores, so `VLLM_LTX2_DIT_FLASH_ATTN` is now three-way with FA-2 unset-default and `=flash` the arm section 10.7 measured. Nothing in 10.7 is invalidated - it is a same-binary naive-against-flash pair at a recorded binary sha256, and it is the denominator #1551's own ratio is taken against. What moves is the reach of the QUESTION: section 10.4 wrote V4 as self-calibrating precisely so "the FA-2 arm (#1551) can take this same criterion", and #1743 now asks it of a rung this row never rendered. A pixel A/B of the FA-2 arm is owed and is not this row's. THIRD, the complete CI rollup is recorded at its exact SHA: 18 pass, 6 skip, 3 fail on `3f31bf0d4`, every failure characterised BY TEXT. Both Windows lanes print `Windows portability contract OK` and then die at STATUS_STACK_BUFFER_OVERRUN in test_openai_api_server.exe after the length-40 refusal, which is #584 and not #1649 - #1701 rescoped that gate. `agent-record` prints `agent record OK: ENGINE=170 ...` and then FileNotFoundError: 'hugo', which is #1722. The twelve lanes green here that are not inherited are named, so a future red on any of them is attributable rather than arguable. The merge authority is recorded as a gate rather than assumed. It stood PENDING while an agent relayed "merge it", because a relayed instruction is not developer consent, and the refusal cost time and was correct. The developer gave it directly on 2026-08-23. `.agents/developer-preferences.md` carries it scoped to this row and explicitly not as blanket authority. The `## Owed` entry for FA-2 head_dim 128 takes main's HANDED OFF wording; the two bullets main does not carry are preserved. Every unrelated key in `docs/ENVIRONMENT.md` is byte-for-byte main's, with only the one knob row re-edited. The row stays ACTIVE, not DONE, while #1743 is open. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --- .agents/specs/ltx25-dit-attn-flash.md | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/.agents/specs/ltx25-dit-attn-flash.md b/.agents/specs/ltx25-dit-attn-flash.md index e083e8a1e..930d8714a 100644 --- a/.agents/specs/ltx25-dit-attn-flash.md +++ b/.agents/specs/ltx25-dit-attn-flash.md @@ -23,6 +23,19 @@ correct to within its committed tolerance; what moves the picture is the render' **Whether a 7.112x arm that renders a different video should stay the default is a product decision this row does not take**, and #1743 is where it is argued. +**AND THE FLASH ARM IS NO LONGER THE DEFAULT.** +[#1551](https://github.com/mudler/vllm.cpp/issues/1551) landed as `a4f2a9585` +while this row was in review: `vt::AttentionDenseFa2` is instantiated at +head_dim 128 and the DiT now routes onto its tensor cores, so +`VLLM_LTX2_DIT_FLASH_ATTN` is a three-way knob whose unset default is FA-2 and +whose `=flash` value is what §10.7 measured. **Nothing in §10.7 is invalidated +by that** -- it is a same-binary naive-against-flash pair at a recorded binary +sha256, and it remains the denominator #1551's own ratio is taken against. What +changes is the reach of the QUESTION: §10.4 wrote V4 as self-calibrating +precisely "why the FA-2 arm (#1551) can take this same criterion", and #1743 now +asks it of a rung this row never rendered. A pixel A/B of the FA-2 arm is owed +and is not this row's. + The diagnosis is confirmed against the tree and the change is scoped to **one production call site**. It was once scoped to that call site plus a shared-memory cap repair in `LaunchAttentionDenseFlash`; that repair is @@ -547,9 +560,11 @@ never a synonym for "probably fine". | full preflight | `scripts/agent-preflight.sh` | **PASS at HEAD** — and it was NOT before: `documentation-checkpoint` was red on two of this branch's own commits (see below) | | `documentation-checkpoint` | CI, and locally over the branch range | **PASS at HEAD, RED before it, and the red was THIS BRANCH's** — `2aa78c69b` and `2f39a9426` each recorded a measurement in `.agents/benchmark-record.md` without writing the public projection that then existed. The control on the main-only range `4c193bd55..5d548d003` is rc 0, so it was not inherited. Both commits were replaced by one that writes the surfaces together when the branch was rebuilt, and the checker is re-run at each head rather than trusted to have stayed fixed — a job that has stopped appearing in a failing set is not the same fact as a job that passes. **THE COUPLING THAT PRODUCED THAT RED NO LONGER EXISTS, and the row is corrected rather than left to mislead:** `1db7e59cf` deleted `docs/STATUS.md` from the tree, deleted `scripts/check-doc-checkpoint.py` and `scripts/check-public-doc-tables.py`, and reduced this job to `check-now-current.py` plus `check-role-discipline.py`. A measurement now owes the row spec's `## Now` and nothing under `docs/` unless it adds a benchmark ID, which this one does not. An earlier revision of this row named `docs/STATUS.md` as the repair, and a reader who followed it would have recreated a file that `scripts/check-site.py` reds on for want of a `nav.yaml` entry | | `build-newest-gcc` | CI | **PASS, and now green on `main` too** — it was red on `main` on `::getpid` in `test_qwen3_dflash2_gguf.cpp:547`, a file this change does not touch; [#1581](https://github.com/mudler/vllm.cpp/pull/1581) fixed it and this branch carries that fix through the merge. A red here after the merge is therefore this row's, not inherited | -| `build-test-cpu`, `sanitize-cpu` (both) | CI | **INHERITED** — all three fail on the same single case, `test_runner.cpp:1557`, from #1273; owned by [#1602](https://github.com/mudler/vllm.cpp/issues/1602) and [#1608](https://github.com/mudler/vllm.cpp/issues/1608). Verified against `main` with `scripts/main-baseline.py`, not by reading a push run: those are all cancelled (#274) | +| `build-test-cpu`, `sanitize-cpu` (both) | CI | **PASS, and they were INHERITED-RED until this branch's last merge.** All three used to fail on one case, `test_runner.cpp:1557` from #1273, owned by [#1602](https://github.com/mudler/vllm.cpp/issues/1602) and [#1608](https://github.com/mudler/vllm.cpp/issues/1608). [PR #1700](https://github.com/mudler/vllm.cpp/pull/1700) fixed #1608 on `main` by declaring `ROCM_ATTN`'s `MultipleOf(16)` block sizes, and merging `main` carried the fix here: on head `3f31bf0d4` all three report `pass`, and `main` itself now reports `success` on `build-test-cpu` and `sanitize-cpu (thread)`. **The row is corrected rather than left standing, because an inherited-red claim that has gone green is a record contradicting its own tree** — it would have excused a future red on three lanes that are now clean | | `windows-msvc-cpu` / `-vulkan` | CI | **INHERITED, baseline-less lane, and the attribution is READ FROM THIS BRANCH'S OWN LOG rather than carried forward.** Job `97122086205` on head `36e596f22` prints `Windows portability contract OK` and then dies in `test_openai_api_server.exe` with `exited with status -1073740791` (`STATUS_STACK_BUFFER_OVERRUN`), immediately after `The decoder prompt (length 40) is longer than the maximum model length of 32`. **That is [#584](https://github.com/mudler/vllm.cpp/issues/584), NOT [#1649](https://github.com/mudler/vllm.cpp/issues/1649)**: #1649 is the `/W4 /WX` negated-by-`/w` refusal that fires BEFORE compilation, and [#1701](https://github.com/mudler/vllm.cpp/pull/1701) rescoped that gate, so the lane now gets past it and fails somewhere else. A post-#1701 red citing #1649 is a false attribution to an issue whose message the log no longer contains. A markdown-only control PR (#1295) fails the identical step; #584/#965 own it. **`windows-msvc-vulkan` on that head reads `cancelled`, which `gh pr checks` renders as `fail`** and which is not a verdict at all (#274) | | `agent-record` | CI | **INHERITED, and it is NOT the checker its name suggests** — `scripts/check-agent-record.py` and `scripts/audit-live-rows.py` both return rc 0 locally on `main` and on this head, so the job's red is elsewhere inside it. Read by failure text rather than by job name: the job also runs `tests/scripts/test_check_site.py`, whose `test_rendered_benchmark_index_links_resolve_to_emitted_pages` shells out to `hugo` and **ERRORs with `FileNotFoundError: [Errno 2] No such file or directory: 'hugo'`** on a runner that has no `hugo`. That is the #1661 shape exactly — a guard that probes a `returncode` which a missing binary never produces — one restructure later. Newly inherited because `1db7e59cf` added the test. **The local-against-CI asymmetry is the whole reason this was misreadable**: `hugo` is on this developer's `PATH`, so `test_check_site.py` runs 7/7 `OK` here and every checker in the job returns rc 0, while the same job reds on a runner that lacks the binary. A job name is not a diagnosis, and "the checkers pass locally" was true and did not explain it. Already owned and in flight: [#1722](https://github.com/mudler/vllm.cpp/issues/1722), with fix [PR #1726](https://github.com/mudler/vllm.cpp/pull/1726). Not repaired here, because it is another row's open pull request and duplicating it is the failure that check exists to avoid. **[#1754](https://github.com/mudler/vllm.cpp/issues/1754) is a DUPLICATE of #1722** — same defect, same file, same job, filed later and while #1726 was already open — and both are open as of 2026-08-23. Cite #1722, which is the one carrying the fix; the duplicate is recorded here rather than silently preferred, because two open issues for one red is how a fix gets written twice | +| complete CI rollup | CI, exact SHA `3f31bf0d4` | **TAKEN, 18 pass / 6 skip / 3 fail**, and every failure characterised BY TEXT rather than by job name: `windows-msvc-cpu` and `-vulkan` both print `Windows portability contract OK` and then die in `test_openai_api_server.exe` at `STATUS_STACK_BUFFER_OVERRUN` after the length-40 refusal ([#584](https://github.com/mudler/vllm.cpp/issues/584)), and `agent-record` prints `agent record OK: ENGINE=170 ...` and then `FileNotFoundError: 'hugo'` ([#1722](https://github.com/mudler/vllm.cpp/issues/1722)). Green on this head and NOT inherited: `build-newest-gcc`, `cuda-fat-build`, `device-leakage`, `vulkan-spirv-freshness`, `documentation-checkpoint`, `commit-protocol-tag`, `pr-size`, `last-gated-commit`, both `verify` lanes, `build-test-vulkan`, `build-test-cpu-arm64`, `cuda-arch-features` | +| merge authority | `.agents/developer-preferences.md` | **GRANTED by the developer for this row, 2026-08-23**, in their own words in session. It was `PENDING` before that and the merge was refused on it: an agent relaying "merge it" is not developer consent, and this row waited rather than treating a relayed instruction as the value. The grant is scoped to `LTX25-DIT-ATTN-FLASH` and is not blanket authority | **A side effect of that red is worth recording, because it was invisible.** The `documentation-checkpoint` job runs `set -eu` and this checker is the FIRST of