feat(MODEL-MM-dots3-note): W4b-2 — the sliding arm on the decode path (#699)

[localai-bot]

W4b-2 of the dots3-note row (#699). The 33 sliding_attention layers and the
PADDED physical KV row are on the decode path, reached from
ModelRegistry::Forward. W4b-1 landed the sliding maths as host code with no
production call site and named W4b-2 as the row that would wire it; this is that
wiring, plus the two windowed kernels it needed.

Upstream re-derived at vLLM origin/main = bc2d63e650.
git diff --stat d9fbe526c0 -- vllm/models/dots3_note/ is EMPTY against
bc2d63e650 and against origin/main alike, measured in a local vLLM checkout
rather than assumed, so every §2.3 anchor W4b-1 corrected holds unchanged at the
newer head.

The window is a kernel bound here and a gather upstream, and that is the same function

vt::MlaDecodeAttention and vt::MlaPrefillAttention each grow an optional
AttentionWindow — the (left, right) pair this tree already uses on
PagedAttentionArgs, and literally the pair upstream hands FlashAttention
(run_sliding_window(..., causal=True, window_size=(sliding_window - 1, 0)),
attention.py:300). Upstream's decode gathers [max(seq_len - GATHER_LEN, 0), …)
into a workspace with GATHER_LEN rounded up to 8 and then masks the scores
(:76-79, :151-152); the gather is a SUPERSET and the mask makes it exact,
because Triton needs a power-of-two tile. Walking the paged block table directly
over the same key range reaches the identical set with no gather and no mask.

std::nullopt is the ABSENT state and a NOT-TAKEN branch, not a wide window: the
CPU decode's loop START moves, the CUDA decode's kv_start moves in both split
stages, the FA-2 prefill reuses the paged launcher's own
is_causal = causal && !is_local normalization, and the CPU prefill's lower
bound moves. Both op gates prove that bit-for-bit rather than to a tolerance:
a window at least as wide as the sequence produces byte-identical output to no
window, which a mask applied afterwards could not.

The padded row needed zero vt changes

Tensor::Slice(2, 0, logical) shrinks shape[2] and keeps both leading strides,
every MLA cache op sources its strides from the tensor, and that IS upstream's
kv_cache[..., : self.head_size] (_logical_cache, attention.py:700-702). The
narrowing is one line in Dots3NoteModel::ForwardDevice. This is the correction
W4b-1 recorded, executed.

The evidence is the RAW cache bytes after a real forward, not an argument: lanes
[6, 10) of every slot a FULL layer wrote are still zero, with the CONTROL that
the same lanes on the sliding layers carry 28 non-zero values.

Two of W4a's three refusals are lifted, the third is narrowed, one is new

refusal	now
any sliding_attention layer	LIFTED
a PADDED physical latent row	LIFTED
a KV cache row disagreeing with the config	KEPT, now compared against the PHYSICAL row
seq_len > index_topk	KEPT, NARROWED — asked only of a config that HAS a full layer, because a sliding layer sets self.indexer = None (model.py:432-434)
a windowed prefill with chunked CONTEXT	NEW, in the seam — upstream never merges context chunks under a window, so there is nothing to mirror
a NEGATIVE MlaBlockDims::sliding_window	NEW, and it shipped untested; the review repair gave it a case (below)
MoE / nextn	unchanged (W5 / W10)

The RELEASED dots-studio/dots3-note-prev config still refuses, now at layer 1's
MoE rather than layer 2's sliding attention, so nothing a user can run changed.

The gate

test_dots3_note_attn 36 cases / 3028 assertions (30/2418 at W4b-1),
CPU-only, no GPU, no checkpoint, no speed claim. A MIXED config
({full, sliding, full}, dense MLPs, physical row 10 against the full arm's
logical 6) is loaded through the REAL registry and run through
ModelRegistry::Forward TWICE against one cache pool — a 6-token PREFILL then a
DECODE of the seventh, over a SHUFFLED block table — and compared against a
whole-model double reference that dispatches per layer kind into W3's
ref::Forward and W4b-1's sref::Forward.

Residue 0.0254 relative; bound 6e-2. Three ratios, kept SEPARATE because merging
them is the row's own finding F1: bound/residue 2.36x,
nearest-mechanism/bound 2.63x, nearest-mechanism/residue 6.22x. A port
with no window at all lands at 0.819 — 13.6x the bound.

The fixture was retuned twice, and a measurement forced both. The first draft
(four layers, thetas 41 vs 137) measured a residue of 0.119 with the nearest
mechanism at 0.106 — the nearest defect sat UNDER the quantisation floor. The fix
was the fixture, not the bound: thetas orders apart the way the released 5e4 vs
8e7 are, and three layers instead of four.

test_ops_mla_attn 15 cases / 246290 assertions (11 / 197113 at the base
SHA, measured by checking those two test files out at the base, rebuilding and
running, then restoring them byte-for-byte); test_ops_mla_prefill
7 cases / 329772 assertions (6 / 329772 before the review repair; 4 / 242156
at the base). That +1 case is +0 assertions on purpose, and the count is the
statement: the case the repair added is a CUDA one, doctest scores a case that
returns before its first assertion as PASSED, and this box has no device. It is a
skip wearing a pass, it is written down as one, and it must not be read as new
coverage executed here. It WAS executed later, on a leased GPU — see the CUDA
section below.

Their oracle is the op itself on a DIFFERENT input — a windowed paged call
against an unwindowed call over the truncated key list — rather than a second
copy of the same arithmetic.

The mutation pass, through the harness this tree ships

27 rows through scripts/mutation-harness.py, all re-measured at the final
36/3028 baseline, plus six more for the review repair. Highlights, with the full
tables in spec §4.8:

• Two reachability rows. Deleting the DECODE call site
  (impl.sliding_window = dims.sliding_window) and the PREFILL one
  (…, dims.sliding_window)) each take the gate RED, so the window comes from
  the config through the real loader and the shared seam.
• FIVE of the twenty-seven failed to BUILD on their first attempt, each on
  -Werror=unused-variable. Nearly one row in five would have read as a passing
  test in a hand-driven pass; the compiler exit is printed beside every row and
  each was re-run behind ((void)x, …).
• M16 found a real GATE GAP. Deleting the new windowed-prefill-with-context
  refusal left the gate green, because the case asserting it never made it out
  of the draft. It is in the gate now with two controls, and M16b reds.
• M19/M20 map which gate sees what. The same leaked window is invisible to
  test_deepseek_v2_forward and RED on test_mla_attention_block. Together
  they say impl.sliding_window really is on the DeepSeek path and its 0 is
  load-bearing — which the byte-identity table needs and cannot supply alone,
  since identical output could also mean nothing ever read the field.

What a fresh review changed, and what it corrected in the record

Nine findings, all discharged. Six are code and three are record; the record half
needed the mutation numbers the code half makes measurable, which is why it is a
second commit.

The FA-2 MLA-prefill windowed path had no test on any device. The record
named both CUDA files together and then described ONE parity case, which reads as
covering both halves. It did not: the windowed CUDA case was the DECODE sibling
in test_ops_mla_attn, and test_ops_mla_prefill's only HasCuda() cases are
pre-existing and unwindowed. A later lease would have discharged the decode half
against a record that read as covering both. test_ops_mla_prefill now carries
CUDA MLA prefill: the sliding window matches the CPU reference, comparing the
windowed device call against the windowed CPU op, against the unwindowed device
call, and against the file's expanded single-query oracle. It deliberately does
NOT assert the CPU case's wide-window bit-identity: a finite window sets
is_causal = false and dispatches FA-2's LOCAL template, so on the GPU that
agreement is numerical rather than byte-exact. Both entries in ## Owed now name
the two halves separately, because a lease closes one at a time.

A divergence this brick creates and never recorded. Upstream's
MLAAttention.get_kv_cache_spec branches on the window and returns
SlidingWindowMLASpec(..., sliding_window=self.sliding_window)
(vllm/model_executor/layers/attention/mla_attention.py:1215-1219 @
bc2d63e650), and every sliding layer sets one
(vllm/models/dots3_note/nvidia/model.py:457). MakeDots3NoteKVCache emits one
uniform v1::MLAAttentionSpec for all 46. There is no correctness
consequence — the window is applied on READ and the gate above proves it — but
33 of 46 layers then hold a full-length latent cache where upstream caps a
windowed layer near the window, 513 blocks against 524288 on the released
config. It is the single largest memory property of this architecture, and a
token gate structurally cannot see it, which is exactly why porting.md asks for
the memory format to be compared with the oracle explicitly. THREE pieces are
missing rather than one, so it is scoped to W4b-3 rather than fixed here. While this
branch sat in review, SlidingWindowMLASpec LANDED on main (KV-DSV4-MULTICACHE
W1, #1960, c1e6f3fb9), so the merge commit's own include/vllm/v1/kv_cache_interface.h
DEFINES the type this branch's copy still lists as omitted. The divergence is
unchanged — we still emit one uniform spec for all 46 layers — but W4b-3 now
inherits the spec type and owes only the per-layer wiring, which is less than was
owed when the paragraph above was written. Verified at merge time rather than
assumed: struct SlidingWindowMLASpec : SlidingWindowSpec at
include/vllm/v1/kv_cache_interface.h:396 on origin/main.

A byte-identity claim narrowed to what was actually measured. The record said
§4.6's six-arm fingerprint table "is reproducible from outside the session that
produced it". Two of six arms reproduce; four do not, and the first draft
generalised from the first two. The diagnosis is measured on three legs and is
not a behaviour change: exactly ONE commit touches the five shared MLA files
between the two bases and it is W4a itself; grep -r finds the fingerprints only
in the spec, so neither probe was ever committed; and q_lora_scale did not
exist at §4.6's base (grep -c is 0 there and 2 at §4.8's), so §4.6's probe
could not have set the field its own arms 2-3 are said to cover. The byte COUNTS
agree on all six arms, so the two probes agreed on shapes and differed on values.
The transferable rule is the reason this is in the record rather than quietly
corrected: a fingerprint from an uncommitted, hand-written probe is not a
cross-session reproducible quantity. Two probes sharing a prose label are two
instruments, and comparing their outputs measures the authors, not the code.
Committing the probe is what fixes it; neither scratch tree survives and a third
hand-written probe would be a third instrument, so it goes to ## Owed.

A mutation's stated mechanism was wrong, and the conclusion survives it. The
record explained M19's green by saying test_deepseek_v2_forward's CPU forward
drives the PREFILL half while impl.sliding_window reaches only the decode MQA —
which implies the prefill half WOULD be caught. Measured: leaking the window into
the prefill call instead is EXIT 0, 11 cases / 1052 assertions, zero failures,
SURVIVED. It misses a window leak on BOTH halves, because those CPU cases have no
value oracle for the attention output at all — each compares the model against
itself under another configuration, or against nothing.

A refusal that shipped untested now has a case.
MlaBlockDims::sliding_window < 0 is load-bearing, because the ops read the
window as > 0, so a negative value throws nowhere and silently degrades a
windowed layer to full attention. It has controls on both sides of the boundary
(0 is ABSENT and legal, 513 is legal) so it cannot pass on an implementation that
refused every window. test_mla_attention_block 2247715 -> 2247718
assertions, 12 cases unmoved.

One refusal is DECLINED rather than tested, on a stronger reason than the
review's. The per-step ld.head_size() <= physical_row check is UNREACHABLE,
not merely untested: physical_latent_row() IS swa.latent_row(), so on a
sliding layer the comparison is an identity, and on a full layer
ParseDots3NoteParams has already refused the violating config at load. R3 is
the measurement that turns that from an argument into a gate — deleting the
CONFIG-level refusal reds test_dots3_note_scaffold, so the closure is gated
rather than assumed. The review's reading, that Tensor::Slice backstops it,
understates it: the backstop is not reached either. It is kept as the executable
spelling of upstream's assert physical_head_size >= self.head_size, the site
says so, and ## Owed carries it as an untested assertion.

A dead assignment and an uncalled accessor are deleted.
step.rope_cache = &rope_full had no reader in its TU and would have handed a
future one the full arm's rope on a sliding layer, or an empty Tensor on a pure
SWA config; a null fails at the first read instead.
MlaBlockDims::has_sliding_window() had no caller anywhere and no call site was
invented for it, because every consumer wants the value and
ForwardMlaAttentionBlock assigns it unconditionally precisely so a 0 cannot be
skipped. Two stale comments describing the pre-retune four-layer fixture go with
them.

The seam is byte-identical on all six arms, measured within this session

Six arms across the seam's branch space (q_lora present/absent, both rope
layouts, both dtypes), BASE in a separate git archive tree at 925a4a587 with
a byte-identical probe (md5sum equal), its own configure and its own build.
Six for six identical, base against head. That is the claim this table was
built to make, and it is the claim it supports; the cross-session comparison it
also attempted is the one narrowed above.

test_mla_attention_block 12 / 2247718 (12 / 2247715 before the review
repair added the negative-window case) and test_deepseek_v2_forward
11 / 1052; test_deepseek_v2_decode_graph_seam 3/230, test_ops_mla_cache
9/2947, test_dots3_note_scaffold 26 / 110819.

NOT run and named: the SACRED DeepSeek-V2-Lite e2e token gate needs a ~29.26 GiB
checkpoint on a CUDA host; this brick ran CPU-only on a box with neither.

## Nothing lands dead

Everything this PR adds is reached from ModelRegistry::Forward, and two
reachability mutations say so rather than the sentence: deleting the decode call
site (impl.sliding_window = dims.sliding_window) and the prefill one
(… , dims.sliding_window)) each take the gate RED.

What is NOT reached, declared: W4b-1's dots3_note_attn.{h,cpp} sliding
functions — ForwardSlidingAttention, GatherSwaKv, ApplySwaScoreMask,
BuildSlidingWindowMetadata, WritePaddedMlaCache, NarrowLogicalCacheRows —
still have no production call site and did not gain one here, because the device
path reaches the same key set through the paged block table instead of
upstream's Triton gather-plus-mask. They are the gate's oracle, the status W3's
ForwardFullAttention has had since W4a. Owning row
MODEL-MM-dots3-note-dots3-note-for-causal-lm, issue #699, listed under
## Owed in the spec.

The CUDA half is compiled and EXECUTED, on sm_87

The two CUDA files this PR changes — src/vt/cuda/cuda_mla_attn.cu and
src/vt/cuda/cuda_flash_attn_fa2.cu — were written on a box with no GPU and no
nvcc, and for most of this row's life the record had to say "written, not
compiled, not run". An rc run lease on orin:gpu0 (Jetson AGX Orin, sm_87,
~36 minutes of device time) closed both halves of that. The SHA was PROVEN rather
than asserted: the job cloned in-container and refused to build unless
git rev-parse HEAD equalled 53424910dfa31fbd10bcb3296a12401eaed8ee54 with
git status --porcelain empty.

Compiled, on two toolchains. Both objects were deleted first, gencode was
read from compile_commands.json, and real per-arch SASS was confirmed with
cuobjdump --list-elf rather than a PTX leg.

TU	CUDA 12.6 / sm_87	CUDA 13.0 + the full CI arch list
cuda_mla_attn.cu	sm_87.cubin, 1.56 MB	rc=0, 10 cubins — 80, 86, 87, 89, 90a, 100a, 103a, 110, 120a, 121a
cuda_flash_attn_fa2.cu	sm_87.cubin, 514 KB	rc=0, 6 cubins — 80, 86, 87, 89, 120a, 121a

The second column reproduces what CI's cuda-fat-build asks, on the toolchain it
uses. FA-2 being ON was MEASURED three ways rather than inferred from the default
— CUDA feature fa2: ENABLED for [87], VLLM_CPP_FLASH_ATTN:BOOL=ON in
CMakeCache, and the generated manifest VLLM_CPP_CUDA_FA2_COMPILED_ARCHS "87" —
which matters because cuda_flash_attn_fa2.cu enters the archive only when
VLLM_CPP_FLASH_ATTN (default ON), CUTLASS headers and a non-empty
VT_FA2_ARCHS all hold, and cuda-fat-build supplies the second with
-DVLLM_CPP_CUTLASS_FETCH=ON and the third through an arch list that intersects
the fa2 feature set at 80, 86, 87, 89, 120a and 121a.

Executed — and the assertion counts are the proof. Same binaries in both
columns; the control is CUDA_VISIBLE_DEVICES="", so the delta is the device and
not the build.

run	cases	assertions
windowed decode alone, no device	1	0
windowed decode alone, on device	1	49,158
windowed prefill alone, no device	1	0
windowed prefill alone, on device	1	467,010

1 case / 0 assertions / SUCCESS! is exactly the shape these two cases have worn
until today, and the right-hand column is the first execution either has ever
had. Whole-binary figures on device: test_ops_mla_attn 246,290 -> 2,401,528
and test_ops_mla_prefill 329,772 -> 2,931,678. Every filter matched exactly one
case, so no zero-match false green; all exit codes 0 and no timeout.

Numerically correct. Windowed decode MaxAbsDiff(gpu, cpu) is 2.38e-07,
2.68e-07 and 2.68e-07 across the three split arms, against a < 1e-3 bar.
Windowed prefill, with 475 (query, key) pairs dropped across 57 queries: gpu_win
vs cpu_win 0.00294137, gpu_none vs cpu_none 0.00294137 and gpu_win vs
expanded 0.00294137, all against < 3e-2 — and the decisive one, gpu_win
vs gpu_none = 1.06055 against a > 1e-2 bar. The window demonstrably BITES
on the device, so the FA-2 launcher is not dropping window_size on the floor.
That is the exact defect the is_local normalization exists to prevent, now
measured rather than argued.

The wrinkle, stated rather than tidied away. Three of those four prefill
numbers are byte-identical at 0.00294137. The likely reason is that the global
max-error element sits on an UNWINDOWED row, where gpu_win/gpu_none and
cpu_win/cpu_none/expanded each coincide and all three become the same
subtraction — but that element was not instrumented, so this is inference and not
a measurement. It is not degeneracy: if the CPU arms agreed wholesale the second
assertion would read about 1.06, not 0.0029.

Two limits. Execution is proven on sm_87 only; the ten-arch result is
COMPILE-ONLY, because CUDA 13 cannot run against that box's NVRM 540.4.0 driver
(cudaGetDeviceCount rc=35). And this is KERNEL-level parity on two ops. It is
not the e2e model gate, which is unrelated and still owed.

The two red CI lanes are pre-existing and unrelated, with the evidence to re-derive that

Neither red is attributable to this change, and both are named here rather than
left for a reader to assume.

• build-test-cpu fails at the TEST step, not the build:
  627 of 628 passed, and the single failure is test_cpu_threadpool at
  tests/vt/test_cpu_threadpool.cpp:539, CHECK(ratio < 100.0) reading
  2 threads 0.48 us, 5 threads 52.638 us, ratio 109.662. It is a fixed
  wall-clock ratio guard whose denominator is the small-thread leg, so it fires
  on GitHub's 4-vCPU runner while needing roughly a 37x blowup to catch a real
  regression on a large box. Issue
  #631 owns it and it is
  deliberately unfixed, because raising a threshold to green a gate is a scope
  change that needs its own spec. It touches no MLA, attention or model path.
• windows-msvc-cpu and windows-msvc-vulkan are the baseline-less lane,
  issues #503 and
  #584. Controlled rather than
  asserted: they fail identically on open PRs record(ORACLE-EXLLAMAV3-GATEABILITY): the pin does not build on aarch64, so the EXL3 arm still has no token oracle (#1901) #1967, docs: align public feature counts with the registry #1956 and fix(BACKEND-ROCM): charge the expanded size when every keep flag is off #1935, and ZERO
  of the last 12 windows runs produced a main verdict, so there is no green
  baseline for this branch to have regressed from.

The merge that built clean and threw

Integrating origin/main a second time merged clean, COMPILED clean, and took
test_dots3_note_attn from 36/3028 green to five cases throwing
resident weight: EMPTY tensor has no host bytes to alias. The incoming change
is #1952's review finding #1953: ResidentWeight now refuses an empty weight,
because an empty one aliases a null host pointer no downstream op can detect.
The refusal is right and this branch was on the wrong side of it — it made BOTH
rope caches resident up front while the materializer deliberately leaves the
unused one empty (each is 64 MiB at 524288 positions). One guard per cache fixes
it, and the comment now says the guard is the CONTRACT rather than an
optimization.

The two branches touched different files, so nothing about this was visible to
the merge: it is "merge-tree CLEAN is not merge-tree BUILDS" one step further
along, where it builds too. Re-running the focused gate set AFTER the merge and
BEFORE the push is the only thing that caught it. Spec §4.8 records it.

What is owed, and the split I am proposing

The row is NOT finished, and the sm_87 result above does not narrow this list.

W4b-3, and the line is real: the DSA lightning indexer's SELECTION shares
nothing with the sliding window. The sliding layers have no indexer at all; the
indexer is on the FULL layers, and lifting seq_len > index_topk needs indexer
weights on device, a logits kernel, a top-k and a SPARSE MLA attention kernel on
both backends. Bundling it here would have doubled a PR that already carries two
new kernel bounds and a seam extension. W4b-3 carries it, together with the
windowed-prefill-with-context refusal and the SlidingWindowMLASpec divergence
above.

.agents/specs/dots3-note.md is STALE against this body at the merge commit,
in three places, and W4b-3 owns every one. The spec is the row's authoritative
record, so a merge that leaves it disagreeing with its own commit message is a
debt and is written down as one. It is not fixed on this branch because a push
now would cancel a cuda-fat-build this change has already had destroyed twice,
and the fresh reviewer sanctioned folding record-only corrections into W4b-3:

• §4.8 reads test_mla_attention_block 12 / 2247715 and calls it "unmoved
  from the numbers §4.6 recorded". The measured value at this SHA is
  12 / 2247718 — the implementer, a fresh reviewer and the operator each
  measured it independently and agree. 2247718 appears nowhere in the spec.
  This body says 2247718 and the spec contradicts it.
• §4.8's #### The #1969 REVIEW-REPAIR rows section says "Five more rows" and
  its table lists six: R1, R1-control, R2, R3, R4 and R4b. This body says
  six.
• The most misleading of the three. §4.8 and ## Owed still say the CUDA
  lease "cannot currently be taken" and name thor:gpu0 as the host that
  discharges it. Both statements are now false. The compile and the execution
  above were discharged on orin:gpu0, and thor could never have discharged
  the PREFILL half at any point, because VT_CUDA_FEATURE_TABLE's fa2 row is
  8.0,8.6,8.7,8.9,12.0a,12.1a — sm_87 carries FA-2 and thor's sm_110 does not.
  Left alone, the record claims blocked work that has in fact been done, on a
  host that could not have done it.

Two comment-level corrections also go to W4b-3, named here so the miss is in
the history rather than only in a review thread. They are not fixed on this
branch for the same reason, and neither is worth a gate run of its own:

• src/vllm/model_executor/models/dots3_note_device.cpp's upstream anchors for
  the deepseek_v2.py diff are off by three. The lines are 460 and 498
  at bc2d63e650, not 457 and 495 — 457 is qk_nope_head_dim: int, and 495 is
  a bare ). The substance of that correction is right; only the two numbers
  are wrong.
• The new CUDA prefill case's comment says the CPU fixture is "unchanged". It is
  not literally one experiment: the CUDA case uses h = kHeadsLite (16) where
  the CPU case uses h = 4. Lengths, window and seeds do match.

Still open, and blocked on things this row cannot buy. The end-to-end parity
gate against vLLM — token-exact or the ratified near-tie form, chosen by
measurement — plus every throughput, latency and memory axis, and the quantized
arms. No host this project owns runs the dots3_note oracle at any published
precision, and the row is beyond the parity pin. ## Owed in the spec carries
each of these with its owner and #699.

Row stays SPIKE. Under §6.4 option B the CPU gate remains a consistency gate
against an independent reference, not a correctness gate against an oracle: no
vLLM instance for dots3_note runs on any host this project owns.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [Claude Code]