You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Full-model numerical readiness remains BLOCKED. The local source-correct candidate now passes 239/240 approved short decoder comparisons, with 232/232 cache comparisons passing. Cached case 3 mtp_seed still fails (NRMSE 0.035922356240143276 > 0.03). The separate 257-token generation control also fails. Thresholds and references have not changed.
These results are from an unpublished local source proposal, not the unchanged PR head e3d01a7712f38deb962e7107752da7fb668a7054. The full-eager attention control was rejected (138/240 approved comparisons fail); no production remedy was adopted from it.
The description below preserves earlier implementation details and historical validation results; the status above supersedes its older numerical scorecard.
Summary
Full-model numerical readiness remains BLOCKED. The latest user-supplied H100 run (2026-10-11, export checkout a72b658f) reports normalization, source fidelity, exact eight-token generation and conditional MTP passing, but full decoder parity still fails 141/240 comparisons. Matching short token sequences does not clear full-logit/state gates or establish speculative runtime support.
Add pinned Lightning BF16/NVFP4-source recipes, strict explicit ModelOpt-to-dense-BF16 reconstruction and paired target/MTP graphs via build(..., task="nemotron-h-mtp") / --task nemotron-h-mtp. Outputs: decoder/model.onnx and mtp/model.onnx, with all 270 MTP tensors accounted for and zero package-level MTP omissions. Default NextN refusal and explicit target-only named omission accounting remain. Native NVFP4 stays fail-closed; reconstruction is not preserved NVFP4. No runnable Olive integration or implemented speculative-generation loop is claimed.
MTP follows pinned vLLM draft and parent layers, checked against NVIDIA config and tensor inventories.
Both graphs use NVIDIA checkpoint weights; Mobius constructs ONNX directly with onnxscript.nn/onnx_ir, not by tracing upstream PyTorch. Decoder emits post-final-norm mtp_seed; draft pairs h_i with t_(i+1) to predict t_(i+2), owns independent KV and has no RoPE/Mamba. Target-shared checkpoint embedding/head sources are copied into physically dedicated draft tables.
Published head e3d01a7712f38deb962e7107752da7fb668a7054, same PR branch, signed-off/coauthored linear history. MTP child a10651bd integrated as 897a0e05; norm child d3c7a51a as c5d47af4; census/signature/report redaction followed in 4ad49ac/4ba865c/a72b658. New interface review fixes are 089c161b; diagnostic child 1059ab1a integrated as 49d5f2c8, documentation follow-up e3d01a77. No production precision, recurrence or acceptance-threshold changes in this latest diagnostic follow-up.
Compatibility evidence and limits
BF16 revision a9904d24bcc1d289a1950fa9d2b978c47cf903b9; NVFP4 revision bee7596271d1495f6992ae224aefde4410e816b8. Context 262144/1048576; index tensor bytes 65842365568/21559589596 across 14/52 shards. Metadata pins do not authenticate every payload/tokenizer or prove memory fit. Six source hashes at dc179846... remain historical evidence, not current dependency locks.
Both pinned sources contain MTP shared-expert up/down projections, shapes [3712,2688]/[2688,3712]; the earlier absence inference was false. Independently bounded-read headers confirm 270 identical MTP names/shapes, BF16 except FP32 correction bias, zero quantized MTP modules/scales. Target has 6243 names; NVFP4 profiles contain 46 FP8/5935 NVFP4 projections, 46 input/12 KV scales. Full config/header checks after the norm correction: 40176 decoder/1839 MTP nodes. Separate GGUF census: 43272 raw/40222 optimized nodes, 6257 initializers, 6028 MatMul, 60 inputs/59 outputs. These are graph/source checks, not numerical acceptance.
GroupedGatedRMSNorm is wired only to Nemotron mamba.norm, preserving its namespace. It avoids faulty 2D-scale RMSNormalization and matches source cast-before-gamma semantics; other Mamba defaults and the legacy flag are unchanged. Local paths remain transient in lazy bindings/overlap guards, not persisted nested reports/ONNX metadata.
Latest supplied H100 observations — not locally re-executed full-weight CUDA results:
Supersedes earlier generation FAIL only for this bounded run
Conditional MTP
PASS: 12 comparisons, max NRMSE .0047783538309905, min cosine .9999894304916273
Same supplied target conditioning; not proof that target states are correct
Speculative integration
NOT_RUN
External acceptance/verification/rollback not implemented here
All four baseline full-logit comparisons still fail NRMSE (.0457179/.0531702/.0425463/.0418169), despite matching argmax. First prefill state failure is layer-4 conv state; first hidden failure is layer 34. All 192 identical-input Mamba substage measurements are below unchanged gates: recurrence/state at the same dt agree near 6.1e-7/1.1e-7 NRMSE. All 12 same-input native FP32 router expert sets agree; independently carried FP32 inputs change selections at layer-34 cases 0/1. Layer-4 norm agrees on identical exposed rounded BF16 residual, qualified because fused SkipNorm may normalize before residual-output rounding. These facts separate upstream precision amplification from demonstrated operator errors; they do not justify an invented recurrence patch.
Native FP32 CPU-fallback audit matches prefill (112 comparisons, max NRMSE 1.55e-6) but differs on cached steps (75/96/102 failures). A reference-only diagnostic removing the cached dt minimum clamp matches native fallback; baseline export/runner were unchanged. The original pinned native source SHA256 5417d6f8dcbee945d4f1c4ae5327b0d1f443b29cc99098c99ec7befb9c9afcd8 matches the reported module. Creator independently invoked its original fallback bodies on CPU, removing only dispatch decorators: synthetic bounded prefill output/state NRMSE 7.13e-8/4.61e-8; cached bounded-versus-unbounded contrasts up to .00714 (below the full-model gate). Native BF16 source rounding is reproduced exactly; state-readout and early-FP32-dt contrasts measure .001765/.003323. These are synthetic source-contract diagnostics, not a full-model fix or permission to replace the acceptance reference.
Earlier 897a0e05 report remains historical: original decoder 232/240 failures, max NRMSE 2.10802144/min cosine -.0396307; scratch decomposition 130/240 failures, .1194279/.9928551; generation failed. Earlier scoped MTP SHA256 1bd1e3f7fb717d02cc16dd4a48a3f6148bb090d3d4dbdedd99ee82e51fd9e31e and 58 scoped comparisons do not describe the new artifact. Prior reported fused batch-2 cached multi-token, hole-mask and local CausalConv provider-assignment limits are not proven cleared by the new norm test.
The complete 8593-line report was preserved and parsed (6401 leaves), SHA256 27F1FEE27ACEF0C007341D787BD7E989D5BDCCA274AED6EED0DC972AB53534C0. Its /home/mobius/... paths name remote evidence; they are not accessible scripts/arrays here. Needed portable bundle: original run_validation.py, independent reference/audit helpers and command/backend definitions; optimized graph/provider profiles; case-0 layers 0–4 pre-round residual operands, exposed residual/norm/xBC/conv arrays; case-1 identical state/dt/C/readout arrays; actual native kernel dispatch and dt_limit. Compare full corrected portable/fused decoder using identical tokens/weights/cache provenance before altering production semantics.
Runtime remains external: shifted token/hidden pairing, independent draft KV, target verification/sampling, accepted-prefix commit, rejected draft rollback and target KV plus Mamba conv/SSM restoration/replay. Metadata is runtime_unvalidated. Native NVFP4, full reconstructed-checkpoint parity, Olive and performance remain unvalidated. Full weighted/GPU recipe commands remain session-scoped NOT_RUN; externally supplied execution is labeled separately.
Validation
Local Windows ARM64/Adreno with isolated x64 CPU Python; Torch 2.14.1+cpu, Transformers 5.18.0, ONNXScript .7.2, ONNXIR 1.0.0, ORT 1.30.0. No new large-weight downloads/GPU packages/paid GPU work by this session.
Six new cases cover two-row prefill plus three reused-state steps, grouped B/C, active/inactive dt clamp, output/conv/FP32 SSM and FP16/BF16 cast boundaries against independent equations. They protect current export semantics rather than silently replacing the reference. CPU source probe rerun in session artifacts; no repository copy of upstream source or generated diagnostics.
Historical creator results: norm integration 177 passed/7 skipped; census 23 passed/2 unchanged privilege cases deselected plus generated-docs check; redaction/streaming 105 passed/4 existing skips; earlier MTP integration 545 passed/2 skipped and package/runtime/task 421 passed/6 unchanged privilege cases deselected. Not aggregate unique counts or full new-head reruns. Full pinned config/header/ten-hash checks previously passed; no invented full-model CPU/CUDA pass.
At a72b658f, six OS/Python unit jobs, lint/lintrunner/CodeQL, L1/L3/fast integration/GenAI/build passed. L4/L5 check surfaces show failure; job/step details say cancelled during golden execution at one hour, not passing or established assertion failures. CLA remains pending. New-head CI must be inspected live.
Both new review threads addressed and resolved in 089c161b: PRRT_kwDORmWK4M6qkxbe now correctly declares dedicated draft tables/null shared-runtime pointers; PRRT_kwDORmWK4M6qkxcH makes draft private, with only public paired wrapper advertising the task. Full metadata/paired-task regressions pass. Prior nine threads were already resolved; no duplicate review requests. Full-model correctness is a release gate, not a thread to blanket-resolve.
Remaining gate: inspect portable runner/reference/backend and earliest-boundary data, retain unchanged acceptance thresholds and all 240 comparisons, then diagnose and verify full decoder/state parity. Generation PASS and norm PASS are scoped progress, not full-model or speculative-runtime acceptance.
Code relationship graphs
Graph source commit 089c161; subsequent 49d5f2c8/e3d01a77 change tests/docs only, not displayed production relationships. 53 locally AST-verified edges. Light theme/source-module grouping; blue functions, purple classes, green methods, gray external APIs; solid calls/construction, dashed type/config use. Scope: loader/numerics/cache/safety/target/MTP/grouped norm and immediate dependencies. Omitted: exhaustive branches/full math/weights, actual CUDA dispatch/performance, speculative verification/rollback and unrelated models. These are static relationships, not runtime execution. Scripts/data/searchable zoomable neighbor-highlight HTML remain in session artifacts. Authenticated attachment SHA256 verified; public access and persisted embeds verified after publication.
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution. You have signed the CLA already but the status is still pending? Let us recheck it.
Implement explicit, scale-aware mixed FP8/NVFP4 reconstruction with bounded-row temporaries, exact per-expert source accounting, lazy header revalidation and truthful provenance. Validate activation and KV-cache scales without claiming their quantization is preserved. Keep native formats and unsupported NextN/MTP contracts blocked; no tensors are silently discarded. Correct FP32 router/accumulation and preserve the declared SSM cache dtype. Add offline numeric, public-builder/CLI, serialization, source-accounting and cache regressions, and update the pinned recipes to identify the remaining full-Lightning MTP gate. Full checkpoints, CUDA/native kernels, generation and runtime packaging are not claimed.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Signed-off-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Address automated lint/code-quality findings by using the established direct dtype-marker assignment with its intentional dynamic-attribute type annotation, and avoid binding unused checkpoint-fixture return values. Numerical behavior and represented code relationships are unchanged. Targeted model/public-builder regressions and the B010 lint gate pass.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Signed-off-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Keep router parameters FP32 before BF16 dtype conversion, not only during graph computation. The pinned NVFP4 checkpoint stores its unquantized router in FP32, while the BF16 variant stores it in BF16. Independent source fixtures and a non-BF16-representable value catch premature rounding across streaming and serialization paths.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Signed-off-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
CI's exact census regression exposed stale graph-only evidence: FP32 expert accumulation raises the raw/optimized counts to measured 43180/40109 nodes. Retain strict graph, initializer, MatMul and state census assertions and regenerate the catalog/matrix. Distinguish this local graph refresh from historical ORT 1.29 runtime inspection; no runtime support is claimed.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Signed-off-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Wire the user-approved target_decoder_only API and --target-decoder-only CLI option through both pinned floating and opt-in reconstructed ModelOpt paths. Retain the source NextN config, default refusal and native NVFP4 block; persist variant provenance and every omitted mtp.* name, source shape/dtype and reason. Keep all non-MTP inventory/shape/scale failures strict and distinguish graph-only uninspected inventories. Cover API/CLI/default/native/errors/save/reload and update pinned unexecuted recipe candidates.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Signed-off-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
rui-ren
changed the title
Add strict Nemotron Lightning BF16 reconstruction and compatibility gates
Add explicit Nemotron Lightning target-only BF16 exports and strict reconstruction
Oct 8, 2026
For local builds, the streaming report's source is the literal model_id path, and assigning it here persists that private filesystem path both in weight-loading-report.json and the existing mobius.weight_loading metadata. The established GPT-OSS path replaces local sources with local-safetensors-checkpoint (_gptoss_weights.py:318-323); apply the same sanitization before attaching this report and refresh the metadata from the sanitized report.
🧠 Review effort: Balanced
Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.
Resolve shard paths once and register lazy shard/config/index files plus snapshot and resolved blob parent directories with ModelPackage's existing save guard. Share the directory-identity helper with GPT-OSS instead of duplicating it. Add source-directory, indexed/unindexed, floating/reconstructed and hard-link collision regressions that prove no source mutation before lazy reads, with successful fresh-output serialization. Correct reconstructed NVFP4 recipe output precision to bf16 without claiming execution.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Signed-off-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Implement a separate nemotron-h-mtp task that preserves the pinned Lightning attention/MoE NextN weights, exports a post-final-norm target seed, and gives the draft its own KV cache. Keep default refusal, target-only omission reporting, and ModelOpt reconstruction opt-ins unchanged. Account for both graph source inventories and shared embedding/head copies with strict streaming and lazy-source protection.
Validate tiny independent CPU equations across prefill, cached decode, padding, source precisions and the actual target-to-draft bridge. Record pinned header/config evidence and explicit full-checkpoint, BF16/CUDA and generation-runtime limitations; this is not speculative runtime integration or native NVFP4.
Signed-off-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Signed-off-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
rui-ren
changed the title
Add explicit Nemotron Lightning target-only BF16 exports and strict reconstruction
Add Nemotron Lightning target/MTP graph exports and explicit BF16 reconstruction
Oct 8, 2026
Keep gating and per-group normalization in FP32, then cast the normalized activation before applying native-dtype gamma, matching the pinned HF Zamba2RMSNormGated implementation. Wire only Nemotron to the explicit component so it avoids the demonstrated CUDA RMSNormalization two-dimensional-scale kernel failure without altering other Mamba defaults or the existing opt-in workaround.
Add CPU FP32/FP16 numerical coverage, BF16 graph precision and CUDA-gated probes, production group dimensions, optimized-graph checks, and unchanged weight/cache contract regressions. Full H100 decoder parity and generation remain blocked pending first-divergence diagnostics.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Signed-off-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Measure raw and optimized graphs using the pinned GGUF config and existing CPU export route. Preserve exact census assertions, including the two additional decomposition constants, and regenerate the linked catalog/evidence. Runtime correctness remains blocked by the reported full-model decoder mismatch.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Signed-off-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Accept and forward the base layer's optional attention, position and cache parameters while retaining the draft's existing two-argument call. Add a positional forwarding regression; independent MTP parity and cached decode tests remain passing.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Signed-off-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Use a portable local-safetensors-checkpoint source label in dense streaming reports and ONNX metadata. Transient lazy bindings and overlap protections retain actual paths. Cover reconstructed target exports and both MTP component reports through save/reload, preserving remote IDs.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Signed-off-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Declare physically dedicated draft tables despite shared checkpoint sources and keep the draft implementation private behind the public paired wrapper. Add metadata/public-task regressions. Document the user-supplied norm and generation passes while retaining failed full decoder parity and reference-contract qualifications; no recurrence or threshold changes.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Signed-off-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Exercise portable Mamba2 prefill and three reused-state steps with an independent FP32 recurrence, grouped B/C expansion, source-precision grouped normalization, and timesteps below the Lightning minimum. Preserve the distinction between bounded export semantics and the native fallback's unbounded cached timestep path instead of silently replacing the reference.
Protect FP16/BF16 export cast boundaries for timestep addition, FP32 SSM state and readout/D-skip rounding. These regressions do not change production recurrence or establish full-model H100 numerical parity.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Signed-off-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Full-model readiness remains BLOCKED. The local source-correct candidate now passes 239/240 approved short decoder comparisons, versus the previously reported 99/240. This is progress, not numerical acceptance.
Source qualification: the published PR head remains e3d01a7712f38deb962e7107752da7fb668a7054. These improved results belong to the preserved local v13 candidate and complete 29-file source proposal, not the unchanged published head. The complete proposal is production-proposal-complete-v15.patch, SHA256 dd7eebbc3b4049d11b69ab91ee2eb0a08272d7b3e2b1e98cee1469c24b535c3a. It includes all eight new helpers/tests omitted by the earlier tracked-only v14 patch. A fresh exact-base application verified every proposed file byte-for-byte.
Area
Status
Evidence / qualification
Normalization
PASS
Preserved 15-case run, including all five BF16 CUDA cases; not a new rerun today
Source fidelity
PASS, qualified
Prior independent source-weight validation preserved; current 6,515 binding-identity checks pass against that verified storage, not a new checkpoint arithmetic comparison
Approved short decoder parity
FAIL
239/240 pass; the remaining failure is cached case 3 mtp_seed
Approved short cache comparisons
PASS
232/232 pass
Original eight-token greedy generation
PASS
Preserved exact-token result; does not clear decoder parity
Conditional actual-target MTP
PASS, conditional
12/12 comparisons; not end-to-end speculative acceptance
Extended FP32-reference comparisons
FAIL
44/360 fail for the current candidate; retained separately from matched-precision diagnostics
Native-BF16 generation controls
FAIL overall
129-token control passes; 257-token control fails
Speculative integration
NOT_RUN
Ordinary decoder correctness remains a prerequisite
The sole approved short failure has NRMSE 0.035922356240143276, cosine 0.9993554903780785, and maximum absolute error 0.48468780517578125. Gates remain NRMSE <= 0.03, cosine >= 0.999, finite outputs, and exact generation-token identity. Explicit CUDA use_tf32=0 reproduces the same failure.
Independent receiving-session control
The existing opus-sonnet-5-5 session completed a fresh full portable/eager-attention export on H100 GPU 7, with identical short feeds, references, gates and TF32 disabled. 138/240 approved-FP32 comparisons fail (maximum NRMSE 0.157073591459086); 140/240 native-BF16 diagnostic comparisons fail (maximum NRMSE 0.156140821531311). The control is rejected as a fix. No production change was adopted. Its saved manifest was independently checked after completion.
This result does not prove fused attention cannot contribute to drift. An expert-set flip after layer 15 is inferred from the variant's localized jump, not directly observed in a router trace.
Remaining investigation
The approved short blocker is reused-state decoding, whereas the 257-token blocker is prefill; they must not be conflated. Same-input Mamba, router, normalization, final-head and eager-attention audits have not established another production source-contract defect. A source-correct sigmoid changes an early BF16 rounding boundary and subsequent cache/routing history; reverting it or adding arbitrary casts to fit a gate is not an acceptable remedy.
Native BF16 itself fails 103/240 approved FP32 comparisons, which qualifies the diagnostic oracle but does not waive the original acceptance failure. Native cold and restored-warm executions differ across execution contexts; repeated runs within the tested contexts are stable, and restoring the archived warm-up reproduces all 112 archived arrays bitwise. This is not evidence of uncontrolled native non-repeatability.
The next focused handoff is the cached case-3 seed: explicitly map cache/feed provenance, localize the first causal divergence through the reused-state trajectory, and propose a production patch only for a demonstrated same-input source-contract mismatch. The existing references and thresholds remain unchanged. The 257-token generation blocker remains open separately.
Evidence is preserved on the H100 machine under /home/mobius/validation_lightning_h100_parity_fix_20261011, including opus-sonnet-5-5-continuation/REPORT.md, raw arrays, metrics, source patches, runtime/provider identities and SHA256 manifests. Runtime: Python 3.12.15, Torch 2.14.1+cu130, Transformers 5.19.0, ORT 1.28.0.dev20260722004, CUDA 13.0, driver 580.105.08. Model revision remains a9904d24bcc1d289a1950fa9d2b978c47cf903b9.
H100 follow-up: remaining failure localized, no new production fix established
The unpublished local v13 candidate still does not meet full readiness. The approved short decoder comparison remains 239/240, with cached case3 mtp_seed NRMSE 0.035922356240143276 against the unchanged limit0.03. The separate257-token generation failure remains open. No new production code has been committed or pushed in this follow-up.
All360 original mandatory tensors preserved bitwise; all six original-port NPZ files have identical hashes to the uninstrumented results
This capture is usable for those output paths; historical feed-byte provenance remains qualified
Independent RMS arithmetic
Original-checkpoint, ONNX and captured gamma values agree at six layers; all36 residual additions reproduced exactly; both native and fused results have isolated one-BF16-ULP differences from direct FP64-rounded RMS
Identifies the standalone kernel; not a full-graph kernel trace
Independent full-native FP32 comparison
Same case3 seed remains the sole failure, NRMSE0.03189153827692811
The remaining failure is not solely caused by the corrected reference approximation; this is a diagnostic, not a replacement acceptance reference
Hidden-stage localization
First case3 hidden gate failure at MoE layer24, NRMSE0.04211967661127825
An upstream trajectory difference becomes materially larger at this boundary
Identical-input layer24 router replay
Verified original HF norm/router/correction-bias values; ONNX and natural native unsorted router select the same expert set [87,42,123,78,93,16]; choice scores match exactly
No local selection mismatch on this sample; differing returned expert order is not a set mismatch
Layer24 normalization attribution
Native BF16 normalization on the captured source residual reproduces the captured norm exactly; FP32 normalization on that same residual still selects expert16
Layer24 normalization rounding alone does not cause the observed expert change
Independent FP64 router arithmetic
Confirms expert16 on source inputs with cutoff margin0.0017302731, but expert19 on the native-FP32 residual with margin0.0034974230; native/source routing-weight error versus FP64 below7e-8
The expert change follows the different incoming residual; it is not explained by the measured same-input router error
Complete same-input MoE24 replay
Original HF expert/router/shared-expert values verified; native BF16 output is bitwise identical to the captured ONNX MoE output; FP32 arithmetic on the same numeric input has NRMSE0.0036765736
No local complete-MoE discrepancy on the first failing layer's actual source input
Fresh native reference loader fidelity
BF16 and two-GPU FP32 each verify all6,243 target tensors, covering all401 native state tensors; zero value mismatches or uncovered elements; all270 MTP source tensors explicitly excluded
Loaded reference weights and strict FP32 correction-bias buffers are faithful at the pinned loader/source; this does not certify historical in-memory objects or numerical readiness
Final case3 decomposition against full-native FP32: pre-final residual NRMSE0.0257901662, mathematically normalized unweighted activation0.0244578619, and gamma-weighted activation0.0318642164. Actual BF16 output versus its own mathematical normalization has NRMSE0.0016030992. This supports upstream drift plus learned-gamma weighting, not an established final-RMS kernel defect.
The receiving opus-sonnet-5-5 session completed cache interventions without a production patch. Substituting all approved reference caches still failed the seed gate; those substitutions round BF16 state ports and are not fully identical-input FP32-reference executions. They do not prove a universal BF16 error floor. Its subsequent weight-audit continuation was blocked by CLI Python permissions, so the lead completed both full weight audits directly; no receiving CLI remains running.
One new standalone MoE probe was rejected because a bare-module dtype conversion accidentally BF16-rounded the strict FP32 correction-bias buffer. The corrected probe restores original checkpoint buffer values after dtype conversion and matches the captured ONNX BF16 MoE output exactly. The failed probe is preserved and explicitly marked rejected; this was a diagnostic implementation error, not a defect in the preserved pretrained references, whose complete buffer/value audits pass.
Acceptance is unchanged: approved decoder FAIL; short cache comparisons PASS232/232; original eight-token generation PASS, preserved—not rerun here; conditional actual-target MTP PASS12/12, preserved—not end-to-end acceptance;257-token generation FAIL; speculative integration NOT_RUN. No tolerances were relaxed, no reference authority was silently changed, and no ORT patch is justified by these results yet.
New router evidence is preserved under case3-router24-lead-v1 and case3-router24-fp64-lead-v1 in /home/mobius/validation_lightning_h100_parity_fix_20261011. Native source SHA256 remains 5417d6f8dcbee945d4f1c4ae5327b0d1f443b29cc99098c99ec7befb9c9afcd8; source decoder SHA256 remains 28602cd75259febafc7ca0a519b1066a97b19f1b8a378f2519116e20b347cf47.
CUDA runtime onboarding follow-up: microsoft/onnxruntime#33312 and microsoft/onnxruntime-genai#2710 now have zero unresolved code-review threads, and GenAI's exact6979e63 head passes the public Linux x64/ARM64 and Windows CUDA checks. This does not clear this export's previously reported full-checkpoint numerical/generation gates.
I bounded the next numerical probe to real Mamba layer0 at this unchanged e3d01a7 export and checkpoint revision a9904d24bcc1d289a1950fa9d2b978c47cf903b9. Decoder SHA7036d267 is unchanged. All eight mixer parameters match checkpoint values exactly. Both implementations receive identical normalized BF16 hidden inputs; CUDA TF32 is disabled on both sides. Reference is the actual unmodified transformers5.19 NemotronHMamba2Mixer, real DynamicCache and PyTorch fallback, not a substituted equation model.
For reproduction, extract the original layer0 mixer with normalized hidden, past conv and past SSM as its three inputs, and mixer output/present conv/present SSM as its three outputs. Compare prefill plus the three cached single-token cases with both identical upstream incoming caches and independently carried ONNX caches. Translate upstream conv cache by taking its last three of four entries, and transpose SSM carry from[B,H,D,N] to[B,H,N,D]. Capture the norm's inputs and out-projection input with passive hooks.
Result:18/24 comparisons pass; all six cached mixer-hidden comparisons fail unchanged NRMSE<=0.03/cosine>=0.999 gates. Identical-upstream-cache hidden NRMSE is0.05568/0.08278/0.04784; independent-native-carry NRMSE is0.05471/0.12395/0.15029. Convolution carry and gates match bitwise. Upstream SSM carry is indeed FP32; its fallback scan is FP32 on prefill but BF16 on single-token cached forwards.
At corrected ORT provider SHA068c67babdffb1e14a6ed0c5d32be352ac0bc1be8f3d1de8ca1e87e5dfb812f2, native LinearAttention itself matches an independently evaluated float64 gated recurrence with float32 per-step state in8/8 output/state comparisons (max NRMSE4.4e-7). Adding these observations leaves baseline hidden/cache outputs bitwise unchanged. The unchanged upstream grouped gated norm also matches the exported norm bitwise in4/4 cases when supplied identical native BF16 scan/gate. The cached discrepancy is therefore before that normalization, not the tested native recurrence law or matched-input norm.
The export casts dt/B/C/value processing to FP32 before its native attention; upstream's PyTorch fallback rounds some dt/discretization/state-readout operations in BF16 while keeping FP32 carry. That is a concrete precision-contract difference, but a BF16-state-readout-only control holding upstream updated state fixed does not reproduce the entire observed scan (maxabs1/1.5/16), so it is not a complete root cause or a proposed fix. Optimized-kernel-reference behavior has not been measured by this probe. No export/acceptance workaround or weakened tolerance was added.
This is reduced first-layer attribution, not full-model parity or readiness. The intended upstream optimized-kernel versus fallback precision contract needs to be established and the remaining pre-normalization path resolved before claiming full-model support; merging the runtime correction alone must not be assumed to fix these already-patched-provider failures.
This branch has not been deployed
No deployments
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
Latest H100 status (2026-10-12)
Full-model numerical readiness remains BLOCKED. The local source-correct candidate now passes 239/240 approved short decoder comparisons, with 232/232 cache comparisons passing. Cached case 3
mtp_seedstill fails (NRMSE 0.035922356240143276 > 0.03). The separate 257-token generation control also fails. Thresholds and references have not changed.These results are from an unpublished local source proposal, not the unchanged PR head
e3d01a7712f38deb962e7107752da7fb668a7054. The full-eager attention control was rejected (138/240 approved comparisons fail); no production remedy was adopted from it.Detailed verified results, source qualification and focused follow-up.
The description below preserves earlier implementation details and historical validation results; the status above supersedes its older numerical scorecard.
Summary
Full-model numerical readiness remains BLOCKED. The latest user-supplied H100 run (2026-10-11, export checkout
a72b658f) reports normalization, source fidelity, exact eight-token generation and conditional MTP passing, but full decoder parity still fails 141/240 comparisons. Matching short token sequences does not clear full-logit/state gates or establish speculative runtime support.Add pinned Lightning BF16/NVFP4-source recipes, strict explicit ModelOpt-to-dense-BF16 reconstruction and paired target/MTP graphs via
build(..., task="nemotron-h-mtp")/--task nemotron-h-mtp. Outputs:decoder/model.onnxandmtp/model.onnx, with all 270 MTP tensors accounted for and zero package-level MTP omissions. Default NextN refusal and explicit target-only named omission accounting remain. Native NVFP4 stays fail-closed; reconstruction is not preserved NVFP4. No runnable Olive integration or implemented speculative-generation loop is claimed.Implementation and source provenance:
Both graphs use NVIDIA checkpoint weights; Mobius constructs ONNX directly with
onnxscript.nn/onnx_ir, not by tracing upstream PyTorch. Decoder emits post-final-normmtp_seed; draft pairsh_iwitht_(i+1)to predictt_(i+2), owns independent KV and has no RoPE/Mamba. Target-shared checkpoint embedding/head sources are copied into physically dedicated draft tables.Published head
e3d01a7712f38deb962e7107752da7fb668a7054, same PR branch, signed-off/coauthored linear history. MTP childa10651bdintegrated as897a0e05; norm childd3c7a51aasc5d47af4; census/signature/report redaction followed in4ad49ac/4ba865c/a72b658. New interface review fixes are089c161b; diagnostic child1059ab1aintegrated as49d5f2c8, documentation follow-upe3d01a77. No production precision, recurrence or acceptance-threshold changes in this latest diagnostic follow-up.Compatibility evidence and limits
BF16 revision
a9904d24bcc1d289a1950fa9d2b978c47cf903b9; NVFP4 revisionbee7596271d1495f6992ae224aefde4410e816b8. Context 262144/1048576; index tensor bytes 65842365568/21559589596 across 14/52 shards. Metadata pins do not authenticate every payload/tokenizer or prove memory fit. Six source hashes atdc179846...remain historical evidence, not current dependency locks.Both pinned sources contain MTP shared-expert up/down projections, shapes [3712,2688]/[2688,3712]; the earlier absence inference was false. Independently bounded-read headers confirm 270 identical MTP names/shapes, BF16 except FP32 correction bias, zero quantized MTP modules/scales. Target has 6243 names; NVFP4 profiles contain 46 FP8/5935 NVFP4 projections, 46 input/12 KV scales. Full config/header checks after the norm correction: 40176 decoder/1839 MTP nodes. Separate GGUF census: 43272 raw/40222 optimized nodes, 6257 initializers, 6028 MatMul, 60 inputs/59 outputs. These are graph/source checks, not numerical acceptance.
GroupedGatedRMSNormis wired only to Nemotronmamba.norm, preserving its namespace. It avoids faulty 2D-scale RMSNormalization and matches source cast-before-gamma semantics; other Mamba defaults and the legacy flag are unchanged. Local paths remain transient in lazy bindings/overlap guards, not persisted nested reports/ONNX metadata.Latest supplied H100 observations — not locally re-executed full-weight CUDA results:
a72b658f[6993,1046,1256,1010,1784,8961,1307,10787]exactlyAll four baseline full-logit comparisons still fail NRMSE (.0457179/.0531702/.0425463/.0418169), despite matching argmax. First prefill state failure is layer-4 conv state; first hidden failure is layer 34. All 192 identical-input Mamba substage measurements are below unchanged gates: recurrence/state at the same dt agree near 6.1e-7/1.1e-7 NRMSE. All 12 same-input native FP32 router expert sets agree; independently carried FP32 inputs change selections at layer-34 cases 0/1. Layer-4 norm agrees on identical exposed rounded BF16 residual, qualified because fused SkipNorm may normalize before residual-output rounding. These facts separate upstream precision amplification from demonstrated operator errors; they do not justify an invented recurrence patch.
Native FP32 CPU-fallback audit matches prefill (112 comparisons, max NRMSE 1.55e-6) but differs on cached steps (75/96/102 failures). A reference-only diagnostic removing the cached dt minimum clamp matches native fallback; baseline export/runner were unchanged. The original pinned native source SHA256
5417d6f8dcbee945d4f1c4ae5327b0d1f443b29cc99098c99ec7befb9c9afcd8matches the reported module. Creator independently invoked its original fallback bodies on CPU, removing only dispatch decorators: synthetic bounded prefill output/state NRMSE 7.13e-8/4.61e-8; cached bounded-versus-unbounded contrasts up to .00714 (below the full-model gate). Native BF16 source rounding is reproduced exactly; state-readout and early-FP32-dt contrasts measure .001765/.003323. These are synthetic source-contract diagnostics, not a full-model fix or permission to replace the acceptance reference.Earlier
897a0e05report remains historical: original decoder 232/240 failures, max NRMSE 2.10802144/min cosine -.0396307; scratch decomposition 130/240 failures, .1194279/.9928551; generation failed. Earlier scoped MTP SHA2561bd1e3f7fb717d02cc16dd4a48a3f6148bb090d3d4dbdedd99ee82e51fd9e31eand 58 scoped comparisons do not describe the new artifact. Prior reported fused batch-2 cached multi-token, hole-mask and local CausalConv provider-assignment limits are not proven cleared by the new norm test.The complete 8593-line report was preserved and parsed (6401 leaves), SHA256
27F1FEE27ACEF0C007341D787BD7E989D5BDCCA274AED6EED0DC972AB53534C0. Its/home/mobius/...paths name remote evidence; they are not accessible scripts/arrays here. Needed portable bundle: originalrun_validation.py, independent reference/audit helpers and command/backend definitions; optimized graph/provider profiles; case-0 layers 0–4 pre-round residual operands, exposed residual/norm/xBC/conv arrays; case-1 identical state/dt/C/readout arrays; actual native kernel dispatch and dt_limit. Compare full corrected portable/fused decoder using identical tokens/weights/cache provenance before altering production semantics.Runtime remains external: shifted token/hidden pairing, independent draft KV, target verification/sampling, accepted-prefix commit, rejected draft rollback and target KV plus Mamba conv/SSM restoration/replay. Metadata is
runtime_unvalidated. Native NVFP4, full reconstructed-checkpoint parity, Olive and performance remain unvalidated. Full weighted/GPU recipe commands remain session-scoped NOT_RUN; externally supplied execution is labeled separately.Validation
Local Windows ARM64/Adreno with isolated x64 CPU Python; Torch 2.14.1+cpu, Transformers 5.18.0, ONNXScript .7.2, ONNXIR 1.0.0, ORT 1.30.0. No new large-weight downloads/GPU packages/paid GPU work by this session.
Six new cases cover two-row prefill plus three reused-state steps, grouped B/C, active/inactive dt clamp, output/conv/FP32 SSM and FP16/BF16 cast boundaries against independent equations. They protect current export semantics rather than silently replacing the reference. CPU source probe rerun in session artifacts; no repository copy of upstream source or generated diagnostics.
Historical creator results: norm integration 177 passed/7 skipped; census 23 passed/2 unchanged privilege cases deselected plus generated-docs check; redaction/streaming 105 passed/4 existing skips; earlier MTP integration 545 passed/2 skipped and package/runtime/task 421 passed/6 unchanged privilege cases deselected. Not aggregate unique counts or full new-head reruns. Full pinned config/header/ten-hash checks previously passed; no invented full-model CPU/CUDA pass.
At
a72b658f, six OS/Python unit jobs, lint/lintrunner/CodeQL, L1/L3/fast integration/GenAI/build passed. L4/L5 check surfaces show failure; job/step details say cancelled during golden execution at one hour, not passing or established assertion failures. CLA remains pending. New-head CI must be inspected live.Both new review threads addressed and resolved in
089c161b:PRRT_kwDORmWK4M6qkxbenow correctly declares dedicated draft tables/null shared-runtime pointers;PRRT_kwDORmWK4M6qkxcHmakes draft private, with only public paired wrapper advertising the task. Full metadata/paired-task regressions pass. Prior nine threads were already resolved; no duplicate review requests. Full-model correctness is a release gate, not a thread to blanket-resolve.Remaining gate: inspect portable runner/reference/backend and earliest-boundary data, retain unchanged acceptance thresholds and all 240 comparisons, then diagnose and verify full decoder/state parity. Generation PASS and norm PASS are scoped progress, not full-model or speculative-runtime acceptance.
Code relationship graphs
Graph source commit 089c161; subsequent
49d5f2c8/e3d01a77change tests/docs only, not displayed production relationships. 53 locally AST-verified edges. Light theme/source-module grouping; blue functions, purple classes, green methods, gray external APIs; solid calls/construction, dashed type/config use. Scope: loader/numerics/cache/safety/target/MTP/grouped norm and immediate dependencies. Omitted: exhaustive branches/full math/weights, actual CUDA dispatch/performance, speculative verification/rollback and unrelated models. These are static relationships, not runtime execution. Scripts/data/searchable zoomable neighbor-highlight HTML remain in session artifacts. Authenticated attachment SHA256 verified; public access and persisted embeds verified after publication.