[Burntgogi]

Maintainer continuation and contributor handoff

The maintainer merged current main into this branch as 35f9c80, made it ready for review, and reported A6000 gates and serving checks, including a Korean prompt. The previous 182012b3 conflicts are resolved in that newer head. The maintainer plans to merge after CI and review; check GitHub for their live state. The maintainer explicitly labeled the cc12.1 pins provisional pending a GB10 scan.

I could not find a confirmed solution to the earlier-source CPU/GPU greedy-output disagreement and am handing off this investigation. In my reply, I asked the maintainer or another contributor to investigate whether it persists on the new head and, if it is a defect under the intended contract, find a fix or otherwise decide its disposition. I did not run the aligned CPU-text comparison on 35f9c80b, so neither persistence nor resolution there is claimed. The numerical evidence below is source-scoped historical evidence, not a current-head result.

Historical contributor single-attempt disposition on 182012b (before maintainer merge)

The then-current source 182012b independently executed the complete model on GPU0 through the actual Korean natural HTTP endpoint, using21-prefix-prefill/last-prompt-next, ctx1024 per slot/two slots, temperature0 and the same22 prompt IDs as the pinned mainline CPU natural-v2 run. GPU output reached150003 EOG after231 tokens and the owned server shut down0. The CPU reached150003 after121 tokens. Their first actual token disagreement is predict position29: CPU100, GPU230. Both the original old CPU-text assertion and the aligned independent natural CPU text comparison remain false; successful execution/EOG is not numerical acceptance. The new serve binary SHA256 is 0e86410befc7b21783222498b963eecb679c5533a8dec1e63a66bdfb793a2769.

A current182 clean forced trace using the actual HTTP-generated IDs matched the natural HTTP's first eight argmax values; clean tap/no-tap full logits, placement and loaded expert slots were byte-identical. A separately identified four-file observer overlay at the first divergent prefix produced full target logits, final norm and all49 layer rows bit-identical to the clean trace. The CPU target logits/norm/layer0 and natural norm/delta controls matched their original reference byte-for-byte, as did both CPU/GPU residual identities. Observer changes were reversed, the tracked source was clean afterward, no owned process remained, and GPU1's existing service remained active/healthy.

At that first-prefix layer0, the actual same-GPU-normalized-input CPU/GPU FFN deltas differ by relative L2 0.006633730568. The relevant Q8_0 GPU path intentionally multiplies F32 activations; the pinned mainline CPU's Q8_0 path quantizes activations. An independent F64 calculation over the original packed weights, same actual GPU norm and no production change matched the GPU FFN to relative L2 1.8720e-7; with both activation Q8_0 roundtrips through the pinned CPU trait it matched the CPU FFN to 1.5974e-7. Four sampled actual packed gate rows matched GGML trait-dequant F64 dots exactly. This is strong evidence about the two FFN arithmetic paths, not proof that either violates its contract or that this layer alone caused the final model's first token divergence.

The controlled head counterfactual for token100−230 margins is CPU head/CPU norm +0.39533424, CPU head/GPU norm −0.61946678, GPU head/GPU norm −0.67440081. The CPU head's complete150144-logit natural control was byte-identical. Thus head arithmetic alone cannot reverse this first divergence; GPU/CPU layer0 input already differs upstream. No single responsible operator or whole-model accuracy acceptance is established.

Requested maintainer disposition: with the repository's deliberately F32-activation GPU contract versus mainline CPU's Q8_0-activation path, which existing model-oracle/quality acceptance applies to this exact greedy-text disagreement? Please do not interpret this as a proposed tolerance/precision/tie/expected-text change. No production numerical change was made because a contract-violating defect was not demonstrated. Actual numerical diagnostics, exact-source IDs, raw input hashes, controls and the initial trace-only rejected-environment error are retained in the contributor evidence archive.

As separately observed, #781 was maintainer-merged at 4e05b5fd and #632 closed by the maintainer; their state does not prove this Draft's correctness. git merge-tree against that pinned main reported conflicts in crates/gpu/src/host/handoff.rs, docs/research/flint-attach.md, and tools/unsafe-ratchet.txt; this branch was not silently merged or history replaced. #781's cc12.1 rows are explicitly maintainer-provisional copies of cc8.6 values until physical GB10 measurement. The older 30-row CI result below is a historical 182012b3 run, not a claim about later main. #782 is on maintainer head 56b15d0d with new-head disposition pending; #783 was reconciled as 9e7ab5aa, received a new exact-head approval, lint success and completed T0 success.

Earlier source-scoped record

Original 182 integration record

Current head 182012b preserves the ordinary histories of #781/#782/#783 and merges pinned upstream 0df4076 after observed conflicts blocked CI. No force push, replacement branch, new issue or new PR.

Closure changes

• Shared typed projection/activation owners, protected prequantized expert inputs, mixed card/tier and empty-tier allocation contracts retained.
• All five gpu: size mixed expert scratch from selected resident forms #783 review repairs retained; the real Qwen candidate-form failure is fixed by model-derived accepts::leg_down_forms in both public wrappers and every caller.
• Integrated explicit reserve API retains input_scales:bool: Qwen/MiniMax/common paths passfalse; FlInt's input-restoration scales remain true. The component API intentionally has no such bool.
• Upstream Q5_0 packed-32 dispatch is retained; no new format contribution was started here.
• The shared llama serve recipe includes its required flint feature; no diagnostic was suppressed.

Actual host proof

At 695018d and current 182012b, formatting check, the discovered model unit/integration targets’ non-ignored cases (cargo test -p bloomery-model --tests), and workspace all-target Clippy including the real FlInt feature union exited0. The existing Qwen bp contract previously reproduced2failed/1passed and now passes3/0 with unchanged placement expectations. The separate actual public-planner budget smoke selected only the Q8Act layer and left the over-budget supported Q8_0 layer on the host; reserve2169472,zero violations.

At prior fbf77f7, actual PR-check40435532684 completed lint success, check-recipes exit0/613s, T0 36/37; only check-ptx-names was red for30 genuine physical-cc12.1 rows. This cleared the recipe bug, not full T0 or the resource gate. The local full recipe invocation's12 pre-existing locale-sensitive shell-output failures are separately preserved, not re-pinned. The prior CI is not relabeled as182012b3.

Actual NVIDIA execution — keep the sources separate

Current public182012b3090d2e00051bc736052ef6f583636921 was separately built and executed on GPU0: tests::hw_full_file_capture_slots_reset_and_ring_boundary 1passed/0failed/0ignored,1filtered,38.38s, case and terminal0. This uses the complete25,054,688,704-byte model for tensor forward, resident-slot reset and ring boundary, not a header-prefix stand-in. The completed c143 binary was archived before reusing its idle target cache; neither approved component source nor the original695 source/target was changed.

The earlier, separately pinned 695018d completed protected-input1pass, mixed/empty1pass, shared DS41 library40pass and full-real-file1pass, all0failed/ignored; attention/router and the actual Cargo-owned serve/JIT/extractor builds also exited0. These are not relabeled as182 executions.

695 actual HTTP remains exit1, only korean: oracle_text_equal and korean: chat_parts_match_oracle. English19tokens and Korean231tokens reached natural150003 EOG. Actual SSE/nonstream equality, thinking/content splitting, two concurrent resident-slot clients, required/auto tools, and a natural question without a function name or use-tool instruction selecting get_weather passed. Four-stop coverage is tokenizer-only, not four naturally generated stop types. Owned server shutdown0. The original expected CPU string was not re-pinned.

Controlled numerical boundary, not an accuracy waiver

An independent mainline a37cc46104eb50900d5ab4a9ba02a0abdac039db CPU natural-v2 run used the same22prompt IDs and21-prefix-prefill/last-prompt-next boundary,4threads,1slot,ctx total/per-slot1024,batch/ubatch32,KVf16,flash auto,no offload and unchanged strict argmax/tie policy. No generated reference IDs were read or fed. It reached150003 EOG after121tokens. Its full call16 logit row is bit-identical to the original forced CPU control. It still differs from actual695 HTTP: first predicted position29,CPU100/GPU230. Earlier38/96 disagreements from different GPU placement are historical, not this run's first divergence.

The695+named four-file observer's complete target logits/norm/all49layers are bit-identical to clean695; eager/graph × tap/no-tap four-arm full arrays/placement/loaded-slot controls also match within695. The actual CPU/GPU residual identities match every byte. Before either CPU counterfactual, the natural CPU norm/delta8192-byte controls matched exactly. With the same actual normalized input, the packed-Q8_0 up/gate/SwiGLU/down chain still differs (max_abs0.001177542,relativeL2 0.624272%). Feeding the slightly different CPU/GPU norms into the CPU FFN yielded identical8192-byte outputs, so normalization alone does not explain that target's FFN difference. This forced call16 experiment is not a first-kernel/quantization-defect finding, nor proof of the natural position29 cause or whole-trajectory acceptance. No precision, tolerance, band, tie or compiler workaround.

Actual physical8.6 JIT/add-class evidence

At the same toolchain on physicalcc8.6, 95a483f →695018db, generate_ds41 has 458→468entries,10added,0removed,0changed, ptx-compare added-only/exit0, methoddecl1/archsm_86. Both actual JIT scans exited0. The original coordinator exited2 because its stdout-only comparison omitted the scanner's authentic bundle provenance on stderr. A separately recorded CPU-only comparison of all original stderr+stdout bytes, each input hashed, produced the valid verdict; no GPU/JIT/build was replayed and no provenance row invented. The original2 is retained, not rewritten as0. This is not a current182 full-PTX or physical12.1 proof; actual13.3.33 is not called the full13.3.73 pin environment.

All owned workers are now finished, no owned source process remains and GPU0 has no compute process. GPU1's existing service was never stopped and remains active/healthok.

Remaining closure gates

• Resolve/appropriately establish the aligned mainline numerical oracle and the first operation inside the observed FFN difference; do not turn either the original or matched CPU disagreement into a pass by changing expectations/precision/bands.
• Complete affected current-source HTTP and unexecuted model/residency/tier/NVMe acceptance after the numerical change. The current182 full-file GPU smoke is real but not all-runtime coverage; already completed controls are not work to repeat.
• Measure the 30 genuine physicalcc12.1 rows, including gpu: support six-slot F32 tier handoff #781's3 (not33). Deeper serial inventory of the allowed02/03 OS views found no NVIDIA PCI device/node or alternate libcuda path, so no usable registered lane was established. No blank/fake row,cc8.6 substitution,driver install or permission bypass.
• Fresh current-code review and maintainer merge only after the gates pass. gpu: support ordinary IQ4_XS shared sites #782/gpu: size mixed expert scratch from selected resident forms #783 are separately Ready/current-bot-APPROVED and should not be moved for nonblocking warnings. This contributor has read-only upstream permissions. No new issue/PR/duplicate comment,ROCm or performance acceptance claim. Flint: attach head128 ring attention and hosted runtime (needs GPU) #799/model: Flint (flint, 256 experts, sliding-window attention with sinks) #632 remain open.

Current182 native accounting: 587passed,0failed,125ignored,0filtered. Those125 were ignored by that native command; the separate header/GPU runs are named explicitly. Unexecuted fixture/real-model cases are not claimed, and this is not all-model runtime acceptance.

Current182 independent static integration review found no reachable defect in the admitted-form repair, explicit false/true scale split, empty-tier bytes or upstream packed-Q5_0 restore path. The previously ignored current-source real-file hw_flint_header was separately run: 1passed,0failed/ignored/filtered,779tensors/49layers,ctx512/1024/4096×slots1/3; this remains header/placement-only, not tensor-payload/GPU proof.

Actual current182 PR-check40448378247 has completed: lint success, check-recipes0/680s, T0 36/37,red only check-ptx-names for30 genuine physical12.1 rows. This is a fresh current-head result, not the earlier fb run.

License provenance checked against current primary sources: both RecursiveSI/flint and unsloth/flint-GGUF currently declare apache-2.0, and both repositories’ actual LICENSE files contain Apache2.0. The pinned GGUF header’s other/flint-preview text is preserved, not rewritten; source statements and cached metadata remain distinguished. No weights/header payload was redistributed here, and no new license grant is asserted.

[midagedev]

Thank you. I merged main (now with #781, #782 and #783) into this branch so Flint can land in 0.3.10. Maintainer changes in the merge:

• tools/ref/ptx-shapes/{sm_86,sm_121,sm_121a}-cc12.1.tsv: the nine Flint entries' 27 rows, at the cc 8.6 scan's values under PIN(2026-10-27) until a GB10 scan runs (as gpu: support six-slot F32 tier handoff #781's six-slot row).
• tools/recipes.py: the host nightly's host_left_out expectation is back, with bloomery-gpu-flint in it.
• tools/hf-arch-check.py: "a glm5nextx file is refused by name" is back, with flint in the seat list.
• docs/cards/flintclip-tape.card: the lease card of the recorded run below.

Box (RTX A6000): just check and just lint 0 warnings; gate-flint-meta, gate-gpu-flint-attn, gate-gpu-flint-router, gate-gpu-flint-lib, gate-gpu-ds41-tier, gate-ops, gate-gpu-lib green. bloomery-serve on /models/flint-GGUF/flint-UD-Q4_K_XL.gguf at the serving defaults passed chat, streaming, tool calls, --parallel 2 and a Korean prompt. Recorded run: https://tape.midagedev.com/r/c9garawzei5e7hkfw4fq

I will squash-merge after CI and the review.

[Burntgogi]

Thank you for merging main and running the A6000 gates and serving checks. I am handing off the remaining investigation: I could not find a confirmed solution to the CPU/GPU greedy-output disagreement. Would you or another contributor please investigate whether it still occurs on the new head and, if it represents a defect under the intended numerical contract, find a fix or otherwise determine the appropriate disposition?

For clarity, my actual natural HTTP comparison was on the earlier head 182012b3090d2e00051bc736052ef6f583636921, not your new 35f9c80be22d816381c8c3ab287e0262b51ccbe4. With the same 22 prompt IDs and aligned prefill/next boundary, its first mismatch was predict position 29: CPU token 100 vs GPU token 230. GPU generation ended normally, but the strict CPU text comparison failed. I have not run that comparison on your new head and cannot say whether the mismatch persists there; your A6000 serving success is separately scoped to your recorded run.

The earlier-head same-input diagnostics in the PR body found GPU Q8_0 weights with F32 activations and CPU Q8_0 activation roundtrips each close to their corresponding independent F64 calculations; the head-only counterfactual does not account for the first divergence. I could not prove a single causal operator or a numerical contract violation. I did not change the expected text, tolerances, precision or tie handling. Please use your judgment on whether and how to address this evidence; I am stepping back rather than proposing another numerical patch or test waiver.

[github-actions]

Verdict: MUST_FIX: one test is deleted with no dated reason, and the new flint required feature breaks the documented bloomery-serve build commands.

Must fix

1. A test is removed without a dated reason

crates/gpu/src/card_experts.rs:223 card_downs is unchanged and still used at line 373. The PR deletes the_leg_buffers_hold_the_card_downs_forms, its only test (base card_experts.rs:1256). Its mutant was "buffers back on the 128-value form alone": an NVFP4 down finds no q8 blocks of 32 and is refused at its first run. After this PR, nothing turns red if card_downs returns Q8Act only. The PR text does not give a reason for the removal.
Cost: the contract lost its guard while its code stays live.
Fix: restore the test in the tests module of card_experts.rs, unchanged:

    #[test]
    fn the_leg_buffers_hold_the_card_downs_forms() {
        let q8 = DownForms::of(DownAct::Q8Act);
        let b32 = DownForms::of(DownAct::Q8Blocks32);
        assert_eq!(card_downs(DownForms::NONE), q8);
        assert_eq!(card_downs(q8), q8);
        assert_eq!(card_downs(b32), b32);
        assert!(!card_downs(b32).holds(DownAct::Q8Act));
        let both = q8.with(DownAct::Q8Blocks32);
        assert_eq!(card_downs(both), both);
    }

If the removal is intended, add a dated reason in the commit message and PR body.

2. Documented bloomery-serve builds now fail

crates/gpu-gates/Cargo.toml:176 adds flint to required-features of bloomery-serve. The justfile, tools/release/build.sh and tools/ref/dense-logits.py get flint. These docs do not: docs/BUILD.md:88,259,262,313, docs/contrib/dgx-spark.md:59, docs/serving.md:702. Each lists ...solar2,minimaxm3,gemma-embedding2,hf with --bin bloomery-serve. Cargo refuses that build ("requires the features: flint"), so a contributor who follows the docs gets an error.
Cost: the same edit is needed later, after a contributor reports it.
Fix: insert flint, after minimaxm3, in each of those commands, as build.sh:194 does.

Non-blocking

⚠ Known CPU/GPU greedy-text disagreement: the PR body reports the Korean HTTP run at 695018db fails oracle_text_equal and chat_parts_match_oracle, and first diverges at predict position 29 (CPU 100, GPU 230). No run on the current head is reported. The author hands the question to the maintainer. A merge decision should state which oracle gate applies to this model.

⚠ Protected expert path adds launches: crates/gpu/src/card_leg.rs:534 with AmaxF16, each routed layer runs normalize and restore, and a separate quantize replaces the fused swiglu-quant. Net +2 launches per routed layer per decode step, about +96 for 49 layers [derived]. Direct models are unchanged. The class "launch-count change" needs a timed A/B on the contributor's card; the PR body has none.

⚠ Lost tensor name in a refusal: crates/gpu/src/card_experts.rs:366 replaced down_act, which named layer.names[2] in the error, with GpuError::state, which names no tensor. Use GpuError::Tensor { what, name: layer.names()[2].to_string(), need } as before, if GpuError::Tensor still takes those fields.

Checked

• Duplicates: about 25 new items checked. union_plan moved from the gate to placement (net fewer lines). The dryer pairs flint.rs:37-46, minimaxm3/mod.rs:50-59 and solar2/mod.rs:51-60 are the same open_placed trait method per family; they hold no shared knowledge. The dryer pairs in shared_routed.rs are shape only. The ring kernels follow the existing per-variant kernel family.
• Complexity: the crapper list is new gate, trace and kernel-launch bodies. Nothing grew by a branch in an existing function.
• Silent failure: 9 sites followed. rule.unwrap_or(Act::SiluMul) is paired with the rule.is_some() flag. ExpertInput::protect panics on non-finite input. Ring live_keys returns 0, and the merge kernel raises the fault and writes NaN. The _ => {} at the Ffn match is followed by hosted().
• Ring safety: append chunks are cut to capacity - window + 1 (chunk_rows). Rollback is refused once the ring wrapped (cut_keeps). Body.at is raised before a prefill can fail. The refute step found no failure path.
• Tests: the new hw_prequantized_leg_preserves_protection_and_replay compares protected and ordinary outputs at amplitude 8192, so a dropped restore should turn it red (read, not run). The removed test: see item 1.
• Gates: the justfile, ptx-shapes and unsafe-ratchet edits are mechanical. PIN lines carry dates. No band moved.
• Escape catalog: no tools/*.sh change adds a C1 to C4 pattern.

Notes

• The PR body has no nvidia-smi line and no just gate-* output; its GPU runs are described in prose. gpu.json says no evidence.
• The cc12.1 ptx-shapes rows are provisional copies of cc8.6 rows, as the PR text says.

_{Card check: needs-gpu. This pull request changes crates/app/Cargo.toml, crates/app/src/arch/flint.rs, crates/app/src/arch/minimaxm3/mod.rs … and carries no gate run on a card (CONTRIBUTING's evidence list); a maintainer adds gpu-verified after one.}

_{🔨 Shingler, bloomery's review bot: approves a clean pull request and requests changes on a must-fix (how it works)}

[midagedev]

The review's two must-fix items are in one more maintainer commit: the_leg_buffers_hold_the_card_downs_forms is back in card_experts.rs unchanged, and the documented bloomery-serve builds (docs/BUILD.md, docs/contrib/dgx-spark.md, docs/serving.md) name flint.

On the Korean greedy-text gap (CPU 100 vs GPU 230 at predict position 29): Flint has no CPU or llama.cpp referee on our side yet, so for 0.3.10 the gate is the card path's own gates and the serving checks above. A CPU/GPU oracle gate for Flint is a follow-up.

[github-actions]

Verdict: CLEAN: no failure path survived in the hunks I read, and no duplicate, extra element or moved gate was proven. This diff is about 560 KB, so I read the riskiest parts and not every file (see Notes).

Non-blocking

⚠ Third copy of the expert-form match: crates/model/src/ops.rs:5156 adds ExpertForm::combine_into. The same Gated/GatedOai/Relu2 dispatch already exists in Combines::slot (ops.rs:5446) and in InputCombine (ops.rs:710 to 734). Its only caller, moe.rs:1104, matches run.form once to pick gate and up, and combine_into matches it again. Direction: let Combines::slot call combine_into too, or drop the method and do the combine inside the existing match in serve_resolved. Do this only if the net element count falls.

Checked

• Duplicates: about 20 new items Grepped (ExpertInput, Protection, Slots, combine_into, ring kernels, live_keys, the seat adapter). One hit, the ⚠ above. The three dryer pairs between flint.rs, minimaxm3/mod.rs and solar2/mod.rs are impl Batch shells over different types, so they share a shape only. The two shared_routed.rs pairs were not read in depth.
• Complexity: crapper flagged only new functions. The gates (loaded_case, valid_case) and trace_flint::run are long by design. I read none of them for splitting. Body::load (37) and Binding::load (34) are new loaders, and I found no branch added to a shared function.
• Silent failure: 12 sites followed.
  • rule.unwrap_or(Act::SiluMul) in expert_input.rs:7455 is safe: the rule.is_some() flag is passed and activate=0 leaves u unread.
  • _ => {} over Ffn at body.rs (about line 1842 of the diff) lets dense layers through to the binding.
  • .ok() on usize::try_from lands in an error branch.
  • The ExpertInput::protect assert and the device bad flag turn a non-finite activation into a named panic or a QuantColumn fault, with NaN output.
  • The ring append kernel faults CachePos and writes NaN when p >= ctx.
  • The merge kernel faults KeyCount.
• Boundaries I tried and found blocked:
  • Ring append, two tokens on one row: the host refuses m > rows, and the unsafe fn contract names distinct positions modulo rows.
  • expert_input::slot with counts[1] > len: the subtraction wraps, but every read stays below len.
  • AmaxF16 selected without load-time scale storage: serve_resolved and UnionCall::expert_input return a named error. The GPU normalize launch contract requires scales.len() >= slots.
  • Empty and zero activation: ln(0) gives scale 1.
• Tests: read hw_prequantized_leg_preserves_protection_and_replay and the new gate_flint_attn loaded case.
  • Mutant "restore skipped after down": the want/got comparison turns red.
  • Mutant "scale not republished on changed-input replay": the changed-input replay compares against an ordinary run, so it turns red.
  • I did not read the two-card and full-file tests.
• Gates: BAND = 1e-2 in gate_flint_attn is unchanged. The unsafe-ratchet.txt rows (gpu, gpu-deepseek41, gpu-flint, gpu-gates) and the model-tax.tsv rows each carry a dated PIN with a reason. No tolerance or KNOWN_DIVERGENCE moved.
• Escape catalog: C1 to C4 do not match. The justfile and release/build.sh changes only add the flint feature. I did not check that every recipe building bloomery-serve got it.
• Performance: hot paths touched are the decode step (card_leg, route_body) and the prompt path. The protection path adds one normalize and one restore launch per routed block only when AmaxF16 is selected. Direct keeps the old fused kernels. Scale buffers are allocated at load. No per-step allocation found. The PR body has no timed A/B. This is a question about the class "launch count change", not a finding, because the default path is unchanged.

Notes

• Not read in full: gpu-flint/src/body.rs, step.rs, prefill.rs, two_card_tests.rs, examples/trace_flint.rs, route_body.rs, gate_flint_router.rs, placement/*, docs/research/flint-attach.md. A defect there would not show in this review.
• The PR body states an open CPU/GPU greedy-text disagreement (first difference at predict position 29) that was not re-run on this head, and asks the maintainer for a disposition. This review does not cover numerical acceptance. ExpertInput::protect (host) and normalize (device) use ln/exp from different libraries, so the scales can differ by an ulp.
• The PR text has no nvidia-smi line and no gate output in its own files, only links to a maintainer comment. gpu.json says evidence: false.

_{Card check: needs-gpu. This pull request changes crates/app/Cargo.toml, crates/app/src/arch/flint.rs, crates/app/src/arch/minimaxm3/mod.rs … and carries no gate run on a card (CONTRIBUTING's evidence list); a maintainer adds gpu-verified after one.}

_{🔨 Shingler, bloomery's review bot: approves a clean pull request and requests changes on a must-fix (how it works)}