Repository navigation
Conversation
github-actions
Bot
added
gpu
|
Timed sitting (
pp4096 falls inside the derived band (1,000–3,000). pp512 falls below it (3,000–5,800). Two stages still run in 8-row chunks: the latent absorb ( |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Verdict: MUST_FIX: Wide dispatch rejects supported weights, shared-block scratch can be too small, and the tap gate accepts isolated NaNs.
Must fix
1. Preserve the pass path for formats the wide arm cannot run
crates/gpu/src/arch/raon1/rows.rs:336 selects wide execution for every prompt of nine or more ids. mla.rs:75,208 admits Q6_K attn_output; a nine-id prompt reaches wide.rs:608, where only_blocks32 rejects it. The existing pass handles that projection through eight ids. Dense/shared downs (wide.rs:801) and Q4_K/Q5_K routed downs (wide.rs:923) also reject formats the loader admits.
Resolve eligibility at load and retain the existing pass loop for other weights. This prevents a prompt-length-dependent failure before more models use the path.
Add pub(super) wide_ok: bool, to Body. In its initializer, before layers is moved, add:
wide_ok: layers.iter().all(|layer| {
let mixer = match &layer.mixer {
Mixer::Rda { sites, .. } =>
sites.out.ty() == SiteTy::Q8_0 && sites.f_b.ty() == SiteTy::F32,
Mixer::Mla { sites, .. } =>
sites.sites()[3..].iter().all(|s| s.ty() == SiteTy::Q8_0),
};
let ffn = match &layer.ffn {
Ffn::Dense(g) => matches!(
g.down.ty(), SiteTy::Q4K | SiteTy::Q5K | SiteTy::Q6K
),
Ffn::Routed(r) => {
let gate_up = match (w.get(&r.stacks[0]), w.get(&r.stacks[1])) {
(Some(DevWeight::KQuant { ty: g, .. }),
Some(DevWeight::KQuant { ty: u, .. })) =>
g == u && crate::gemm::GemmWeight::from_ggml(*g).is_ok(),
_ => false,
};
gate_up && r.shared.down.ty() == SiteTy::Q8_0
&& matches!(w.get(&r.stacks[2]),
Some(DevWeight::Q5_0 { .. }) |
Some(DevWeight::KQuant {
ty: gguf::GgmlType::Q8_0 | gguf::GgmlType::Q5_1
| gguf::GgmlType::IQ4_NL, ..
}))
}
};
mixer && ffn
}),Replace the wide loop header with:
while self.wide_ok && n - done >= WIDE_FROM {Gate nine ids with a Q6_K output projection; removing self.wide_ok must turn it red.
2. Reject non-finite tap distances before taking percentiles
crates/gpu-gates/src/bin/gate_raon1_e2e.rs:700 discards the maximum. shared/e2e.rs:19 converts a NaN tap to an infinite distance. Among eight verify rows, the median and p90 select indices 4 and 6, so one corrupted row at index 7 leaves both checks passing. verify_ok feeds the clause verdict at line 729.
Checking finiteness now preserves the fault contract without tightening the rounding bands. Replace line 700 with:
if v.iter().any(|r| !r.is_finite()) { return false; }
let (med, p90, _) = spread(v);Mutation: replace one verify tap value with NaN. The current checks stay green; this check returns false.
3. Include the shared block in the scratch width
crates/gpu/src/arch/raon1/body.rs:202 omits e.shared_ff(). The metadata reader accepts seven shared experts of width 256 with dense width 1024. wide.rs:263 then allocates 256 × 1536 floats for each gate/up buffer. A 256-id prompt with Q8_0 shared projections needs 256 × 1792 floats and fails the GEMM buffer check. The existing pass allocation includes the shared width (body.rs:658).
Fixing the common size formula also fixes the reserve calculation. Replace the field initializer with:
ff_rows: hp.dense_ff.max(e.shared_ff()).max(ffn::sigmoid_128_6::N_USED * e.ff),Non-blocking
⚠ Publish the timing proof: docs/cards/raonpf-pp.card:5 changes prompt geometry and float sum order. Supply owning-gate output and a same-lease pp512/pp4096 A/B on the contributor's card. Also reconcile the prediction: 512 ids in 50–57 ms gives 9.0–10.2k tok/s [derived, this card's RTX 3090/model conditions], not 4,300.
Optimization headroom
- G. Per-launch and per-sync fixed cost: crates/gpu/src/arch/raon1/wide.rs:583 rebuilds the same head route at each Q6_K MLA layer. For L ≥ 1 such layers, fill it once in the existing unit setup; save
(L − 1) / mroute launches per prompt token for a unit of m rows.
Checked
Validation: traced loader admission through wide dispatch and the shared-block allocation to their error checks. A numeric reproduction confirms that seven zero distances plus infinity pass the current tap predicate and fail the proposed guard. Both inline replacements target added lines. No findings conflict with the exclusions, simple-design scope or escape catalog; no ledger lessons apply.
Notes
CUDA gates were not executed here. The PR lacks the card/driver line and gate command outputs required for GPU evidence.
Card check: needs-gpu. This pull request changes crates/gpu-gates/src/bin/gate_raon1_e2e.rs, crates/gpu/src/arch/raon1/body.rs, crates/gpu/src/arch/raon1/mla.rs … and carries no gate run on a card (CONTRIBUTING's evidence list); a maintainer adds gpu-verified after one.
🔨 Shingler, bloomery's review bot: approves a clean pull request and requests changes on a must-fix (how it works)
| let pass = prompt && verify; | ||
| let rels = row_rels(&sunk, &stepped, w, (from, layers.len())); | ||
| let within = |v: Vec<f64>| { | ||
| let (med, p90, _) = spread(v); |
There was a problem hiding this comment.
Reject NaN-derived infinite distances before percentiles can discard them.
| let (med, p90, _) = spread(v); | |
| if v.iter().any(|r| !r.is_finite()) { return false; } | |
| let (med, p90, _) = spread(v); |
| dense_ff: hp.dense_ff, | ||
| shared_ff: e.shared_ff(), | ||
| expert_ff: e.ff, | ||
| ff_rows: hp.dense_ff.max(ffn::sigmoid_128_6::N_USED * e.ff), |
There was a problem hiding this comment.
Include the shared width so a full 256-row unit fits the gate/up buffers.
| ff_rows: hp.dense_ff.max(ffn::sigmoid_128_6::N_USED * e.ff), | |
| ff_rows: hp.dense_ff.max(e.shared_ff()).max(ffn::sigmoid_128_6::N_USED * e.ff), |
There was a problem hiding this comment.
Since the last review:
- Wide eligibility: fixed at crates/gpu/src/arch/raon1/rows.rs:339.
- Non-finite taps: fixed at crates/gpu-gates/src/bin/gate_raon1_e2e.rs:659.
- Shared-block scratch: fixed at crates/gpu/src/arch/raon1/body.rs:240.
- Timing evidence: still open; the prediction arithmetic was corrected.
Verdict: MUST_FIX: The new fixture duplicates existing weight generators and tensor-size facts.
Must fix
1. Reuse the existing random-weight generators
crates/gpu/src/arch/raon1/synth.rs:38 copies the LCG at crates/gpu/src/arch/raon1/rda.rs:596. Its values also repeats that module's F32, Q8_0 and Q6_K generators. The LCG body and Q6_K generation expression are identical (R14).
Reuse them now so fixture fixes have one owner and this addition removes three definitions.
Delete the local Lcg and its implementation and the Q8Block import. Replace the existing rda::tests import with:
use super::rda::tests::{Lcg, f32s, q6k, q8_0, same};In rda.rs, change the existing method header to pub(crate) fn next(&mut self) -> u64 { and replace the tuple declaration with:
pub(crate) struct Part(String, GgmlType, Vec<usize>, pub(crate) Vec<u8>);Replace values with:
fn values(r: &mut Lcg, ty: GgmlType, n: usize, range: (f32, f32)) -> Vec<u8> {
let name = String::new();
let dims = vec![n];
match ty {
GgmlType::F32 => f32s(r, name, dims, range).3,
GgmlType::Q8_0 => q8_0(r, name, dims).3,
GgmlType::Q6_K => q6k(r, name, dims).3,
GgmlType::Q4_K => (0..n / 256 * 144)
.map(|i| match i % 144 {
1 | 3 => 0x14,
_ => r.next() as u8,
})
.collect(),
GgmlType::Q5_0 => (0..n / 32 * 22)
.map(|i| if i % 22 == 1 { 0x14 } else { r.next() as u8 })
.collect(),
other => panic!("the synthetic file holds no {other:?} tensor"),
}
}2. Read tensor sizes from the existing type table
crates/gpu/src/arch/raon1/synth.rs:256 repeats five block sizes already owned by crates/gguf/src/qtype.rs:46 and its rows (R14). Both declaration sizing and the byte assertion now depend on this second table.
Remove the copied facts before another fixture type extends both tables. Replace the definition with:
fn nbytes(ty: GgmlType, n: usize) -> usize {
let row = ty.row().expect("a known fixture tensor type");
let n = u64::try_from(n).expect("fixture element count fits u64");
let bytes = gguf::tensor_bytes(row.blck, row.bytes, &[n])
.expect("fixture byte count fits u64");
usize::try_from(bytes).expect("fixture byte count fits usize")
}Non-blocking
⚠ Publish current GPU and timing proof: wide.rs:718 changes absorption quantization; mla.rs:957 changes attention geometry. At 256 rows with Q6_K values, absorb launches fall from 32 to 4; for a multi-segment cache, flash launches fall from 64 to 16 [derived, per MLA layer/unit]. Attach current owning-gate output, PTX comparison including raon_row_window, and the same-lease pp512/pp4096 A/B on the contributor's card.
Checked
Validation: both duplicates and their existing owners read. Both must-fix sites are added in this incremental diff, and the fixes reuse existing owners with fewer definitions. The proposed generator access and tensor-sizing APIs exist. Eligibility, shared-width and finiteness fixes have corresponding assertions. No exclusion or escape-catalog pattern applies; no ledger lessons are in force.
Notes
CUDA gates were not executed here. The PR lacks the card/driver line, both run hashes and gate command outputs required for GPU evidence.
Card check: needs-gpu. This pull request changes crates/gpu-gates/src/bin/gate_raon1_e2e.rs, crates/gpu/src/arch/raon1/body.rs, crates/gpu/src/arch/raon1/mla.rs … and carries no gate run on a card (CONTRIBUTING's evidence list); a maintainer adds gpu-verified after one.
🔨 Shingler, bloomery's review bot: approves a clean pull request and requests changes on a must-fix (how it works)
There was a problem hiding this comment.
Since the last review:
- Random-weight generators: still open (crates/gpu/src/arch/raon1/synth.rs:38).
- Tensor-size table: still open (crates/gpu/src/arch/raon1/synth.rs:256).
- GPU and timing proof: still open (crates/gpu/src/arch/raon1/wide.rs:718).
Verdict: MUST_FIX: Both previously reported fixture duplications remain.
Must fix
1. Reuse the existing random-weight generators
crates/gpu/src/arch/raon1/synth.rs:38 duplicates the LCG at crates/gpu/src/arch/raon1/rda.rs:596. Its values also copies that module's F32, Q8_0 and Q6_K generators (R14, kind B).
Reuse the existing owners before fixture changes require parallel edits.
Delete the local Lcg and its implementation and the Q8Block import. Replace the existing rda::tests import with:
use super::rda::tests::{Lcg, f32s, q6k, q8_0, same};In rda.rs, change the existing method header to pub(crate) fn next(&mut self) -> u64 { and replace the tuple declaration with:
pub(crate) struct Part(String, GgmlType, Vec<usize>, pub(crate) Vec<u8>);Replace values with:
fn values(r: &mut Lcg, ty: GgmlType, n: usize, range: (f32, f32)) -> Vec<u8> {
let name = String::new();
let dims = vec![n];
match ty {
GgmlType::F32 => f32s(r, name, dims, range).3,
GgmlType::Q8_0 => q8_0(r, name, dims).3,
GgmlType::Q6_K => q6k(r, name, dims).3,
GgmlType::Q4_K => (0..n / 256 * 144)
.map(|i| match i % 144 {
1 | 3 => 0x14,
_ => r.next() as u8,
})
.collect(),
GgmlType::Q5_0 => (0..n / 32 * 22)
.map(|i| if i % 22 == 1 { 0x14 } else { r.next() as u8 })
.collect(),
other => panic!("the synthetic file holds no {other:?} tensor"),
}
}2. Read tensor sizes from the existing type table
crates/gpu/src/arch/raon1/synth.rs:256 repeats five block-size facts owned by crates/gguf/src/qtype.rs:67 and its following rows. Declaration sizing and the byte assertion consume this second table (R14, kind B).
Remove it before another fixture type requires extending both tables. Replace the definition with:
fn nbytes(ty: GgmlType, n: usize) -> usize {
let row = ty.row().expect("a known fixture tensor type");
let n = u64::try_from(n).expect("fixture element count fits u64");
let bytes = gguf::tensor_bytes(row.blck, row.bytes, &[n])
.expect("fixture byte count fits u64");
usize::try_from(bytes).expect("fixture byte count fits usize")
}Non-blocking
⚠ Publish current GPU and timing proof: wide.rs:718 changes absorb precision; mla.rs:957 changes flash geometry. At 256 rows with Q6_K values, absorb launches fall from 32 to 4 and multi-segment flash launches from 64 to 16 [derived, per MLA layer/unit]. Attach owning-gate output, PTX comparison including raon_row_window, and same-lease pp512/pp4096 A/B. Include the card/driver line, both hashes and model file so the reported gains can be verified.
Checked
Duplicates: both carried defects and their existing owners confirmed. Both dryer pairs are module-specific bundle loads.
Complexity: the interdiff changes only unsafe-ratchet pins; no function growth.
Silent failure: the ignored fixture-file cleanup result does not alter model output.
Tests: no test changes in the interdiff. Removing wide_ok() at rows.rs:339 routes the synthetic Q6_K prompt into the wide arm's refusal; synth.rs:335 then fails.
Gates: rebased pins preserve 247 functions and add four unsafe blocks, 2112 → 2116, with dated reasons.
Escape catalog: C1–C4 do not match. No ledger lessons apply. Both proposed fixes reuse existing owners and remove code.
Notes
CUDA gates were not executed here. The PR body and prediction cards lack the complete GPU evidence required by CONTRIBUTING.md.
Card check: needs-gpu. This pull request changes crates/gpu-gates/src/bin/gate_raon1_e2e.rs, crates/gpu/src/arch/raon1/body.rs, crates/gpu/src/arch/raon1/mla.rs … and carries no gate run on a card (CONTRIBUTING's evidence list); a maintainer adds gpu-verified after one.
🔨 Shingler, bloomery's review bot: approves a clean pull request and requests changes on a must-fix (how it works)
|
The two must-fix items are in dfbef09: synth.rs reuses rda::tests' generators (Lcg, f32s, q8_0, q6k) and gguf's type table (GgmlType::row + gguf::tensor_bytes). The synthetic file's bytes are unchanged. gate-gpu-lib green (fixture tier, RTX 3090, 331 s). |
There was a problem hiding this comment.
Since the last review:
- Random-weight generators: fixed; synth.rs:42 reuses the existing generators.
- Tensor-size table: fixed; synth.rs:223 reads the existing type table.
Verdict: CLEAN: Both earlier must-fix findings are resolved, and the incremental changes introduce no new defect.
Checked
Duplicates: 0 new items. Both earlier duplicates now reuse their owners. Both dryer pairs read: distinct device-module bundle loads.
Complexity: crapper's nine flagged functions are unchanged in this increment; values and nbytes lose branches.
Silent failure: 0 added sites. Unsupported fixture types and byte-count overflow raise named panics.
Tests: no tests or assertions changed. Inspection confirms identical RNG draws and F32/Q8_0/Q6_K encoding. The existing fallback test rejects dropping wide_ok() at rows.rs:339. CUDA gates were not run here.
Gates: no pins, thresholds or cases moved in this increment.
Escape catalog: C1–C4 do not match.
Card check: needs-gpu. This pull request changes crates/gpu-gates/src/bin/gate_raon1_e2e.rs, crates/gpu/src/arch/raon1/body.rs, crates/gpu/src/arch/raon1/mla.rs … and carries no gate run on a card (CONTRIBUTING's evidence list); a maintainer adds gpu-verified after one.
🔨 Shingler, bloomery's review bot: approves a clean pull request and requests changes on a must-fix (how it works)
…GEMM (gemm_q5_0p), so a Q5_0 down has a prompt route Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…EMM for every projection, the common KDA mixer's chunked path for the RDA layers, the route tables for the experts (#616) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…itions of one prompt bit-identical, and the long set's taps against the steps Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…l entry Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… pp4096, RTX 3090) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…_0 planes, and its flash attend takes 32 tokens a launch; check_flash takes any m >= 1, the partials sized by the caller The absorb quantizes its activations to int8 for the 32-value GEMM, which moves the e2e gate's D set from 5.32e-2 to 7.61e-2 against the vendor (band 8.0e-2). The flash change moves no bit: 32 tokens a launch equal every cut by bits. raon_row_window is a new kernel entry (add class). The card bytes added are 185,753,600 (partials, unit buffers, form); the e2e gate's resident bytes rose by exactly that. docs/cards/raonpf2-pp.card holds the prediction for the prompt sitting. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…14 ms at P = 512 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…o a file it cannot run keeps the rows pass at every prompt length; non-finite taps fail the band check; the wide gate rows hold the shared block - Body::wide_ok: the load runs the wide arm's own refusals (wide::rda_runs, MlaSites::wide_runs, wide::ffn_runs) over every layer. A file with, for instance, a Q6_K attn_output runs a 9-id prompt on the rows pass. Routed gate and up stacks of two types are refused by name. - gate_raon1_e2e: a tap distance that is not finite fails within_bands before the percentiles (taps_band_separation), and wide_runs_the_file holds the real file on the wide arm. - wide_ff_rows includes the shared experts' width, as the rows pass does. - synth.rs: a lib test builds a three-layer file at load, a Q6_K output projection against a Q8_0 control. FAIL-first: the loop without wide_ok turns the synthetic test red; the old ff_rows formula turns a_wide_units_gate_rows_hold_the_shared_block red. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The synthetic file's bytes are unchanged: F32, Q8_0 and Q6_K come from rda::tests' generators, which hold the same expressions and call order. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Since the last review:
- Random-weight generators: fixed; crates/gpu/src/arch/raon1/synth.rs:42 still reuses the existing generators.
- Tensor-size table: fixed; crates/gpu/src/arch/raon1/synth.rs:223 still reads the existing type table.
Verdict: CLEAN: The rebase reconciles the documented unsafe pins with the new base and introduces no new defect.
Checked
Duplicates: 0 new items checked in the interdiff. Both dryer pairs read: loaders for distinct device bundles; unchanged since the last review.
Complexity: all nine crapper-flagged functions are unchanged in the interdiff; no new complexity growth.
Silent failure: 0 new sites; the interdiff changes only unsafe-count pins and their dated explanations.
Tests: no tests changed; mutation checks are not applicable. Independent source recount matches base 2115/247/10 and head 2119/247/10 blocks/functions/implementations. CUDA gates were not run here.
Gates: the two adjusted PIN histories preserve the base's three Q6_K blocks and account for the join's three blocks plus the window copy's one block. No numeric band or gate case changed in the interdiff.
Escape catalog: C1–C4 do not match.
Notes
The PR reports GPU gate passes but lacks the nvidia-smi line, both commit hashes and gate command output required by CONTRIBUTING.md.
Card check: needs-gpu. This pull request changes crates/gpu-gates/src/bin/gate_raon1_e2e.rs, crates/gpu/src/arch/raon1/body.rs, crates/gpu/src/arch/raon1/mla.rs … and carries no gate run on a card (CONTRIBUTING's evidence list); a maintainer adds gpu-verified after one.
🔨 Shingler, bloomery's review bot: approves a clean pull request and requests changes on a must-fix (how it works)
1 participant
Closes #616.
Raon-1-mini's prompt ran through the verify rows pass in chunks of 8 rows, at about 250–300 tok/s on an RTX 3090. It now runs in wide units of up to 256 rows:
linear/mixer_kda.rs) for the RDA layers;The kernel table gains one row: a Q5_0 stack's packed words route through the 32-value GEMM (
gemm_q5_0p). Only Raon's wide unit builds a Q5_0 stack route. The dense sites and the qwen3moe arms do not admit the entry, so other families do not change.Measured, RTX 3090,
depth-serve.sh, one lease (main's rows 312.4 and 298.8 tok/s):The second round (
docs/cards/raonpf2-pp.card, predicted +4..+10 %) read +7.27 % ± 0.60 % at P = 512 and +5.22 % ± 0.60 % at P = 4096; decode moved inside ± 0.5 %. The first round's derivation (4,300 tok/s) missed: a unit costs 105.5 ms against the derived 50..57 ms, because the experts and the dense GEMMs run one after the other on one stream and do not overlap (raonpf-design.mdsection 9).Second round:
attn_k_bQ8_0 planes, and the latent flash takes 32 tokens a launch.check_flashtakes anym >= 1; the caller sizes the partials. The new entry israon_row_window. The card gains 185.8 MB.Review fixes:
Body::wide_okis decided at load from the refusals the wide arm raises. A file it cannot run (for instance a Q6_Kattn_output) keeps the rows pass at every prompt length. Routed gate and up stacks of two types are refused by name.Evidence:
just ptx-compare faa81674f gate_raon1_e2e: added-only. The only new entry israon_join_qkv(spill 0, jit_local 0).prompt_rows_is_steps,verify_is_steps,commit_keeps_steps).gate_raon1_e2ehas new clauses:prompt_wide_is_vendors: the wide prompt against the vendor's logits;prompt_wide_is_partitions: two partitions of one prompt are bit-identical;gate-gpu-raon1-e2e,-dflash,-attnandgate-gpu-libgreen on the real tier.ptx-compareadded-only. The flash callers' gates (gate-gpu-p5,-e2e,-hybrid,-moe,-p10,-p8b) are green on the real tier. FAIL-first per fix: the loop withoutwide_okturns a synthetic-file test red, and the oldff_rowsformula turns the shared-width test red.gate-gpu-raon1-e2e,-dflash,-attn,gate-raon1-metaandgate-raon1-tokenizerare green.just affectedselects: 223 items, 2 red.gate-gpu-glm5next-tpran out of memory: another process held 17.5 GB of the same card.gate-gpu-ds41-dspark-loop's draft-load byte check moved while another process allocated on the card.This work was written with AI help: a Claude subagent wrote the change, and Claude reviewed it.
🤖 Generated with Claude Code