diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index 42cf8d85d..00073cc44 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -37,7 +37,7 @@ | D-DRAM-P3-2 | non-vacuous `project` k4/k16 sparse/mixed cases | Queued | this session; research branch | | D-DRAM-P3-3 | = D-DRAM-1 read-only census | Done: [entry](entries/2026-11-03-d-dram-1-census.md); only `half_add_mask`/`full_add_mask` shipped, `U24x5`/`Abacus` lab-only, no production lease | this session; report to EXO-46 / ndarray owner | | D-DRAM-P3-4 | EXISTS early exit per tile (0.16× DuckDB at 10M) | Queued | this session drafts; mask-risc owner reviews | -| D-DRAM-P3-5 | = D-DRAM-3 k-of-n Abakus bench, A1/A2/A5–A7 | Queued, after P3-3 | this session; no new opcode/type | +| D-DRAM-P3-5 | = D-DRAM-3 k-of-n Abakus bench, A1/A2/A5–A7 | Measured: kill fired, carry words 1.7–4.0× slower than facade byte counters; A1/A2/A6 partial, A5/A7 open | `entries/2026-11-03-d-dram-p3-5-kofn.md`, `crepe-plan-probe/examples/kofn_carry_words.rs` | | D-DRAM-P3-6 | ordered u8-in-u32 compare proposal | Queued | this session proposes; ndarray owner | | D-DRAM-P3-7 | route once per frame, lab measurement | Queued | this session; production GATED EXO-46 | | D-DRAM-P3-8 | = D-DRAM-4 end-to-end, decision memo | GATED | D4 + S-INGRESS (#1788) | diff --git a/.claude/board/entries/2026-11-03-d-dram-p3-5-kofn.md b/.claude/board/entries/2026-11-03-d-dram-p3-5-kofn.md new file mode 100644 index 000000000..14e2f8681 --- /dev/null +++ b/.claude/board/entries/2026-11-03-d-dram-p3-5-kofn.md @@ -0,0 +1,56 @@ +# D-DRAM-P3-5: k-of-n on the shipped carry words loses to facade byte counters + +**Status:** MEASURED-LAB, kill condition fired. Owner: this session (plan `.claude/plans/d-dram-p3-integrated-v1.md`, IDEAS "k-of-n on the shipped carry words"). Probe: `crates/crepe-plan-probe/examples/kofn_carry_words.rs`. + +## What was measured + +The question: per lane, are at least k of n masks set? The arms: +- **carry**: the Abakus register fold, built from the two shipped words only. `full_add_mask` adds two masks into column 0, `half_add_mask` ripples the carry upward, and a constant-k comparator runs over the planes. +- **simd**: native counting on the same facade, one `U8x64` of counters per word, `saturating_add(mask_blend(m, 0, 1))` per mask, then `cmpgt_mask(k - 1)`. +- **lane** and **bytes**: naive per-lane counting. +- **or** and **and**: plain Boolean folds. +- **oracle**: walks set bits with `trailing_zeros` into per-lane `u32` counts. + +Setup: 1024 words (65,536 lanes), median of 31 repetitions, three processes. One host with AVX-512 and no VNNI, built for the default x86-64 target. The facade backend that build selected was not recorded. + +## Result (ns per 64-lane word, ranges over 3 processes) + +| workload | n | simd | carry | carry / simd | +|---|---|---|---|---| +| dense p=7/8 | 3 | 2.8–5.0 | 7.0–13 | 1.7–2.6× | +| dense p=7/8 | 8 | 6.4–6.8 | 15–16 | 2.2–2.4× | +| dense p=7/8 | 32 | 24–25 | 61–65 | 2.4–2.6× | +| random p=1/2 | 3 | 2.6–4.2 | 6.8–7.8 | 1.7–2.9× | +| random p=1/2 | 8 | 6.2–6.8 | 18–20 | 2.8–3.1× | +| random p=1/2 | 32 | 24–38 | 58–100 | 2.3–4.0× | +| sparse one lane | 3 | 2.6–3.0 | 6.7–7.6 | 2.3–2.9× | +| sparse one lane | 8 | 6.1–6.7 | 13–15 | 1.9–2.3× | +| sparse one lane | 32 | 24–43 | 87–130 | 3.0–3.7× | + +Each row covers k = 1, ⌈n/2⌉ and n; the ratio range is taken over all three. The carry arm is slower than facade counting in every cell, dense masks included. The naive lane loop (130–1,300 ns) and byte loop (33–260 ns) are slower than both. The plain OR/AND fold is 2.0–23 ns. + +**Correctness:** in all 3 runs, every arm agrees with the oracle in every cell. The silence twin holds: k = 1 equals OR and k = n equals AND exactly. Anti-vacuity: on random masks the middle k splits the lanes (density between 5% and 95%). + +**Disable runs** (commit first, then patch, then `git checkout`; each patch asserted to apply): +- the comparator drops its `gt` term: 21 FAIL lines; +- the ripple stops after one column: 29 FAIL lines; +- restored: 0. + +## Reading + +The kill condition named in IDEAS fired: on dense masks the carry-word threshold is slower than native. The finding is narrow. Built from the two shipped `u64` words, a bit-sliced counter does not beat a byte counter that the facade already has, for a k-of-n threshold up to n = 32. The likely reason is width. The shipped words act on one `u64` (64 lanes) per call, while `U8x64` counts 64 lanes in one vector add. This is HYPOTHESIS, not measured: no wide carry-word arm was built. + +This does not contradict #402. There, the bit-plane counter beat 64 scalar per-lane adds; here the comparison arm is a vector byte counter. + +## Coverage against the #1949 A-gates (proposal by a GPT session; adopted after review) + +- **A1, partial:** n ∈ {3, 8, 32}, k ∈ {1, ⌈n/2⌉, n}. Not covered: k = 0, k = n + 1, and n ∈ {1, 2, 5, 16, 24}. +- **A2, partial:** dense, random and one-lane masks. Not covered: zero, full, alternating, tail masks; only one host and target. +- **A6, partial:** a misplaced carry is caught by the oracle (disable 2). +- **A5 and A7:** not covered. No NULL/Bag semantics; an isolated microkernel only, with no Quack lowering and no other ISA tier. + +## Open + +- A carry fold over `U64x8` (512 lanes per call) would need a wide adder. That word does not exist, and building one would be a new ndarray word that needs its owner (#1949 §7). It is not built here. +- Repeat with `target-cpu=native` and on AVX2-only and NEON hosts. The facade backend selected is unrecorded. +- The model may still win where byte counters cannot follow: counts above 255 (more than 8 planes), and folds that keep the planes for later arithmetic. Neither was measured. diff --git a/.claude/board/entries/README.md b/.claude/board/entries/README.md index 5e757307a..980fbbcae 100644 --- a/.claude/board/entries/README.md +++ b/.claude/board/entries/README.md @@ -25,7 +25,7 @@ index row, (3) no duplicate entry id. Checks 1 and 2 are deliberately opposite directions; the stranding this convention prevents shows up in exactly one of them, never both. -572 entries, 2026-08-06 .. 2026-11-04. +573 entries, 2026-08-06 .. 2026-11-04. | date | entry id | finding | file | |---|---|---|---| @@ -36,6 +36,7 @@ exactly one of them, never both. | 2026-11-03 | `D-GLOBAL-WHERE-P2` | | [2026-11-03-d-global-where-p2.md](2026-11-03-d-global-where-p2.md) | | 2026-11-03 | `D-GLOBAL-WHERE` | | [2026-11-03-d-global-where-p0.md](2026-11-03-d-global-where-p0.md) | | 2026-11-03 | `D-DRAM-P3` | | [2026-11-03-d-dram-p3-consolidated.md](2026-11-03-d-dram-p3-consolidated.md) | +| 2026-11-03 | `D-DRAM-P3-5` | | [2026-11-03-d-dram-p3-5-kofn.md](2026-11-03-d-dram-p3-5-kofn.md) | | 2026-11-03 | `D-DRAM-0` | | [2026-11-03-d-dram-abacus-address-memoization.md](2026-11-03-d-dram-abacus-address-memoization.md) | | 2026-11-03 | `D-DRAM-1` | | [2026-11-03-d-dram-1-census.md](2026-11-03-d-dram-1-census.md) | | 2026-11-03 | `D-DR-FOLD-MAP` | | [2026-11-03-d-dr-fold-map.md](2026-11-03-d-dr-fold-map.md) | diff --git a/.claude/plans/d-dram-p3-integrated-v1.md b/.claude/plans/d-dram-p3-integrated-v1.md index 903192c8d..6d29b0174 100644 --- a/.claude/plans/d-dram-p3-integrated-v1.md +++ b/.claude/plans/d-dram-p3-integrated-v1.md @@ -84,7 +84,7 @@ Stop on any of: - [ ] P3-2 non-vacuous `project` cases - [x] P3-3 D-DRAM-1 census written (`entries/2026-11-03-d-dram-1-census.md`) - [ ] P3-4 EXISTS early exit, draft PR -- [ ] P3-5 D-DRAM-3 k-of-n bench (A1, A2, A5–A7) +- [x] P3-5 D-DRAM-3 k-of-n bench: kill fired, carry words 1.7–4.0× slower than facade byte counters (`entries/2026-11-03-d-dram-p3-5-kofn.md`); A5/A7 not covered - [ ] P3-6 ordered compare proposal - [ ] P3-7 route-once lab measurement - [ ] P3-8 D4 / S-INGRESS decision memo diff --git a/crates/crepe-plan-probe/examples/kofn_carry_words.rs b/crates/crepe-plan-probe/examples/kofn_carry_words.rs new file mode 100644 index 000000000..86739e23f --- /dev/null +++ b/crates/crepe-plan-probe/examples/kofn_carry_words.rs @@ -0,0 +1,314 @@ +//! D-DRAM-P3-5 (D-DRAM-3): k-of-n threshold over n masks, built from the two +//! shipped ndarray carry words only, against native per-lane counting. +//! +//! Lab only. No public type, no new opcode. The only ndarray words used are +//! `half_add_mask` and `full_add_mask`; `U24x5` / `Abacus` are lab fixtures +//! and are not imported. +//! +//! Question: does the bit-plane carry fold (the Abakus register model) answer +//! "at least k of these n masks are set, per lane" faster than native +//! per-lane counting, and does it agree exactly with an independent oracle? +//! +//! Arms, all producing one result mask word per 64 lanes: +//! - `lane` native: for each of the 64 lanes, count the n bits, compare to k. +//! - `bytes` native: 64 `u8` counters per word, one add per mask, compare. +//! - `simd` native on the facade: one `U8x64` of counters, per mask +//! `saturating_add(mask_blend(m, 0, 1))`, then `cmpgt_mask(k - 1)`. +//! - `carry` the bit-plane counter: `full_add_mask` adds two masks into +//! column 0, `half_add_mask` ripples the carry up; a constant-k comparator +//! over the planes gives the result. The planes stay in registers. +//! - `or` / `and`: native Boolean folds, timed only for k = 1 and k = n, +//! where k-of-n must equal them exactly (the silence twin). +//! +//! The oracle walks the set bits of every mask with `trailing_zeros` into a +//! per-lane `u32` count, so it shares no mechanism with any arm. +//! +//! Run: `cd crates/crepe-plan-probe && cargo run --release --example kofn_carry_words` + +use ndarray::simd::{U8x64, full_add_mask, half_add_mask}; +use std::hint::black_box; +use std::time::Instant; + +const WORDS: usize = 1024; // 65,536 lanes per mask +const REPS: usize = 31; +const MAX_PLANES: usize = 8; + +struct Rng(u64); +impl Rng { + fn next(&mut self) -> u64 { + // splitmix64 + self.0 = self.0.wrapping_add(0x9E37_79B9_7F4A_7C15); + let mut z = self.0; + z = (z ^ (z >> 30)).wrapping_mul(0xBF58_476D_1CE4_E5B9); + z = (z ^ (z >> 27)).wrapping_mul(0x94D0_49BB_1331_11EB); + z ^ (z >> 31) + } +} + +#[derive(Clone, Copy)] +enum Workload { + Dense, + Random, + SparseOneLane, +} + +impl Workload { + fn name(self) -> &'static str { + match self { + Workload::Dense => "dense p=7/8", + Workload::Random => "random p=1/2", + Workload::SparseOneLane => "sparse one lane", + } + } + fn word(self, r: &mut Rng) -> u64 { + match self { + Workload::Dense => r.next() | r.next() | r.next(), + Workload::Random => r.next(), + Workload::SparseOneLane => 1u64 << (r.next() % 64), + } + } +} + +/// `masks[i][w]`: input mask i, word w. +fn make(n: usize, wl: Workload, seed: u64) -> Vec> { + let mut r = Rng(seed); + (0..n) + .map(|_| (0..WORDS).map(|_| wl.word(&mut r)).collect()) + .collect() +} + +fn planes_for(n: usize) -> usize { + (usize::BITS - n.leading_zeros()) as usize +} + +// ── oracle ────────────────────────────────────────────────────────────────── + +fn oracle(masks: &[Vec], k: usize) -> Vec { + let mut counts = vec![0u32; WORDS * 64]; + for m in masks { + for (w, &word) in m.iter().enumerate() { + let mut x = word; + while x != 0 { + let j = x.trailing_zeros() as usize; + counts[w * 64 + j] += 1; + x &= x - 1; + } + } + } + let mut out = vec![0u64; WORDS]; + for (i, &c) in counts.iter().enumerate() { + if c as usize >= k { + out[i / 64] |= 1u64 << (i % 64); + } + } + out +} + +// ── native arms ───────────────────────────────────────────────────────────── + +fn arm_lane(masks: &[Vec], k: usize, out: &mut [u64]) { + for (w, o) in out.iter_mut().enumerate() { + let mut r = 0u64; + for j in 0..64 { + let mut c = 0usize; + for m in masks { + c += ((m[w] >> j) & 1) as usize; + } + r |= ((c >= k) as u64) << j; + } + *o = r; + } +} + +fn arm_bytes(masks: &[Vec], k: usize, out: &mut [u64]) { + let k = k as u8; + for (w, o) in out.iter_mut().enumerate() { + let mut cnt = [0u8; 64]; + for m in masks { + let x = m[w]; + for (j, c) in cnt.iter_mut().enumerate() { + *c += ((x >> j) & 1) as u8; + } + } + let mut r = 0u64; + for (j, &c) in cnt.iter().enumerate() { + r |= ((c >= k) as u64) << j; + } + *o = r; + } +} + +fn arm_simd(masks: &[Vec], k: usize, out: &mut [u64]) { + let (zero, one) = (U8x64::splat(0), U8x64::splat(1)); + let km1 = U8x64::splat((k - 1) as u8); + for (w, o) in out.iter_mut().enumerate() { + let mut cnt = zero; + for m in masks { + cnt = cnt.saturating_add(U8x64::mask_blend(m[w], zero, one)); + } + *o = cnt.cmpgt_mask(km1); + } +} + +fn arm_or(masks: &[Vec], out: &mut [u64]) { + for (w, o) in out.iter_mut().enumerate() { + *o = masks.iter().fold(0, |a, m| a | m[w]); + } +} + +fn arm_and(masks: &[Vec], out: &mut [u64]) { + for (w, o) in out.iter_mut().enumerate() { + *o = masks.iter().fold(!0, |a, m| a & m[w]); + } +} + +// ── the carry-word arm ────────────────────────────────────────────────────── + +/// Adds `carry` (weight 2^from) into the planes, rippling up. +#[inline(always)] +fn ripple(p: &mut [u64; MAX_PLANES], b: usize, from: usize, mut carry: u64) { + let mut j = from; + while carry != 0 && j < b { + let (s, c) = half_add_mask(p[j], carry); + p[j] = s; + carry = c; + j += 1; + } +} + +/// Per lane: plane value >= k, for a constant k, MSB first. +#[inline(always)] +fn ge_const(p: &[u64; MAX_PLANES], b: usize, k: usize) -> u64 { + let mut eq = !0u64; + let mut gt = 0u64; + for j in (0..b).rev() { + if (k >> j) & 1 == 1 { + eq &= p[j]; + } else { + gt |= eq & p[j]; + } + } + gt | eq +} + +fn arm_carry(masks: &[Vec], k: usize, out: &mut [u64]) { + let n = masks.len(); + let b = planes_for(n); + for (w, o) in out.iter_mut().enumerate() { + let mut p = [0u64; MAX_PLANES]; + let mut i = 0; + while i + 1 < n { + let (s, c) = full_add_mask(p[0], masks[i][w], masks[i + 1][w]); + p[0] = s; + ripple(&mut p, b, 1, c); + i += 2; + } + if i < n { + ripple(&mut p, b, 0, masks[i][w]); + } + *o = ge_const(&p, b, k); + } +} + +// ── timing ────────────────────────────────────────────────────────────────── + +fn time(mut f: impl FnMut(&mut [u64])) -> f64 { + let mut out = vec![0u64; WORDS]; + let mut v = Vec::with_capacity(REPS); + for _ in 0..REPS { + let t = Instant::now(); + f(black_box(&mut out)); + black_box(&out); + v.push(t.elapsed().as_nanos() as f64 / WORDS as f64); + } + v.sort_by(|a, b| a.partial_cmp(b).unwrap()); + v[REPS / 2] +} + +fn main() { + let mut failures = 0usize; + println!( + "k-of-n over n masks, {WORDS} words ({} lanes), median of {REPS}, ns per 64-lane word", + WORDS * 64 + ); + println!( + "{:<16} {:>3} {:>3} {:>9} {:>9} {:>9} {:>9} {:>9} {:>10}", + "workload", "n", "k", "lane", "bytes", "simd", "carry", "or/and", "simd/carry" + ); + for wl in [Workload::Dense, Workload::Random, Workload::SparseOneLane] { + for n in [3usize, 8, 32] { + let masks = make(n, wl, 0x5EED ^ n as u64); + for k in [1usize, n.div_ceil(2), n] { + let want = oracle(&masks, k); + let mut got = vec![0u64; WORDS]; + for (name, f) in [ + ("lane", arm_lane as fn(&[Vec], usize, &mut [u64])), + ("bytes", arm_bytes), + ("simd", arm_simd), + ("carry", arm_carry), + ] { + f(&masks, k, &mut got); + if got != want { + failures += 1; + println!( + "FAIL {name} {} n={n} k={k}: disagrees with oracle", + wl.name() + ); + } + } + // Silence twin: k = 1 is OR, k = n is AND, exactly. + let bool_ns = if k == 1 || k == n { + let mut b = vec![0u64; WORDS]; + if k == 1 { + arm_or(&masks, &mut b); + } else { + arm_and(&masks, &mut b); + } + arm_carry(&masks, k, &mut got); + if got != b { + failures += 1; + println!( + "FAIL twin {} n={n} k={k}: carry != {}", + wl.name(), + if k == 1 { "OR" } else { "AND" } + ); + } + Some(time(|o| { + if k == 1 { + arm_or(&masks, o) + } else { + arm_and(&masks, o) + } + })) + } else { + None + }; + // Anti-vacuity: a middle k on random masks must split the lanes. + if matches!(wl, Workload::Random) && k == n.div_ceil(2) { + let ones: u32 = want.iter().map(|w| w.count_ones()).sum(); + let frac = ones as f64 / (WORDS * 64) as f64; + if !(0.05..=0.95).contains(&frac) { + failures += 1; + println!("FAIL vacuity random n={n} k={k}: result density {frac:.3}"); + } + } + let t_lane = time(|o| arm_lane(&masks, k, o)); + let t_bytes = time(|o| arm_bytes(&masks, k, o)); + let t_simd = time(|o| arm_simd(&masks, k, o)); + let t_carry = time(|o| arm_carry(&masks, k, o)); + let b = bool_ns.map_or("-".to_string(), |x| format!("{x:.2}")); + println!( + "{:<16} {n:>3} {k:>3} {t_lane:>9.2} {t_bytes:>9.2} {t_simd:>9.2} {t_carry:>9.2} {b:>9} {:>9.2}x", + wl.name(), + t_simd / t_carry + ); + } + } + } + if failures == 0 { + println!("all arms agree with the oracle; twins k=1 == OR and k=n == AND hold"); + } else { + println!("{failures} FAILURES"); + std::process::exit(1); + } +}