diff --git a/.claude/board/LATEST_STATE.md b/.claude/board/LATEST_STATE.md index 82f73c451..5f203008e 100644 --- a/.claude/board/LATEST_STATE.md +++ b/.claude/board/LATEST_STATE.md @@ -1,3 +1,7 @@ +## 2026-10-21 — D-CMR-2b measured: no win on whole extents; short runs pay per-run cost (in PR) + +- **lance-graph-benches (lab example only, no library change):** `examples/pred_scratch_probe.rs`. On whole extents and 40k-row runs a fused compare-and-count fold does not beat today's tiled `execute_extent`. On `set_lower`'s run shape with 200-row runs, per-run `execute_extent` is 2.5–4× slower than the fold; compiling once cuts it by only 5–18 %. Remedy (fewer edge-tile passes, a run-batched entry point, or a short-run fold) not decided, not built. Entry `entries/2026-10-21-d-cmr-2b-scratch-probe.md`. Contract inventory: unchanged. + ## 2026-10-21 — main after #1654 (merged `d754128`) - **On main:** #1652, #1653, #1654 (D-RA-2 repoint of relalg `symbolic` onto `contract::set_rules`; `Rule::{EmptyMeasure, CountAll}`; #1651 closeout). Contract inventory delta: two `set_rules::Rule` variants (see the D-RA-2 section below). diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index 71c00a051..46baede51 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -256,6 +256,7 @@ Plan: `.claude/plans/d-cmr-concept-chain-normalization-v1.md`. Parent: D-CMR-RES | **D-CMR-0** | Plan, identity matrix, inventory, board rows | Shipped #1532 (docs) | docs only; every inventory row cites code | | **D-CMR-1** | Set-theory IR core in the contract: `Runs` (∩ ∪ ∖, complement), residual `Res`, measures; pure | Shipped (#1644, `c05a313`): `contract::set_algebra::{SetExpr, Res, Runs, normalize}`; measures not yet (`entries/2026-10-20-d-cmr-1-set-algebra.md`) | algebra laws vs materialized bitsets as oracle | | **D-CMR-2** | Lowering: runs → extents, residual → predicates per run; measurement levels geometry / optics / etch | Shipped (#1650, `f031089`): `lance_graph::fold::set_lower::{SetPlan, SortedBasis, Level, Measured}`; geometry reads nothing, optics/etch run the residual per run via `execute_extent`. The scratch-free `Pred` fold in mask-risc is D-CMR-2b, not built (`entries/2026-10-21-d-cmr-2-lowering.md`) | == row oracle; `Count` without residual reads zero rows | +| **D-CMR-2b** | Scratch-free `Pred` fold in mask-risc (lab first) | Measured (`lance-graph-benches/examples/pred_scratch_probe.rs`; `entries/2026-10-21-d-cmr-2b-scratch-probe.md`): no win on whole extents or 40k-row runs; on 200-row runs per-run `execute_extent` is 2.5–4× slower than the fold, compile-once cuts A by 5–18 %. Remedy not decided, not built | answers == oracle before timing; interleaved paired timing with a null arm | | **D-CMR-3** | Quack frontend: `Filter` → set IR (extent hoisting falls out) | Queued | identical answers on the quack suite; other-basis range stays residual | | **D-CMR-4** | Two dimensions: `Rect` over composite coordinates, row-major or Morton runs | Queued | rectangle ∩ == bitset oracle | | **D-CMR-5** | **BLOCKED: lawful design proposed (operator 2026-10-21).** Preimage `f⁻¹(S)` as a co-addressed transposed tile fold over a version-witnessed projection of `f`, built in preparation by a sequential same-table scan with frozen-codebook endpoint translation (ADMISSION B; reusable across queries). `S` enters only as a membership plane translated into the same frozen coordinates. No seek or load of `f` keyed by `S`'s values or a previous fold (Q1 = C), no `Gather`, no scatter terminal. R1 (`rmap` `rebind` / `discover` escape-driven seeks) stays option C. ⊘ Was: "VIOLATION — GATHER. DO NOT BUILD" (entry `2026-10-21-d-cmr-5-rescope.md`) | BLOCKED — needs a contract-level attested witness for the frozen codebook (`rmap::Witness` is a probe-local stand-in); no preimage operator exists; not admitted, not ready to build | == row oracle; program checked Gather-free with a silence twin; attest / projection / query costs reported apart; a many-to-one `f` gives a valid distinct-source preimage (tested) and a bag-sensitive Count/Sum over it is refused (tested); refusal on a stale or missing witness | diff --git a/.claude/board/entries/2026-10-21-d-cmr-2b-scratch-probe.md b/.claude/board/entries/2026-10-21-d-cmr-2b-scratch-probe.md new file mode 100644 index 000000000..12f9675e0 --- /dev/null +++ b/.claude/board/entries/2026-10-21-d-cmr-2b-scratch-probe.md @@ -0,0 +1,23 @@ +# 2026-10-21 — D-CMR-2b phase 1: a scratch-free `Pred` fold does not pay on whole extents; on `set_lower`'s short runs, per-run `execute_extent` is 2.5–4× slower + +**Status:** MEASURED (lab probe, no production change). **Basis:** D-CMR-2's open item (`entries/2026-10-21-d-cmr-2-lowering.md`); operator authorization 2026-10-21; conditions of `gpt-ada-01/007`; claim EXO-50 `session_013z2e1YXkm6o6HNrDn5BPWA/023`. + +- **Dependencies:** ndarray sibling at `origin/master` `37a3b59` (workspace `[patch]`); `ndarray::simd` has every compare→mask primitive (`*_i32_to_mask[_under]`, `*_u32_to_mask[_under]`), `mask_ternlog_popcount`, `popcount_batch_u64`; no fused compare-and-count primitive. mask-risc lib 47/47 against it. Host AVX-512 (Xeon @ 2.10 GHz, 4 cores), `target-cpu=native`; the probe prints the tier. +- **What scratch is today:** caller-owned, `slots × TILE_WORDS` (256 words = 2 KiB per slot), allocated once per `Scratch::for_program`, never population-sized. So the question was per-op tile write-back / re-read and the separate `And` + count passes, not heap memory. +- **Probe:** `crates/lance-graph-benches/examples/pred_scratch_probe.rs`, run with `cargo run --release -p lance-graph-benches --example pred_scratch_probe`. Shapes `S1` Count(a), `S2` Count(a∧b), `S2u` b gated `under` a, `S3` Count(a∧b∧plane) over `i32` lanes; ~1 / 50 / 99 % selectivity. Two workloads: **whole** (one extent `0..n`, n = 16k / 1M / 4M) and **runs** (`set_lower`'s shape: over n = 1M, K non-overlapping runs of 200, 3,000 or 40,000 rows at odd starts, about a quarter of the rows). Arms: **A** `execute_extent` per extent with reused scratch; **A+alloc** with `Scratch::for_program` once per call (what `SetPlan::count` does); **Ac** A with the program compiled once (`Program::compile` + `execute_compiled`); **B** compares into fresh chunks + one `mask_ternlog_popcount`, plane shifted into a window when a run starts mid-word; **B′** the same over reused chunks. Every configuration asserted against a bit-serial oracle before timing. Timing interleaved per rep with rotating order, median of 101 per-rep ratios, plus a null arm (A timed twice). +- **Result (3 runs; ranges over the 3 selectivities × 3 runs):** + + | workload | null A′/A | A+alloc/A | Ac/A | B′/A | + |---|---|---|---|---| + | whole 16k (1 tile) | 0.989–0.999 | 1.026–1.090 | 0.979–1.005 | S1 1.40–1.46, S2/S2u/S3 1.06–1.26 | + | whole 1M | 0.957–1.012 | 0.971–1.027 | 0.971–1.033 | S1 1.04–1.20, S2/S2u/S3 0.95–1.04 | + | whole 4M | 0.984–1.014 | 0.989–1.023 | 0.987–1.011 | 0.970–1.028 | + | 1,310 runs × 200 | 0.952–1.033 | 0.981–1.023 | 0.821–0.948 | **0.236–0.524** (S3 0.24–0.27, S2 0.30–0.33) | + | 87 runs × 3,000 | 0.971–0.997 | 0.989–1.023 | 0.905–1.027 | S2/S3 0.67–0.80, S2u 0.86–0.93, S1 0.74–1.06 | + | 6 runs × 40,000 | 0.971–0.999 | 0.985–1.029 | 0.993–1.029 | 1.00–1.45 | + +- **Reading, whole extents and long runs:** no win for the fold; it is equal or slower. The earlier version of this probe (same fold, whole extents only, before the move to `lance-graph-benches`) measured B′/A 0.95–1.05 at 16k; this version measures 1.06–1.46 with the same A. The cause is not found. 64-byte alignment of the chunks was tried and does not change it. Either way the fold does not beat A there. +- **Reading, short runs:** at 200 rows per run, per-run `execute_extent` costs about 170–520 ns a run (S1 → S3) and the fold is 2.5–4× faster. Compiling once (`Ac`) cuts A by only 5–18 %, so re-deriving the lowering per call is a small part of the gap. From the code (not attributed by measurement): `extent_tiles` executes an unaligned extent as a head-edge word, a body and a tail-edge word, each running every op and the terminal, and `execute_compiled` validates on every call. Per-call allocation (`A+alloc`) costs nothing measurable. +- **Decision input:** a `Pred` `Lowering` arm for whole extents (phase 2 as first framed) is not supported. The measured gap is per-run cost on short runs, which is what `set_lower` produces at optics/etch. Not decided which remedy (fewer passes per edge tile in `execute_extent`, a run-batched entry point, or a short-run fold); not built. + +OPEN: (1) where the per-run cost of `execute_extent` goes on a 200-row run — not attributed between edge-tile passes, validation and op dispatch; (2) why this probe's fold is slower at full tiles than the earlier version. Not measured: `u32` / strided lanes, more than two comparisons, `Keep` terminals, other tiers (AVX2, NEON), real `set_lower` run distributions. diff --git a/.claude/board/entries/README.md b/.claude/board/entries/README.md index 4f8798545..b5018bf36 100644 --- a/.claude/board/entries/README.md +++ b/.claude/board/entries/README.md @@ -25,7 +25,7 @@ index row, (3) no duplicate entry id. Checks 1 and 2 are deliberately opposite directions; the stranding this convention prevents shows up in exactly one of them, never both. -406 entries, 2026-08-06 .. 2026-10-21. +407 entries, 2026-08-06 .. 2026-10-21. | date | entry id | finding | file | |---|---|---|---| @@ -37,6 +37,7 @@ exactly one of them, never both. | 2026-10-21 | `D-RA-2` | | [2026-10-21-d-ra-2-relalg-symbolic-repoint.md](2026-10-21-d-ra-2-relalg-symbolic-repoint.md) | | 2026-10-21 | `D-CMR-6` | | [2026-10-21-d-cmr-6-rule-catalogue.md](2026-10-21-d-cmr-6-rule-catalogue.md) | | 2026-10-21 | `D-CMR-5` | | [2026-10-21-d-cmr-5-rescope.md](2026-10-21-d-cmr-5-rescope.md) | +| 2026-10-21 | `D-CMR-2b` | | [2026-10-21-d-cmr-2b-scratch-probe.md](2026-10-21-d-cmr-2b-scratch-probe.md) | | 2026-10-21 | `D-CMR-2` | | [2026-10-21-d-cmr-2-lowering.md](2026-10-21-d-cmr-2-lowering.md) | | 2026-10-20 | `D-RU-1` | | [2026-10-20-u1-jc-batch.md](2026-10-20-u1-jc-batch.md) | | 2026-10-20 | `thought-git-replay-api-write-rubicon` | | [2026-10-20-thought-git-replay-api-write-rubicon.md](2026-10-20-thought-git-replay-api-write-rubicon.md) | diff --git a/.claude/plans/d-cmr-concept-chain-normalization-v1.md b/.claude/plans/d-cmr-concept-chain-normalization-v1.md index d7dd8525d..da34c45ac 100644 --- a/.claude/plans/d-cmr-concept-chain-normalization-v1.md +++ b/.claude/plans/d-cmr-concept-chain-normalization-v1.md @@ -211,6 +211,7 @@ when the first floating-point or quantized column arrives, not before. |---|---|---| | D-CMR-1 | **Set IR core** in the contract: `Runs` with `∩ ∪ ∖` and complement, `Res`, measures; pure, no lowering | algebra laws (closure, distributivity, De Morgan, `∅`) checked against materialized bitsets as oracle on random and edge-case runs | | D-CMR-2 | **Lowering** to mask-risc: runs → extents, residual → predicates within each run, the three measurement levels. Level 2 for residual comparisons needs predicate fusion in mask-risc (`fused_ternlog` declines `Pred` today) | == row oracle; a `Count` with no residual reads zero rows (instrumented); disabling level 1 or 2 keeps answers | +| D-CMR-2b | **Scratch-free `Pred` fold** (lab first, phase 2 only on a measured win) | interleaved paired timing against today's tiled `execute_extent` with a null arm; measured 2026-10-21: no win on whole extents; on `set_lower`'s 200-row runs per-run `execute_extent` is 2.5–4× slower, remedy open (`entries/2026-10-21-d-cmr-2b-scratch-probe.md`) | | D-CMR-3 | **Quack normalization pass:** over `Filter`, computing set values (ranges on a proven basis → `Runs`, other leaves → `Res`, `sql_where` first). Extent hoisting falls out of this (Experiment C) | answers identical to today's `lower` path on the quack suite; a range on another basis stays residual (negative control) | | D-CMR-4 | **Two dimensions:** `Rect` over composite coordinates, linearized row-major or in Morton order | rectangle `∩` == bitset oracle; run count within the stated bound | | D-CMR-5 | **BLOCKED: lawful design proposed (operator 2026-10-21).** Preimage `f⁻¹(S)` as a co-addressed transposed tile fold over a version-witnessed projection of `f`, built in preparation by a sequential same-table scan with frozen-codebook endpoint translation (ADMISSION B; reusable across queries). `S` enters only as a membership plane translated into the same frozen coordinates. No seek or load of `f` keyed by `S`'s values or a previous fold (Q1 = C), no `Gather`, no scatter terminal. R1 (`rmap` `rebind` / `discover` escape-driven seeks) stays option C. ⊘ Was: "VIOLATION — GATHER. DO NOT BUILD" (entry `2026-10-21-d-cmr-5-rescope.md`) | == row oracle; program checked Gather-free with a silence twin; attest / projection / query costs reported apart; a many-to-one `f` gives a valid distinct-source preimage (tested) and a bag-sensitive Count/Sum over it is refused (tested); refusal on a stale or missing witness | diff --git a/crates/lance-graph-benches/Cargo.toml b/crates/lance-graph-benches/Cargo.toml index 35c1ecf2f..79eead2ac 100644 --- a/crates/lance-graph-benches/Cargo.toml +++ b/crates/lance-graph-benches/Cargo.toml @@ -24,6 +24,10 @@ datafusion = { workspace = true } serde_json = "1" lance-graph-quack = { path = "../lance-graph-quack" } lance-graph-mask-risc = { path = "../lance-graph-mask-risc" } +# `examples/pred_scratch_probe.rs` (D-CMR-2b) calls the `ndarray::simd` +# compare and fold primitives directly for its scratch-free arm. Same package +# as mask-risc's git coordinate through the workspace `[patch]`. +ndarray = { path = "../../../ndarray", default-features = false, features = ["std"] } [[bench]] name = "graph_execution" diff --git a/crates/lance-graph-benches/examples/pred_scratch_probe.rs b/crates/lance-graph-benches/examples/pred_scratch_probe.rs new file mode 100644 index 000000000..1edac7e39 --- /dev/null +++ b/crates/lance-graph-benches/examples/pred_scratch_probe.rs @@ -0,0 +1,452 @@ +//! D-CMR-2b, phase 1 (lab only): what does a residual comparison's scratch +//! actually cost, and does a scratch-free fold win? +//! +//! `fused_ternlog` declines `Pred`, so a program with a comparison runs tiled: +//! each op writes its destination slot for one tile (`TILE_WORDS` words), +//! later ops re-read it, and the terminal counts the last slot. Scratch is +//! caller-owned and tile-sized (`slots × TILE_WORDS` words), never +//! population-sized, so the question is not heap memory. It is whether the +//! per-op tile write-back and re-read, and the separate `And` + count passes, +//! cost anything measurable against a fold that compares into chunks and +//! counts them in one fused `mask_ternlog_popcount` call. +//! +//! Arms, on the same data and the same programs: +//! +//! - **A** `execute_extent` with a `Scratch` built once outside the timing. +//! - **A+alloc** the same with `Scratch::for_program` once per call, inside the +//! timing (what `fold::set_lower`'s `SetPlan::count` does: one scratch per +//! call, then one `execute_extent` per run). +//! - **B** a lab fold: per chunk of at most `TILE_WORDS` words, each comparison +//! into a fresh stack chunk via the same `ndarray::simd` compare primitive +//! the executor uses, then ONE `mask_ternlog_popcount` over the chunks (and +//! a resident plane, shifted into a window when a run starts mid-word). +//! - **B′** the same fold over chunks reused across calls, which separates the +//! value of the fused count from the cost of initialising the chunks. +//! - **Ac** arm A with the program compiled once outside the timing +//! (`Program::compile` + `execute_compiled`), which separates the per-call +//! recognition cost of `execute_extent` from the tile work itself. +//! +//! Shapes (`a = lane0 > ta`, `b = lane1 < tb`, `p = plane 0`): +//! +//! - `S1` Count(a) +//! - `S2` Count(a ∧ b) +//! - `S2u` Count(a ∧ b) with `b` gated `under` `a` (the executor's own fused +//! compare-under form; arm B uses `*_to_mask_under` likewise) +//! - `S3` Count(a ∧ b ∧ p) +//! +//! Two workloads: +//! +//! - **whole**: one extent `0..n` (n = 16k, 1M, 4M). +//! - **runs**: `set_lower`'s shape. Over n = 1M, K non-overlapping runs of +//! length L (200, 3,000, 40,000 rows) at unaligned starts, together about a +//! quarter of the rows. Arm A calls `execute_extent` once per run; arm B folds +//! each run. +//! +//! Every configuration is checked against a bit-serial row oracle and arm A +//! before anything is timed. Timing is interleaved per rep in a rotating +//! order, the median of 101 per-rep ratios is reported, and a null arm (A timed +//! twice) gives the method's own noise. +//! +//! `cargo run --release -p lance-graph-benches --example pred_scratch_probe` + +use std::cell::RefCell; +use std::ops::Range; +use std::time::Instant; + +use lance_graph_mask_risc::exec::{execute_compiled, execute_extent, Scratch, TILE_WORDS}; +use lance_graph_mask_risc::{ + words_for, Foreign, LaneRef, MaskOp, Operand, Out, Planes, Pred, Program, Terminal, Value, +}; +use ndarray::simd::{gt_i32_to_mask, lt_i32_to_mask, lt_i32_to_mask_under, mask_ternlog_popcount}; + +/// `a & b & c` in the VPTERNLOG input convention. +const AND3: i32 = 0x80; + +fn lcg(seed: &mut u64) -> u64 { + *seed = seed + .wrapping_mul(6364136223846793005) + .wrapping_add(1442695040888963407); + *seed >> 11 +} + +fn time(f: &mut impl FnMut() -> T) -> f64 { + let t = Instant::now(); + std::hint::black_box(f()); + t.elapsed().as_nanos() as f64 +} + +fn med(mut v: Vec) -> f64 { + v.sort_by(|a, b| a.total_cmp(b)); + v[v.len() / 2] +} + +/// Interleaved, paired timing: every rep runs A, A again (the null arm), +/// A+alloc, B, B′ and Ac back to back, in an order that rotates per rep, so +/// drift and frequency changes hit all arms alike. Returns the median ns of A +/// and the medians of the per-rep ratios A′/A, A+alloc/A, B/A, B′/A and Ac/A. +fn paired( + reps: usize, + mut a: impl FnMut() -> usize, + mut aa: impl FnMut() -> usize, + mut b: impl FnMut() -> usize, + mut b2: impl FnMut() -> usize, + mut ac: impl FnMut() -> usize, +) -> (f64, [f64; 5]) { + let mut ta = vec![]; + let mut ratios: [Vec; 5] = Default::default(); + for r in 0..reps { + let mut t = [0f64; 6]; + for k in 0..6 { + let i = (k + r) % 6; + t[i] = match i { + 0 | 1 => time(&mut a), + 2 => time(&mut aa), + 3 => time(&mut b), + 4 => time(&mut b2), + _ => time(&mut ac), + }; + } + ta.push(t[0]); + for j in 0..5 { + ratios[j].push(t[j + 1] / t[0]); + } + } + let [n, al, b, b2, ac] = ratios; + (med(ta), [med(n), med(al), med(b), med(b2), med(ac)]) +} + +#[derive(Clone, Copy, PartialEq, Eq, Debug)] +enum Shape { + S1, + S2, + S2u, + S3, +} + +const SHAPES: [Shape; 4] = [Shape::S1, Shape::S2, Shape::S2u, Shape::S3]; + +fn program(shape: Shape, ta: i32, tb: i32) -> Program { + let a = Pred::GtI32 { lane: 0, t: ta }; + let b = Pred::LtI32 { lane: 1, t: tb }; + let pred = |pred, under, dst| MaskOp::Pred { pred, under, dst }; + let and = |a, b, dst| MaskOp::And { a, b, dst }; + let (ops, last) = match shape { + Shape::S1 => (vec![pred(a, None, 0)], 0), + Shape::S2 => ( + vec![ + pred(a, None, 0), + pred(b, None, 1), + and(Operand::Scratch(0), Operand::Scratch(1), 2), + ], + 2, + ), + Shape::S2u => ( + vec![pred(a, None, 0), pred(b, Some(Operand::Scratch(0)), 1)], + 1, + ), + Shape::S3 => ( + vec![ + pred(a, None, 0), + pred(b, None, 1), + and(Operand::Scratch(0), Operand::Scratch(1), 2), + and(Operand::Scratch(2), Operand::Plane(0), 3), + ], + 3, + ), + }; + Program::new( + ops, + Terminal::Count { + mask: Operand::Scratch(last), + }, + ) +} + +/// The bit-serial reference over the rows of `runs`. +fn oracle( + shape: Shape, + l0: &[i32], + l1: &[i32], + plane: &[u64], + ta: i32, + tb: i32, + runs: &[Range], +) -> usize { + runs.iter() + .flat_map(|r| r.clone()) + .filter(|&r| { + let a = l0[r] > ta; + let b = l1[r] < tb; + let p = plane[r / 64] >> (r % 64) & 1 == 1; + match shape { + Shape::S1 => a, + Shape::S2 | Shape::S2u => a && b, + Shape::S3 => a && b && p, + } + }) + .count() +} + +/// Chunks a fold works in: two comparison masks and a plane window. +struct Chunks { + ca: [u64; TILE_WORDS], + cb: [u64; TILE_WORDS], + cp: [u64; TILE_WORDS], +} + +impl Chunks { + fn new() -> Self { + Chunks { + ca: [0; TILE_WORDS], + cb: [0; TILE_WORDS], + cp: [0; TILE_WORDS], + } + } +} + +/// `plane`'s bits `r0..r0 + 64·out.len()`, shifted to start at bit 0. +fn window(plane: &[u64], r0: usize, out: &mut [u64]) { + let (w0, s) = (r0 / 64, r0 % 64); + for (i, o) in out.iter_mut().enumerate() { + let lo = plane.get(w0 + i).copied().unwrap_or(0) >> s; + let hi = if s == 0 { + 0 + } else { + plane.get(w0 + i + 1).copied().unwrap_or(0) << (64 - s) + }; + *o = lo | hi; + } +} + +/// Arm B / B′: the fold over the rows of `runs`, in chunks of at most +/// `TILE_WORDS` words. A comparison mask has zero bits past its chunk's rows, +/// so the plane window needs no tail masking under `AND3`. +#[allow(clippy::too_many_arguments)] +fn fold( + shape: Shape, + l0: &[i32], + l1: &[i32], + plane: &[u64], + ta: i32, + tb: i32, + runs: &[Range], + c: &mut Chunks, +) -> usize { + let mut total = 0u64; + for run in runs { + let mut r0 = run.start; + while r0 < run.end { + let r1 = (r0 + TILE_WORDS * 64).min(run.end); + let tw = words_for(r1 - r0); + let (ca, cb) = (&mut c.ca[..tw], &mut c.cb[..tw]); + gt_i32_to_mask(&l0[r0..r1], ta, ca); + total += match shape { + Shape::S1 => mask_ternlog_popcount::(ca, ca, ca), + Shape::S2 => { + lt_i32_to_mask(&l1[r0..r1], tb, cb); + mask_ternlog_popcount::(ca, cb, ca) + } + Shape::S2u => { + lt_i32_to_mask_under(&l1[r0..r1], tb, ca, cb); + mask_ternlog_popcount::(cb, cb, cb) + } + Shape::S3 => { + lt_i32_to_mask(&l1[r0..r1], tb, cb); + if r0.is_multiple_of(64) { + mask_ternlog_popcount::(ca, cb, &plane[r0 / 64..r0 / 64 + tw]) + } else { + let cp = &mut c.cp[..tw]; + window(plane, r0, cp); + mask_ternlog_popcount::(ca, cb, cp) + } + } + }; + r0 = r1; + } + } + total as usize +} + +/// Op passes over one chunk: (arm A, arm B). +fn passes(shape: Shape) -> (usize, usize) { + match shape { + Shape::S1 => (2, 2), + Shape::S2 => (4, 3), + Shape::S2u => (3, 3), + Shape::S3 => (5, 3), + } +} + +/// K non-overlapping runs of length `len` at unaligned starts over `0..n`, +/// together about a quarter of the rows. +fn runs_for(n: usize, len: usize, seed: &mut u64) -> Vec> { + let k = (n / 4 / len).max(1); + let slot = n / k; + (0..k) + .map(|i| { + let start = (i * slot + lcg(seed) as usize % (slot - len)) | 1; + start..(start + len).min(n) + }) + .collect() +} + +struct Data { + n: usize, + l0: Vec, + l1: Vec, + plane: Vec, +} + +fn data(n: usize, seed: &mut u64) -> Data { + let l0 = (0..n).map(|_| (lcg(seed) % 1000) as i32).collect(); + let l1 = (0..n).map(|_| (lcg(seed) % 1000) as i32).collect(); + let mut plane: Vec = (0..words_for(n)) + .map(|_| lcg(seed) ^ lcg(seed) << 21) + .collect(); + if !n.is_multiple_of(64) { + let last = plane.len() - 1; + plane[last] &= (1u64 << (n % 64)) - 1; + } + Data { n, l0, l1, plane } +} + +/// Check every arm against the oracle, then time them; print one row. +fn measure(label: &str, d: &Data, runs: &[Range], shape: Shape, ta: i32, tb: i32, sel: u32) { + let masks: [&[u64]; 1] = [&d.plane]; + let lanes = [LaneRef::I32(&d.l0), LaneRef::I32(&d.l1)]; + let planes = Planes { + n_rows: d.n, + masks: &masks, + lanes: &lanes, + }; + let p = program(shape, ta, tb); + let want = oracle(shape, &d.l0, &d.l1, &d.plane, ta, tb, runs); + let run_a = |s: &mut Scratch<'_>| -> usize { + runs.iter() + .map(|r| { + match execute_extent(&p, &planes, &Foreign::NONE, s, Out::None, r.clone()) + .expect("execute") + { + Value::Count(c) => c, + v => panic!("unexpected {v:?}"), + } + }) + .sum() + }; + let compiled = p.compile(); + let run_ac = |s: &mut Scratch<'_>| -> usize { + runs.iter() + .map(|r| { + match execute_compiled(&compiled, &planes, &Foreign::NONE, s, Out::None, r.clone()) + .expect("execute") + { + Value::Count(c) => c, + v => panic!("unexpected {v:?}"), + } + }) + .sum() + }; + let scratch = RefCell::new(Scratch::for_program(&p, d.n).expect("scratch")); + let chunks = RefCell::new(Chunks::new()); + let fold_fresh = || { + let mut c = Chunks::new(); + fold(shape, &d.l0, &d.l1, &d.plane, ta, tb, runs, &mut c) + }; + let fold_reused = || { + fold( + shape, + &d.l0, + &d.l1, + &d.plane, + ta, + tb, + runs, + &mut chunks.borrow_mut(), + ) + }; + assert_eq!( + run_a(&mut scratch.borrow_mut()), + want, + "A {label} {shape:?} sel={sel}" + ); + assert_eq!( + run_ac(&mut scratch.borrow_mut()), + want, + "Ac {label} {shape:?} sel={sel}" + ); + assert_eq!(fold_fresh(), want, "B {label} {shape:?} sel={sel}"); + assert_eq!(fold_reused(), want, "B' {label} {shape:?} sel={sel}"); + + let (a_ns, [null, alloc, b, b2, ac]) = paired( + 101, + || run_a(&mut scratch.borrow_mut()), + || { + let mut s = Scratch::for_program(&p, d.n).expect("scratch"); + run_a(&mut s) + }, + fold_fresh, + fold_reused, + || run_ac(&mut scratch.borrow_mut()), + ); + let scratch_b = p.scratch_slots as usize * TILE_WORDS.min(words_for(d.n)) * 8; + let (pa, pb) = passes(shape); + println!( + "{label:>14} {:>4} {sel:>4} | {scratch_b:>9} | {pa:>2}/{pb:<2} | {a_ns:>11.0} | {null:>6.3} {alloc:>8.3} {b:>6.3} {b2:>6.3} {ac:>6.3}", + format!("{shape:?}") + ); +} + +fn main() { + let tier = if cfg!(target_feature = "avx512f") { + "avx512f" + } else if cfg!(target_feature = "avx2") { + "avx2" + } else { + "scalar/other" + }; + println!("D-CMR-2b phase 1 probe; build tier: {tier}; TILE_WORDS = {TILE_WORDS}"); + println!( + "{:>14} {:>4} {:>4} | {:>9} | {:>5} | {:>11} | {:>6} {:>8} {:>6} {:>6} {:>6}", + "workload", + "shp", + "sel%", + "scratch_B", + "pass", + "A_ns", + "null", + "alloc/A", + "B/A", + "B'/A", + "Ac/A" + ); + // Thresholds giving roughly 1 %, 50 % and 99 % per comparison. + let sels = [(989, 10, 1), (499, 500, 50), (9, 990, 99)]; + let mut seed = 0x9E37_79B9_7F4A_7C15u64; + for &n in &[16_384usize, 1 << 20, 4 << 20] { + let d = data(n, &mut seed); + let whole = 0..n; + for &(ta, tb, sel) in &sels { + for shape in SHAPES { + measure( + &format!("whole {n}"), + &d, + std::slice::from_ref(&whole), + shape, + ta, + tb, + sel, + ); + } + } + } + let d = data(1 << 20, &mut seed); + for &len in &[200usize, 3_000, 40_000] { + let runs = runs_for(d.n, len, &mut seed); + let label = format!("{}x{len}", runs.len()); + for &(ta, tb, sel) in &sels { + for shape in SHAPES { + measure(&label, &d, &runs, shape, ta, tb, sel); + } + } + } +}