D-RU-1 (U1): JC statistics batch over shared power-sum passes (lab probe)

.claude/board/STATUS_BOARD.md

@@ -5,7 +5,7 @@ Plan: `.claude/plans/d-relalg-unified-v1.md`. Decisions recorded there: ADMISSIO
 | D-id | scope | status | gate / falsifier |
 |---|---|---|---|
 | **D-RU-0** | decisions: ADMISSION = B, D-RA-2 = A | DECIDED (operator, 2026-10-20) | — |
-| **D-RU-1** | U1: JC statistical batch, distinct (selection, key, lane pair) triples over shared power-sum terminals | Queued (authorized; first) | rows traversed and terminal passes below the per-request baseline; results bit-exact on integer lanes; overflow refused, not wrapped; R references |
+| **D-RU-1** | U1: JC statistical batch, distinct (selection, key, lane pair) triples over shared power-sum terminals | MEASURED (`crates/jc-batch-probe`; `entries/2026-10-20-u1-jc-batch.md`): 252 requests → 2 selection + 42 cross passes; at 1M rows 6.4× faster, 11.5× fewer rows traversed, 8.5× fewer values read, 126× fewer mask bytes; bit-identical to per-request; no R reference (R absent) | rows traversed and terminal passes below the per-request baseline; results bit-exact on integer lanes; overflow refused, not wrapped; R references |
 | **D-RU-2** | U2: `let`-bound planes on the contract IR, immutable `Plane(i)`, versioned coordinates, no DAG inside a fold | Queued (after D-CMR-1) | rows traversed and planes materialized against recompute; width test ≤ 3 slots; answers and versions unchanged |
 | **D-RU-3** | cross-rule CSE in crepe-plan-probe (256-rule benchmark, separate from the hop benchmark) | Queued (after D-RU-2) | answers == crepe and ascent; traversals and materializations measured |
 | **D-RU-4** | co-addressed join operator over `JoinDomain` | Queued (after D-RU-2) | parity with the star path; no position taken from another table's values |

.claude/board/entries/2026-10-20-u1-jc-batch.md

@@ -0,0 +1,54 @@
+# 2026-10-20 — D-RU-1 (U1): a JC statistics batch over shared power-sum passes
+
+**Status:** MEASURED (lab probe `crates/jc-batch-probe`). Plan: `.claude/plans/d-relalg-unified-v1.md` §3 and the stage table (U1).
+
+## What landed
+
+- `answer_separately`: each request evaluates its relalg selection into a keep mask, then runs one power or cross pass over it. This is the baseline.
+- `answer_batched`: requests are reduced to the distinct selections after `relalg::normalize`. For each selection, each distinct unordered lane pair gets one `GroupCrossPowerSumsI32` pass keyed by the group lane. A lane that no pair covers gets one `GroupPowerSumsI32` pass.
+  - An ungrouped answer merges the groups with `checked_merge`.
+  - A univariate answer reads the `x` or `y` side of the cross sums.
+  - A reversed pair swaps the sides.
+- Statistics: mean, variance, one-way ANOVA, covariance, Pearson, R², and the simple regression line in both directions.
+- Both paths refuse a request built for another version (`BatchError::VersionMismatch`).
+- `Cost` is counted from the programs that actually run: programs, selection passes, sum passes, rows traversed, values read, mask bytes.
+- `examples/bench.rs` (`BENCH_N`, `BENCH_REPS`).
+
+## Measured
+
+Battery: 252 requests over 2 selections and 7 `I32` lanes. It reduces to 2 selection passes and 42 cross passes; no univariate pass is needed. The table gives medians of 5 warm runs on 4 cores, load below 1.
+
+| rows | separate ms | batched ms | time | rows traversed | values read | mask bytes |
+|---|---|---|---|---|---|---|
+| 100,000 | 40.2 | 5.8 | 6.9× | 11.5× | 8.5× | 126× |
+| 1,000,000 | 424.5 | 66.2 | 6.4× | 11.5× | 8.5× | 126× |
+| 4,000,000 | 1,818.9 | 279.1 | 6.5× | 11.5× | 8.5× | 126× |
+
+At 1M rows the counts are:
+- programs: 504 → 44;
+- rows traversed: 504M → 44M;
+- values read: 1,092M → 129M;
+- mask bytes: 31.5 MB → 250 kB.
+
+The time ratio is below the values-read ratio because a cross pass folds five sums per row.
+
+**Correctness.** The batched answers equal the separate answers bit for bit. They also agree with an independent f64 oracle over directly selected rows: JC's `pearson` and `anova_one_way`, and two-pass mean and variance.
+
+**Integer range.**
+- At `i32::MIN`/`i32::MAX` values, `Σx²` passes `u64` and `Σxy` passes `i64`; the test asserts both. Mean and variance still equal the exact `i128` values.
+- At an offset of 10⁹, the naive one-pass f64 Pearson is off by more than 10⁻³. The exact path is within 10⁻¹² of the two-pass value.
+
+**Tests and disables.** 8 tests pass. Six disable runs each turned red, with every disable patch's anchor asserted:
+- swap the reversed pair;
+- read the `y` side as `x`;
+- drop the pair dedup;
+- skip `normalize`;
+- merge only the first group;
+- drop the version check.
+
+## OPEN
+
+- No R reference: R is not installed in this container. The reference is JC's slice functions plus exact integer sums.
+- A merge overflow cannot happen below the 2³²-row cap the terminals enforce, so `MergeOverflow` is untested; the cap itself is mask-risc's test.
+- The timing of one battery shape. Batches with few requests per pair, so less sharing, are not measured, and neither is the break-even request count.
+- Selections are shared only when they are equal after normalization; overlapping selections (U2's let-planes) are not shared yet.

.claude/board/entries/README.md

@@ -25,10 +25,11 @@ index row, (3) no duplicate entry id. Checks 1 and 2 are deliberately
 opposite directions; the stranding this convention prevents shows up in
 exactly one of them, never both.
 
-395 entries, 2026-08-06 .. 2026-10-20.
+396 entries, 2026-08-06 .. 2026-10-20.
 
 | date | entry id | finding | file |
 |---|---|---|---|
+| 2026-10-20 | `D-RU-1` |  | [2026-10-20-u1-jc-batch.md](2026-10-20-u1-jc-batch.md) |
 | 2026-10-20 | `thought-git-replay-api-write-rubicon` |  | [2026-10-20-thought-git-replay-api-write-rubicon.md](2026-10-20-thought-git-replay-api-write-rubicon.md) |
 | 2026-10-20 | `ADR-2026-10-20-SIX` |  | [2026-10-20-six-decisions.md](2026-10-20-six-decisions.md) |
 | 2026-10-20 | `relstore-quack-petgraph-plan` |  | [2026-10-20-relstore-quack-petgraph-plan.md](2026-10-20-relstore-quack-petgraph-plan.md) |

.claude/plans/d-relalg-unified-v1.md

@@ -296,4 +296,4 @@ One workload, run under three toggles. Each toggle changes one mechanism only.
 | U7 | prefetch and staging of Arm B ranges | U6 or a non-recursive multi-stage plan | critical-path overlap measured; bytes read unchanged |
 | U8 | Wikidata and SAP-shaped fixtures; completeness refusals | U6, U7 | refusal tests; witness carried end to end |
 
-U1 needs nothing new and comes first. U2 builds on D-CMR-1. Both are accepted only on measured row traversals, materializations and total cost (§ Decisions), not on operator counts.
+U1 needs nothing new and comes first. **U1 MEASURED (2026-10-20, `crates/jc-batch-probe`, `entries/2026-10-20-u1-jc-batch.md`):** 252 requests reduce to 2 selection and 42 cross passes; at 1M rows the batch is 6.4× faster with 11.5× fewer rows traversed, 8.5× fewer values read and 126× fewer mask bytes, and its answers equal the per-request answers bit for bit. The R reference is replaced by JC's slice functions because R is not installed. U2 builds on D-CMR-1. Both are accepted only on measured row traversals, materializations and total cost (§ Decisions), not on operator counts.

.github/workflows/jc-batch-probe.yml

@@ -0,0 +1,54 @@
+name: JC batch probe
+# `crates/jc-batch-probe` (D-RU-1 / U1) is excluded from the workspace and has
+# its own [workspace], so no workspace job builds it. It path-deps jc, relalg
+# and mask-risc. jc and relalg name ndarray as a sibling checkout
+# (`../../../ndarray` from their directory), so the job clones it first.
+on:
+  pull_request:
+    paths:
+      - 'crates/jc-batch-probe/**'
+      - 'crates/jc/**'
+      - 'crates/lance-graph-relalg/**'
+      - 'crates/lance-graph-r2il/**'
+      - 'crates/lance-graph-mask-risc/**'
+      - 'crates/lance-graph-quack/**'
+      - 'crates/lance-graph-contract/**'
+      - '.github/workflows/jc-batch-probe.yml'
+      - 'rust-toolchain.toml'
+  push:
+    branches: [main]
+    paths:
+      - 'crates/jc-batch-probe/**'
+      - 'crates/jc/**'
+      - 'crates/lance-graph-relalg/**'
+      - 'crates/lance-graph-r2il/**'
+      - 'crates/lance-graph-mask-risc/**'
+      - 'crates/lance-graph-quack/**'
+      - 'crates/lance-graph-contract/**'
+      - '.github/workflows/jc-batch-probe.yml'
+      - 'rust-toolchain.toml'
+  workflow_dispatch:
+permissions:
+  contents: read
+jobs:
+  probe:
+    runs-on: ubuntu-latest
+    timeout-minutes: 45
+    defaults:
+      run:
+        working-directory: crates/jc-batch-probe
+    env:
+      CARGO_PROFILE_DEV_DEBUG: "0"
+      CARGO_INCREMENTAL: "0"
+    steps:
+      - uses: actions/checkout@v4
+      - name: Setup rust toolchain (pinned by rust-toolchain.toml)
+        run: rustup show
+      - name: Sibling ndarray checkout
+        run: git clone --depth 1 https://github.com/AdaWorldAPI/ndarray ../../../ndarray
+      - run: cargo fmt --check
+      - run: cargo clippy --release --all-targets -- -D warnings
+      - run: cargo test --release
+      # The benchmark asserts that both paths agree before it times them.
+      - name: Benchmark smoke run
+        run: BENCH_N=100000 BENCH_REPS=1 cargo run --release --example bench

Cargo.toml

@@ -174,6 +174,10 @@ exclude = [
     # -> MORK query_multi -> relalg over the D-MPS-8 table. Own workspace; run
     # from its directory so `.cargo/config.toml` applies.
     "crates/crepe-plan-probe",
+    # D-RU-1 (U1) JC statistical batch: one battery of means/variances/ANOVA/
+    # covariance/Pearson/regression answered from shared power-sum passes vs
+    # request by request. Lab probe, own workspace: `cd crates/jc-batch-probe && cargo test --release`.
+    "crates/jc-batch-probe",
     # D-TGB-0 thought-git pre-registration: fixtures, the from-scratch oracle,
     # and signal checks over lance-graph-contract only. Lab probe; run from
     # its directory: `cd crates/thought-git-probe && cargo test`.

crates/jc-batch-probe/Cargo.toml

@@ -0,0 +1,27 @@
+[package]
+name = "jc-batch-probe"
+version = "0.1.0"
+edition = "2021"
+publish = false
+description = "Lab probe (D-RU-1 / U1): a batch of JC statistical requests answered from shared power-sum passes, against one pass per request. Results only; no production path."
+
+# Its own workspace root (it path-deps relalg, which git-deps OGAR). Run with
+# `cargo test --manifest-path crates/jc-batch-probe/Cargo.toml`.
+[workspace]
+
+[dependencies]
+# The selection algebra and its lowering.
+lance-graph-relalg = { path = "../lance-graph-relalg" }
+# The only bulk engine: the power-sum terminals.
+lance-graph-mask-risc = { path = "../lance-graph-mask-risc" }
+# Statistical semantics: projections from the sums.
+jc = { path = "../jc" }
+# The coordinate mask-risc uses; patched below to the sibling jc builds on,
+# so `PowerSums` is one type across all three.
+ndarray = { git = "https://github.com/AdaWorldAPI/ndarray", branch = "master", default-features = false, features = ["std"] }
+
+[patch."https://github.com/AdaWorldAPI/lance-graph"]
+lance-graph-contract = { path = "../lance-graph-contract" }
+
+[patch."https://github.com/AdaWorldAPI/ndarray"]
+ndarray = { path = "../../../ndarray" }

crates/jc-batch-probe/examples/bench.rs

@@ -0,0 +1,161 @@
+//! U1 benchmark: one statistical battery answered request by request versus
+//! from shared power-sum passes. Run in release from this crate's directory:
+//!
+//! ```text
+//! cargo run --release --example bench
+//! ```
+//!
+//! `BENCH_N` rows (default 1,000,000), 7 `I32` statistic lanes, a selection
+//! lane, a category lane, an 8-value group lane and a zero lane. The battery
+//! is the one of `tests/batch.rs` over two selections: every mean, variance
+//! and ANOVA, and every covariance, Pearson, R² and regression line (both
+//! directions) of every pair: 252 requests. Both paths must return equal
+//! answers before either is timed. Times are medians of `BENCH_REPS`
+//! (default 5) warm runs.
+
+use std::hint::black_box;
+use std::time::{Duration, Instant};
+
+use jc_batch_probe::{answer_batched, answer_separately, Request, Stat, Table};
+use lance_graph_mask_risc::LaneRef;
+use lance_graph_relalg::{Cmp, Rel};
+
+const STAT_LANES: u16 = 7;
+const SEL: u16 = 7;
+const CAT: u16 = 8;
+const GROUP: u16 = 9;
+const ZERO: u16 = 10;
+const GROUPS: usize = 8;
+const VERSION: u64 = 1;
+
+fn env(k: &str, d: usize) -> usize {
+    std::env::var(k)
+        .ok()
+        .and_then(|v| v.parse().ok())
+        .unwrap_or(d)
+}
+
+fn lcg(s: &mut u64) -> u64 {
+    *s = s
+        .wrapping_mul(6364136223846793005)
+        .wrapping_add(1442695040888963407);
+    *s >> 33
+}
+
+fn battery(mut select: impl FnMut() -> Rel) -> Vec<Request> {
+    let mut r = Vec::new();
+    let mut push = |stat| {
+        r.push(Request {
+            version: VERSION,
+            select: select(),
+            stat,
+        })
+    };
+    for x in 0..STAT_LANES {
+        push(Stat::Mean(x));
+        push(Stat::Variance(x));
+        push(Stat::Anova(x));
+    }
+    for x in 0..STAT_LANES {
+        for y in (x + 1)..STAT_LANES {
+            push(Stat::Covariance(x, y));
+            push(Stat::Pearson(x, y));
+            push(Stat::RSquared(x, y));
+            push(Stat::Regression(x, y));
+            push(Stat::Regression(y, x));
+        }
+    }
+    r
+}
+
+fn median(mut v: Vec<Duration>) -> Duration {
+    v.sort();
+    v[v.len() / 2]
+}
+
+fn main() {
+    let n = env("BENCH_N", 1_000_000);
+    let reps = env("BENCH_REPS", 5);
+    let mut s = 0xBE7C_u64;
+    let base: Vec<i32> = (0..n)
+        .map(|_| (lcg(&mut s) % 2_001) as i32 - 1_000)
+        .collect();
+    let stats: Vec<Vec<i32>> = (0..STAT_LANES as i32)
+        .map(|j| {
+            base.iter()
+                .map(|&b| b * j + (lcg(&mut s) % 401) as i32 - 200)
+                .collect()
+        })
+        .collect();
+    let sel: Vec<i32> = (0..n).map(|_| (lcg(&mut s) % 100) as i32).collect();
+    let cat: Vec<u32> = (0..n).map(|_| (lcg(&mut s) % 4) as u32).collect();
+    let group: Vec<u32> = (0..n)
+        .map(|_| (lcg(&mut s) % GROUPS as u64) as u32)
+        .collect();
+    let zero = vec![0u32; n];
+    let mut lanes: Vec<LaneRef<'_>> = stats.iter().map(|v| LaneRef::I32(v)).collect();
+    lanes.extend([
+        LaneRef::I32(&sel),
+        LaneRef::U32(&cat),
+        LaneRef::U32(&group),
+        LaneRef::U32(&zero),
+    ]);
+    let t = Table {
+        version: VERSION,
+        n_rows: n,
+        lanes: &lanes,
+        group: GROUP,
+        groups: GROUPS,
+        zero: ZERO,
+    };
+
+    let low = || Rel::i32(SEL, Cmp::Lt, 60);
+    let mut reqs = battery(low);
+    reqs.extend(battery(|| Rel::And(vec![low(), Rel::eq_u32(CAT, 2)])));
+
+    let (a_sep, c_sep) = answer_separately(&t, &reqs).unwrap();
+    let (a_bat, c_bat, red) = answer_batched(&t, &reqs).unwrap();
+    assert_eq!(a_sep, a_bat, "the two paths must agree before timing");
+
+    let time = |f: &dyn Fn()| {
+        let v: Vec<Duration> = (0..reps)
+            .map(|_| {
+                let st = Instant::now();
+                f();
+                st.elapsed()
+            })
+            .collect();
+        median(v)
+    };
+    let t_sep = time(&|| {
+        black_box(answer_separately(&t, &reqs).unwrap());
+    });
+    let t_bat = time(&|| {
+        black_box(answer_batched(&t, &reqs).unwrap());
+    });
+
+    println!("rows {n}, requests {}, {red:?}", reqs.len());
+    println!(
+        "{:>10} {:>9} {:>9} {:>9} {:>14} {:>14} {:>12}",
+        "path", "ms", "programs", "sum pass", "rows trav.", "values read", "mask bytes"
+    );
+    for (name, ms, c) in [("separate", t_sep, c_sep), ("batched", t_bat, c_bat)] {
+        println!(
+            "{:>10} {:>9.1} {:>9} {:>9} {:>14} {:>14} {:>12}",
+            name,
+            ms.as_secs_f64() * 1e3,
+            c.programs,
+            c.sum_passes,
+            c.rows_traversed,
+            c.values_read,
+            c.mask_bytes
+        );
+    }
+    println!(
+        "ratio separate/batched: time {:.1}x, rows {:.1}x, values {:.1}x, masks {:.1}x",
+        t_sep.as_secs_f64() / t_bat.as_secs_f64(),
+        c_sep.rows_traversed as f64 / c_bat.rows_traversed as f64,
+        c_sep.values_read as f64 / c_bat.values_read as f64,
+        c_sep.mask_bytes as f64 / c_bat.mask_bytes as f64
+    );
+}