relalg/r2il: width costs no scratch slots

.claude/board/entries/2026-10-20-crepe-rule-optimizer-spike.md

@@ -19,5 +19,5 @@
 - Dynamic MeTTa rules need crepe's analysis as a library: crepe is a proc-macro crate (follow-up A, `crepe-core` split).
 - Semijoin reduction built (`Options::semijoin`): laid-out rows 36 at every size instead of growing with n; preparation 3.47 ms → 0.28 ms at 50,000. The scan stays O(n).
 - ndarray for the row filter, measured (`examples/row_filter.rs`): over the row-major store the strided compare is slower than the scalar test at every needle count, so it is not wired; over a contiguous column it is 5–6 µs per needle at 50,000 rows and beats the scalar test up to 64 needles; at 128 needles two runs disagree because of benchmark noise (the scalar time moved 1.8× between runs), so that crossover is not pinned (re-measured after the #1632 Codex fix). Next step: a column-major relation store.
-- Stage 3 over Lance 13 / lancedb 0.40 built (`lance_store`): each side read under a relalg address mask with `with_row_addr_prefilter`; every masked read scans exactly its mask (P1–P4 green). The relalg membership mask holds 4 centers at most: `k` centers take `2k - 1` scratch slots (no slot reuse inside a fold), and `MAX_SLOTS = 8` is an r2il setting, not a hard limit (mask-risc addresses 65,536). Above that a scalar test builds it.
+- Stage 3 over Lance 13 / lancedb 0.40 built (`lance_store`): each side read under a relalg address mask with `with_row_addr_prefilter`; every masked read scans exactly its mask (P1–P4 green). The relalg membership mask held 4 centers at most (`2k - 1` scratch slots against `MAX_SLOTS = 8`). `FoldDialect::call_owned` now releases consumed slots and relalg lowers `Or`/`And`/`Not` through it, so any width takes 3 slots and the scalar fallback is gone.
 - Arm B built (`addressed`): lanes in key order, attested with `SealedFacetLane::attest_sorted`, read only at the `seek_from` ranges of the demanded keys. At 50,000 rows it scans `3k` rows where Arm A scans at least 150,000. It is 2.5–12× faster than Arm A up to 1% of centers and pays back its 7 ms attest after 2–3 queries. From 10% of centers up a full scan wins, and with all centers Arm B never breaks even. Open: the selectivity crossover between 1% and 10%; multi-fragment lanes (an append currently forces a compaction); a second lane for demands on the second column.

.claude/plans/crepe-rule-optimizer-spike-v1.md

@@ -118,7 +118,7 @@ DECISION (operator, 2026-10-20): "ja, bau Stufe 3 mit Lance 13 und CI-Workflow"
 The relations are stored column-major in Lance: one table per relation, columns `c0`, `c1` (`UInt32`), written through lancedb 0.40 and read through lance 13. Preparation becomes H14d's stages 2 and 3 per side:
 
 1. Read the one column the address mask needs (the restriction column, or the key column for the semijoin), with `_rowaddr`.
-2. Build the address mask `R` through relalg over that column as a `U32` lane: `eq_u32(col, k)` for a restriction, `Or` of `eq_u32(key, c)` for the semijoin centers (lowered by mask-risc onto `ndarray::simd::eq_u32_to_mask`). Above 4 centers the mask is built by the scalar membership test instead. relalg lowers `Or(a1..ak)` left-associatively, and every atom and every `OR` takes a scratch slot of its own (no reuse inside a fold), so `k` centers take `2k - 1` slots. One fold may use `lance_graph_r2il::MAX_SLOTS = 8`, a setting the crate calls "sized with headroom", so 5 centers (9 slots) fail with `ScratchBufferTooSmall`. mask-risc addresses up to 65,536 slots. Either a larger arena or slot reuse inside the fold (an `Or` chain needs only the accumulator and the current atom) would lift the limit; both are r2il/relalg changes. This is not the §2c speed crossover, which is ndarray's row filter.
+2. Build the address mask `R` through relalg over that column as a `U32` lane: `eq_u32(col, k)` for a restriction, `Or` of `eq_u32(key, c)` for the semijoin centers (lowered by mask-risc onto `ndarray::simd::eq_u32_to_mask`). This works for any number of centers. Until 2026-10-20 it held only 4: relalg lowered `Or(a1..ak)` left-associatively and every atom and every `OR` took a scratch slot of its own, so `k` centers took `2k - 1` slots against `lance_graph_r2il::MAX_SLOTS = 8` (5 centers failed with `ScratchBufferTooSmall`), and above 4 a scalar membership test built the mask. Now `FoldDialect::call_owned` releases the slots an `OR`/`AND`/`NOT` consumed, and relalg, which consumes each subexpression's value exactly once, lowers through it: an `Or` of any width needs 3 slots, and the scalar fallback is removed. This is not the §2c speed crossover, which is ndarray's row filter.
 3. Read the side's columns with `with_row_addr_prefilter(R)`.
 4. Hand those rows to `JoinTable::prepare_star`, as the in-memory path does.
 

.github/workflows/crepe-plan-probe.yml

@@ -3,13 +3,14 @@ name: Crepe plan probe
 # git deps on the crepe and MORK forks, lance 13 / lancedb 0.40), so no
 # workspace job builds it. It runs from its own directory so the cargo config
 # above it and its own `.cargo/config.toml` apply. It path-deps relalg,
-# mask-risc, mork-pathmap-probe and lance-graph-contract (the ordered lane and
+# mask-risc, r2il, mork-pathmap-probe and lance-graph-contract (the ordered lane and
 # facet keys of arm B), so changes there run it too.
 on:
   pull_request:
     paths:
       - 'crates/crepe-plan-probe/**'
       - 'crates/lance-graph-relalg/**'
+      - 'crates/lance-graph-r2il/**'
       - 'crates/lance-graph-contract/**'
       - 'crates/lance-graph-mask-risc/**'
       - 'crates/mork-pathmap-probe/**'
@@ -20,6 +21,7 @@ on:
     paths:
       - 'crates/crepe-plan-probe/**'
       - 'crates/lance-graph-relalg/**'
+      - 'crates/lance-graph-r2il/**'
       - 'crates/lance-graph-contract/**'
       - 'crates/lance-graph-mask-risc/**'
       - 'crates/mork-pathmap-probe/**'
@@ -49,3 +51,12 @@ jobs:
       - run: cargo fmt --check
       - run: cargo clippy --all-targets -- -D warnings
       - run: cargo test
+      # relalg is its own workspace and patches ndarray to a sibling checkout
+      # (`../../../ndarray` from its directory); no other job runs its tests.
+      - name: relalg tests and clippy
+        working-directory: crates/lance-graph-relalg
+        run: |
+          git clone --depth 1 https://github.com/AdaWorldAPI/ndarray ../../../ndarray
+          cargo fmt --check
+          cargo test
+          cargo clippy --all-targets -- -D warnings

crates/crepe-plan-probe/examples/walk_mask.rs

@@ -0,0 +1,232 @@
+//! crepe's `graph_walking` benchmark program, evaluated twice: once by the
+//! code `crepe!` generates (hash sets and hash-map indexes, tuple at a time)
+//! and once over dense bit planes with `ndarray::simd` mask ops. Both runs
+//! must give the same tuple counts. Run in release from this crate's
+//! directory:
+//!
+//! ```text
+//! cargo run --release --example walk_mask
+//! ```
+//!
+//! The program (copied from `crepe/benches/graph_walking.rs`):
+//!
+//! ```text
+//! Node(x) <- Edge(x, _, _);   Node(x) <- Edge(_, x, _);
+//! Walk(x, x, 0) <- Node(x);
+//! Walk(x, z, l1 + l2) <- Edge(x, y, l1), Walk(y, z, l2), (l1 + l2 <= 50);
+//! NoWalk(x, y) <- Node(x), Node(y), !Walk(x, y, _);
+//! ```
+//!
+//! The mask form needs what the macro does not know: node ids dense in
+//! `0..n`, and path lengths in `0..=MAX_PATH_LEN`. `Walk` is one `n x n` bit
+//! plane per length; an edge of length `w` from `x` to `y` makes
+//! `walk[l + w][x] |= delta[l][y]`, the semi-naive step is
+//! `new &= !walk; walk |= new`, and the fixpoint is "every new plane is
+//! zero". `NoWalk` is `node x node & !(OR over l of walk[l])`.
+
+use std::hint::black_box;
+use std::time::{Duration, Instant};
+
+use crepe::crepe;
+use ndarray::simd::{mask_andnot_assign, mask_any, mask_or_assign, popcount_batch_u64};
+
+const MAX_PATH_LEN: u32 = 50;
+const REPS: usize = 9;
+
+crepe! {
+    @input
+    struct Edge(i32, i32, u32);
+
+    @output
+    struct Walk(i32, i32, u32);
+
+    @output
+    struct NoWalk(i32, i32);
+
+    struct Node(i32);
+
+    Node(x) <- Edge(x, _, _);
+    Node(x) <- Edge(_, x, _);
+
+    Walk(x, x, 0) <- Node(x);
+    Walk(x, z, len1 + len2) <-
+        Edge(x, y, len1),
+        Walk(y, z, len2),
+        (len1 + len2 <= MAX_PATH_LEN);
+
+    NoWalk(x, y) <- Node(x), Node(y), !Walk(x, y, _);
+}
+
+fn edges(n: i32) -> Vec<(i32, i32, u32)> {
+    let mut e = Vec::new();
+    for i in 0..n {
+        for j in 0..n {
+            if (i + j) % 50 < 2 {
+                e.push((i, j, 5));
+            }
+        }
+    }
+    e
+}
+
+fn crepe_walk(e: &[(i32, i32, u32)]) -> (usize, usize) {
+    let mut rt = Crepe::new();
+    rt.extend(e.iter().map(|&(a, b, w)| Edge(a, b, w)));
+    let (walk, nowalk) = rt.run();
+    (walk.len(), nowalk.len())
+}
+
+/// Bits per row of an `n`-wide plane.
+fn words(n: usize) -> usize {
+    n.div_ceil(64)
+}
+
+fn set(row: &mut [u64], i: usize) {
+    row[i / 64] |= 1 << (i % 64);
+}
+
+/// The same program over bit planes. Node ids must be dense in `0..n`.
+fn mask_walk(n: usize, e: &[(i32, i32, u32)]) -> (usize, usize) {
+    let w = words(n);
+    let lens = MAX_PATH_LEN as usize + 1;
+    let row = |plane: usize, x: usize| (plane * n + x) * w;
+
+    // Edges grouped by length: adjacency rows `x -> {y}`.
+    let mut by_len: Vec<Option<Vec<u64>>> = vec![None; lens];
+    let mut node = vec![0u64; w];
+    for &(a, b, l) in e {
+        let (a, b, l) = (a as usize, b as usize, l as usize);
+        set(&mut node, a);
+        set(&mut node, b);
+        if l < lens {
+            let adj = by_len[l].get_or_insert_with(|| vec![0u64; n * w]);
+            set(&mut adj[a * w..a * w + w], b);
+        }
+    }
+
+    // walk[l][x] and the delta planes; Walk(x, x, 0) for every node.
+    let mut walk = vec![0u64; lens * n * w];
+    let mut delta = vec![0u64; lens * n * w];
+    let mut live = vec![false; lens];
+    for x in 0..n {
+        if node[x / 64] >> (x % 64) & 1 == 1 {
+            set(&mut walk[row(0, x)..row(0, x) + w], x);
+            set(&mut delta[row(0, x)..row(0, x) + w], x);
+        }
+    }
+    live[0] = true;
+
+    let mut new = vec![0u64; lens * n * w];
+    loop {
+        new.fill(0);
+        let mut new_live = vec![false; lens];
+        for (l1, adj) in by_len.iter().enumerate() {
+            let Some(adj) = adj else { continue };
+            for l2 in (0..lens).filter(|&l2| live[l2]) {
+                let l = l1 + l2;
+                if l >= lens {
+                    continue;
+                }
+                for x in 0..n {
+                    let ar = &adj[x * w..x * w + w];
+                    let dst = row(l, x);
+                    for (wi, &word) in ar.iter().enumerate() {
+                        let mut bits = word;
+                        while bits != 0 {
+                            let y = wi * 64 + bits.trailing_zeros() as usize;
+                            bits &= bits - 1;
+                            let src = row(l2, y);
+                            // `new` and `delta` are different buffers.
+                            mask_or_assign(&mut new[dst..dst + w], &delta[src..src + w]);
+                        }
+                    }
+                }
+                new_live[l] = true;
+            }
+        }
+        let mut any = false;
+        for (l, is_live) in new_live.iter_mut().enumerate() {
+            if !*is_live {
+                continue;
+            }
+            let span = row(l, 0)..row(l, 0) + n * w;
+            mask_andnot_assign(&mut new[span.clone()], &walk[span.clone()]);
+            if mask_any(&new[span.clone()]) {
+                mask_or_assign(&mut walk[span.clone()], &new[span.clone()]);
+                any = true;
+            } else {
+                *is_live = false;
+            }
+        }
+        if !any {
+            break;
+        }
+        std::mem::swap(&mut delta, &mut new);
+        live = new_live;
+    }
+
+    let walk_count = popcount_batch_u64(&walk) as usize;
+    // NoWalk: for each node x, nodes not reached at any length.
+    let mut reach = vec![0u64; w];
+    let mut nowalk = 0usize;
+    for x in 0..n {
+        if node[x / 64] >> (x % 64) & 1 == 0 {
+            continue;
+        }
+        reach.fill(0);
+        for l in 0..lens {
+            mask_or_assign(&mut reach, &walk[row(l, x)..row(l, x) + w]);
+        }
+        let mut miss = node.clone();
+        mask_andnot_assign(&mut miss, &reach);
+        nowalk += popcount_batch_u64(&miss) as usize;
+    }
+    (walk_count, nowalk)
+}
+
+fn median(mut v: Vec<Duration>) -> Duration {
+    v.sort();
+    v[v.len() / 2]
+}
+
+fn time(mut f: impl FnMut() -> (usize, usize)) -> (Duration, (usize, usize)) {
+    let mut t = Vec::with_capacity(REPS);
+    let mut out = (0, 0);
+    for _ in 0..REPS {
+        let s = Instant::now();
+        out = black_box(f());
+        t.push(s.elapsed());
+    }
+    (median(t), out)
+}
+
+fn main() {
+    println!(
+        "{:>5} | {:>10} {:>10} | {:>9} | {:>9} {:>9}",
+        "n", "crepe ms", "mask ms", "speed-up", "Walk", "NoWalk"
+    );
+    for n in [128usize, 256, 512, 1024] {
+        let e = edges(n as i32);
+        let (tm, mask) = time(|| mask_walk(n, &e));
+        let (tc, crep) = if n <= 512 {
+            time(|| crepe_walk(&e))
+        } else {
+            (Duration::ZERO, mask)
+        };
+        assert_eq!(mask, crep, "n={n}: mask and crepe disagree");
+        let speed = if tc.is_zero() {
+            "-".to_string()
+        } else {
+            format!("{:.1}x", tc.as_secs_f64() / tm.as_secs_f64())
+        };
+        println!(
+            "{:>5} | {:>10.3} {:>10.3} | {:>9} | {:>9} {:>9}",
+            n,
+            tc.as_secs_f64() * 1e3,
+            tm.as_secs_f64() * 1e3,
+            speed,
+            mask.0,
+            mask.1
+        );
+    }
+}

crates/crepe-plan-probe/src/lance_store.rs

@@ -27,21 +27,6 @@ use mork_pathmap_probe::join::for_each_set;
 use crate::logic::RelId;
 use crate::native::{PreparedStar, StarPlan, prepare_from_rows};
 
-/// Up to this many semijoin centers, the address mask is an `Or` of `eq_u32`
-/// atoms through relalg; above it, the scalar membership test builds it.
-///
-/// The limit comes from a setting, not from the algebra and not from a speed
-/// crossover. relalg lowers `Or(a1..ak)` left-associatively, and each atom
-/// and each `OR` takes a scratch slot of its own (slots are not reused
-/// inside a fold), so `k` centers take `2k - 1` slots. One fold may use
-/// `lance_graph_r2il::MAX_SLOTS` (8, "sized with headroom") of them. Measured:
-/// 4 centers fit (7 slots); 5 need 9 and fail with
-/// `ScratchBufferTooSmall { need_words: 145, have_words: 129 }` at 1,000 rows.
-/// mask-risc itself addresses up to 65,536 slots. A larger arena, or slot
-/// reuse in the fold, lifts this. (The 64-needle crossover in plan §2c is
-/// ndarray's row filter, a different mechanism.)
-pub const SEMIJOIN_ATOMS_MAX: usize = 4;
-
 #[derive(Debug)]
 pub enum LanceStoreError {
     Lance(String),
@@ -260,22 +245,16 @@ pub fn address_mask(values: &[u32], rel: &Rel) -> Result<Vec<u64>, RelError> {
     Ok(keep)
 }
 
-/// Rows whose value is one of `centers` (sorted, deduplicated).
+/// Rows whose value is one of `centers` (sorted, deduplicated): an `Or` of
+/// `eq_u32` atoms through relalg, for any number of centers. relalg releases
+/// each consumed slot (`FoldDialect::call_owned`), so the fold needs three
+/// scratch slots however wide the `Or` is.
 pub fn membership_mask(values: &[u32], centers: &[u32]) -> Result<Vec<u64>, RelError> {
     if centers.is_empty() {
         return Ok(vec![0; words_for(values.len())]);
     }
-    if centers.len() <= SEMIJOIN_ATOMS_MAX {
-        let rel = Rel::Or(centers.iter().map(|&c| Rel::eq_u32(0, c)).collect());
-        return address_mask(values, &rel);
-    }
-    let mut keep = vec![0u64; words_for(values.len())];
-    for (i, v) in values.iter().enumerate() {
-        if centers.binary_search(v).is_ok() {
-            keep[i / 64] |= 1 << (i % 64);
-        }
-    }
-    Ok(keep)
+    let rel = Rel::Or(centers.iter().map(|&c| Rel::eq_u32(0, c)).collect());
+    address_mask(values, &rel)
 }
 
 fn row_addr_mask(mask: &[u64], n: usize, addr: &[u64]) -> (RowAddrMask, usize) {

crates/crepe-plan-probe/tests/pipeline.rs

@@ -836,9 +836,7 @@ fn n_semijoin_intersects_two_restricted_sides() {
 
 mod stage3 {
     use super::*;
-    use crepe_plan_probe::lance_store::{
-        LanceStore, SEMIJOIN_ATOMS_MAX, SideIo, membership_mask, prepare_lance,
-    };
+    use crepe_plan_probe::lance_store::{LanceStore, SideIo, membership_mask, prepare_lance};
     use crepe_plan_probe::pipeline::evaluate_lance;
 
     async fn lance_family(labels: &Labels, f: &Family) -> (tempfile::TempDir, LanceStore) {
@@ -1001,19 +999,12 @@ mod stage3 {
         ));
     }
 
-    /// The semijoin mask is built through relalg up to `SEMIJOIN_ATOMS_MAX`
-    /// centers and by the scalar test above it; both give the same rows.
+    /// The semijoin mask is built through relalg for any number of centers,
+    /// past the 4 that fit before relalg released consumed slots.
     #[test]
-    fn the_membership_mask_agrees_on_both_sides_of_the_crossover() {
+    fn the_membership_mask_is_exact_for_any_number_of_centers() {
         let values: Vec<u32> = (0..1_000u32).map(|i| (i * 37) % 500).collect();
-        for k in [
-            0usize,
-            1,
-            7,
-            SEMIJOIN_ATOMS_MAX,
-            SEMIJOIN_ATOMS_MAX + 1,
-            200,
-        ] {
+        for k in [0usize, 1, 4, 5, 64, 200, 500] {
             let centers: Vec<u32> = (0..k as u32).map(|c| c * 3).collect();
             let got = membership_mask(&values, &centers).unwrap();
             for (i, v) in values.iter().enumerate() {