diff --git a/.claude/board/entries/2026-10-20-crepe-rule-optimizer-spike.md b/.claude/board/entries/2026-10-20-crepe-rule-optimizer-spike.md index f1e980392..6e8b39823 100644 --- a/.claude/board/entries/2026-10-20-crepe-rule-optimizer-spike.md +++ b/.claude/board/entries/2026-10-20-crepe-rule-optimizer-spike.md @@ -19,5 +19,5 @@ - Dynamic MeTTa rules need crepe's analysis as a library: crepe is a proc-macro crate (follow-up A, `crepe-core` split). - Semijoin reduction built (`Options::semijoin`): laid-out rows 36 at every size instead of growing with n; preparation 3.47 ms → 0.28 ms at 50,000. The scan stays O(n). - ndarray for the row filter, measured (`examples/row_filter.rs`): over the row-major store the strided compare is slower than the scalar test at every needle count, so it is not wired; over a contiguous column it is 5–6 µs per needle at 50,000 rows and beats the scalar test up to 64 needles; at 128 needles two runs disagree because of benchmark noise (the scalar time moved 1.8× between runs), so that crossover is not pinned (re-measured after the #1632 Codex fix). Next step: a column-major relation store. -- Stage 3 over Lance 13 / lancedb 0.40 built (`lance_store`): each side read under a relalg address mask with `with_row_addr_prefilter`; every masked read scans exactly its mask (P1–P4 green). The relalg membership mask holds 4 centers at most: `k` centers take `2k - 1` scratch slots (no slot reuse inside a fold), and `MAX_SLOTS = 8` is an r2il setting, not a hard limit (mask-risc addresses 65,536). Above that a scalar test builds it. +- Stage 3 over Lance 13 / lancedb 0.40 built (`lance_store`): each side read under a relalg address mask with `with_row_addr_prefilter`; every masked read scans exactly its mask (P1–P4 green). The relalg membership mask held 4 centers at most (`2k - 1` scratch slots against `MAX_SLOTS = 8`). `FoldDialect::call_owned` now releases consumed slots and relalg lowers `Or`/`And`/`Not` through it, so any width takes 3 slots and the scalar fallback is gone. - Arm B built (`addressed`): lanes in key order, attested with `SealedFacetLane::attest_sorted`, read only at the `seek_from` ranges of the demanded keys. At 50,000 rows it scans `3k` rows where Arm A scans at least 150,000. It is 2.5–12× faster than Arm A up to 1% of centers and pays back its 7 ms attest after 2–3 queries. From 10% of centers up a full scan wins, and with all centers Arm B never breaks even. Open: the selectivity crossover between 1% and 10%; multi-fragment lanes (an append currently forces a compaction); a second lane for demands on the second column. diff --git a/.claude/plans/crepe-rule-optimizer-spike-v1.md b/.claude/plans/crepe-rule-optimizer-spike-v1.md index 0b2e2db9a..3bdd23da4 100644 --- a/.claude/plans/crepe-rule-optimizer-spike-v1.md +++ b/.claude/plans/crepe-rule-optimizer-spike-v1.md @@ -118,7 +118,7 @@ DECISION (operator, 2026-10-20): "ja, bau Stufe 3 mit Lance 13 und CI-Workflow" The relations are stored column-major in Lance: one table per relation, columns `c0`, `c1` (`UInt32`), written through lancedb 0.40 and read through lance 13. Preparation becomes H14d's stages 2 and 3 per side: 1. Read the one column the address mask needs (the restriction column, or the key column for the semijoin), with `_rowaddr`. -2. Build the address mask `R` through relalg over that column as a `U32` lane: `eq_u32(col, k)` for a restriction, `Or` of `eq_u32(key, c)` for the semijoin centers (lowered by mask-risc onto `ndarray::simd::eq_u32_to_mask`). Above 4 centers the mask is built by the scalar membership test instead. relalg lowers `Or(a1..ak)` left-associatively, and every atom and every `OR` takes a scratch slot of its own (no reuse inside a fold), so `k` centers take `2k - 1` slots. One fold may use `lance_graph_r2il::MAX_SLOTS = 8`, a setting the crate calls "sized with headroom", so 5 centers (9 slots) fail with `ScratchBufferTooSmall`. mask-risc addresses up to 65,536 slots. Either a larger arena or slot reuse inside the fold (an `Or` chain needs only the accumulator and the current atom) would lift the limit; both are r2il/relalg changes. This is not the §2c speed crossover, which is ndarray's row filter. +2. Build the address mask `R` through relalg over that column as a `U32` lane: `eq_u32(col, k)` for a restriction, `Or` of `eq_u32(key, c)` for the semijoin centers (lowered by mask-risc onto `ndarray::simd::eq_u32_to_mask`). This works for any number of centers. Until 2026-10-20 it held only 4: relalg lowered `Or(a1..ak)` left-associatively and every atom and every `OR` took a scratch slot of its own, so `k` centers took `2k - 1` slots against `lance_graph_r2il::MAX_SLOTS = 8` (5 centers failed with `ScratchBufferTooSmall`), and above 4 a scalar membership test built the mask. Now `FoldDialect::call_owned` releases the slots an `OR`/`AND`/`NOT` consumed, and relalg, which consumes each subexpression's value exactly once, lowers through it: an `Or` of any width needs 3 slots, and the scalar fallback is removed. This is not the §2c speed crossover, which is ndarray's row filter. 3. Read the side's columns with `with_row_addr_prefilter(R)`. 4. Hand those rows to `JoinTable::prepare_star`, as the in-memory path does. diff --git a/.github/workflows/crepe-plan-probe.yml b/.github/workflows/crepe-plan-probe.yml index 73699ee93..1f7eec0e9 100644 --- a/.github/workflows/crepe-plan-probe.yml +++ b/.github/workflows/crepe-plan-probe.yml @@ -3,13 +3,14 @@ name: Crepe plan probe # git deps on the crepe and MORK forks, lance 13 / lancedb 0.40), so no # workspace job builds it. It runs from its own directory so the cargo config # above it and its own `.cargo/config.toml` apply. It path-deps relalg, -# mask-risc, mork-pathmap-probe and lance-graph-contract (the ordered lane and +# mask-risc, r2il, mork-pathmap-probe and lance-graph-contract (the ordered lane and # facet keys of arm B), so changes there run it too. on: pull_request: paths: - 'crates/crepe-plan-probe/**' - 'crates/lance-graph-relalg/**' + - 'crates/lance-graph-r2il/**' - 'crates/lance-graph-contract/**' - 'crates/lance-graph-mask-risc/**' - 'crates/mork-pathmap-probe/**' @@ -20,6 +21,7 @@ on: paths: - 'crates/crepe-plan-probe/**' - 'crates/lance-graph-relalg/**' + - 'crates/lance-graph-r2il/**' - 'crates/lance-graph-contract/**' - 'crates/lance-graph-mask-risc/**' - 'crates/mork-pathmap-probe/**' @@ -49,3 +51,12 @@ jobs: - run: cargo fmt --check - run: cargo clippy --all-targets -- -D warnings - run: cargo test + # relalg is its own workspace and patches ndarray to a sibling checkout + # (`../../../ndarray` from its directory); no other job runs its tests. + - name: relalg tests and clippy + working-directory: crates/lance-graph-relalg + run: | + git clone --depth 1 https://github.com/AdaWorldAPI/ndarray ../../../ndarray + cargo fmt --check + cargo test + cargo clippy --all-targets -- -D warnings diff --git a/crates/crepe-plan-probe/examples/walk_mask.rs b/crates/crepe-plan-probe/examples/walk_mask.rs new file mode 100644 index 000000000..a53b90c4a --- /dev/null +++ b/crates/crepe-plan-probe/examples/walk_mask.rs @@ -0,0 +1,232 @@ +//! crepe's `graph_walking` benchmark program, evaluated twice: once by the +//! code `crepe!` generates (hash sets and hash-map indexes, tuple at a time) +//! and once over dense bit planes with `ndarray::simd` mask ops. Both runs +//! must give the same tuple counts. Run in release from this crate's +//! directory: +//! +//! ```text +//! cargo run --release --example walk_mask +//! ``` +//! +//! The program (copied from `crepe/benches/graph_walking.rs`): +//! +//! ```text +//! Node(x) <- Edge(x, _, _); Node(x) <- Edge(_, x, _); +//! Walk(x, x, 0) <- Node(x); +//! Walk(x, z, l1 + l2) <- Edge(x, y, l1), Walk(y, z, l2), (l1 + l2 <= 50); +//! NoWalk(x, y) <- Node(x), Node(y), !Walk(x, y, _); +//! ``` +//! +//! The mask form needs what the macro does not know: node ids dense in +//! `0..n`, and path lengths in `0..=MAX_PATH_LEN`. `Walk` is one `n x n` bit +//! plane per length; an edge of length `w` from `x` to `y` makes +//! `walk[l + w][x] |= delta[l][y]`, the semi-naive step is +//! `new &= !walk; walk |= new`, and the fixpoint is "every new plane is +//! zero". `NoWalk` is `node x node & !(OR over l of walk[l])`. + +use std::hint::black_box; +use std::time::{Duration, Instant}; + +use crepe::crepe; +use ndarray::simd::{mask_andnot_assign, mask_any, mask_or_assign, popcount_batch_u64}; + +const MAX_PATH_LEN: u32 = 50; +const REPS: usize = 9; + +crepe! { + @input + struct Edge(i32, i32, u32); + + @output + struct Walk(i32, i32, u32); + + @output + struct NoWalk(i32, i32); + + struct Node(i32); + + Node(x) <- Edge(x, _, _); + Node(x) <- Edge(_, x, _); + + Walk(x, x, 0) <- Node(x); + Walk(x, z, len1 + len2) <- + Edge(x, y, len1), + Walk(y, z, len2), + (len1 + len2 <= MAX_PATH_LEN); + + NoWalk(x, y) <- Node(x), Node(y), !Walk(x, y, _); +} + +fn edges(n: i32) -> Vec<(i32, i32, u32)> { + let mut e = Vec::new(); + for i in 0..n { + for j in 0..n { + if (i + j) % 50 < 2 { + e.push((i, j, 5)); + } + } + } + e +} + +fn crepe_walk(e: &[(i32, i32, u32)]) -> (usize, usize) { + let mut rt = Crepe::new(); + rt.extend(e.iter().map(|&(a, b, w)| Edge(a, b, w))); + let (walk, nowalk) = rt.run(); + (walk.len(), nowalk.len()) +} + +/// Bits per row of an `n`-wide plane. +fn words(n: usize) -> usize { + n.div_ceil(64) +} + +fn set(row: &mut [u64], i: usize) { + row[i / 64] |= 1 << (i % 64); +} + +/// The same program over bit planes. Node ids must be dense in `0..n`. +fn mask_walk(n: usize, e: &[(i32, i32, u32)]) -> (usize, usize) { + let w = words(n); + let lens = MAX_PATH_LEN as usize + 1; + let row = |plane: usize, x: usize| (plane * n + x) * w; + + // Edges grouped by length: adjacency rows `x -> {y}`. + let mut by_len: Vec>> = vec![None; lens]; + let mut node = vec![0u64; w]; + for &(a, b, l) in e { + let (a, b, l) = (a as usize, b as usize, l as usize); + set(&mut node, a); + set(&mut node, b); + if l < lens { + let adj = by_len[l].get_or_insert_with(|| vec![0u64; n * w]); + set(&mut adj[a * w..a * w + w], b); + } + } + + // walk[l][x] and the delta planes; Walk(x, x, 0) for every node. + let mut walk = vec![0u64; lens * n * w]; + let mut delta = vec![0u64; lens * n * w]; + let mut live = vec![false; lens]; + for x in 0..n { + if node[x / 64] >> (x % 64) & 1 == 1 { + set(&mut walk[row(0, x)..row(0, x) + w], x); + set(&mut delta[row(0, x)..row(0, x) + w], x); + } + } + live[0] = true; + + let mut new = vec![0u64; lens * n * w]; + loop { + new.fill(0); + let mut new_live = vec![false; lens]; + for (l1, adj) in by_len.iter().enumerate() { + let Some(adj) = adj else { continue }; + for l2 in (0..lens).filter(|&l2| live[l2]) { + let l = l1 + l2; + if l >= lens { + continue; + } + for x in 0..n { + let ar = &adj[x * w..x * w + w]; + let dst = row(l, x); + for (wi, &word) in ar.iter().enumerate() { + let mut bits = word; + while bits != 0 { + let y = wi * 64 + bits.trailing_zeros() as usize; + bits &= bits - 1; + let src = row(l2, y); + // `new` and `delta` are different buffers. + mask_or_assign(&mut new[dst..dst + w], &delta[src..src + w]); + } + } + } + new_live[l] = true; + } + } + let mut any = false; + for (l, is_live) in new_live.iter_mut().enumerate() { + if !*is_live { + continue; + } + let span = row(l, 0)..row(l, 0) + n * w; + mask_andnot_assign(&mut new[span.clone()], &walk[span.clone()]); + if mask_any(&new[span.clone()]) { + mask_or_assign(&mut walk[span.clone()], &new[span.clone()]); + any = true; + } else { + *is_live = false; + } + } + if !any { + break; + } + std::mem::swap(&mut delta, &mut new); + live = new_live; + } + + let walk_count = popcount_batch_u64(&walk) as usize; + // NoWalk: for each node x, nodes not reached at any length. + let mut reach = vec![0u64; w]; + let mut nowalk = 0usize; + for x in 0..n { + if node[x / 64] >> (x % 64) & 1 == 0 { + continue; + } + reach.fill(0); + for l in 0..lens { + mask_or_assign(&mut reach, &walk[row(l, x)..row(l, x) + w]); + } + let mut miss = node.clone(); + mask_andnot_assign(&mut miss, &reach); + nowalk += popcount_batch_u64(&miss) as usize; + } + (walk_count, nowalk) +} + +fn median(mut v: Vec) -> Duration { + v.sort(); + v[v.len() / 2] +} + +fn time(mut f: impl FnMut() -> (usize, usize)) -> (Duration, (usize, usize)) { + let mut t = Vec::with_capacity(REPS); + let mut out = (0, 0); + for _ in 0..REPS { + let s = Instant::now(); + out = black_box(f()); + t.push(s.elapsed()); + } + (median(t), out) +} + +fn main() { + println!( + "{:>5} | {:>10} {:>10} | {:>9} | {:>9} {:>9}", + "n", "crepe ms", "mask ms", "speed-up", "Walk", "NoWalk" + ); + for n in [128usize, 256, 512, 1024] { + let e = edges(n as i32); + let (tm, mask) = time(|| mask_walk(n, &e)); + let (tc, crep) = if n <= 512 { + time(|| crepe_walk(&e)) + } else { + (Duration::ZERO, mask) + }; + assert_eq!(mask, crep, "n={n}: mask and crepe disagree"); + let speed = if tc.is_zero() { + "-".to_string() + } else { + format!("{:.1}x", tc.as_secs_f64() / tm.as_secs_f64()) + }; + println!( + "{:>5} | {:>10.3} {:>10.3} | {:>9} | {:>9} {:>9}", + n, + tc.as_secs_f64() * 1e3, + tm.as_secs_f64() * 1e3, + speed, + mask.0, + mask.1 + ); + } +} diff --git a/crates/crepe-plan-probe/src/lance_store.rs b/crates/crepe-plan-probe/src/lance_store.rs index b1d9a7c33..1d33253d0 100644 --- a/crates/crepe-plan-probe/src/lance_store.rs +++ b/crates/crepe-plan-probe/src/lance_store.rs @@ -27,21 +27,6 @@ use mork_pathmap_probe::join::for_each_set; use crate::logic::RelId; use crate::native::{PreparedStar, StarPlan, prepare_from_rows}; -/// Up to this many semijoin centers, the address mask is an `Or` of `eq_u32` -/// atoms through relalg; above it, the scalar membership test builds it. -/// -/// The limit comes from a setting, not from the algebra and not from a speed -/// crossover. relalg lowers `Or(a1..ak)` left-associatively, and each atom -/// and each `OR` takes a scratch slot of its own (slots are not reused -/// inside a fold), so `k` centers take `2k - 1` slots. One fold may use -/// `lance_graph_r2il::MAX_SLOTS` (8, "sized with headroom") of them. Measured: -/// 4 centers fit (7 slots); 5 need 9 and fail with -/// `ScratchBufferTooSmall { need_words: 145, have_words: 129 }` at 1,000 rows. -/// mask-risc itself addresses up to 65,536 slots. A larger arena, or slot -/// reuse in the fold, lifts this. (The 64-needle crossover in plan §2c is -/// ndarray's row filter, a different mechanism.) -pub const SEMIJOIN_ATOMS_MAX: usize = 4; - #[derive(Debug)] pub enum LanceStoreError { Lance(String), @@ -260,22 +245,16 @@ pub fn address_mask(values: &[u32], rel: &Rel) -> Result, RelError> { Ok(keep) } -/// Rows whose value is one of `centers` (sorted, deduplicated). +/// Rows whose value is one of `centers` (sorted, deduplicated): an `Or` of +/// `eq_u32` atoms through relalg, for any number of centers. relalg releases +/// each consumed slot (`FoldDialect::call_owned`), so the fold needs three +/// scratch slots however wide the `Or` is. pub fn membership_mask(values: &[u32], centers: &[u32]) -> Result, RelError> { if centers.is_empty() { return Ok(vec![0; words_for(values.len())]); } - if centers.len() <= SEMIJOIN_ATOMS_MAX { - let rel = Rel::Or(centers.iter().map(|&c| Rel::eq_u32(0, c)).collect()); - return address_mask(values, &rel); - } - let mut keep = vec![0u64; words_for(values.len())]; - for (i, v) in values.iter().enumerate() { - if centers.binary_search(v).is_ok() { - keep[i / 64] |= 1 << (i % 64); - } - } - Ok(keep) + let rel = Rel::Or(centers.iter().map(|&c| Rel::eq_u32(0, c)).collect()); + address_mask(values, &rel) } fn row_addr_mask(mask: &[u64], n: usize, addr: &[u64]) -> (RowAddrMask, usize) { diff --git a/crates/crepe-plan-probe/tests/pipeline.rs b/crates/crepe-plan-probe/tests/pipeline.rs index a1bfbda8d..11687b182 100644 --- a/crates/crepe-plan-probe/tests/pipeline.rs +++ b/crates/crepe-plan-probe/tests/pipeline.rs @@ -836,9 +836,7 @@ fn n_semijoin_intersects_two_restricted_sides() { mod stage3 { use super::*; - use crepe_plan_probe::lance_store::{ - LanceStore, SEMIJOIN_ATOMS_MAX, SideIo, membership_mask, prepare_lance, - }; + use crepe_plan_probe::lance_store::{LanceStore, SideIo, membership_mask, prepare_lance}; use crepe_plan_probe::pipeline::evaluate_lance; async fn lance_family(labels: &Labels, f: &Family) -> (tempfile::TempDir, LanceStore) { @@ -1001,19 +999,12 @@ mod stage3 { )); } - /// The semijoin mask is built through relalg up to `SEMIJOIN_ATOMS_MAX` - /// centers and by the scalar test above it; both give the same rows. + /// The semijoin mask is built through relalg for any number of centers, + /// past the 4 that fit before relalg released consumed slots. #[test] - fn the_membership_mask_agrees_on_both_sides_of_the_crossover() { + fn the_membership_mask_is_exact_for_any_number_of_centers() { let values: Vec = (0..1_000u32).map(|i| (i * 37) % 500).collect(); - for k in [ - 0usize, - 1, - 7, - SEMIJOIN_ATOMS_MAX, - SEMIJOIN_ATOMS_MAX + 1, - 200, - ] { + for k in [0usize, 1, 4, 5, 64, 200, 500] { let centers: Vec = (0..k as u32).map(|c| c * 3).collect(); let got = membership_mask(&values, ¢ers).unwrap(); for (i, v) in values.iter().enumerate() { diff --git a/crates/lance-graph-r2il/src/lib.rs b/crates/lance-graph-r2il/src/lib.rs index 0ed1ed4ca..813685434 100644 --- a/crates/lance-graph-r2il/src/lib.rs +++ b/crates/lance-graph-r2il/src/lib.rs @@ -267,6 +267,12 @@ pub struct FoldDialect<'p> { /// The next unused scratch slot WITHIN the current `ops` accumulation. /// Reset to 0 every time a fold finalizes and runs. next_slot: u16, + /// Slots below [`Self::next_slot`] whose value is dead, released by + /// [`Self::call_owned`] and handed out again by [`Self::fresh`] before + /// a new slot is minted. Cleared with `next_slot` at every fold. Only + /// `call_owned` ever releases a slot, so a caller that never uses it sees + /// exactly the old numbering. + free: Vec, /// The current execution epoch — incremented every time a fold /// finalizes and runs, at the same point [`Self::next_slot`] resets to /// 0. Every [`Val::Slot`] minted carries this value at mint time; a @@ -333,6 +339,7 @@ impl<'p> FoldDialect<'p> { foreign, ops: Vec::new(), next_slot: 0, + free: Vec::new(), epoch: 0, scratch: vec![0u64; cap], group_sink: &mut [], @@ -552,11 +559,67 @@ impl<'p> FoldDialect<'p> { /// The next unused scratch slot within the current `ops` accumulation, /// advancing [`Self::next_slot`] by one. fn fresh(&mut self) -> u16 { + if let Some(s) = self.free.pop() { + return s; + } let s = self.next_slot; self.next_slot += 1; s } + /// [`Dialect::call`] for a caller that OWNS the masks it combines. + /// + /// For `IntOr`, `IntAnd` and `IntNot` in [`BoolMode::InPlace`], the slots + /// of the consumed operands are released once the op is emitted (unless + /// one of them is the result), and [`Self::fresh`] hands them out again. + /// An `Or` or `And` chain over `k` atoms then needs three slots, not + /// `2k - 1`. Every other call, and every call in [`BoolMode::Fused`], is + /// plain [`Dialect::call`]. + /// + /// # Contract + /// + /// The caller guarantees that no other [`Val`] anywhere (on this stack, on + /// another stack, or held aside) names the consumed slots. [`Val`] is + /// `Copy`, so the dialect cannot check this: a copy that survives the call + /// would read whatever a later op writes into the released slot. A + /// lowering that consumes each subexpression's value exactly once, as + /// `lance-graph-relalg` does, meets the contract by construction. A + /// stack program that can duplicate values must use `call`. + pub fn call_owned( + &mut self, + f: FnIndex, + v: [u8; 3], + stack: &mut Vec, + ) -> Result<(), FoldError> { + let arity = match f { + INT_OR | INT_AND => 2, + INT_NOT => 1, + _ => return self.call(f, v, stack), + }; + if self.mode != BoolMode::InPlace || stack.len() < arity { + return self.call(f, v, stack); + } + let epoch = self.epoch; + let consumed: Vec = stack[stack.len() - arity..] + .iter() + .filter_map(|v| match *v { + Val::Slot { slot, epoch: e } if e == epoch => Some(slot), + _ => None, + }) + .collect(); + self.call(f, v, stack)?; + let result = match stack.last() { + Some(&Val::Slot { slot, epoch: e }) if e == self.epoch => Some(slot), + _ => None, + }; + for s in consumed { + if Some(s) != result && !self.free.contains(&s) { + self.free.push(s); + } + } + Ok(()) + } + /// Survivor gating (verbatim from W0C, restated over `Operand`): if `b` /// was produced by the op emitted LAST and that op is an ungated `Pred`, /// gate it under `a` instead of spending a facade pass on `And`. @@ -611,6 +674,7 @@ impl<'p> FoldDialect<'p> { fn finalize_and_run(&mut self, terminal: Terminal, out: OutKind) -> Result { let ops = std::mem::take(&mut self.ops); self.next_slot = 0; + self.free.clear(); self.epoch += 1; self.exprs.clear(); self.lowered.clear(); diff --git a/crates/lance-graph-r2il/tests/dialect.rs b/crates/lance-graph-r2il/tests/dialect.rs index 87a24ddfb..03a689e1a 100644 --- a/crates/lance-graph-r2il/tests/dialect.rs +++ b/crates/lance-graph-r2il/tests/dialect.rs @@ -358,3 +358,65 @@ fn a_fold_wider_than_max_slots_is_refused_not_truncated() { "a fold wider than MAX_SLOTS must be refused, got {wide:?}" ); } + +/// FAILS IF: `call_owned` stops releasing the slots an `IntOr` / `IntNot` +/// consumed (so a chain of `k` ranges needs `2k - 1` slots again), or if plain +/// `call` starts releasing them too. Plain `call` must keep the old numbering: +/// its caller may hold copies of a `Val::Slot`, so it is refused past +/// `MAX_SLOTS` exactly as before. +#[test] +fn call_owned_reuses_consumed_slots_and_call_does_not() { + let values = lane(); + let lanes = [LaneRef::I32(&values)]; + let planes = Planes { + n_rows: N, + masks: &[], + lanes: &lanes, + }; + let foreign = Foreign::NONE; + let width = 4 * MAX_SLOTS; + + // `[0,1) | [1,2) | … | [width-1, width)`, negated twice: `width` rows. + let run = |owned: bool| { + let mut d = FoldDialect::new(&planes, &foreign); + let mut stack = vec![]; + let op = |d: &mut FoldDialect<'_>, f: FnIndex, v: [u8; 3], s: &mut Vec| { + if owned { + d.call_owned(f, v, s) + } else { + call(d, f, v, s) + } + }; + for i in 0..width { + op(&mut d, RANGE, [i as u8, i as u8 + 1, 0], &mut stack).expect("RANGE"); + if i > 0 { + op(&mut d, INT_OR, [0; 3], &mut stack).expect("OR"); + } + } + op(&mut d, INT_NOT, [0; 3], &mut stack).expect("NOT"); + op(&mut d, INT_NOT, [0; 3], &mut stack).expect("NOT"); + let r = call(&mut d, POP_COUNT, [0; 3], &mut stack); + let slots = d.last_program().map(|p| p.scratch_slots); + (r, stack.last().copied(), slots) + }; + + let (r, top, slots) = run(true); + assert_eq!(r, Ok(())); + assert_eq!(top, Some(Val::Scalar(width as i64))); + assert_eq!( + slots, + Some(3), + "an owned chain needs the accumulator, one atom and one result" + ); + + let (r, _, _) = run(false); + assert!( + matches!( + r, + Err(FoldError::Exec( + lance_graph_mask_risc::ExecError::ScratchBufferTooSmall { .. } + )) + ), + "plain call must keep one slot per result, got {r:?}" + ); +} diff --git a/crates/lance-graph-relalg/src/lower.rs b/crates/lance-graph-relalg/src/lower.rs index 401090299..cf3309b2e 100644 --- a/crates/lance-graph-relalg/src/lower.rs +++ b/crates/lance-graph-relalg/src/lower.rs @@ -180,13 +180,13 @@ fn lower_rel(d: &mut FoldDialect<'_>, s: &mut Vec, rel: &Rel) -> Result<(), Rel::Empty => empty(d, s), Rel::All => { empty(d, s)?; - call(d, INT_NOT, [0; 3], s) + owned(d, INT_NOT, s) } Rel::Plane(p) => load(d, s, SPACE_PLANE, *p), Rel::Atom(a) => lower_atom(d, s, *a), Rel::Not(c) => { lower_rel(d, s, c)?; - call(d, INT_NOT, [0; 3], s) + owned(d, INT_NOT, s) } Rel::And(cs) => fold(d, s, cs, INT_AND, &Rel::All), Rel::Or(cs) => fold(d, s, cs, INT_OR, &Rel::Empty), @@ -206,11 +206,19 @@ fn fold( lower_rel(d, s, first)?; for c in rest { lower_rel(d, s, c)?; - call(d, op, [0; 3], s)?; + owned(d, op, s)?; } Ok(()) } +/// A mask op whose operands this lowering owns: every subexpression's value +/// is consumed exactly once, so the dialect may release the consumed slots +/// (the contract of `FoldDialect::call_owned`). Width is then free and only +/// depth costs scratch slots. +fn owned(d: &mut FoldDialect<'_>, f: FnIndex, s: &mut Vec) -> Result<(), FoldError> { + d.call_owned(f, [0; 3], s) +} + fn lower_atom(d: &mut FoldDialect<'_>, s: &mut Vec, a: Atom) -> Result<(), FoldError> { match a { Atom::U32 { lane, eq, v } => { @@ -218,7 +226,7 @@ fn lower_atom(d: &mut FoldDialect<'_>, s: &mut Vec, a: Atom) -> Result<(), s.push(Val::Scalar(i64::from(v))); call(d, INT_EQUAL, [0; 3], s)?; if !eq { - call(d, INT_NOT, [0; 3], s)?; + owned(d, INT_NOT, s)?; } Ok(()) } @@ -244,7 +252,7 @@ fn lower_atom(d: &mut FoldDialect<'_>, s: &mut Vec, a: Atom) -> Result<(), } } if matches!(op, Cmp::Ge | Cmp::Le | Cmp::Ne) { - call(d, INT_NOT, [0; 3], s)?; + owned(d, INT_NOT, s)?; } Ok(()) } diff --git a/crates/lance-graph-relalg/tests/rewrites.rs b/crates/lance-graph-relalg/tests/rewrites.rs index cdfa58f35..f79d134e3 100644 --- a/crates/lance-graph-relalg/tests/rewrites.rs +++ b/crates/lance-graph-relalg/tests/rewrites.rs @@ -11,6 +11,10 @@ use lance_graph_relalg::{ const N: usize = 1_000; +// Measured, see `a_wide_conjunction_is_one_interval_after_the_rewrite`. +const PASSES_REWRITTEN: usize = 1; +const PASSES_AS_WRITTEN: usize = 10; + // Lane indexes. const X: u16 = 0; // i32, (i % 50) - 20: negatives, zero, positives const Y: u16 = 1; // i32, (i * 7) % 31 @@ -460,11 +464,16 @@ fn negation_reaches_the_leaves() { count(&fixture(), rel); } -/// FAILS IF: the rewrite stops shrinking a wide conjunction — as written it -/// needs more scratch slots than one fold has and is refused; rewritten it is -/// one interval and runs. +/// FAILS IF: the rewrite stops shrinking a wide conjunction. Rewritten, the +/// ten comparisons on one lane are one interval and one pass. As written +/// each comparison is its own pass (survivor gating absorbs the `And`s). +/// +/// Until the as-written lowering released consumed slots, this description +/// was refused as written (ten comparisons needed more than the dialect's 8 +/// slots) and that refusal was the assertion. Width now costs no slots, so +/// both run, and the rewrite's gain is pinned by the passes it saves. #[test] -fn a_wide_conjunction_runs_only_after_the_rewrite() { +fn a_wide_conjunction_is_one_interval_after_the_rewrite() { let rel = Rel::And((0..10).map(|i| Rel::i32(X, Cmp::Lt, 20 - i)).collect()); let fx = fixture(); let q = Query { @@ -472,8 +481,14 @@ fn a_wide_conjunction_runs_only_after_the_rewrite() { measure: Measure::Count, }; let ((r1, _), (r2, _)) = both(&fx, &q); - assert!(r2.as_ref().is_err_and(out_of_slots), "as written: {r2:?}"); - assert_eq!(r1.expect("rewritten").answer, oracle(&fx, &q).0); + let (r1, r2) = (r1.expect("rewritten"), r2.expect("as written")); + let want = oracle(&fx, &q).0; + assert_eq!(r1.answer, want); + assert_eq!(r2.answer, want); + assert_eq!( + (r1.passes, r2.passes), + (PASSES_REWRITTEN, PASSES_AS_WRITTEN) + ); } /// FAILS IF: a `!=` on an `I32` lane is not reconciled with the intervals @@ -506,3 +521,123 @@ fn an_inequality_in_a_union_meets_the_intervals() { let (r, _) = count(&fx, wide); assert_eq!(r.programs, 0); } + +// ── wide junctions: width is free, only depth costs (D-SRAM-2 follow-up) ─── + +/// A lane whose value is its row: `eq_u32(0, c)` keeps exactly row `c`. +fn wide_planes(f: impl FnOnce(&Planes<'_>, &[u32]) -> T) -> T { + let ids: Vec = (0..N as u32).collect(); + let lanes = [LaneRef::U32(&ids)]; + let planes = Planes { + n_rows: N, + masks: &[], + lanes: &lanes, + }; + f(&planes, &ids) +} + +fn kept(rows: impl Fn(usize) -> bool) -> Vec { + plane(rows) +} + +/// FAILS IF: an `Or` of `k` atoms takes a scratch slot per atom and per `OR` +/// (`2k - 1`), so a wide disjunction is refused once `2k - 1` passes the +/// dialect's 8 slots — at 5 atoms. Children fold into the first child's +/// slot, as Quack's in-place lowering does. +#[test] +fn a_wide_disjunction_keeps_its_rows_in_constant_slots() { + for k in [1usize, 4, 5, 64, 1_000] { + let centers: Vec = (0..k as u32).map(|c| (c * 7) % N as u32).collect(); + let q = Query { + rel: Rel::Or(centers.iter().map(|&c| Rel::eq_u32(0, c)).collect()), + measure: Measure::Keep, + }; + let want = kept(|r| centers.contains(&(r as u32))); + wide_planes(|planes, _| { + for written in [false, true] { + let mut keep = vec![0u64; words_for(N)]; + let got = if written { + run_as_written(&q, planes, Some(&mut keep)) + } else { + run(&q, planes, Some(&mut keep)) + }; + assert!(got.is_ok(), "k={k} written={written}: {got:?}"); + assert_eq!(keep, want, "k={k} written={written}"); + } + let mut keep = vec![0u64; words_for(N)]; + let (_, program) = + lance_graph_relalg::run_with_program(&q, planes, Some(&mut keep)).unwrap(); + if let Some(p) = program { + assert!(p.scratch_slots <= 3, "k={k}: {} slots", p.scratch_slots); + } + }); + } +} + +/// FAILS IF: a wide conjunction of negations takes a slot per `NOT` and per +/// `AND`. Each `NOT` is written in place over its own atom's slot. +#[test] +fn a_wide_conjunction_of_negations_keeps_its_rows_in_constant_slots() { + for k in [5usize, 64, 1_000] { + let centers: Vec = (0..k as u32).map(|c| (c * 3) % N as u32).collect(); + let q = Query { + rel: Rel::And( + centers + .iter() + .map(|&c| Rel::Not(Box::new(Rel::eq_u32(0, c)))) + .collect(), + ), + measure: Measure::Count, + }; + let want = (0..N).filter(|r| !centers.contains(&(*r as u32))).count() as u64; + wide_planes(|planes, _| { + let got = run_as_written(&q, planes, None); + assert!( + matches!(got, Ok(ref r) if r.answer == Answer::Count(want)), + "k={k}: {got:?}" + ); + let got = run(&q, planes, None); + assert!( + matches!(got, Ok(ref r) if r.answer == Answer::Count(want)), + "k={k}: {got:?}" + ); + }); + } +} + +/// Paired: depth still costs. A slot held by an outer junction stays live +/// while its inner one is built, so a nested description needs more slots +/// than a flat one of the same width. (If it did not, the constant above +/// would be vacuous.) +#[test] +fn depth_still_costs_slots() { + let flat = Query { + rel: Rel::Or((0..8u32).map(|c| Rel::eq_u32(0, c)).collect()), + measure: Measure::Keep, + }; + let deep = Query { + rel: Rel::Or(vec![ + Rel::eq_u32(0, 1), + Rel::And(vec![ + Rel::Not(Box::new(Rel::eq_u32(0, 2))), + Rel::Or(vec![Rel::eq_u32(0, 3), Rel::eq_u32(0, 4)]), + ]), + ]), + measure: Measure::Keep, + }; + wide_planes(|planes, _| { + let slots = |q: &Query| { + let mut keep = vec![0u64; words_for(N)]; + lance_graph_relalg::run_with_program(q, planes, Some(&mut keep)) + .unwrap() + .1 + .map_or(0, |p| p.scratch_slots) + }; + assert!( + slots(&deep) > slots(&flat), + "deep={} flat={}", + slots(&deep), + slots(&flat) + ); + }); +}