diff --git a/.claude/board/LATEST_STATE.md b/.claude/board/LATEST_STATE.md index 366a29240..9b722f505 100644 --- a/.claude/board/LATEST_STATE.md +++ b/.claude/board/LATEST_STATE.md @@ -1,3 +1,8 @@ +## 2026-10-21 — D-CMR-2 lowering in core (in PR) + +- **Core (new module, no contract change):** `lance_graph::fold::set_lower::{SetPlan, SortedBasis, Level, Measured, LowerError}`. A `SetExpr` is normalized (contract) and split into `Runs ∩ residual`: comparisons on an attested sorted `i32` lane (`SortedBasis`, one attestation scan) become runs by counted binary-search seeks; the rest compiles once to a mask-risc `Program` run over each run with `execute_extent`. Levels: geometry (no residual, `Count = Σ(hi−lo)`, nothing read), optics (`Count` terminal), etch (`Keep` into `Out::Mask`, outside bits cleared). `SetPlan::unsplit` is the geometry-off arm. Entry `entries/2026-10-21-d-cmr-2-lowering.md`. +- Open: D-CMR-2b, a scratch-free `Pred` fold in mask-risc (`fused_ternlog` declines `Pred`); residual comparisons run tiled with per-tile scratch. + ## 2026-10-21 — main after #1644 (merged `c05a313`) - **On main:** #1641 (D-RELALG-UNIFIED plan), #1643 (U1 `jc-batch-probe`, walk_mask follow-up), #1644 (D-CMR-1 / D-RA-2, `contract::set_algebra`; inventory delta below under 2026-10-20). diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index dd6964d13..a33af65e5 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -255,10 +255,10 @@ Plan: `.claude/plans/d-cmr-concept-chain-normalization-v1.md`. Parent: D-CMR-RES |---|---|---|---| | **D-CMR-0** | Plan, identity matrix, inventory, board rows | Shipped #1532 (docs) | docs only; every inventory row cites code | | **D-CMR-1** | Set-theory IR core in the contract: `Runs` (∩ ∪ ∖, complement), residual `Res`, measures; pure | Shipped (#1644, `c05a313`): `contract::set_algebra::{SetExpr, Res, Runs, normalize}`; measures not yet (`entries/2026-10-20-d-cmr-1-set-algebra.md`) | algebra laws vs materialized bitsets as oracle | -| **D-CMR-2** | Lowering: runs → extents, residual → predicates per run; measurement levels geometry / optics / etch | Queued (after D-CMR-1; lowering reaches mask-risc through core via `lance_graph::fold`, #1537 — there is no contract → mask-risc edge) | == row oracle; `Count` without residual reads zero rows | +| **D-CMR-2** | Lowering: runs → extents, residual → predicates per run; measurement levels geometry / optics / etch | In PR (branch `claude/eager-albattani-iufugq`): `lance_graph::fold::set_lower::{SetPlan, SortedBasis, Level, Measured}`; geometry reads nothing, optics/etch run the residual per run via `execute_extent`. The scratch-free `Pred` fold in mask-risc is D-CMR-2b, not built (`entries/2026-10-21-d-cmr-2-lowering.md`) | == row oracle; `Count` without residual reads zero rows | | **D-CMR-3** | Quack frontend: `Filter` → set IR (extent hoisting falls out) | Queued | identical answers on the quack suite; other-basis range stays residual | | **D-CMR-4** | Two dimensions: `Rect` over composite coordinates, row-major or Morton runs | Queued | rectangle ∩ == bitset oracle | -| **D-CMR-5** | **⛔ VIOLATION — GATHER. DO NOT BUILD. CORRECT: D-HOLO-H14c / H14d.** Preimage `f⁻¹(S)` under a proven mapping (the hop) | Queued | == row oracle; many-to-one image refused | +| **D-CMR-5** | **BLOCKED: lawful design proposed (operator 2026-10-21).** Preimage `f⁻¹(S)` as a co-addressed transposed tile fold over a version-witnessed projection of `f`, built in preparation by a sequential same-table scan with frozen-codebook endpoint translation (ADMISSION B; reusable across queries). `S` enters only as a membership plane translated into the same frozen coordinates. No seek or load of `f` keyed by `S`'s values or a previous fold (Q1 = C), no `Gather`, no scatter terminal. R1 (`rmap` `rebind` / `discover` escape-driven seeks) stays option C. ⊘ Was: "VIOLATION — GATHER. DO NOT BUILD" (entry `2026-10-21-d-cmr-5-rescope.md`) | BLOCKED — needs a contract-level attested witness for the frozen codebook (`rmap::Witness` is a probe-local stand-in); no preimage operator exists; not admitted, not ready to build | == row oracle; program checked Gather-free with a silence twin; attest / projection / query costs reported apart; a many-to-one `f` gives a valid distinct-source preimage (tested) and a bag-sensitive Count/Sum over it is refused (tested); refusal on a stale or missing witness | | **D-CMR-6** | Rule catalogue + fixpoint driver: complement, relational difference | Queued | Experiments A, D, E; each guard removal fails | | **D-CMR-7** | NAL frontend: aperture → `Runs` | Queued (after NAL lands in core) | NAL W2 tests unchanged | | **D-CMR-8** | Equality-saturation probe (`egg`/`egglog` as references) | Queued (only if rules compete) | compared against D-CMR-6; no dependency without evidence | diff --git a/.claude/board/entries/2026-10-21-d-cmr-2-lowering.md b/.claude/board/entries/2026-10-21-d-cmr-2-lowering.md new file mode 100644 index 000000000..d43b0a8f9 --- /dev/null +++ b/.claude/board/entries/2026-10-21-d-cmr-2-lowering.md @@ -0,0 +1,21 @@ +# 2026-10-21 — D-CMR-2: the contract set algebra lowered onto mask-risc + +**Status:** TEST-PINNED (in PR, branch `claude/eager-albattani-iufugq`). **Basis:** plan `d-cmr-concept-chain-normalization-v1.md` §3.4 and the D-CMR-2 row; ADMISSION B (`2026-10-20-six-decisions.md` §1) for the seeks. + +- **Where:** `crates/lance-graph/src/fold/set_lower.rs` (`lance_graph::fold::set_lower`). Core already depends on the contract and mask-risc unconditionally; no new dependency. +- **Split:** `SetPlan::new(&SetExpr, n_rows, Option<&SortedBasis>)` normalizes, then takes `Runs` from comparisons on the attested sorted lane (an exact basis-only expression, or the basis conjuncts of a top-level `And`) and leaves the rest as the residual. Mixed `Or`s stay residual and spend no seeks. +- **Costs apart:** `SortedBasis::attest_reads` (one scan, n keys), `SetPlan::seeks` (one binary search per boundary), `Measured::{extents, extent_rows}` (query side). +- **Levels:** geometry answers from the runs alone; optics runs the residual program (`Count`) per run via `execute_extent`; etch runs it with `Keep` into `Out::Mask`, bits outside the runs cleared. +- **Tests (6):** 400 random nested expressions over a sorted basis lane, an unsorted lane, a code lane and two planes (N = 1000, not a word multiple): count and set equal the row oracle at every level and for `unsplit`, both levels exercised, runs narrow the extents (anti-vacuity). Geometry runs against `Planes` with no lanes and no masks and reports 0 extents (≤ 6 seeks). Experiment C (basis range + unsorted predicate → one run + residual, extent under half the rows). Non-basis range stays residual (silence twin). Unsorted basis and row-count mismatches refused. +- **Disable runs** (`cargo test -p lance-graph --lib set_lower`): + + | probe | red | + |---|---| + | geometry off (no residual still runs a program) | `geometry_reads_no_plane_lane_or_extent` | + | split off (`new` = `unsplit`) | geometry, Experiment C, oracle anti-vacuity | + | basis-lane check off (any lane's comparison becomes runs) | Experiment C, silence twin, oracle | + | `keep` does not clear the destination | oracle (`keep`) | + +- **Not done:** D-CMR-2b, a scratch-free `Pred` fold in mask-risc. `fused_ternlog` declines `Pred`, so a residual comparison runs tiled with per-tile scratch (bounded by `TILE_WORDS`, never population-sized). No rows/words-read counter exists in mask-risc; the geometry claim is proven by running against empty planes instead. + +OPEN: D-CMR-2b (mask-risc fused predicate fold; changes `ir.rs` / `exec.rs`, coordinate on EXO-50 first); a `u32` / facet sorted basis (`SealedFacetLane`) instead of an `i32` lane. diff --git a/.claude/board/entries/2026-10-21-d-cmr-5-rescope.md b/.claude/board/entries/2026-10-21-d-cmr-5-rescope.md new file mode 100644 index 000000000..1ef1a4f9f --- /dev/null +++ b/.claude/board/entries/2026-10-21-d-cmr-5-rescope.md @@ -0,0 +1,17 @@ +# 2026-10-21 — D-CMR-5 re-scoped: preimage via frozen coordinates, still BLOCKED + +**Status:** DECISION (operator, 2026-10-21, in session `session_013z2e1YXkm6o6HNrDn5BPWA`; also relayed on EXO-50 as `gpt-ada-01/003`). **Basis:** read-only check of D-CMR-5 against the global→local registers (this session; `crates/mork-pathmap-probe/src/rmap.rs`, `crates/crepe-plan-probe/examples/walk_tile.rs`), the ADMISSION ruling (`2026-10-20-six-decisions.md` §1). + +Operator's words: "Der entscheidende Unterschied ist, ob ein Wert nur in eine bereits festgelegte Koordinate übersetzt wird oder darüber bestimmt, welche Zeile einer Relation gelesen wird." + +| question | ruling | +|---|---| +| Q1: seek `f`'s attested sorted key with range bounds taken from `S` (or a previous fold) | **C, not admitted.** Foreign values steer the reads of `f`; a sorted index and doing it in preparation change nothing. Independent literal / intrinsic same-table key-range seeks on `f` stay B. | +| Q2: translate `S`'s global ids into tile coordinates through a frozen codebook | **B, conditionally.** Pure representation translation, provided it only produces an `S` membership plane, triggers no relational access to `f`, and the mapping is attested, version-bound and fully accounted. | + +- **Design direction (not an admission):** a once-built, version-bound projection of `f` (sequential same-table scan, frozen-codebook endpoint translation) may be reused across queries; `S` is translated into the same coordinates; `f⁻¹(S)` is a co-addressed transposed fold. Reading `f` selectively by the values found in `S` stays forbidden. +- **Two existing registers:** R1 (`rmap` `Focus` + `Csr` + `rebind`) seeks with escapes / values from another column, so it stays option C and is not legalized. R2 (`walk_tile`, D-RU-6) has the admissible shape, but its witness is the probe-local `rmap::Witness`. +- **Many-to-one:** not an error for the set-valued preimage; `Count(f⁻¹(S))` over distinct source coordinates is valid. Refuse a bag-sensitive aggregation that set deduplication would destroy, unless a separately proven multiplicity-preserving aggregation exists. +- **Still BLOCKED:** no contract-level attestation for the codebook; no preimage operator exists (`rmap::hop` computes the image). The plan row's former "CORRECT: H14c/H14d" pointed at the co-addressing principle, not a hop implementation. + +OPEN: the contract-level codebook witness (version, map generation / seal, coordinate-domain identity, scope / completeness, invalidation). diff --git a/.claude/board/entries/README.md b/.claude/board/entries/README.md index 35c5fa127..2a43cc4a9 100644 --- a/.claude/board/entries/README.md +++ b/.claude/board/entries/README.md @@ -25,12 +25,14 @@ index row, (3) no duplicate entry id. Checks 1 and 2 are deliberately opposite directions; the stranding this convention prevents shows up in exactly one of them, never both. -399 entries, 2026-08-06 .. 2026-10-21. +401 entries, 2026-08-06 .. 2026-10-21. | date | entry id | finding | file | |---|---|---|---| | 2026-10-21 | `D-RU-6-LAB-1` | | [2026-10-21-d-ru-6-tile-walk-probe.md](2026-10-21-d-ru-6-tile-walk-probe.md) | | 2026-10-21 | `D-RU-6-LAB-2` | | [2026-10-21-d-ru-6-sparse-compute.md](2026-10-21-d-ru-6-sparse-compute.md) | +| 2026-10-21 | `D-CMR-5` | | [2026-10-21-d-cmr-5-rescope.md](2026-10-21-d-cmr-5-rescope.md) | +| 2026-10-21 | `D-CMR-2` | | [2026-10-21-d-cmr-2-lowering.md](2026-10-21-d-cmr-2-lowering.md) | | 2026-10-20 | `D-RU-1` | | [2026-10-20-u1-jc-batch.md](2026-10-20-u1-jc-batch.md) | | 2026-10-20 | `thought-git-replay-api-write-rubicon` | | [2026-10-20-thought-git-replay-api-write-rubicon.md](2026-10-20-thought-git-replay-api-write-rubicon.md) | | 2026-10-20 | `ADR-2026-10-20-SIX` | | [2026-10-20-six-decisions.md](2026-10-20-six-decisions.md) | diff --git a/.claude/plans/d-cmr-concept-chain-normalization-v1.md b/.claude/plans/d-cmr-concept-chain-normalization-v1.md index 440e0cbdd..7142f9272 100644 --- a/.claude/plans/d-cmr-concept-chain-normalization-v1.md +++ b/.claude/plans/d-cmr-concept-chain-normalization-v1.md @@ -213,7 +213,7 @@ when the first floating-point or quantized column arrives, not before. | D-CMR-2 | **Lowering** to mask-risc: runs → extents, residual → predicates within each run, the three measurement levels. Level 2 for residual comparisons needs predicate fusion in mask-risc (`fused_ternlog` declines `Pred` today) | == row oracle; a `Count` with no residual reads zero rows (instrumented); disabling level 1 or 2 keeps answers | | D-CMR-3 | **Quack normalization pass:** over `Filter`, computing set values (ranges on a proven basis → `Runs`, other leaves → `Res`, `sql_where` first). Extent hoisting falls out of this (Experiment C) | answers identical to today's `lower` path on the quack suite; a range on another basis stays residual (negative control) | | D-CMR-4 | **Two dimensions:** `Rect` over composite coordinates, linearized row-major or in Morton order | rectangle `∩` == bitset oracle; run count within the stated bound | -| D-CMR-5 | **⛔ VIOLATION — GATHER. CORRECT: `d-holo-h14c-thought-first-v1.md` / `d-holo-h14d-logic-mask-read-v1.md`.** **Preimage** `f⁻¹(S)` under a proven mapping (the hop) | == row oracle; a many-to-one image is refused | +| D-CMR-5 | **BLOCKED: lawful design proposed (operator 2026-10-21).** Preimage `f⁻¹(S)` as a co-addressed transposed tile fold over a version-witnessed projection of `f`, built in preparation by a sequential same-table scan with frozen-codebook endpoint translation (ADMISSION B; reusable across queries). `S` enters only as a membership plane translated into the same frozen coordinates. No seek or load of `f` keyed by `S`'s values or a previous fold (Q1 = C), no `Gather`, no scatter terminal. R1 (`rmap` `rebind` / `discover` escape-driven seeks) stays option C. ⊘ Was: "VIOLATION — GATHER. DO NOT BUILD" (entry `2026-10-21-d-cmr-5-rescope.md`) | == row oracle; program checked Gather-free with a silence twin; attest / projection / query costs reported apart; a many-to-one `f` gives a valid distinct-source preimage (tested) and a bag-sensitive Count/Sum over it is refused (tested); refusal on a stale or missing witness | | D-CMR-6 | **Rule catalogue and fixpoint driver**: complement, relational difference `Sum(A) − Sum(A∧B) → Sum(A∧¬B)` | Experiments A, D, E; each guard removal fails | | D-CMR-7 | **NAL normalization pass:** aperture → `Runs` over the R2IL program, R2IL route unchanged (after NAL lands in core) | NAL W2 tests unchanged | | D-CMR-8 | **Equality-saturation probe** (`egg`, `egglog` as references only) | only if rules compete; compared against D-CMR-6; no dependency without evidence | diff --git a/crates/lance-graph/src/fold.rs b/crates/lance-graph/src/fold.rs index 733f59a59..340744cad 100644 --- a/crates/lance-graph/src/fold.rs +++ b/crates/lance-graph/src/fold.rs @@ -11,5 +11,10 @@ //! A capability that is missing here is missing in one of the three crates. pub use lance_graph_mask_risc as mask_risc; + +/// D-CMR-2: the contract's set algebra lowered onto mask-risc (runs as +/// extents, residual as a program, three measurement levels). A lowering, +/// not a second algebra: the rules live in `lance_graph_contract::set_algebra`. +pub mod set_lower; pub use lance_graph_quack as quack; pub use lance_graph_r2il as r2il; diff --git a/crates/lance-graph/src/fold/set_lower.rs b/crates/lance-graph/src/fold/set_lower.rs new file mode 100644 index 000000000..612fa83af --- /dev/null +++ b/crates/lance-graph/src/fold/set_lower.rs @@ -0,0 +1,826 @@ +//! D-CMR-2: lowering of the contract's set algebra onto mask-risc. +//! +//! A [`SetExpr`] (`lance_graph_contract::set_algebra`) says WHAT rows are +//! selected. This module decides HOW mask-risc reads them, in the normal form +//! `Runs ∩ residual`: +//! +//! - **Runs** come from comparisons on an attested sorted lane (a +//! [`SortedBasis`]): each comparison is one counted seek, never a scan. That +//! is the same-table, attested-key seek the 2026-10-20 ADMISSION ruling +//! (option B) admits in preparation. +//! - **The residual** is everything else. It is compiled once into a +//! mask-risc [`Program`] and run over each run as an execution extent. +//! +//! Three measurement levels, cheapest first (plan +//! `d-cmr-concept-chain-normalization-v1.md` §3.4); [`SetPlan`] picks the +//! first that applies: +//! +//! | level | when | what runs | +//! |---|---|---| +//! | [`Level::Geometry`] | no residual | `Count = Σ(hi − lo)`; no plane, lane or extent is read | +//! | [`Level::Optics`] | residual, a measure is wanted | the residual program over each run as an extent (`execute_extent`, `Count` terminal) | +//! | [`Level::Etch`] | the set itself is wanted | the residual program with a `Keep` terminal over each run; bits outside the runs are zero | +//! +//! **Known limit, stated rather than hidden:** mask-risc's fused paths decline +//! a `Pred` (`Program::fused_ternlog`, `ir.rs`), so a residual comparison runs +//! tiled and writes per-tile scratch (at most `TILE_WORDS` words, never a +//! population-sized mask). A scratch-free predicate fold is a mask-risc change +//! (D-CMR-2b), not part of this lowering. +//! +//! Costs are reported apart, as the ADMISSION ruling requires: the attestation +//! scan ([`SortedBasis::attest_reads`]), the preparation seeks +//! ([`SetPlan::seeks`]) and the query's extents ([`Measured`]). +//! +//! Not here: joins, gathers, positional addressing across tables, preimage +//! (D-CMR-5), two-dimensional coordinates (D-CMR-4). + +use lance_graph_contract::set_algebra::{normalize, Cmp, Res, Runs, SetExpr}; +use lance_graph_mask_risc::{ + execute_extent, words_for, ExecError, Foreign, MaskOp, Operand, Out, Planes, Pred, Program, + Scratch, Terminal, Value, +}; + +/// Why a set could not be lowered or measured. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum LowerError { + /// The basis lane is not non-decreasing in row order; `row` is the first + /// row whose key is smaller than its predecessor's. + BasisNotSorted { row: usize }, + /// The basis has a different number of rows than the plan. + BasisRows { basis: usize, plan: usize }, + /// The planes span a different number of rows than the plan. + PlaneRows { planes: usize, plan: usize }, + /// The `Keep` destination is not `words_for(n_rows)` long. + OutLen { got: usize, want: usize }, + /// A `Sym` id does not fit a mask-risc plane index. + SymOutOfRange(u32), + /// The residual program needs more scratch slots than mask-risc addresses. + TooManySlots, + /// mask-risc refused the residual program. + Exec(ExecError), +} + +impl From for LowerError { + fn from(e: ExecError) -> Self { + LowerError::Exec(e) + } +} + +/// An attested sorted `i32` lane: keys are non-decreasing in row order, so a +/// comparison on this lane selects one contiguous row range (or, for `!=`, +/// at most two), found by binary search. +/// +/// Attestation is one sequential scan of the keys, counted in +/// [`SortedBasis::attest_reads`]; it is preparation cost, reported apart from +/// seeks and query reads. The keys must be the same values the planes carry +/// for `lane` (`Planes::lanes[lane]`), because a basis comparison that stays +/// in a residual (inside a mixed `Or`) is evaluated on the planes. +#[derive(Debug, Clone, Copy)] +pub struct SortedBasis<'a> { + lane: u16, + keys: &'a [i32], +} + +impl<'a> SortedBasis<'a> { + /// Attest `keys` as the sorted basis for `lane`, or refuse with the first + /// row out of order. + pub fn attest(lane: u16, keys: &'a [i32]) -> Result { + if let Some(i) = keys.windows(2).position(|w| w[1] < w[0]) { + return Err(LowerError::BasisNotSorted { row: i + 1 }); + } + Ok(SortedBasis { lane, keys }) + } + + /// The lane this basis describes. + pub fn lane(&self) -> u16 { + self.lane + } + + /// Keys read by the attestation scan (one per row). + pub fn attest_reads(&self) -> u64 { + self.keys.len() as u64 + } + + /// The rows where `key op t`, as runs. Every boundary is one binary + /// search (one seek, counted in `seeks`). + fn rows(&self, op: Cmp, t: i32, seeks: &mut u64) -> Runs { + let n = self.keys.len() as u64; + let mut seek = |strict: bool| { + *seeks += 1; + if strict { + self.keys.partition_point(|&k| k < t) as u64 + } else { + self.keys.partition_point(|&k| k <= t) as u64 + } + }; + match op { + Cmp::Lt => Runs::from_intervals([(0, seek(true))]), + Cmp::Ge => Runs::from_intervals([(seek(true), n)]), + Cmp::Le => Runs::from_intervals([(0, seek(false))]), + Cmp::Gt => Runs::from_intervals([(seek(false), n)]), + Cmp::Eq => { + let lo = seek(true); + Runs::from_intervals([(lo, seek(false))]) + } + Cmp::Ne => { + let lo = seek(true); + let hi = seek(false); + Runs::from_intervals([(0, lo), (hi, n)]) + } + } + } +} + +/// Which measurement level a plan uses for a measure. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum Level { + /// Answered from the runs alone; nothing is read. + Geometry, + /// The residual program runs over each run, producing a measure. + Optics, + /// The residual program runs over each run, writing the set. + Etch, +} + +/// A measure plus what producing it cost. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct Measured { + pub value: T, + pub level: Level, + /// Extents the residual program was executed over (0 at geometry). + pub extents: u64, + /// Rows those extents span in total (0 at geometry). + pub extent_rows: u64, +} + +/// A [`SetExpr`] split into `Runs ∩ residual` over a population of `n_rows`, +/// ready to measure against [`Planes`]. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct SetPlan { + n_rows: usize, + runs: Runs, + /// `None` when the runs alone are the set (no residual). + residual: Option, + seeks: u64, +} + +impl SetPlan { + /// Normalize `e` (the contract's [`normalize`]) and split it. Basis + /// comparisons become runs; the rest becomes the residual. Without a + /// basis the runs are the whole population. + pub fn new( + e: &SetExpr, + n_rows: usize, + basis: Option<&SortedBasis<'_>>, + ) -> Result { + if let Some(b) = basis { + if b.keys.len() != n_rows { + return Err(LowerError::BasisRows { + basis: b.keys.len(), + plan: n_rows, + }); + } + } + let n = normalize(e); + let mut seeks = 0; + let full = Runs::full(n_rows as u64); + let (runs, residual) = match exact_runs(&n, basis, n_rows, &mut seeks) { + Some(r) => (r, None), + None => match &n { + SetExpr::And(cs) => { + let mut runs = full; + let mut rest = Vec::new(); + for c in cs { + match exact_runs(c, basis, n_rows, &mut seeks) { + Some(r) => runs = runs.intersect(&r), + None => rest.push(c.clone()), + } + } + let residual = match rest.len() { + 0 => None, + 1 => rest.pop(), + _ => Some(SetExpr::And(rest)), + }; + (runs, residual) + } + other => (full, Some(other.clone())), + }, + }; + Ok(SetPlan { + n_rows, + runs, + residual, + seeks, + }) + } + + /// `e` with no split at all: one run over the whole population and the + /// normalized expression as the residual. The disable arm of the + /// geometry level; it must give the same answers. + pub fn unsplit(e: &SetExpr, n_rows: usize) -> SetPlan { + let n = normalize(e); + let (runs, residual) = match n { + SetExpr::Empty => (Runs::empty(), None), + SetExpr::All => (Runs::full(n_rows as u64), None), + other => (Runs::full(n_rows as u64), Some(other)), + }; + SetPlan { + n_rows, + runs, + residual, + seeks: 0, + } + } + + /// The runs the residual is evaluated over. + pub fn runs(&self) -> &Runs { + &self.runs + } + + /// The residual, if any. + pub fn residual(&self) -> Option<&SetExpr> { + self.residual.as_ref() + } + + /// Basis seeks spent building the runs (preparation cost). + pub fn seeks(&self) -> u64 { + self.seeks + } + + /// The level a measure (not `Keep`) is answered at. + pub fn measure_level(&self) -> Level { + if self.residual.is_none() || self.runs.is_empty() { + Level::Geometry + } else { + Level::Optics + } + } + + /// The number of selected rows. + pub fn count(&self, planes: &Planes<'_>) -> Result, LowerError> { + self.check_planes(planes)?; + let residual = match (&self.residual, self.runs.is_empty()) { + (Some(r), false) => r, + _ => { + return Ok(Measured { + value: self.runs.count(), + level: Level::Geometry, + extents: 0, + extent_rows: 0, + }) + } + }; + let program = compile( + residual, + Terminal::Count { + mask: Operand::Plane(0), + }, + )?; + let mut scratch = Scratch::for_program(&program, planes.n_rows)?; + let mut total = 0u64; + for &(lo, hi) in self.runs.intervals() { + let v = execute_extent( + &program, + planes, + &Foreign::NONE, + &mut scratch, + Out::None, + lo as usize..hi as usize, + )?; + if let Value::Count(c) = v { + total += c as u64; + } + } + Ok(Measured { + value: total, + level: Level::Optics, + extents: self.runs.intervals().len() as u64, + extent_rows: self.runs.count(), + }) + } + + /// Write the selected set into `out` (`words_for(n_rows)` words). Every + /// bit outside the selection is cleared first. + pub fn keep(&self, planes: &Planes<'_>, out: &mut [u64]) -> Result, LowerError> { + self.check_planes(planes)?; + let want = words_for(self.n_rows); + if out.len() != want { + return Err(LowerError::OutLen { + got: out.len(), + want, + }); + } + out.fill(0); + let residual = match (&self.residual, self.runs.is_empty()) { + (Some(r), false) => r, + _ => { + for &(lo, hi) in self.runs.intervals() { + set_bits(out, lo as usize, hi as usize); + } + return Ok(Measured { + value: (), + level: Level::Geometry, + extents: 0, + extent_rows: 0, + }); + } + }; + let program = compile( + residual, + Terminal::Keep { + mask: Operand::Plane(0), + }, + )?; + let mut scratch = Scratch::for_program(&program, planes.n_rows)?; + for &(lo, hi) in self.runs.intervals() { + execute_extent( + &program, + planes, + &Foreign::NONE, + &mut scratch, + Out::Mask(out), + lo as usize..hi as usize, + )?; + } + Ok(Measured { + value: (), + level: Level::Etch, + extents: self.runs.intervals().len() as u64, + extent_rows: self.runs.count(), + }) + } + + fn check_planes(&self, planes: &Planes<'_>) -> Result<(), LowerError> { + if planes.n_rows != self.n_rows { + return Err(LowerError::PlaneRows { + planes: planes.n_rows, + plan: self.n_rows, + }); + } + Ok(()) + } +} + +/// The runs `e` selects, when `e` is built only from comparisons on the +/// basis lane (and `All` / `Empty`); `None` otherwise. +fn exact_runs( + e: &SetExpr, + basis: Option<&SortedBasis<'_>>, + n_rows: usize, + seeks: &mut u64, +) -> Option { + match e { + SetExpr::All => Some(Runs::full(n_rows as u64)), + SetExpr::Empty => Some(Runs::empty()), + SetExpr::Res(Res::I32 { lane, op, t }) => { + let b = basis.filter(|b| b.lane == *lane)?; + Some(b.rows(*op, *t, seeks)) + } + SetExpr::And(cs) | SetExpr::Or(cs) => { + // Decide purity first, so a mixed junction spends no seeks. + if !cs.iter().all(|c| is_basis_only(c, basis)) { + return None; + } + let mut parts = cs + .iter() + .map(|c| exact_runs(c, basis, n_rows, seeks).expect("checked basis-only")); + let first = parts.next()?; + Some(match e { + SetExpr::And(_) => parts.fold(first, |a, b| a.intersect(&b)), + _ => parts.fold(first, |a, b| a.union(&b)), + }) + } + _ => None, + } +} + +fn is_basis_only(e: &SetExpr, basis: Option<&SortedBasis<'_>>) -> bool { + match e { + SetExpr::All | SetExpr::Empty => true, + SetExpr::Res(Res::I32 { lane, .. }) => basis.is_some_and(|b| b.lane == *lane), + SetExpr::And(cs) | SetExpr::Or(cs) => cs.iter().all(|c| is_basis_only(c, basis)), + _ => false, + } +} + +fn to_pred(r: Res) -> Pred { + match r { + Res::I32 { lane, op, t } => match op { + Cmp::Lt => Pred::LtI32 { lane, t }, + Cmp::Le => Pred::LeI32 { lane, t }, + Cmp::Gt => Pred::GtI32 { lane, t }, + Cmp::Ge => Pred::GeI32 { lane, t }, + Cmp::Eq => Pred::EqI32 { lane, v: t }, + Cmp::Ne => Pred::NeI32 { lane, v: t }, + }, + Res::U32 { lane, eq: true, v } => Pred::EqU32 { lane, v }, + Res::U32 { lane, eq: false, v } => Pred::NeU32 { lane, v }, + } +} + +/// Compile a residual into one program whose terminal reads its result. +fn compile(residual: &SetExpr, terminal: Terminal) -> Result { + let mut emit = Emit { + ops: Vec::new(), + next: 0, + }; + let result = emit.expr(residual)?; + let terminal = match terminal { + Terminal::Count { .. } => Terminal::Count { mask: result }, + Terminal::Keep { .. } => Terminal::Keep { mask: result }, + other => other, + }; + Ok(Program::new(emit.ops, terminal)) +} + +struct Emit { + ops: Vec, + next: u16, +} + +impl Emit { + fn slot(&mut self) -> Result { + let s = self.next; + self.next = self.next.checked_add(1).ok_or(LowerError::TooManySlots)?; + Ok(s) + } + + fn expr(&mut self, e: &SetExpr) -> Result { + Ok(match e { + SetExpr::Sym(i) => { + Operand::Plane(u16::try_from(*i).map_err(|_| LowerError::SymOutOfRange(*i))?) + } + SetExpr::Res(r) => { + let dst = self.slot()?; + self.ops.push(MaskOp::Pred { + pred: to_pred(*r), + under: None, + dst, + }); + Operand::Scratch(dst) + } + SetExpr::Not(x) => { + let a = self.expr(x)?; + let dst = self.slot()?; + self.ops.push(MaskOp::Not { a, dst }); + Operand::Scratch(dst) + } + SetExpr::And(cs) | SetExpr::Or(cs) => { + let and = matches!(e, SetExpr::And(_)); + let mut it = cs.iter(); + let mut acc = match it.next() { + Some(c) => self.expr(c)?, + None => return self.constant(and), + }; + for c in it { + let b = self.expr(c)?; + let dst = self.slot()?; + self.ops.push(if and { + MaskOp::And { a: acc, b, dst } + } else { + MaskOp::Or { a: acc, b, dst } + }); + acc = Operand::Scratch(dst); + } + acc + } + SetExpr::All => self.constant(true)?, + SetExpr::Empty => self.constant(false)?, + }) + } + + /// A constant mask, built as a contradiction or tautology over row 0's + /// range: `Range{0, 0}` is empty; its negation is everything. + fn constant(&mut self, all: bool) -> Result { + let dst = self.slot()?; + self.ops.push(MaskOp::Pred { + pred: Pred::Range { lo: 0, hi: 0 }, + under: None, + dst, + }); + if !all { + return Ok(Operand::Scratch(dst)); + } + let neg = self.slot()?; + self.ops.push(MaskOp::Not { + a: Operand::Scratch(dst), + dst: neg, + }); + Ok(Operand::Scratch(neg)) + } +} + +fn set_bits(out: &mut [u64], lo: usize, hi: usize) { + for r in lo..hi { + out[r / 64] |= 1u64 << (r % 64); + } +} + +#[cfg(test)] +mod tests { + use super::*; + use lance_graph_mask_risc::LaneRef; + + const N: usize = 1000; // not a multiple of 64 + + struct Fx { + basis: Vec, + other: Vec, + codes: Vec, + p0: Vec, + p1: Vec, + } + + fn fx(seed: u64) -> Fx { + let mut s = seed | 1; + let mut next = move || { + s ^= s << 13; + s ^= s >> 7; + s ^= s << 17; + s + }; + let mut basis: Vec = (0..N).map(|_| (next() % 50) as i32 - 10).collect(); + basis.sort_unstable(); + let other = (0..N).map(|_| (next() % 21) as i32 - 10).collect(); + let codes = (0..N).map(|_| (next() % 5) as u32).collect(); + let mut plane = |p: u64| { + let mut w = vec![0u64; words_for(N)]; + for r in 0..N { + if next() % p == 0 { + w[r / 64] |= 1 << (r % 64); + } + } + w + }; + let p0 = plane(3); + let p1 = plane(2); + Fx { + basis, + other, + codes, + p0, + p1, + } + } + + // lane 0 = basis (sorted i32), lane 1 = other i32, lane 2 = u32 codes; + // masks 0, 1. + fn with_planes(f: &Fx, k: impl FnOnce(&Planes<'_>) -> T) -> T { + let lanes = [ + LaneRef::I32(&f.basis), + LaneRef::I32(&f.other), + LaneRef::U32(&f.codes), + ]; + let masks: [&[u64]; 2] = [&f.p0, &f.p1]; + k(&Planes { + n_rows: N, + masks: &masks, + lanes: &lanes, + }) + } + + /// The row oracle: evaluates the expression row by row, by definition. + fn holds(f: &Fx, e: &SetExpr, r: usize) -> bool { + match e { + SetExpr::All => true, + SetExpr::Empty => false, + SetExpr::Sym(i) => { + let p = if *i == 0 { &f.p0 } else { &f.p1 }; + p[r / 64] >> (r % 64) & 1 == 1 + } + SetExpr::Res(Res::I32 { lane, op, t }) => { + let x = if *lane == 0 { f.basis[r] } else { f.other[r] }; + match op { + Cmp::Lt => x < *t, + Cmp::Le => x <= *t, + Cmp::Gt => x > *t, + Cmp::Ge => x >= *t, + Cmp::Eq => x == *t, + Cmp::Ne => x != *t, + } + } + SetExpr::Res(Res::U32 { eq, v, .. }) => (f.codes[r] == *v) == *eq, + SetExpr::Not(c) => !holds(f, c, r), + SetExpr::And(cs) => cs.iter().all(|c| holds(f, c, r)), + SetExpr::Or(cs) => cs.iter().any(|c| holds(f, c, r)), + } + } + + fn oracle(f: &Fx, e: &SetExpr) -> (u64, Vec) { + let mut w = vec![0u64; words_for(N)]; + let mut c = 0; + for r in 0..N { + if holds(f, e, r) { + c += 1; + w[r / 64] |= 1 << (r % 64); + } + } + (c, w) + } + + struct Gen(u64); + impl Gen { + fn next(&mut self) -> u64 { + self.0 = self + .0 + .wrapping_mul(6364136223846793005) + .wrapping_add(1442695040888963407); + self.0 >> 33 + } + fn leaf(&mut self) -> SetExpr { + const OPS: [Cmp; 6] = [Cmp::Lt, Cmp::Le, Cmp::Gt, Cmp::Ge, Cmp::Eq, Cmp::Ne]; + match self.next() % 7 { + 0..=2 => SetExpr::i32( + 0, + OPS[(self.next() % 6) as usize], + (self.next() % 60) as i32 - 15, + ), + 3 => SetExpr::i32( + 1, + OPS[(self.next() % 6) as usize], + (self.next() % 25) as i32 - 12, + ), + 4 => SetExpr::Res(Res::U32 { + lane: 2, + eq: self.next().is_multiple_of(2), + v: (self.next() % 5) as u32, + }), + _ => SetExpr::Sym((self.next() % 2) as u32), + } + } + fn expr(&mut self, depth: u32) -> SetExpr { + if depth == 0 || self.next().is_multiple_of(3) { + return self.leaf(); + } + match self.next() % 4 { + 0 => SetExpr::negated(self.expr(depth - 1)), + 1 | 2 => SetExpr::And( + (0..1 + self.next() % 3) + .map(|_| self.expr(depth - 1)) + .collect(), + ), + _ => SetExpr::Or( + (0..1 + self.next() % 3) + .map(|_| self.expr(depth - 1)) + .collect(), + ), + } + } + } + + /// FAILS IF: any level gives a different count or set than the row + /// oracle, or the unsplit plan (geometry disabled) disagrees. Random + /// nested expressions over a sorted basis lane, an unsorted lane, a code + /// lane and two planes, N not a word multiple. + #[test] + fn every_level_equals_the_row_oracle() { + let mut g = Gen(0xD2C2); + let (mut geometry, mut optics, mut split_rows_saved) = (0, 0, 0u64); + for k in 0..400 { + let f = fx(1 + k % 5); + let e = g.expr(4); + let (want_c, want_w) = oracle(&f, &e); + let basis = SortedBasis::attest(0, &f.basis).unwrap(); + let plan = SetPlan::new(&e, N, Some(&basis)).unwrap(); + let flat = SetPlan::unsplit(&e, N); + with_planes(&f, |p| { + let c = plan.count(p).unwrap(); + assert_eq!(c.value, want_c, "{e:?}"); + assert_eq!(flat.count(p).unwrap().value, want_c, "unsplit {e:?}"); + let mut w = vec![!0u64; words_for(N)]; + plan.keep(p, &mut w).unwrap(); + assert_eq!(w, want_w, "keep {e:?}"); + let mut w2 = vec![!0u64; words_for(N)]; + flat.keep(p, &mut w2).unwrap(); + assert_eq!(w2, want_w, "unsplit keep {e:?}"); + match c.level { + Level::Geometry => geometry += 1, + _ => optics += 1, + } + let fc = flat.count(p).unwrap(); + split_rows_saved += fc.extent_rows.saturating_sub(c.extent_rows); + }); + } + // Anti-vacuity: both levels are exercised, and the runs really + // narrow the extents somewhere. + assert!( + geometry > 20 && optics > 20, + "geometry {geometry}, optics {optics}" + ); + assert!(split_rows_saved > 0); + } + + /// FAILS IF: a set with no residual reads anything. The planes carry no + /// lanes and no masks at all, so any program run would be refused; the + /// geometry level must answer without one. + #[test] + fn geometry_reads_no_plane_lane_or_extent() { + let f = fx(7); + let basis = SortedBasis::attest(0, &f.basis).unwrap(); + let e = SetExpr::Or(vec![ + SetExpr::And(vec![ + SetExpr::i32(0, Cmp::Ge, 0), + SetExpr::i32(0, Cmp::Lt, 10), + ]), + SetExpr::i32(0, Cmp::Gt, 30), + ]); + let plan = SetPlan::new(&e, N, Some(&basis)).unwrap(); + assert!(plan.residual().is_none()); + assert!(plan.runs().intervals().len() >= 2, "a multi-run set"); + let empty = Planes { + n_rows: N, + masks: &[], + lanes: &[], + }; + let c = plan.count(&empty).unwrap(); + assert_eq!(c.level, Level::Geometry); + assert_eq!((c.extents, c.extent_rows), (0, 0)); + assert_eq!(c.value, oracle(&f, &e).0); + let mut w = vec![0u64; words_for(N)]; + assert_eq!(plan.keep(&empty, &mut w).unwrap().level, Level::Geometry); + assert_eq!(w, oracle(&f, &e).1); + // Preparation: one seek per boundary, never a scan. + assert!(plan.seeks() <= 6, "{} seeks", plan.seeks()); + assert_eq!(basis.attest_reads(), N as u64); + } + + /// Experiment C (plan §7). FAILS IF: a sorted-basis range plus an + /// unsorted predicate does not become runs plus a residual, or the + /// residual runs over more rows than the range. + #[test] + fn a_range_plus_an_unsorted_predicate_splits_into_runs_and_residual() { + let f = fx(3); + let basis = SortedBasis::attest(0, &f.basis).unwrap(); + let e = SetExpr::And(vec![ + SetExpr::i32(0, Cmp::Ge, 5), + SetExpr::i32(0, Cmp::Le, 20), + SetExpr::i32(1, Cmp::Gt, 0), + ]); + let plan = SetPlan::new(&e, N, Some(&basis)).unwrap(); + assert_eq!(plan.residual(), Some(&SetExpr::i32(1, Cmp::Gt, 0))); + assert_eq!(plan.runs().intervals().len(), 1); + with_planes(&f, |p| { + let c = plan.count(p).unwrap(); + assert_eq!(c.value, oracle(&f, &e).0); + assert_eq!(c.level, Level::Optics); + assert_eq!(c.extent_rows, plan.runs().count()); + assert!(c.extent_rows < N as u64 / 2, "{} rows", c.extent_rows); + }); + } + + /// Silence twin. FAILS IF: comparisons on a lane that is not the basis + /// are turned into runs. With the basis on lane 0, a lane-1 range stays + /// residual and the runs are the whole population. + #[test] + fn a_non_basis_range_stays_residual() { + let f = fx(4); + let basis = SortedBasis::attest(0, &f.basis).unwrap(); + let e = SetExpr::And(vec![ + SetExpr::i32(1, Cmp::Ge, 0), + SetExpr::i32(1, Cmp::Lt, 5), + ]); + let plan = SetPlan::new(&e, N, Some(&basis)).unwrap(); + assert_eq!(plan.runs(), &Runs::full(N as u64)); + assert!(plan.residual().is_some()); + assert_eq!(plan.seeks(), 0); + } + + /// FAILS IF: an unsorted lane is accepted as a basis. + #[test] + fn an_unsorted_lane_is_refused_as_basis() { + let f = fx(5); + assert!(f.other.windows(2).any(|w| w[1] < w[0])); + assert!(matches!( + SortedBasis::attest(1, &f.other), + Err(LowerError::BasisNotSorted { .. }) + )); + } + + /// FAILS IF: a mismatched population is measured instead of refused. + #[test] + fn row_count_mismatches_are_refused() { + let f = fx(6); + let basis = SortedBasis::attest(0, &f.basis).unwrap(); + assert!(matches!( + SetPlan::new(&SetExpr::All, N - 1, Some(&basis)), + Err(LowerError::BasisRows { .. }) + )); + let plan = SetPlan::new(&SetExpr::Sym(0), N, Some(&basis)).unwrap(); + let wrong = Planes { + n_rows: N + 1, + masks: &[], + lanes: &[], + }; + assert!(matches!( + plan.count(&wrong), + Err(LowerError::PlaneRows { .. }) + )); + with_planes(&f, |p| { + let mut short = vec![0u64; 3]; + assert!(matches!( + plan.keep(p, &mut short), + Err(LowerError::OutLen { .. }) + )); + }); + } +}