diff --git a/.claude/board/entries/2026-11-03-d-global-where-p2.md b/.claude/board/entries/2026-11-03-d-global-where-p2.md new file mode 100644 index 000000000..b604a0b64 --- /dev/null +++ b/.claude/board/entries/2026-11-03-d-global-where-p2.md @@ -0,0 +1,38 @@ +# D-GLOBAL-WHERE-P2: criteria-aware WHERE, DR composition, Lance selective read, three parity verdicts + +**Status:** MEASURED, lab only. No production change. +**Probe:** `.claude/probes/2026-11-03-d-global-where-p1/` on `main` holds the docs. The probe code, DuckDB scripts and raw results are on the research branch `research/d-global-where-p2` (commit `1422b2b92`). Read `REPORT.md` for numbers and verdicts, `ENGINEERING-PARITY.md` for the literature matrix, and `LANCE-TRACE.md` for the source-backed caller path. The Lance arm lives in `crates/crepe-plan-probe/examples/where_lance_arm.rs`. + +**Semantic parity: reached.** +- Every count and EXISTS answer equals both the oracle and DuckDB 2.0.0, at 1M and 10M, threads 1 and 4. +- Every Lance arm (full, allow/block prefilter, attested range read) equals the oracle and the full-projection oracle at 1M and 10M. +- ClassView → physical column resolution goes through `predicate_iri`. It refuses AbsentField, Unauthorized, UnknownProperty, AmbiguousProperty and StaleSchema, each paired with a success control. +- Two traps prove a shortcut would be caught: an ordinal shortcut is wrong on 16/16 positions, and a label shortcut finds no column on 16/16. + +**Performance parity: split.** +- Count beats DuckDB by 14.2× (1M, 1 thread) down to 1.96× (10M, 4 threads), weighted 34/46/14/6. +- EXISTS loses at 10M, 0.43× and 0.16×. `Terminal::Any` has no per-tile early exit (`exec.rs:1793`). +- Lance at 1M: a clustered selection reads 0.1–9% of the full scan's bytes. +- A scattered 0.1% selection already reads 33% of the bytes at k=1 and 99.8% at k=16. At 10–50% the masked read is 1.4–11× slower than a full scan, and `ReadPolicy::DEFAULT` already falls back to a full read there. +- At 10M a clustered 0.1% read is 112× faster than the full scan. +- `ReadPolicy::DEFAULT` decides on covered rows alone, so it also sends a single contiguous 10% range to a full scan: 3.8× slower at 1M, 4.6× at 10M. Range count, not coverage, separates clustered from scattered demand. + +**Engineering parity: reached on representation, not on strategy.** +- Missing wiring: EXISTS early exit, and routing once instead of per query (break-even at Q ≥ 2; 158,198 → 1,012 µs per query at Q=256). +- Missing capability: choosing evaluation direction, and an ordered u8-in-u32 compare word. Composing existing words costs one pass per 2-bit prefix, up to 5× slower than a hand loop. + +**Costs measured, never assumed free.** +- RowAddrMask ingress is a copy, and scales with ranges, not rows: 0.4–5.4 µs for 1 range; about 4,600 µs and 126 KB for 91,527 scattered ranges; 123,162 µs for 3.26M ranges. One tree is built per read, and its size is measured outside the timer. The first run built two trees per C read and doubled those figures (Codex finding on #1950); the corrected numbers are the ones recorded. +- Lane routing costs about 70–90 ns per id. + +**Boundary:** `BLOCKED_D4`. No witness binds a DR Frame to a Lance version. Arm C takes its ids as given and composes no DR ordinal with `_rowid`. No third coordinate map was built. + +**Found:** Lance 13 truncates `max_rows_per_file` with `as u32` (`write.rs:1077`). `1 << 40` became 0 and produced one fragment per batch. #1633's NotContiguous guard caught it at 10M. + +**OPEN:** +- S-INGRESS / D4: one version-checked DR → `_rowid` authority (owner decision). +- EXISTS early break (mask-risc owner). +- The ordered u8 compare kernel (ndarray / mask-risc owner). +- Why the borrowed lane read is 2.3–2.5× slower at 50% selectivity. +- `plan_read` should weigh the range count (lab #1633 owner). +- Every Lance byte count was served from the page cache, not disk. diff --git a/.claude/board/entries/README.md b/.claude/board/entries/README.md index c9a7d735b..73ec68041 100644 --- a/.claude/board/entries/README.md +++ b/.claude/board/entries/README.md @@ -25,13 +25,14 @@ index row, (3) no duplicate entry id. Checks 1 and 2 are deliberately opposite directions; the stranding this convention prevents shows up in exactly one of them, never both. -567 entries, 2026-08-06 .. 2026-11-03. +569 entries, 2026-08-06 .. 2026-11-03. | date | entry id | finding | file | |---|---|---|---| | 2026-11-03 | `D-XLOG-META-WIRE-P0` | | [2026-11-03-d-xlog-meta-wire-p0.md](2026-11-03-d-xlog-meta-wire-p0.md) | | 2026-11-03 | `D-GRAMMAR-WHEN-1` | | [2026-11-03-d-grammar-when-1.md](2026-11-03-d-grammar-when-1.md) | | 2026-11-03 | `D-GRAMMAR-CONSTRAINTS-0` | | [2026-11-03-d-grammar-constraints-0.md](2026-11-03-d-grammar-constraints-0.md) | +| 2026-11-03 | `D-GLOBAL-WHERE-P2` | | [2026-11-03-d-global-where-p2.md](2026-11-03-d-global-where-p2.md) | | 2026-11-03 | `D-GLOBAL-WHERE` | | [2026-11-03-d-global-where-p0.md](2026-11-03-d-global-where-p0.md) | | 2026-11-03 | `D-DRAM-0` | | [2026-11-03-d-dram-abacus-address-memoization.md](2026-11-03-d-dram-abacus-address-memoization.md) | | 2026-11-03 | `D-DR-FOLD-MAP` | | [2026-11-03-d-dr-fold-map.md](2026-11-03-d-dr-fold-map.md) | diff --git a/.claude/probes/2026-11-03-d-global-where-p1/ENGINEERING-PARITY.md b/.claude/probes/2026-11-03-d-global-where-p1/ENGINEERING-PARITY.md new file mode 100644 index 000000000..ebfc25f27 --- /dev/null +++ b/.claude/probes/2026-11-03-d-global-where-p1/ENGINEERING-PARITY.md @@ -0,0 +1,170 @@ +# Engineering parity: DuckDB literature against DR / Quack / mask-risc / Lance 13 + +> Paths under `probe/`, `duckdb/` and `results/` refer to the research branch [`research/d-global-where-p2`](https://github.com/AdaWorldAPI/lance-graph/tree/research/d-global-where-p2/.claude/probes/2026-11-03-d-global-where-p1) (commit `1422b2b92`); they are not on `main`. + +Date: 2026-11-03. Lab research, READ-ONLY. This replaces the earlier draft (scope change: literature first). + +Grades: +- **VERIFIED-IN-CODE:** I read the file:line myself. +- **CROSSWALK:** taken from the #1788 crosswalk, `.claude/plans/duckdb-dr-quack-machine-room-reuse-v1.md` Appendix A, and not re-read. +- **CLAIMED:** a publication says so; not checked in source. +- **UNVERIFIED:** I could not check it. + +DuckDB C++ source is **not on disk** (`selection_vector.hpp` not found). DuckDB symbols are therefore CROSSWALK at @1641475. + +## Sources actually read + +| # | publication | status | what I read | +|---|---|---|---| +| P1 | DuckDB docs, "Vector" internals (`/internals/vector` redirects to `docs/2.0/internals/vector.html`) | **SHIPPED-FEATURE** (2.0 docs) | the page | +| P2 | "Optimizers: The Low-Key MVP" (blog, 2024-11-14) | **SHIPPED-FEATURE** (blog about the shipped optimizer) | the page | +| P3 | Liu et al., *Selective Late Materialization*, PVLDB 18(11):4616–4628 | **PUBLISHED-RESEARCH-PROTOTYPE** | the PDF, §1–§2.3 and §4.4–§5.1 (pp. 4616–4621) | +| P4 | Qiao & Zhang, *Data Chunk Compaction*, SIGMOD 2025 | **PUBLISHED-RESEARCH-PROTOTYPE** (shipping version UNVERIFIED) | library-page abstract only | + +Notes on P3 and P4: +- P3's implementation is the fork `github.com/yhliu918/duckdb@latest` on DuckDB v1.1.0 (p. 4616, §5.1.2). +- Shipped DuckDB does have a *different* late-materialization rule, `optimizer/late_materialization.cpp:159` `TryLateMaterialization` (CROSSWALK A.2 #15). It must not stand in for P3. +- P4's PDF returned 403 from ACM and from the authors' site. Its mechanism details are UNVERIFIED. + +## Engineering-assimilation matrix (one row per principle) + +### 1. Reusable selection (P1; P3 §2.2) + +- **(a) Problem.** Avoid copying survivors after a filter. +- **(b) DuckDB mechanism and lifetime.** DuckDB keeps a selection vector or dictionary over a child vector. P3 §2.2 says the selection vector holds "DataChunk-local row IDs", shared by columns. Its lifetime is **one chunk** (2,048 rows, `STANDARD_VECTOR_SIZE`, P1). P1 does not state ownership; that part is UNVERIFIED. +- **(c) Our equivalent.** + - Packed `u64` masks plus the sealed `u16` map (CROSSWALK A.2 #1/#2). + - The DRF lease, lifetime **one version × scope × seal** (CROSSWALK A.1). + - mask-risc tiles, `TILE_WORDS = 256` (`mask-risc/src/exec.rs:110`). +- **(d) Status.** ALREADY WORKS, and lives longer: it is reused across queries, not rebuilt per chunk. There is no `Vec` by design (CROSSWALK #1). +- **(e) Falsifier.** D-DR-FOLD part B (`REPORT-BENCH.md`, "B: amortization"): + - the split policy reaches near-hot cost at Q≈256 queries per generation; + - witnessing a generation costs 502–758 µs. + - If generations churn faster than about 1/Q, the reuse advantage disappears. + +### 2. Late materialization (P3) + +- **(a) Problem.** Avoid copying a payload through joins and filters when few rows survive. P3 §2.2–2.3: in modern engines the only remaining cost is replacing virtual IDs with values. Plain late materialization (LM) is no longer a win on average (§1). +- **(b) DuckDB mechanism and lifetime.** P3 makes the choice **per attribute, per query, at plan time**. For each attribute it picks one point between the scan and first use (Alg. 1, §3), using a trained cost model of fetch cost plus memory-copy cost. It carries row IDs and `disable_column` until that point. It fetches in batches at row-group granularity (120K rows) (§4.5). +- **(c) Our equivalent.** + - Lance projection plus `with_row_addr_prefilter` (CROSSWALK A.3). + - DRF MAP / BORROW / GATHER (CROSSWALK A.4). + - There is no hash join carrying a row payload (CROSSWALK #14). +- **(d) Status.** INTENTIONALLY DIFFERENT. P3's biggest win, a smaller hash-join payload (§2.2, Fig. 2), has no target here. What remains missing is the **cost-based choice** between BORROW and GATHER and the masked Lance fetch. That choice is today a measured crossover (2 folds, DRF-P2), not a planner decision. +- **(e) Falsifier.** + - H14c: a uniform 1 % mask reads 1.00× the bytes; a clustered one reads 0.24× (CROSSWALK A.3). This is P3's "out-of-order probing" in storage form. + - P3 §5: LM wins only at high selectivity or with long payloads. + +### 3. Evaluation direction / join order (P2) + +- **(a) Problem.** Avoid huge intermediate results. + - P2: an unoptimized join produced about 5·10¹² intermediate rows; the optimized one ran in 0.769 s. + - P2: join filter pushdown, i.e. pushing the build side's min/max into the probe scan, gave about 10×. +- **(b) DuckDB mechanism and lifetime.** Join order is chosen **per query** from table statistics. P2 does not describe the build-side rule (UNVERIFIED). +- **(c) Our equivalent.** + - Frontier masks vs CSR vs a d-anchored gather, all as lab arms (P0 `REPORT.md:115,122`; D-DR-FOLD). + - There is no chooser in Quack (D-DR-FOLD Recommendation 1: "a lowering that chooses evaluation direction … from presence density"). +- **(d) Status.** **MISSING CAPABILITY (strategy, not kernel).** DR does have presence density as a versioned, popcount-cheap fact. That is a better input than a sampled estimate. +- **(e) Falsifiers.** + - P0, `multi_skewed` W4: DuckDB produced 3,726,779 join rows for an answer of 1,164 (`REPORT.md:99`). + - D-DR-FOLD: top-down frontier won 54/54 sweep cells against the anchored gather. + - P0: `csr` loses 2–14× on dense data (`REPORT.md:115,122`). Picking the wrong direction is the cost. + +### 4. Short-circuit for EXISTS (P2 filter pushdown and LIMIT; the P2-probe evidence) + +- **(a) Problem.** Stop as soon as the answer is known. +- **(b) DuckDB mechanism and lifetime.** EXISTS ran in about 900–2,700 µs, flat in K (`results/where/n1000000/compare_where.md:25-51`). This suggests a scan that stops at the first qualifying chunk. **Lifetime: per query.** The mechanism is CLAIMED-from-timing; I did not read the operator. +- **(c) Our equivalent.** + - `Terminal::Any` (`mask-risc/src/ir.rs:292`). + - The executor loops over every tile (`exec.rs:1632`) and does `any |= mask_any(..)` with **no break** (`exec.rs:1793-1794`). VERIFIED-IN-CODE: the loop and the OR; I did not trace every Any path. + - ndarray `mask_any` has no early exit (local `ndarray/src/simd_masking_ops.rs:4864-4870`; this is gap G8, CROSSWALK). +- **(d) Status.** **MISSING WIRING.** It is one break per tile, after the first tile with a set bit. No new capability is needed. +- **(e) Falsifier.** P2 probe, `results/where/n1000000/compare_where.md`: + - At K ≥ 8 our `any` costs about the same as our `count` (1,400–3,900 µs), while DuckDB EXISTS costs 880–2,700 µs. + - 17 LOSS rows, e.g. `k24_large_and` at a ratio of 0.36. + - After the break, `any` should fall with the position of the first hit, not with N. + +### 5. Physical-read avoidance (P2 filter pushdown and statistics propagation; CROSSWALK #12; Lance 13) + +- **(a) Problem.** Avoid reading blocks or rows that cannot qualify. +- **(b) DuckDB mechanism and lifetime.** + - Filters move into the scan. DuckDB's own count plan shows `"Projections": ""` with 8 pushed filters (P2-probe `results/where/n1000000/explain/k8_medium_and.txt`). + - Zone maps are **per row group, persisted** (`RowGroup::CheckZonemap`, CROSSWALK #12). + - Statistics folding answers `c0<1000 OR c3<5` with no operator at all (`duckdb/bench_where.py:98-101`). + - P2 never mentions zone maps. +- **(c) Our equivalent.** + - Lance prunes storage, at a chunk of ≤4,096 values (mini-block) (CROSSWALK A.3). + - `with_row_addr_prefilter` applies a DR mask to the read. + - Resident data needs no pruning: it is already in memory. +- **(d) Status.** ALREADY WORKS (storage), with **MISSING WIRING**: + - one coordinate→`_rowid` authority, plus bulk word→`RowAddrTreeMap` (CROSSWALK A.5; S-INGRESS OPEN); + - statistics folding for tautologies or contradictions: the case `adv_contradictory_17` lands at PARITY rather than a win (`results/where/n1000000/compare_where.md:61`). +- **(e) Falsifier.** + - H14d: an empty R reads 0 B; clustered 1 % reads 0.19×; uniform 1 % reads 1.00× (CROSSWALK A.3). + - The integrated DR64→Lance→Quack path is **unmeasured**. + +### 6. Chunk compaction (P4) + +- **(a) Problem.** After a selection or a join, chunks become sparse and vector loops run underfull. +- **(b) DuckDB mechanism and lifetime.** Compaction is triggered per chunk, with a threshold learned at run time. "Logical compaction" in hash joins minimizes data movement (abstract only). The article reports up to 63 % speedup. +- **(c) Our equivalent.** + - A mask fold costs per word, not per surviving row, so a sparse mask does not run underfull. It costs N/64 words regardless of density. + - `materialize_rows` is egress only (CROSSWALK #16). +- **(d) Status.** INTENTIONALLY DIFFERENT: the representation removes the problem. The open question is the sparse end. With very few survivors, whole-word passes are wasted: D-DR-FOLD found "A's per-row lane term" and that C wins when "the chain dies early". That remedy is item 3, not compaction. +- **(e) Falsifier.** At very low survivor density, a fold of K predicates costs K·N/64 words while survivors≪N/64. The `tiny` rows in `results/where/n1000000/compare_where.md` show our count flat in population (e.g. k24 tiny 3,800 µs vs large 4,200 µs). Prediction: compaction would not help that; switching direction (item 3) would. + +### Routing (P2-probe composition) + +- **Evidence.** The `route` column dominates: for example, 35,914 µs out of a 50,178 µs total at sel=50,000, D=1 (`results/comp_both.log`, N=1M table). +- **DuckDB.** It builds a per-query hash table, which it does not reuse across queries (UNVERIFIED for shipped DuckDB). +- **Ours.** + - Arm C, the derived sorted `(global, coord)` index with a `StaleIndex` refusal (SPEC.md:38-49). + - Status: **MISSING WIRING**, not MISSING CAPABILITY. + - Its reuse lifetime is the frame (one generation), which is longer than DuckDB's single query. + +## WHEN vs WHETHER + +DuckDB, both shipped and in P3, optimizes **when** a property is recalled. It already avoids **whether** in one case: COUNT on pushed filters projects nothing (`"Projections": ""`). It still has to read every *filter* column on every query. + +We can skip the recall entirely where the answer is a fact about presence, not about values: + +1. **COUNT / EXISTS / ALL over an already-bound DR mask.** For a predicate that is already a versioned plane, there is no value read at all. `Terminal::Count`/`Any` read a mask only (`ir.rs:290-292`). DRF's MAP-only path holds no value (CROSSWALK A.4, path 1). DuckDB re-scans the filter columns every query (`rows_scanned: 1000000` in the k8 EXPLAIN). The gain lasts for as long as the plane's generation does. +2. **Field-presence questions.** `FieldMask`/`WideFieldMask` are "pure presence bits" (`contract/src/class_view.rs:60-70`, `:243`), so the question "has property p" never touches p's value. +3. **Empty or contradictory demand.** An empty R reads 0 B from Lance (H14d). If an empty mask skipped `with_row_addr_prefilter` (G8), we would never open the storage at all. +4. **DISTINCT over a sealed map.** DISTINCT is the population of the distinct ordinals in the domain mask (CROSSWALK A.1, `membership.rs:103-125`); no value is needed. + +We **cannot** skip it: + +- **The first evaluation of a value predicate.** The plane has to be computed once; that is the P2 `discover` scan, N-bound by design (`comp_both.log`). +- **SUM / MIN / GROUP BY on a payload.** These need BORROW or GATHER. +- **Any non-resident source.** It needs a Lance masked read with projection, so the cost is bytes per chunk touched (H14c), not density. +- **After a generation change.** Witness or rebuild: 502–758 µs per generation (D-DR-FOLD B). + +**Falsifier for WHETHER.** Re-run the 17+ K-group `count` cases with the K predicate planes pre-bound at the same version (a resident plane per criterion). The cost should drop to K·N/64 word ANDs plus a popcount, with no lane reads, and stay flat across repeated queries. DuckDB's cost should stay per query: `rows_scanned` equals N each time. If ours does not stay flat across repeats, the "versioned fact" claim fails. + +## Ranked assimilation candidates (producer → missing edge → consumer → test → owner gate) + +1. `Terminal::Any` → per-tile early break (`exec.rs:1793`) → Quack EXISTS → the K≥8 EXISTS cases in `results/where/*/compare_where.md`, plus a can-stay-silent test on an empty mask → mask-risc owner. +2. Presence density (popcount of resident planes) → choice of evaluation direction in the lowering → Quack multi-hop / W4 → P0 `multi_skewed`/`multi_dense` W4 + the D-DR-FOLD sweep → Quack owner (D-DR-FOLD D1 ruling). +3. P1 Arm C derived index → a frame-scoped binding reused across queries → composition / route stage → `comp_both.log` route column at sel 50k → DRF/D-GLOBAL-WHERE owner. +4. DR mask → one coordinate→`_rowid` authority plus bulk `RowAddrTreeMap` → Lance masked fetch → H14d cells + the integrated path (unmeasured) → Lance-seam owner, gated on the S-INGRESS ruling. +5. Plane statistics (empty/full per tile) → constant-fold of tautologies and contradictions at bind time → Quack → `adv_contradictory_17` (PARITY today) → Quack owner. Rejected as a resident zone map (CROSSWALK #12). + +## Draft verdict + +**Engineering parity is reached by representation on selection reuse and on compaction.** A versioned DR mask outlives DuckDB's per-chunk selection vector, and word folds remove the problem of underfull chunks. + +**It is not reached on strategy.** DuckDB's per-query optimizer chooses join order, pushes filters into scans, folds statistics and short-circuits EXISTS. Every P2 benchmark loss maps to one of these, never to a kernel: +- EXISTS at K≥8 (no break, `exec.rs:1793`); +- W4 wrong-direction arms; +- the route stage. + +The assimilation is to feed DR's *already-versioned* facts (presence density, an empty mask, a bound plane) into decisions the lowering does not yet make. DuckDB's representations are not to be copied. P3's per-attribute late materialization is a research prototype on a v1.1.0 fork. Its main payoff, a smaller hash-join payload, has no target here. + +## Not verified + +- P4's mechanism beyond the abstract. +- The build-side rule and EXISTS operator in DuckDB (no source on disk). +- Whether shipped DuckDB includes P4. +- Every `Terminal::Any` code path, beyond the loop I read. +- The integrated DR→Lance read. diff --git a/.claude/probes/2026-11-03-d-global-where-p1/LANCE-TRACE.md b/.claude/probes/2026-11-03-d-global-where-p1/LANCE-TRACE.md new file mode 100644 index 000000000..82985090d --- /dev/null +++ b/.claude/probes/2026-11-03-d-global-where-p1/LANCE-TRACE.md @@ -0,0 +1,112 @@ +# Lance 13 selective-read trace — D-GLOBAL-WHERE vs established contracts + +> Paths under `probe/`, `duckdb/` and `results/` refer to the research branch [`research/d-global-where-p2`](https://github.com/AdaWorldAPI/lance-graph/tree/research/d-global-where-p2/.claude/probes/2026-11-03-d-global-where-p1) (commit `1422b2b92`); they are not on `main`. + +Repo `/home/user/lg-p1`, branch `ccr-85acaff0-oszq40-global-where-p1` (HEAD `3fb41149d`). Read-only. No cargo run. Labels: **VERIFIED-IN-CODE** (read first-hand, file:line), **VERIFIED-IN-DOC** (read in a committed plan/entry/PR body), **HYPOTHESIS**, **OPEN**. + +## Step 1 — established contracts + +| PR | established | evidence | +|---|---|---| +| #1489 (D-TELEPORT-2) | Lance 13's own scanner/index paths prune source reads; "No new primitive is needed"; remaining seam = "a cost choice between full scan and index take". Shuffled 50 % B-tree is ~9× slower than full scan (scattered take). Index-only COUNT reads no data columns. | PR body (MCP); `crates/lance-graph-benches/examples/lance_source_probe.rs:1-41` | +| #1573 (D-HOLO H14c) | `with_row_addr_prefilter` reads; `CoordMap` (coord→`_rowid` at one version) + `ReadError::StaleMapping`. Uniform 1 %: bytes 1.00×, rows scanned 692/65,536; clustered 1 %: 0.24×; MaterializationStyle changed bytes in 0/120 cells. Stale map → silent omission; stable ids survive update+compaction, not overwrite. | PR body; `crates/holo-lance-masked-io-probe/src/main.rs:146-218`; `.claude/board/entries/2026-10-15-d-holo-h14c-thought-first.md:17-19` | +| #1575 (D-HOLO H14d) | `logic → address mask R (mask-risc) → with_row_addr_prefilter(R) → terminal`. Rows scanned == \|R\| in every cell. Bytes: uniform 1 % 1.00, clustered 1 % 0.19, clustered 10 % 0.99, apertures 0.01–0.12. "Bytes are page-bound, so the saving needs survivors that are page-local." | PR body; `crates/holo-h14d-probe/src/main.rs:449-533,752-770` | +| #1633 (D-SRAM-2) | Stage 3 `lance_store` reads each side under a relalg-built mask; Arm B `addressed` = version-attested ordered lane (`SealedFacetLane::attest_sorted`), ranges via `seek_from`, `insert_range` masks, refusals `Stale`/`OtherDataset`/`NotSorted`/`NotContiguous`/`NotLeadingColumn`. Arm B reads exactly `3k` rows; 2.5–12× faster ≤1 % centers; full scan wins from 10 %. | PR body; `crates/crepe-plan-probe/src/lance_store.rs`; `crates/crepe-plan-probe/src/addressed.rs:1-330` | +| #1788 (D-DQM crosswalk, local `10291a8cc`) | Coordinate domains are four spaces; exactly one version-checked coord→`_rowid` authority required; S-INGRESS open. | `.claude/plans/duckdb-dr-quack-machine-room-reuse-v1.md:146-330` | +| #1789 (D-DQM P1, local `ab1f3a83f`) | Chunk-touched predictor FAILS (8.8 % / 16.4 % cells > 0.10; clustered 1 % predicted 0.44 vs 0.24/0.19 measured; over-predicts 59/73). No estimator proposed. | PR body (MCP) | + +### Verbatim decisions (#1788, plan file) + +- **S-INGRESS** (plan `:256-260`): "**Copy status of the handoff (seam S-INGRESS, OPEN, operator decision).** The words → `RowAddrTreeMap` step re-encodes an elected `Keep` set into a container Lance owns, and Lance copies it twice more. The waben ruling (`waben-fold-execution-loop-v1.md:1967-1979`, "where a set IS elected … the container is a free choice") covers our own carriers; whether it covers a foreign-owned ingress container is not ruled. Until the operator names the seam (as D-DRF-1 did for its cache), nothing on this path is called zero-copy or a fold." +- **A.0-1 correction** (plan `:158-162`): "Building a Lance `RowAddrMask` is an **ingress copy outside the fold** … Nothing on that path may be called zero-copy or a fold (see A.3, seam S-INGRESS)." +- **Single authority** (plan `:182-186`): "`DR ordinal ≠ _rowid ≠ _rowaddr ≠ output position.` No code today composes `ord` with any `_rowid` table (closed search space `crates/**/*.rs`). There must be exactly one version-checked coordinate→`_rowid` authority; today there are two lab ones with different witnesses (`holo-lance-masked-io-probe/src/main.rs:148-156` `CoordMap{version}` + `ReadError::StaleMapping`; DRF `check_seal` on `index_seal`), and the crepe handoff has none (`crepe-plan-probe/src/lance_store.rs:264`, keyed on `_rowaddr`)." +- **Register rows** (plan `:318-320`): ONE coordinate→`_rowid` authority = ADAPT (wiring), duplication risk HIGH ("a third map"); DR ordinal → `_rowid` composition = "look up through both maps (never store `row_ids∘ord` as a third table)", ADAPT gated on S-INGRESS; bulk words → `RowAddrTreeMap` = `insert_fragment` / `from_lsb0_bytes` / `insert_range`, ADAPT gated on S-INGRESS. +- **A.3 headline** (`:254`): "**Logical DR64 density does not predict Lance I/O; chunk locality of the selected `_rowid`s does.**" +- **A.3 "do not rebuild"** (`:228-232`): the API set `with_row_addr_prefilter`, `RowAddrMask::{from_allowed,from_block}`, projection, `filter` refine, `MaterializationStyle`, `scan_stats_callback` is "Already implemented and measured; do **not** rebuild". +- **Timing rule** (`:262-264`): storage results reported as bytes, ranges, requests, rows scanned, not wall time. +- A.4 end-to-end map (`:280-289`): nonresident path = `C_map ord→_rowid via ONE version-checked map [INTEGRATION GAP]` → `C_mask words→RowAddrTreeMap — S-INGRESS [OPEN]` → `C_io with_row_addr_prefilter + projection of demanded columns only (Lance owns)` → re-attach by stored coordinate column. `:311`: "**the integrated DR64 → Lance → Quack query is unmeasured**". + +## Step 2 — the caller path, hop by hop + +| hop | symbol / location | status | +|---|---|---| +| ClassView → WideFieldMask | `lance-graph-contract/src/class_view.rs:243` `WideFieldMask` (contract type exists) | EXISTS (contract) | +| WideFieldMask → required Lance columns | none found. Closed search: `rg '\.project\('` over `crates/**/*.rs` excluding the five lab probes: every hit is `ClassView::project`/`SubstrateView::project`/math `project` (class_view.rs:1122, causal_witness.rs:685, …), not `Scanner::project`. Production Lance scans hard-code column names: `lance-graph/src/graph/cycle_sink.rs:440,608,1079,1162,1211`, `graph/versioned.rs:443`. | **MISSING EDGE** (VERIFIED by closed grep + read of candidate lines; whole-file reads of cycle_sink/versioned NOT done → PARTIAL for those files) | +| WHERE / DR selection → mask words | Quack `lower` + mask-risc `execute_into` (probe `lib.rs:63-64`); relalg `run_with_program` in `crepe-plan-probe/src/lance_store.rs:233-250` | EXISTS (resident) | +| mask words → version-attested row ids | three different lab authorities: (1) H14c `CoordMap{version, id_of}` + `StaleMapping` (`holo-lance-masked-io-probe/src/main.rs:146-218`); (2) H14d `row_ids` positional `Vec`, **no version** (`holo-h14d-probe/src/main.rs:448-471`); (3) crepe Arm A `_rowaddr` column read at the same `Arc`, **no explicit witness** (`lance_store.rs:184-205,264-269`); (4) crepe Arm B `SealedFacetLane` attested at `ds.version()` + `uri`, refuses `Stale`/`OtherDataset`/`NotContiguous` (`addressed.rs:101-116,284-309`). DR side: dt64 `RouteVersion`/`Frame` (`lance-graph-dt64/src/dr.rs:153-176`) — a DR route-log witness, not a Lance dataset version; dt64 has no Lance dependency (`lance-graph-dt64/Cargo.toml:10`). | **COORDINATE GAP**: no code composes DR ordinal → `_rowid` (confirms plan `:182`) | +| → Lance `RowAddrMask` (AllowList) | `RowAddrMask::from_allowed` at `lance_store.rs:268`, `addressed.rs:270`, H14c `main.rs:216`, H14d `main.rs:532`, `lance_source_probe.rs:357`. **BlockList (`from_block`) is called nowhere** in `crates/**/*.rs` (closed grep `from_block`, 0 hits). | EXISTS (lab only) | +| → `Scanner::with_row_addr_prefilter` | `lance_store.rs:217`, H14c `main.rs:234`, H14d `main.rs:490`, `lance_source_probe.rs:379` | EXISTS (lab only) | +| → `Scanner::project` (required values only) | `lance_store.rs:191,215` (`c0`/`c1` literal); H14c `main.rs:231` (`coord` + lane names); H14d `main.rs:488`; `lance_source_probe.rs:373-392`. Columns are literal names, never derived from a FieldMask. | EXISTS, but FIELD selection is hard-coded, not ClassView-driven | + +Closed search space for the Lance calls: `rg 'with_row_addr_prefilter|RowAddrMask|RowAddrTreeMap|from_allowed|from_block|take_rows|with_fragments|scan_stats_callback'` over `crates/**/*.rs` → hits only in the 5 files above (+ `crepe-plan-probe/src/lib.rs:23` doc). All are lab: `crepe-plan-probe`, `holo-lance-masked-io-probe`, `holo-h14d-probe` are in the root `exclude` list (`Cargo.toml:51` onward, entries at `:55,:137,:206`); `lance-graph-benches` is a member but the hit is an example. `take_rows` / `with_fragments`: 0 hits. + +### Field-mask vs row-mask confusion check +- No site passes a WideFieldMask/FieldMask where a row mask is expected, or vice versa (VERIFIED in the files read). In the D-GLOBAL-WHERE P1 probe, `WideFieldMask` appears only as `MethodBinding.on/reads` (field positions) in `falsify_p1.rs:199-201` — correctly field-side. +- **Naming hazard (VERIFIED):** `project.rs:1-62` calls property-layout matching a "projection probe", but it projects nothing through Lance or ClassView; it measures packing 64 logical properties into 32 resident i32/u32 lanes. That is a third meaning of "projection" next to `Scanner::project` and `ClassView::project`. +- crepe `lance_store.rs:282-307`: builds the row mask from a full one-column read (`read_column`, no prefilter) — i.e. a full column scan per side per query to produce the row mask, then a second masked scan. Rows→mask is correct domain use (`_rowaddr`), but the mask's input read is unpruned. + +## Step 3 — answers + +**a) Paths that invoke Lance masked reads (prefilter + projection)** — VERIFIED-IN-CODE, all lab: +1. `crepe-plan-probe::lance_store::read_rows` (`lance_store.rs:209-229`), called from `prepare_lance` (`:274-364`), Arm A. +2. `crepe-plan-probe::addressed::read_side` (`addressed.rs:316-330`) → `read_rows`, Arm B (version-attested, range masks). +3. `holo-lance-masked-io-probe::read_with` / `masked_read` (`main.rs:201-253`). +4. `holo-h14d-probe::read` (`main.rs:482-520`). +5. `lance-graph-benches/examples/lance_source_probe.rs::run` Arm C (`:375-381`). +No production (workspace `src/`) caller of `with_row_addr_prefilter` exists in `crates/**/*.rs`. + +**b) Bypasses** +- **The D-GLOBAL-WHERE probes never touch Lance.** Both `Cargo.toml`s depend only on contract, dt64, quack, mask-risc, ndarray (+ `dr-math-binding-probe` in P0) — `.claude/probes/2026-11-03-d-global-where-p1/probe/Cargo.toml:10-15`, `.../d-global-where/probe/Cargo.toml:10-16`. Closed grep for `lance::|Scanner|RowAddr|_rowid|take_rows|Dataset` over both probe trees: 0 hits. Both SPEC.md files: no `drf|SealedMap|Focus|CoordMap|RowAddr|_rowid|lance::|Dataset` hit. Data is generated in memory: P0 REPORT `:18` ("datasets are deterministic (SplitMix64) and are regenerated by `bench`, not committed"); compose.rs `:92-134` (`where_val`, `a_of`, `b_of`, `Graph::edges_of` closures); `fetch` is "edges … from the generator, standing in for a global relation index" (`compose.rs:38-40`) and its doc disclaims "that `fetch` equals a real relation-index read" (`:57-58`). Lane values reach folds via `WorkingSet::gather_lane(frame, source: Fn(u64)->i32)` (`lib.rs:405-416`) — a copy into a new coordinate-ordered `Vec`, i.e. a GATHER over a closure, not a storage read. +- Inside those probes, shapes the workspace already rules against (VERIFIED-IN-CODE): `discover` extracts mask → `Vec` ids (`compose.rs:175-179`); `fetch_edges` returns `Vec<(u64,u64)>` tuples (`compose.rs:128-134`); `partition` builds `Vec>` (`:206-213`). Per CLAUDE.md these are the tuple/row-id-list shapes; A.2 row 1 marks SelectionVector-style row-id lists INTENTIONALLY ELIMINATED. +- crepe Arm A: full unmasked column scan to build each mask (`lance_store.rs:184-205`, called at `:292,:336`). Unmasked full reads also at `:347-348` when no centers. +- `project.rs:33-41`: packed ordered compares done with a hand-written Rust "extraction loop" because "Quack has no ordered compare over an unsigned lane". The plan's register (`:322`) already names the fix: ndarray `*_u8_to_mask` / `*_u64_to_mask` exist (`simd_masking_ops.rs:3569-3724,3787-3936`, cited by the plan, not re-read here) but have no `Pred` variant — ADAPT (wiring). The probe re-derives the kernel instead. +- No `take`/`take_rows` loops anywhere (0 hits). + +**c) DR → Lance coordinate mapping + version witness: MISSING (not reused, not re-derived)**. The DR side (dt64 `MapTable`/`Frame`/`RouteVersion`) and the Lance side (H14c `CoordMap{version}`, crepe `AddressedRelation{uri, lane.version()}`) never meet. dt64 has no Lance dependency. D-GLOBAL-WHERE adds its own `WorkingSet` + `CoordIndex` (arm C "derived index … built FROM `MapTable` views under one frame", `lib.rs:4-9,34-36`) — derived under the DR frame, but with no Lance version, so it is neither of the existing Lance witnesses. HYPOTHESIS: if this `CoordIndex` were later keyed to `_rowid`s it would be exactly the "third map" the register forbids (plan `:318`). + +**d) Mask construction frequency** (VERIFIED-IN-CODE): +- crepe Arm A: `row_addr_mask` (`lance_store.rs:264-269`, per-bit `for_each_set` → `Vec` → `from_iter`) once per side **per `prepare_lance` call** (per query), after a full column read. +- crepe Arm B: `range_mask` (`addressed.rs:265-271`, `insert_range` per run) once per side per query; `attest` once per table (cold, reused while `ds.version()` matches, `:284-309`). +- H14c: `CoordMap` built once per dataset (`main.rs:338`); `masked_read` rebuilds the `RowAddrMask` from a coord list on every call (`:215-216`), called 5× per cell and once per lane in the independent strategy (`:366-393`). +- H14d: `row_ids` once per dataset (`main.rs:643`); `row_mask` once per (pattern,d,width,k) cell (`:755`), per-bit enumeration (`:523-533`). +- D-GLOBAL-WHERE: no Lance mask at all. + +**e) Bytes attributed to physical layout, not missing filtering** — yes, consistently: +- H14d PR #1575: "Bytes are page-bound, so the saving needs survivors that are page-local." +- Plan A.3 `:251-254`: uniform mask touches a 4,096-row chunk with prob `1 − (1 − d)^4096` (≈1 at 1 %) → "chunk locality of the selected `_rowid`s does" predict I/O. +- `.claude/probes/2026-11-03-d-dr-fold/D-DR-FOLD-MAP.md:47`: "Lance 13 `with_row_addr_prefilter`: a scattered mask cuts rows decoded, not bytes fetched; a page-local mask cuts both (1 % uniform 1.00×, 1 % clustered 0.24×, 0.1 % uniform 0.77×)." — LAB, H14c-4. +- `.claude/probes/2026-11-03-d-dr-fold/REPORT-BENCH.md:129`: "The H14c result stands: a scattered mask cuts rows decoded, not bytes; a clustered one cuts both." (this is the #1945 / D-DR-FOLD quote's source). +- `.claude/board/entries/2026-11-03-d-dr-fold-map.md:5`: "A mask cuts bytes fetched only when presence is page-local. Every recorded `bytes_read` was served from the page cache, not read from disk. The physical read-planning contract is missing and has no owner." +- Rows scanned == |R| in every H14d cell, so filtering is not missing; bytes are layout-bound. +- #1789: chunk-count predictor fails; lance-io merges at 4 KiB on local temp dirs vs 64 KiB cloud default. + +**f) Where time goes** +- Storage: not timed by rule (A.3 `:262-264`); bytes from page cache only (dr-fold-map `:5`). #1633: full read 9,810–30,710 µs, Arm B warm 370–38,980 µs, attest 6,700–9,800 µs (quoted from the PR body, not gated). +- DR routing: P0 REPORT `:112` — cold losses on every `multi_*` dataset come from "**addressing**: … `MapTable::locate` per edge endpoint … O(tiles) per id"; P0 entry `:18`: "Routing once and reusing costs 5 µs against 650–970 µs." P1 `comp_both.log:18-28` (N=1M): `route` 480.6 µs → 282,447.8 µs, `fetch` (in-memory generator) 68–14,086 µs, `fold/set` 60–4,358 µs, `discover` 209–641 µs; FAILED-GATE on 6 of 10 rows. Route dominates every case and grows with scatter (route +7,397.6 µs/set scatter vs +215.2 contig, `:42-43`). +- Mask ingress: never timed on the Lance side except `lance_source_probe` `mask_ms` (`:376-378`); A.4 lists `C_mask` as an OPEN seam. +- Fold: ns/word scale in H14d (2–10 ns/word stage 2, ~6 ns/lane load); in P1 compose, fold/set exceeds 1,000 µs only because sets are 50k–200k ids (`comp_both.log:20-28`). + +## Step 4 — classification + +1. **NOT APPLICABLE (primary)** for the D-GLOBAL-WHERE probes: they never touch Lance (manifests + closed grep, step 3b). They cannot regress or reuse a Lance read path; their numbers say nothing about Lance selective reads. Their costs are DR routing + in-memory generator "fetch". +2. **MISSING WIRING**: (i) DR ordinal → `_rowid` composition (no code; plan `:182`, re-confirmed by dt64's dependency set); (ii) WideFieldMask → `Scanner::project` column set (no caller; all projections literal); (iii) one version-checked authority — still four lab witnesses (H14c `CoordMap`, H14d none, crepe A none, crepe B `SealedFacetLane`). All gated on S-INGRESS / owner decisions D4 (dr-fold-map `:14`). +3. **COORDINATE MISMATCH (latent)**: H14d `row_ids` and crepe Arm A carry no Lance version; the D-GLOBAL-WHERE `CoordIndex` carries a DR frame only. Not an active bug (each is used within one dataset handle), but none can be composed safely across versions. +4. **GENUINE PHYSICAL-I/O LIMITATION**: scattered masks cut rows, not bytes (H14c/H14d, A.3, dr-fold REPORT-BENCH:129). Not a filtering defect; chunk locality decides. +5. **CALLER REGRESSION**: none found. No production caller ever used `with_row_addr_prefilter`, so none regressed. The lab callers from #1573/#1575/#1633 are present and unchanged in this tree. +6. **Re-derivation inside the probe (not Lance)**: `project.rs` extraction loops re-derive ndarray u8/u64 compare kernels (register row `:322`); `gather_lane` re-derives the DRF GATHER (path 3) without DRF's `FocusKey`/`check_seal`; `discover`/`fetch_edges`/`partition` materialize ids and tuples that A.2 row 1 eliminates. + +## Step 5 — minimal corrective proposal (prose only, no edits) + +1. If D-GLOBAL-WHERE is meant to say anything about storage, it must read through an existing Lance path rather than a generator. The reusable one is crepe-plan-probe Arm B: `addressed::attest` (`addressed.rs:101-116`) once per table, then `ranges`+`plan_read`+`read_side` (`:213-330`), which already carries `ReadPolicy` (the #1489 "cost choice") and the `Stale`/`OtherDataset` refusals. Its mask is `insert_range` per run, the register's bulk path. Use it, do not copy it. +2. The missing edge is DR ordinal → Lance row position: H14d's positional `row_ids` (`holo-h14d-probe/src/main.rs:448-471`) shows the lookup shape, but the authority must be ONE witness that records both the DR `Frame`/`RouteVersion` and the Lance `ds.version()` (+uri) — the register's ADAPT row `:318`, looked up through both maps, never stored as a composed table (`:319`). This is gated on S-INGRESS and owner decision D4; it should not be built by a probe. +3. Field side: map the ClassView/WideFieldMask positions a query demands to the column list passed to `Scanner::project` (today literal at `lance_store.rs:215`). That edge is absent everywhere. +4. Inside the probe: replace `project.rs` extraction loops by adding mask-risc `Pred` variants that call the existing ndarray u8/u64 compare kernels (plan `:322`, ADAPT when a consumer asks — this probe is the consumer); stop measuring `route` per query (P0 REPORT already shows route-once costs 5 µs). +5. Report any Lance arm in bytes / rows scanned / ranges, not wall time (A.3 `:262-264`), and state that page-cache bytes are not disk reads. + +## Could not access / not done +- Lance 13 crate source (`scanner.rs:1663`, `mask.rs`, `filtered_read.rs`, `miniblock.rs`): not read; cited from the plan. +- ndarray `simd_masking_ops.rs` lines: cited from the plan, not re-read. +- PR file lists (`get_files`) not fetched; local files read instead. +- PR #1945 not fetched; quote located in local `REPORT-BENCH.md:129` and `D-DR-FOLD-MAP.md:47`. +- Whole-file reads not done for `cycle_sink.rs`, `versioned.rs`, `lance_cache.rs`, `audit.rs`, `lance_vector_search.rs`, `addressed.rs:330-514`, `holo-*` beyond the cited ranges, `where_k.rs`, the P1 `lib.rs` beyond `1-200` and `405-430`. Claims about those files are PARTIAL. +- `drf-focus-probe` source not read; its `SealedMap`/`check_seal` cited from the plan. diff --git a/.claude/probes/2026-11-03-d-global-where-p1/REPORT.md b/.claude/probes/2026-11-03-d-global-where-p1/REPORT.md new file mode 100644 index 000000000..816a6eef5 --- /dev/null +++ b/.claude/probes/2026-11-03-d-global-where-p1/REPORT.md @@ -0,0 +1,463 @@ +# D-GLOBAL-WHERE P2 — report + +Date: 2026-11-03. LAB ONLY. No production crate changed. Nothing here is +authorized as a design; S-INGRESS and D4 stay open. + +**Code and raw data live on the research branch**, not on `main`: +[`research/d-global-where-p2`](https://github.com/AdaWorldAPI/lance-graph/tree/research/d-global-where-p2/.claude/probes/2026-11-03-d-global-where-p1) +(commit `1422b2b92`). It holds the in-memory probe (`probe/`), the DuckDB +scripts (`duckdb/`) and every measurement (`results/`). Any `probe/…`, +`duckdb/…` or `results/…` path cited below refers to that branch. On `main` +are this report, `ENGINEERING-PARITY.md`, `LANCE-TRACE.md`, `SPEC.md` and the +Lance arm (`crates/crepe-plan-probe/examples/where_lance_arm.rs`). + +Labels used on every number: + +- **in-memory**: the P2 probe (`probe/`), data generated in memory, no storage. +- **Lance-backed**: `crates/crepe-plan-probe/examples/where_lance_arm.rs`, a + real Lance 13 dataset on local disk. Bytes are Lance's `bytes_read`, served + from the page cache, not from disk. +- **partial integration**: a stage joined to the next one by a given input + instead of the real producer (the DR→Lance step, see D4). +- **FAILED-GATE**: a warm in-fold number ≥ 1000 µs. Reported, never hidden. +- **COLD/IO (reported)**: prepare, attest, storage reads. Reported, not gated + (operator ruling 2026-10-22, CLAUDE.md scoped exception). + +Answer parity is checked separately from timing: every arm's count and +per-column sums are compared to an oracle before any timing is printed. + +## 1. Frozen evidence, independent of Lance + +Frozen at `49f02414d`; re-checked this pass. + +| probe | what it checks | result | +|---|---|---| +| `falsify_p1` | 37 falsifiers (refusals, composition laws) | 37/37 | +| `where_k` 1M / 10M | WHERE over K criteria, K distribution 34/46/14/6 | every answer equals the oracle; 353 / 648 FAILED-GATE values reported | +| DuckDB 2.0.0 parity (`duckdb/compare_where.py`) | count + EXISTS vs DuckDB, threads 1 and 4 | every DuckDB answer equals the oracle; coverage is enforced in both directions | +| `project` 1M / 10M | 64 logical properties over 32 lanes, layouts U/P/S | all answers match; refusals R1–R3 fire; 23 / 41 FAILED-GATE | +| `compose` 1M / 10M | repeated DR composition beyond 64K | all composed answers equal the oracle; all refusals fire | + +Known gap: the k4/k16 sparse and mixed AND cases in `project` have oracle +count 0, so they cannot catch a wrong non-zero answer. + +DuckDB timing, regular grid (39 cases), weighted by the K distribution: + +| N | threads | count (ours vs DuckDB wall) | EXISTS (ours vs DuckDB wall) | +|---|---|---|---| +| 1M | 1 | 14.17× WIN | 1.27× WIN | +| 1M | 4 | 5.81× WIN | 1.73× WIN | +| 10M | 1 | 7.00× WIN | 0.43× LOSS | +| 10M | 4 | 1.96× WIN | 0.16× LOSS | + +These are answer-parity runs with timings attached. They are not an +engineering-parity verdict; see §6. + +The EXISTS loss at 10M has one cause, verified in code: `Terminal::Any` scans +every tile with no early exit (`mask-risc/src/exec.rs:1633`, `:1793-1794`). +DuckDB stops at the first match. + +## 2. In-memory corrections (`probe/src/bin/corrections.rs`) + +The original timings stay as baselines. Each correction is a separate arm on +the same fixture (`--sel 20000`); every arm matches the oracle. + +**C1 — route once vs per query.** Prepare (routing) is COLD: +162,177 µs at 1M and 164,813 µs at 10M (independent of N, it depends on the +selection). After prepare, a query costs: + +| Q queries | per query, re-routing each time | per query, routing once | +|---|---|---| +| 1 | 163,962 µs | 158,198 µs | +| 16 | 168,809 µs | 10,421 µs | +| 256 | 172,018 µs | 1,012 µs | +| warm, no prepare | — | 381–494 µs | + +Routing once pays off at Q ≥ 2. The stale guard refuses a binding after +`replace_tile`. With the guard disabled the stale binding would be accepted, +so the guard is load-bearing. + +**C2 — existing ndarray kernels vs the hand-written packed u8 compare.** There +is no ordered u8 compare over a packed u32 lane (NO-EXISTING-KERNEL): + +- `lt_u8_to_mask` reads a contiguous `&[u8]`. +- Quack `Cmp` has only `EqU32` / `NeU32` / `MatchU32` and signed `I32` + orderings. + +The kernel arm composes existing words (`ternary_match_u32_to_mask`, +`mask_or_assign`, `mask_not`). It runs one ternary pass for each 2-bit prefix +the threshold needs: + +| predicate | passes | 1M: hand / kernel | 10M: hand / kernel | +|---|---|---|---| +| f0 < 64 | 1 | 289 / 172 µs | 3,219 / 1,949 µs | +| f1 ≥ 64 | 1 | 312 / 161 µs | 3,257 / 2,096 µs | +| f2 ≥ 170 | 4 | 291 / 649 µs | 3,230 / 8,424 µs | +| f3 < 255 | 8 | 336 / 1,300 µs | 3,214 / 16,384 µs | + +The existing kernels win when the threshold needs one pass. They lose by up to +5× when it needs several, because the cost scales with the pass count. Every +10M row is FAILED-GATE: a full 10M-row lane scan is over the µs budget on +either path. The real fix is an ordered `u8`-in-`u32` compare word. That is a +missing kernel, not missing wiring, so it is filed for the mask-risc / ndarray +owner. It was not built here. + +**C3 — borrowed lane reads vs `gather_lane` copies** (sel 20,000; 80,896 B per +copy): + +| threshold | 1M: copy / borrowed | 10M: copy / borrowed | +|---|---|---| +| lt=100 | 43.1 / 43.2 µs | 122.5 / 120.5 µs | +| lt=500 | 60.7 / 148.9 µs | 111.3 / 261.1 µs | +| lt=900 | 44.7 / 59.1 µs | 111.3 / 110.0 µs | + +Borrowing removes the copy and its 80,896 B allocation with no loss at +selectivity 10% and 90%. At 50% the borrowed path is 2.3–2.5× slower. One +possible cause is unpredictable branches in the in-place compare; that is not +yet verified. This is filed open rather than explained. + +## 3. Lance selective-read arm (`where_lance_arm`, Lance-backed) + +The arm reuses the #1633 addressed read; no new read API was added. The edit +to the lab crate `crepe-plan-probe/src/addressed.rs`: + +- makes the existing `ranges` / `range_mask` public; +- adds `range_tree`, so ingress bytes are measured on the tree Lance receives; +- adds `check_fresh`; `current_lane` now calls it, so there is one freshness + rule. + +The example constructs an `AddressedRelation` and calls these directly. + +Fixture: `id` plus `p0..p15` as i32, written as one fragment. Each column +carries a `predicate_iri`. The physical order is permuted (`(5j+3) mod 16`) +and the physical names are not the labels. Attest asserts `_rowid == _rowaddr` +row for row: the prefilter is keyed in `_rowid` space and the lane holds +`_rowaddr`. With stable row ids that assertion would fail, and the arm would +refuse. + +Arms: + +| arm | what it does | +|---|---| +| A-full | full scan of WHERE + demanded columns, fold over the WHERE mask | +| B-allow / B-block | scan the WHERE columns, build a RowAddrMask from the result, masked scan of the demanded columns | +| C-exact | ids given (BLOCKED_D4) → lane ranges (`ranges`) → `range_mask` → masked scan of demanded columns only | +| C-default | as C, with `ReadPolicy::DEFAULT` (coalesce, or full scan when the ranges cover too much) | + +All four arms give the same count and sums at every cell (oracle checked). + +### 1M results (median of 5 warm runs, µs; bytes = Lance `bytes_read`) + +Selected cells (full table: `results/lance_arm/n1000000/lance_arm.log`): + +| layout | sel | k | arm | rows scanned | bytes (% of A) | mask bytes | route | ingress | Lance read | +|---|---|---|---|---:|---:|---:|---:|---:|---:| +| clustered | 0.1% | 1 | A-full | 1,000,000 | 100% | 0 | 0 | 0 | 10,944 | +| clustered | 0.1% | 1 | C-exact | 1,000 | 0.1% | 27 | 65.5 | 0.5 | 620 | +| clustered | 10% | 4 | A-full | 1,000,000 | 100% | 0 | 0 | 0 | 25,139 | +| clustered | 10% | 4 | C-exact | 100,000 | 6.8% | 47 | 6,951 | 1.6 | 2,180 | +| scattered | 0.1% | 1 | C-exact | 988 | 33.3% | 2,124 | 331 | 49.1 | 1,631 | +| scattered | 0.1% | 16 | C-exact | 988 | 99.8% | 2,124 | 487 | 50.3 | 12,930 | +| scattered | 10% | 4 | A-full | 1,000,000 | 100% | 0 | 0 | 0 | 25,411 | +| scattered | 10% | 4 | B-allow | 1,100,164 | 120% | 126,420 | 0 | 3,201 | 67,939 | +| scattered | 10% | 4 | C-exact | 100,164 | 80% | 126,420 | 9,533 | 4,593 | 54,872 | +| scattered | 10% | 16 | C-exact | 100,164 | 100% | 126,420 | 9,136 | 4,572 | 222,968 | +| scattered | 50% | 16 | A-full | 1,000,000 | 100% | 0 | 0 | 0 | 83,343 | +| scattered | 50% | 16 | C-exact | 500,613 | 100% | 377,194 | 43,569 | 10,888 | 945,373 | + +What this shows: + +1. **The mask always cuts rows scanned** to exactly the selection (C) or + n + selection (B, which first scans the WHERE columns). Filtering is not + missing anywhere. +2. **Bytes follow page locality, not selectivity.** Clustered selections read + 0.1–9% of the full scan's bytes. A scattered selection of 0.1% already + reads 33% of the bytes at k=1 and 99.8% at k=16. Every page holding at + least one survivor is read whole. This repeats H14c/H14d (#1573/#1575) on + the #1633 path. It is a physical I/O limit, not a wiring defect. +3. **On scattered selections a masked read is slower than a full scan.** + - At 10% and k=16, C-exact takes 222,968 µs against A's 88,913 µs; B takes + 255,226 µs. + - At 50% and k=16, C-exact takes 945,373 µs against A's 83,343 µs. + - Across scattered 10–50%, C-exact is 1.4–11× slower than A, and B is + 2.4–11× slower. + - `ReadPolicy::DEFAULT` sees this coming and falls back to a full read on + every scattered 10–50% cell. That is the #1489 "cost choice", already + implemented. It still pays for the route stage before deciding. +4. **Mask ingress is a copy, and its cost scales with the number of ranges, + not rows.** + - One clustered range: 27–144 B of tree, 0.4–5.4 µs. + - 91,527 scattered ranges: 126,420 B, ~4,600 µs. + - 326,222 ranges: 377,194 B, ~9,400–10,900 µs. + + Timing covers building the one tree Lance receives. The serialized size is + measured outside the timer. The first run of this report timed two tree + builds per C read and doubled the C ingress figures (Codex finding on + #1950); every number here is from the corrected code. + + It is never zero-copy: `insert_range` builds a `RowAddrTreeMap` that Lance + owns (S-INGRESS). +5. **Route (ids → lane ranges by galloping seek) is the dominant non-I/O + cost.** It is about 70–95 ns per id: 6,900–9,500 µs per 100k ids and + 34,000–44,000 µs per 500k. It is FAILED-GATE on every cell ≥ 10k ids. It is linear in the + number of ids because each id is sought separately. + +### 10M results (median of 3 warm runs, µs) + +Every oracle and refusal check passes. Attest is COLD: 809,110 µs, once per +version. Full table: `results/lance_arm/n10000000/lance_arm.log`. + +| layout | sel | k | arm | rows scanned | bytes (% of A) | mask bytes | route | ingress | Lance read | +|---|---|---|---|---:|---:|---:|---:|---:|---:| +| clustered | 0.1% | 1 | A-full | 10,000,000 | 100% | 0 | 0 | 0 | 133,908 | +| clustered | 0.1% | 1 | C-exact | 10,000 | 0.04% | 27 | 758 | 1.3 | 1,196 | +| clustered | 10% | 4 | A-full | 10,000,000 | 100% | 0 | 0 | 0 | 284,073 | +| clustered | 10% | 4 | C-exact | 1,000,000 | 6.4% | 242 | 73,993 | 11.6 | 16,348 | +| clustered | 10% | 4 | C-default | 10,000,000 | 64.3% | 0 | 71,509 | 0.1 | 319,741 | +| scattered | 0.1% | 1 | C-exact | 9,858 | 33.3% | 20,960 | 4,702 | 583 | 10,024 | +| scattered | 0.1% | 16 | A-full | 10,000,000 | 100% | 0 | 0 | 0 | 1,001,637 | +| scattered | 0.1% | 16 | C-exact | 9,858 | 100% | 20,960 | 4,721 | 560 | 126,264 | +| scattered | 10% | 4 | C-exact | 1,001,669 | 80% | 1,317,366 | 96,773 | 57,455 | 773,128 | +| scattered | 10% | 16 | C-exact | 1,001,669 | 100% | 1,317,366 | 98,639 | 55,939 | 2,814,172 | +| scattered | 50% | 16 | A-full | 10,000,000 | 100% | 0 | 0 | 0 | 1,029,055 | +| scattered | 50% | 16 | C-exact | 5,006,105 | 100% | 4,776,680 | 416,595 | 123,162 | 11,019,172 | + +The 1M pattern holds at 10M, and grows: + +- A clustered 0.1% read is 112× faster than the full scan. +- The worst scattered read (50%, k=16) is 11× slower than the full scan. +- Even a 0.1% scattered selection at k=16 reads 100% of the bytes, yet it + still beats the full scan 7.9× on time, because it decodes 9,858 rows + instead of 10M. +- Ingress for 3.26M ranges is a 4.8 MB tree and 123,162 µs. + +**New: `ReadPolicy::DEFAULT` misroutes clustered demand.** `plan_read` +(`addressed.rs:212-232`) decides on covered rows alone (`covered*1000 >= +n*full_permille`, 10%). It never looks at how many ranges there are, and its +own doc says "Clustered demand … is not measured". A single contiguous 10% +range is therefore sent to a full scan: + +| N | C-default | C-exact | penalty | +|---|---:|---:|---:| +| 1M | 35,606 µs | 9,310 µs | 3.8× | +| 10M | 428,883 µs | 93,994 µs | 4.6× | + +The scattered cells, where the fallback is right, have tens of thousands to +millions of ranges. Range count, not coverage, separates the two. This is +recorded as a probe item and the policy is not changed. It belongs to the +#1633 owner. + +A defect found on the way. Lance 13 compares the file row limit as +`params.max_rows_per_file as u32` (`lance-13.0.0/src/dataset/write.rs:1077`). +The fixture set `1 << 40`, which truncates to 0. Every input batch therefore +closed a file, and the 10M table came out in many fragments. The attest guard +refused it as `NotContiguous` (log kept: `results/lance_arm/n10000000/notcontiguous.log`). +That is the #1633 refusal working as designed. The fixture now uses +`u32::MAX`. This upstream truncation is recorded here and not patched. + +## 4. ClassView → WideFieldMask → physical columns (Lance-backed) + +`resolve()` maps each demanded field position through: + +1. `ClassView::fields(class)` → `FieldRef::predicate_iri`, then +2. the Lance schema field metadata `predicate_iri`, to the physical column name. + +It never maps position to ordinal or label to column name. Two traps prove a +shortcut would be caught: + +- **Ordinal shortcut:** mapping by ordinal gives the wrong column on 16 of 16 + positions. +- **Label shortcut:** mapping by label finds no physical column on 16 of 16. + +Each refusal below is paired with a success control: + +| refusal | trigger | +|---|---| +| AbsentField | the position is not in the presence `FieldMask` | +| Unauthorized | the position is outside the caller's `WideFieldMask` | +| UnknownProperty | no column carries the IRI | +| AmbiguousProperty | two columns carry the IRI (a separate dataset) | +| StaleSchema | the dataset was appended after resolution | + +A refusal returns no column list, so the column list cannot widen. Projected +bytes follow k exactly at A-full: 3.8 MB (k=1), 6.3 MB (k=4), 20.3 MB (k=16). +C reads only the resolved columns. + +Kept distinct throughout: + +| mask | answers | +|---|---| +| WHERE mask | which rows pass | +| `WideFieldMask` | which fields are demanded or permitted | +| DR mask | which identities are selected | +| Lance `RowAddrMask` | which row addresses to read | + +The code never passes one where another is expected. + +## 5. D4 / S-INGRESS boundary + +`BLOCKED_D4`: no type binds a DR `Frame` to a Lance dataset version. + +- `lance-graph-dt64` has no Lance dependency. +- The #1788 single-authority decision is open. + +So arm C takes its ids as a given input, the oracle's sorted identities. It +never composes a DR ordinal with `_rowid`. No third coordinate map was built. + +The freshness checks that do exist are #1633's own, now one function: + +- a stale lane is refused after an append; +- a lane read against another dataset is refused; +- a forged newer-version witness is refused by `seek_from`. + +RowAddrMask construction is measured, not assumed free; see §3 point 4. + +## 6. Cost columns + +One row per stage for three representative 1M cells, k=4, µs. Labels per +stage: **IM** in-memory, **LB** Lance-backed, **PI** partial integration. + +| cell | WHERE evaluation | ClassView resolution | DR routing | mask ingress | Lance read | projection | FOLD | total | +|---|---|---|---|---|---|---|---|---| +| clustered 10%, C-exact | 0 (ids given, PI) | 10.3 LB | 6,951 IM (lane seek; DR→Lance BLOCKED_D4) | 1.6 LB | 2,180 LB | inside Lance read; 6.8% of A's bytes LB | 64.6 LB | 9,310 PI | +| scattered 10%, C-exact | 0 (PI) | 15.7 LB | 9,533 IM | 4,593 LB | 54,872 LB | inside Lance read; 80% of A's bytes LB | 65.9 LB | 72,587 PI | +| scattered 10%, A-full | 790 LB (over loaded columns) | 14.4 LB | 0 | 0 | 25,411 LB | inside Lance read; 100% LB | 2,066 LB | 29,432 LB | + +Not measured in the same cell, because the DR→Lance edge is BLOCKED_D4: + +- in-memory DR WHERE evaluation over resident planes (`where_k`): 138–1,530 µs + at 1M; +- DR routing per composition (`compose`, route column): 35,914 µs out of a + 50,178 µs total at sel 50k, D=1. + +A full end-to-end row, DR selection → Lance read in one process, does not +exist yet. It is the integrated path the #1788 register already lists as +unmeasured. + +## 7. Three parity verdicts + +**Semantic parity: REACHED.** + +- Every count and EXISTS answer equals the oracle and DuckDB at 1M and 10M. +- Every Lance arm equals the oracle and the full-projection oracle at 1M and + 10M. +- Three-valued NULL semantics match. +- Refusals fire and are paired with controls. + +**Performance parity: SPLIT.** + +| operation | ours vs DuckDB | status | +|---|---|---| +| count | 14.2× (1M, 1 thread) down to 1.96× (10M, 4 threads) | WIN | +| EXISTS | 0.43× / 0.16× at 10M | LOSS: no early exit | + +On storage, a clustered selective read beats the full scan by 11.5–17.6× at +1M and 17–112× at 10M. A scattered one at 10–50% loses to the full scan by +1.4–11×, and the existing +`ReadPolicy` already knows to fall back. + +**Engineering parity: NOT REACHED on strategy, reached on representation** +(`ENGINEERING-PARITY.md`; source-backed Lance trace: `LANCE-TRACE.md`): + +- **Selection reuse: ALREADY WORKS.** A versioned DR mask lives longer than + DuckDB's per-chunk selection vector. +- **Chunk compaction: INTENTIONALLY DIFFERENT.** Word folds have no underfull + chunks. +- **Late materialization: INTENTIONALLY DIFFERENT.** +- **Evaluation direction: MISSING CAPABILITY.** +- **EXISTS early exit: MISSING WIRING.** +- **Physical-read avoidance: MISSING** the single `_rowid` authority and + bind-time constant folding. +- **Routing: MISSING WIRING.** Route once, then reuse (C1 shows Q ≥ 2 pays). + +WHEN vs WHETHER: DuckDB optimizes *when* a property is materialized. We can +skip the read entirely for: + +- COUNT / EXISTS / ALL over a bound plane; +- presence questions; +- empty demand; +- DISTINCT over a sealed map. + +We cannot skip: + +- the first evaluation of a value predicate; +- payload aggregates; +- non-resident sources; +- a generation change (502–758 µs). + +## 8. Verdict + +Format per item: producer → missing edge → consumer → test → measured cost → +owner → PR. + +**ALREADY WORKS** + +1. Lance 13 masked read with column projection. + - Path: #1633 `AddressedRelation` → `range_mask` → `with_row_addr_prefilter` + `project` → fold. + - Test: `where_lance_arm`, all oracle checks. + - Cost: clustered 0.1% = 1,176 µs read and 0.1% of the full scan's bytes. + - Owner: lab. PR: this one. +2. `ReadPolicy` cost choice. + - Path: `plan_read` → `Read::Full` on scattered 10–50%. + - Test: C-default cells. + - Cost: avoids the 1.4–11× scattered-read loss. On clustered demand it + picks wrongly; see item 7. + - Owner: lab #1633. + +**WIRE NOW** (lab only, done here) + +3. ClassView / WideFieldMask → physical Lance columns. + - Path: `ClassView::fields` → `predicate_iri` → schema metadata → `Scanner::project`. + - Test: 5 refusals + 2 traps + projected bytes. + - Cost: resolve 7–40 µs. + - Owner: lab, this PR. Production wiring needs an owner decision; not done. + +**PROBE** + +4. EXISTS early exit. + - Path: `Terminal::Any` → break per tile (`exec.rs:1793`) → Quack EXISTS. + - Test: the 10M EXISTS cells, plus a can-stay-silent test on an empty mask. + - Cost: today 0.16× DuckDB at 10M. + - Owner: mask-risc. PR: separate, not opened. +5. Ordered u8-in-u32 compare word. + - Path: ndarray / mask-risc → `Pred` → the `project` layouts. + - Test: C2. + - Cost: up to 5× slower than the hand loop when composed from existing words. + - Owner: ndarray / mask-risc. Not built. +6. Route once, reuse per frame. + - Path: P1 arm C index → a frame-scoped binding → the compose route stage. + - Test: C1. + - Cost: 158,198 → 1,012 µs per query at Q=256. + - Owner: DRF / D-GLOBAL-WHERE. + +7. `ReadPolicy` decides on coverage only. + - Path: `plan_read` → also weigh the range count → C-default. + - Test: the clustered 10% cells. + - Cost: today 3.8× (1M) and 4.6× (10M) slower than the exact read. + - Owner: lab #1633. Not changed here. + +**OWNER DECISION** + +8. One version-checked DR → `_rowid` authority, plus bulk ingress. + - Path: dt64 `Frame` → ONE witness (DR route version + Lance `ds.version()` + uri) → `RowAddrTreeMap`. + - Test: the C arm, with ids from DR instead of given. + - Cost: ingress 0.4–123,162 µs, measured. + - Owner: S-INGRESS / D4. BLOCKED_D4. + +**PARK** + +9. Scattered selective reads on a row-major page layout. + - Physical limit: bytes follow page locality, not selectivity. + - Not a code change. + - Revisit with a clustered layout or a different page size. + +**SKIP** + +10. A new mask store, trie, hash table or paging engine. Nothing measured here + needs one. + +## Not verified + +- The cause of the 50% slowdown on the borrowed path (C3). +- Disk reads: every Lance byte count came from the page cache. +- DuckDB internals: the source is not on disk (see `ENGINEERING-PARITY.md`). diff --git a/.claude/probes/2026-11-03-d-global-where-p1/SPEC.md b/.claude/probes/2026-11-03-d-global-where-p1/SPEC.md new file mode 100644 index 000000000..4eefd3837 --- /dev/null +++ b/.claude/probes/2026-11-03-d-global-where-p1/SPEC.md @@ -0,0 +1,280 @@ +# D-GLOBAL-WHERE P1 probe spec: 64K page amortization, routing arms, cost model (lab only) + +> Paths under `probe/`, `duckdb/` and `results/` refer to the research branch [`research/d-global-where-p2`](https://github.com/AdaWorldAPI/lance-graph/tree/research/d-global-where-p2/.claude/probes/2026-11-03-d-global-where-p1) (commit `1422b2b92`); they are not on `main`. + +Written by the orchestrator; implemented by workers. Root: `/home/user/lg-p1`, branch +`ccr-85acaff0-oszq40-global-where-p1`, cut from `origin/main` `ebbdb27d9`. Edit only +inside `.claude/probes/2026-11-03-d-global-where-p1/`. + +P0 (#1943) is evidence, not a dependency. Its probe is readable with +`git -C /home/user/lg-p1 show ccr-85acaff0-oszq40-global-where:.claude/probes/2026-11-03-d-global-where/probe/src/lib.rs` +(and `.../bin/falsify.rs`, `.../bin/bench.rs`, `.../probe/Cargo.toml`). Copy patterns from it +(counting allocator, SplitMix64, Quack WHERE prepared program, bit helpers); do not depend on it. + +## Hard rules + +- No production change. Consume `lance-graph-dt64` (`dr` module + `BindStore`/`MethodRef`/`admit`), + `lance-graph-quack`, `lance-graph-mask-risc`, `ndarray::simd` unchanged. +- No second identity authority. Every global → coordinate answer comes from `MapTable` / + `TileMap` under a sealed `Frame`. A derived index (arm C) is built FROM `MapTable` views under + one frame, carries that frame, refuses any other frame, and is never consulted for a frame it + was not built for. +- No tuple vectors, XYZ expansion, per-object structs or persisted intermediate relations in a + timed fold. Oracle code may use HashSet/HashMap. +- Units: ns/µs. Any warm or per-query value ≥ 1000 µs prints `FAILED-GATE`; cold/preparation + values ≥ 1000 µs print `COLD (reported, not gate)`. +- Deterministic data: SplitMix64, seed 0x9E3779B97F4A7C15. +- Every binary exits 1 on any failed check or oracle mismatch. + +## Model: three layers + +| layer | here | +|---|---| +| global corpus | u64 global ids, NOT contiguous (pseudo-random over 0..2^32), N up to 10M | +| logical 64K page | ONE `MapTable` (its own frame store and clock) holding up to 256 DR256 tiles, TileIds 0..255: exactly the reservation `Extent::DR256_GLOBAL_X64` names. Page coordinate = `tile_ordinal * 256 + local` (u16 range) | +| active tile | a `TileMap` borrowed from the page under a frame | + +A page holds only the identities a selection demands. Page membership is chosen by the +selection, never by bits of the global id. + +## Routing arms (identical answers, including refusals) + +- **A, baseline:** `MapTable::locate(frame, &page_tiles, g)` per endpoint. +- **B, page-grouped batch:** collect the distinct globals of all endpoints, sort them, locate each + ONCE with `locate`, then resolve endpoints by a merge/binary search over that sorted transient + array (dropped after the call: not an index). Measure locate calls (should equal distinct ids). +- **C, derived index (optional acceleration):** built from `table.map(&frame, t).globals()` for + every page tile: a sorted `Vec<(u64 global, u16 coord)>`. Adjacent equal globals → that id is + `Ambiguous`; absence → `Unmapped`. Stores the `Frame` it was built from; `lookup(frame, g)` + refuses with `StaleIndex` unless `frame.same_issue(&built)`. Building costs one pass over the + page's ids plus a sort; lookup is one binary search (no per-tile scan). + +## Part 1: `src/bin/falsify_p1.rs` + +Fixture: page P of 4 DR256 tiles (1,000 ids), page Q of 2 tiles (400 ids), disjoint except as +stated; edges among P ∪ Q (avg out-degree 4, ~8 % crossing P→Q); lanes `a`,`b` over the ids +(stored as a sparse lookup in the ORACLE only; the folds read lanes indexed by page coordinate +built at page preparation as a borrowed slice of the source lane order: put the fixture's lane +values in an array indexed by the position of the id in a fixture list, and document it). + +Checks F1..F20 (print `Fn PASS/FAIL detail`, exit 1 on failure): + +1. G→L→G and L→G→L parity for every id of P and Q, all three arms. +2. Arms A, B, C give identical coordinates for every edge endpoint of P. +3. `locate` calls: A = 2·edges(P), B = distinct endpoint ids, C = 0 per query after build. +4. An id inserted in two tiles of P: A → `Ambiguous`, B → `Ambiguous`, C → `Ambiguous`. +5. An id in no tile: all arms `Unmapped`. +6. Foreign frame (sealed by Q's table) read against P: A `Stale(ForeignFrame)`; C refuses. +7. Historical frame: seal f1; `advance` replacing tile 0 with a reordered map → f2. Under f1, A and + a C built for f1 return the OLD coordinates; C(f1) queried with f2 → `StaleIndex`; C rebuilt + for f2 == A under f2. +8. Extension: append ids to tile 1 (an extension, `TileMap::extends`); every old id keeps its + coordinate (presence plane built at f_old is still valid for old ids: the answer of the W2 query + restricted to old ids is unchanged); a guard compares generations, not digests. +9. Replacement invalidates the binding: reusing the f1 binding (coords + presence) under f2 without + the frame check gives a WRONG answer (measured); with the check it refuses. +10. Tile removal: an id of a removed tile → `Unmapped` under the new frame, still resolves under + the old frame. +11. Presence-only update: delete an edge; frame and coordinates unchanged; the binding is reused + (0 new locate calls for the fold; the update itself charges endpoint routing via C); the + answer drops by the oracle's amount. An unrelated deletion leaves the answer unchanged. +12. Cross-page edge: x ∈ P, y ∈ Q; B's selection held in Q coordinates; evaluating x's edge via + Q's frame gives the oracle; joining by coordinate coincidence gives a different (wrong) + count. +13. The same global id at different coordinates in two selections/pages; DR-aligned equality. +14. Reused binding == freshly prepared binding (coords and all W2/W3/W4 answers), Q = 64 reuses. +15. EXO-46 shared selection (LAB type `SharedSelection { frame, scope, criteria, mask }`): + a. M1 and M2 (same class, both registered with `BindStore::register`, distinct `MethodRef`s) + both admit the SAME selection when each holds its own permit (a permit = a `Vec` + the probe issues); + b. M2 presenting only M1's permit → refused (no inheritance); + c. a MethodRef of another class → refused (`admit` of dt64); + d. a stale sig rev → refused; + e. another Scope → refused; + f. a stale frame (after `advance`) → refused; + g. a presence update does not change the sealed selection's mask (bit-identical before/after) + while the fold over it reflects the new presence; + h. no mask is passed between programs (structural: the selection owns the mask, folds borrow + it); + i. zero allocation in admit + fold. + Read `crates/lance-graph-dt64/src/lib.rs` for `BindStore`, `MethodBinding`, `MethodRef::of`, + `admit` and its refusal enum; the merged probe + `.claude/probes/2026-11-03-d-xlog-meta-wire/probe/src/{main.rs,bin/v2_compose.rs}` shows a + working registration. +16. AND/XOR/FOLD parity with the oracle on W2 (1-hop EXISTS into B), W3 (2-hop EXISTS), W4 (3-hop + DISTINCT reached in B), W5 (COUNT DISTINCT 1-hop), W6 (Bag 2-path count, checked i64). +17. Folds allocate 0 bytes; no tuple vector. +18. Disable C's duplicate check → F4 fails for C (run as a compiled-in `--disable dupcheck` flag: + the binary takes optional `--disable ` and must then report FAIL on the named check). +19. Disable C's frame check → F7/F9 fail (`--disable framecheck`). +20. Disable binding version guard → F9 gives the wrong answer silently (`--disable bindguard`). + +## Part 2: `src/bin/sweep.rs` (amortization law and plan choice) + +Measure components per configuration, then compute the law, then verify it end to end. + +Configurations: active ids per page n ∈ {64, 256, 4096, 65536} (tiles = ceil(n/256), DR64 when +n ≤ 64 via `Axis::for_survivors`), pages p ∈ {1, 4, 16} (p = 64 only for n ≤ 4096), +distribution ∈ {sparse(deg≈1), medium(deg≈8), dense(deg≈min(64, n/8)), skewed(Zipf, mean 8)}, +cross-page edge fraction ∈ {0.01, 0.30}. Skip a config whose edge count would exceed 40M and say +so. + +Per config, single thread, median of 31 runs: +- `prep_A` / `prep_B` / `prep_C`: build the page tiles (`MapTable::advance`), route all edge + endpoints with the arm, build the coordinate CSR (u32 offsets, u16/u32 targets per page; + cross-page targets carry (page, coord)). +- `route_q_A`, `route_q_B`, `route_q_C`: per-query routing cost if the binding is NOT reused (the + query routes its selection, |A| ids, from scratch). +- `fold_W2`, `fold_W3_csr`, `fold_W3_front`, `fold_W4_csr`, `fold_W4_front`, `fold_W4_tile` (tile + only for total active ≤ 4096: dense rows). +- `where_v`: virtual WHERE over the page's ids (predicate at mapped globals). +- `update`: delete k edges (k = 0.1 %, 1 % of edges) through C: clear presence bits in the CSR's + edge-validity mask; coordinates untouched. +Then C_query(Q) = where_v + route_q(reused? 0 : arm) + fold + prep/Q + maintenance/Q for +Q ∈ {1,4,16,64,256,1024} and change rates {0, 0.1 %, 1 %} per query. Break-even Q for: reuse vs +per-query routing (A, B, C). Verify the law: for Q = 64, run the real loop (prep once, 64 queries +with updates) and compare measured total with the formula (report relative error). + +Plan choice: `fn choose(meta: &Meta) -> Arm` reading ONLY pre-execution metadata (|A|, |B|, n, +p, edges, mean out-degree, max degree, cross-page fraction, hops, binding prepared?). Its +constants are fitted on a CALIBRATION set (seed+1 datasets) and evaluated on the TEST set (seed). +Report per test config: chosen arm, measured best arm, regret = t(chosen)/t(best), choose() cost +in ns. The acceptance test must show the preferred arm changing across configs. + +Output `$OUT/sweep.json` (one record per config × metric) and `$OUT/law.json` (per config × Q × +change rate: predicted cost, components, break-evens) and `$OUT/plan.json`. + +## Part 3: `src/bin/big.rs` (10M corpus, DuckDB comparison data) + +N = 10,000,000 nodes, global ids = a SplitMix64 permutation-free random u32 per node (unique: +regenerate on collision; report) — not 0..N. Lanes a, b uniform 0..1000. Edges: out-degree +Poisson-like mean 2 (skewed variant: 1 % hubs with degree 200) → about 20M edges, global ids. +Write `$OUT/big/nodes.csv` (id,a,b), `edges.csv` (src,dst), `delete.csv`. + +Load (charged, reported separately, like DuckDB's load): a global CSR over node index (offsets +u32[N+1], targets u32 node index), and a global-id → node-index lookup used only for loading. + +Queries (A = a < 1, ~0.1 %; B parameter t varies per run over {100, 200, 300, 400, 500}): +- G1 `count(*) WHERE a < 1` over 10M rows: Quack `Agg::Count` over the 10M lane. +- G2 1-hop EXISTS into B, G3 2-hop EXISTS, G4 3-hop COUNT DISTINCT reached in B, + G5 COUNT DISTINCT 1-hop, G6 Bag 2-path count. +Ours, two modes: +- `direct`: no page binding; walk the global CSR from A every query (value reads through node index). +- `paged`: once per selection A: demanded identities (A ∪ 1/2/3-hop frontier) bound into 64K + pages (MapTable, 256 tiles each, arm C index); page-coordinate CSR; then each query = virtual + WHERE for B on page globals + folds in page coordinates. Report pages, active ids, cross-page + edges, prep cost, and the break-even Q against `direct` and against DuckDB's warm. +- Presence update: delete 0.1 % of active edges (validity bits); re-query. +Every answer against a scalar oracle over the edge list. +`$OUT/big/ours_big.json` records: {query, mode, answer_per_t: [..], cold_us, prep_us, warm_med_us, +warm_p10_us, warm_p90_us, peak_bytes, alloc_query_bytes, pages, active_ids, cross_page_edges, +update_us}. + +## Part 4: `duckdb/bench_big.py`, `duckdb/compare_big.py` + +DuckDB 2.0.0, threads 1 and 4, normal optimizer. Load CSVs (load_us reported). For each query a +prepared statement with parameter $1 = t (G1 has none); cold = PREPARE + first EXECUTE; warm = +median/p10/p90 of 51 EXECUTEs cycling t over {100..500}. Engine time from JSON profiling +`query.total_time` (one execution per t). Do NOT subtract `SELECT 1` as Python overhead: report +it as `trivial_query_wall_us` beside wall and engine. EXPLAIN ANALYZE per query (threads 1) to +`$OUT/big/explain/`. Delete via the delete.csv set, time it, re-run. `duckdb_memory()` totals. +Write `$OUT/big/duckdb_big.json`. `compare_big.py` asserts every answer per t equal (exit 1 +otherwise) and prints a Markdown table: ours direct / paged warm, paged prep, break-even Q vs +DuckDB warm (prep / (duck_warm − ours_warm), ∞ when ours is slower), DuckDB wall/engine/cold/load, +both thread counts, verdict WIN / PARITY (±10 %) / LOSS on wall and engine. Write `compare_big.md`. + +SQL (one dict, identical semantics): +- G1 `SELECT count(*) FROM nodes WHERE a < 1` +- G2 `SELECT count(*) FROM nodes n WHERE n.a < 1 AND EXISTS (SELECT 1 FROM edges e JOIN nodes m ON m.id = e.dst WHERE e.src = n.id AND m.b < $1)` +- G3 `SELECT count(*) FROM nodes n WHERE n.a < 1 AND EXISTS (SELECT 1 FROM edges e1 JOIN edges e2 ON e2.src = e1.dst JOIN nodes m ON m.id = e2.dst WHERE e1.src = n.id AND m.b < $1)` +- G4 `SELECT count(DISTINCT e3.dst) FROM nodes n JOIN edges e1 ON e1.src = n.id JOIN edges e2 ON e2.src = e1.dst JOIN edges e3 ON e3.src = e2.dst JOIN nodes m ON m.id = e3.dst WHERE n.a < 1 AND m.b < $1` +- G5 `SELECT count(DISTINCT e.dst) FROM edges e JOIN nodes n ON n.id = e.src JOIN nodes m ON m.id = e.dst WHERE n.a < 1 AND m.b < $1` +- G6 `SELECT count(*) FROM edges e1 JOIN edges e2 ON e2.src = e1.dst JOIN nodes n ON n.id = e1.src JOIN nodes m ON m.id = e2.dst WHERE n.a < 1 AND m.b < $1` + +--- + +## CORRECTION (operator, 2026-11-03): working-set construction is P0, paging is secondary + +The hypothesis is NOT "partition the corpus into 64K pages". It is: **constructing the right, +query-specific ≤ 64K active working set from an unbounded corpus solves most of the scaling; +the existing DR-local fold algebra then runs unchanged.** The "~98 %" figure is a working +hypothesis, not a measurement. + +Consequences for this spec (they override the text above where they conflict): + +1. **Naming.** The lib type is `WorkingSet` (one `MapTable`, up to 256 DR256 tiles, ≤ 65,536 + identities chosen by the QUERY, not contiguous, not by id bits). "Page" in Parts 1–3 means + this working set. A second working set exists only for bounded continuation. +2. **P0 measured stages, reported separately** (every binary that builds one): + `discover` (global WHERE → candidate ids: scan arm over the corpus lane via Quack Keep, and an + optional pre-built attribute-index arm whose build is charged as preprocessing), + `expand` (reachable demanded ids via the global relation index, bounded at 65,536), + `bind` (DR coordinate binding of exactly those ids: arms A/B/C), + `project` (relation views restricted to the working set: coordinate CSR + presence bits), + `fold`, and `reuse` (Q queries over one binding). +3. **Corpus-size independence is the success criterion.** Part 3 runs N ∈ {1M, 10M} (30M if + disk/RAM allow; say so if not) with the SAME demanded active population. Report each stage per + N. Stages whose cost tracks N (expected: the scan arm of `discover`) are named as such; the + claim holds only if `expand + bind + project + fold` do not grow with N. +4. **Incremental maintenance (P1).** Presence-only update: the binding survives, only presence + bits change. Selection change: B-side parameter changes re-run only `where_v` on the + working-set ids; an A-side change that adds ids EXTENDS the working set (`MapTable::advance` + with `TileMap::extends`, old coordinates kept) instead of rebuilding; measure both against a + full rebuild. +5. **Bounded continuation (P1).** Only when the demanded set exceeds 65,536 (G4 3-hop on the + skewed 10M corpus may): a second working set, cross-set edges resolved through the global id + (`locate` in the other set's frame), Boolean DISTINCT and Bag counts exact. Report its cost + separately. No unbounded intermediate, no second identity authority. +6. **Part 2 (sweep) is reduced** to: active size n ∈ {64, 256, 4096, 65536} × distribution × + cross-set fraction {0, 0.3}; routing arms A/B/C; amortization over Q; update rates; plan choice. + The "pages p" dimension is dropped except p = 2 for the continuation case. +7. **DuckDB (Part 4)** also runs N ∈ {1M, 10M}; report preprocessing on both sides (our global + relation index + optional attribute index; DuckDB load + ANALYZE, and an optional + `CREATE INDEX` arm on edges(src), nodes(id) with its build time). + +--- + +## P2 (operator, 2026-11-03): criteria-aware WHERE and multi-domain DR composition + +This PR carries P1 and P2 together (title P2). Three independent variables, never merged into one +"dataset size": **K** (criteria per WHERE), **population** (survivors), **D** (DR domains composed). + +### Source inventory (orchestrator, read in `crates/lance-graph-quack/src/lib.rs`) +- `Filter::{Cmp, Plane, EqU32Via, Semijoin, And(Vec), Or(Vec), Not}` (`:336`), `Filter::and_by_skip` + (selectivity-ordered AND, `:752`), `Filter::sql_where` three-valued rewrite over validity planes + (`:549`), `lower` (`:1383`) and `lower_fused` (ternlog fusion, `:1397`), executed by mask-risc + `execute_into`. So K-criteria WHERE over resident lanes is PRODUCTION; `EqU32Via`/`Semijoin` are + EXO-140-gated and not used here. + +### Part 5: `src/bin/where_k.rs` + `duckdb/bench_where.py` +Corpus N ∈ {1M, 10M}; columns c0..c15 i32 and nullable c15 (null when `f(id,15) % 10 == 0`). +Values are a closed formula both sides compute identically (no CSV for nodes): +`f(id, j) = (((id * 2654435761 + j * 40503) % 2147483648) * 1103515245 % 2147483648) % 1000` +(all BIGINT, no overflow: operands < 2^31 before each multiply). Correlated column: c1 = +`(c0 + f(id,1) % 20) % 1000`. +Cases: K groups {1, 2–4, 5–16, ≥17 (17 and 24, criteria may reuse columns with different +constants)} × population {tiny ≤ 100, medium ≈ 1 %, large ≈ 50 %} × shape {AND-heavy, OR-heavy, +mixed} plus the adversarial pairs: one criterion ≈ everything; one ≈ nothing; 16 nonselective; 16 +collapsing to one survivor; correlated criteria that do not reduce; contradictory → empty; a +nullable-column case through `sql_where`. Thresholds are SEARCHED by the generator to hit each +population target and the achieved count is reported (never assumed from K). +The generator writes `$OUT/where/cases.json`: each case = predicate tree {op: and|or|not|cmp, +col, cmpop, value} + target + achieved count (oracle). Python builds SQL from the same tree. +Ours per case, single thread, median of 31: `lower_us` (plain and fused), `eval_us` for +`Agg::Count`, `Agg::Any`, `Agg::Rows` (Keep → transient mask), plain vs fused vs `and_by_skip` +order, temporary bytes, and the representation used: `virtual` (Count/Any: no mask survives), +`transient` (Keep mask), `persistent` (never here). Report weighted (34/46/14/6 over K groups) and +unweighted tables. DuckDB: same predicate as `SELECT count(*) FROM nodes WHERE ` and an +`EXISTS` variant (`SELECT EXISTS(SELECT 1 FROM nodes WHERE )`), prepared, threads 1 and 4, +answers must match exactly; EXPLAIN ANALYZE for one case per K group. Output `where_ours.json`, +`where_duckdb.json`, `compare_where.md`. + +### Multi-domain composition (Part 1 additions, F22–F29) +F22 one / two / three DR domains give the same answers as one domain over the union (EXISTS, +COUNT, DISTINCT, Bag). F23 an edge crossing a domain boundary; F24 a path crossing three domains; +F25 the same global id in two selections; F26 a duplicate id across incorrectly configured domains +→ refused (`Ambiguous` when listed together; a global-id union check otherwise); F27 a deleted +cross-domain edge; F28 historical replay across domains; F29 aggregate composition laws: EXISTS = +OR of parts, COUNT = checked sum over DISJOINT contributions, DISTINCT = exact union (and a +falsifier: summing per-domain distinct counts over overlapping domains gives the WRONG answer), +Bag = multiplicities preserved. diff --git a/crates/crepe-plan-probe/examples/where_lance_arm.rs b/crates/crepe-plan-probe/examples/where_lance_arm.rs new file mode 100644 index 000000000..6d813395b --- /dev/null +++ b/crates/crepe-plan-probe/examples/where_lance_arm.rs @@ -0,0 +1,1422 @@ +//! LAB. One integration test of the chain +//! +//! ```text +//! ClassView -> WideFieldMask (demanded property positions) +//! -> WHERE (logical predicate) / DR selection (identities) +//! -> Lance row-address mask (physical row eligibility) +//! -> Scanner::with_row_addr_prefilter -> Scanner::project (resolved columns only) +//! -> fold +//! ``` +//! +//! Run in release from this crate's directory: +//! +//! ```text +//! cargo run --release --example where_lance_arm -- [--n 1000000] [--reps 5] +//! ``` +//! +//! Four masks, kept distinct in type, name and output: +//! - `demanded: WideFieldMask` which PROPERTIES (logical positions of the class); +//! - `where_bits: Vec` which ROWS satisfy the logical predicate (arm A, B); +//! - `dr_ids: Vec` the DR-side identities (arm C; given, not derived); +//! - `RowAddrMask` which PHYSICAL rows Lance may read. +//! +//! Arms (every row labelled "Lance-backed"): +//! - A FULL: one scan of where-columns + resolved demanded columns, WHERE in Rust. +//! - B MASKED PREFILTER: scan the where-columns with `_rowaddr`, WHERE -> row list, +//! build `RowAddrMask` (MASK INGRESS: this COPIES the addresses into Lance's +//! `RowAddrTreeMap`, it is not zero-copy), re-scan with `with_row_addr_prefilter` +//! projecting the resolved demanded columns only. ALLOW always; BLOCK as well at 50%. +//! - C ATTESTED ADDRESSED: `SealedFacetLane::attest_sorted` once per version (cold), +//! ids -> lane ranges with `seek_from`, `addressed::plan_read` under `EXACT` and +//! `DEFAULT`, a range `RowAddrMask`, prefilter scan, fold. +//! +//! WHAT IS CLAIMED: only what the printed table and the oracle checks show on this +//! fixture, on this machine, in this run. The answers of all arms equal a scalar +//! oracle over the generator formula and a full-projection read. +//! +//! WHAT IS NOT CLAIMED: no speed-up in general; no zero-copy for the mask ingress; +//! no DR -> Lance coordinate authority (arm C's ids are GIVEN); no claim that the +//! ClassView resolution covers a production ontology (a 16-field lab class). +//! +//! OWNER GATES, both open: +//! - S-INGRESS (#1788): the mask ingress copies; which coordinate space is the +//! authority for a row address is an operator decision, not made here. +//! - D4 / the joint coordinate witness: no witness binds a DR Frame to a Lance +//! dataset version (see `d4_boundary`, printed as BLOCKED_D4). Nothing here +//! composes DR ordinals with `_rowid` in a stored table. +//! +//! No-ms rule: a warm in-fold value >= 1000 us (WHERE eval, ClassView resolution, +//! DR lane lookup, fold) prints FAILED-GATE (reported). Lance I/O, mask ingress and +//! cold phases print COLD/IO (reported, not gated). +//! +//! Drafted without being compiled; the orchestrator compiles centrally. + +use std::collections::HashMap; +use std::sync::{Arc, Mutex}; +use std::time::Instant; + +use arrow_array::{Int32Array, RecordBatch, RecordBatchIterator, UInt32Array, UInt64Array}; +use arrow_schema::{DataType, Field, Schema}; +use crepe_plan_probe::addressed::{ + AddressedRelation, Read, ReadPolicy, check_fresh, key, leading, plan_read, range_tree, ranges, +}; +use crepe_plan_probe::logic::RelId; +use lance::dataset::scanner::{ExecutionStatsCallback, RowAddrMask, RowAddrTreeMap}; +use lance::dataset::{Dataset, WriteParams}; +use lance_graph_contract::class_view::{ClassId, ClassView, FieldMask, WideFieldMask}; +use lance_graph_contract::ontology::{DisplayTemplate, FieldRef}; +use lance_graph_contract::ordered_lane::{OrderedLaneWitness, SealedFacetLane}; +use mork_pathmap_probe::join::for_each_set; + +const SEED: u64 = 0x9E37_79B9_7F4A_7C15; +const NPROPS: usize = 16; +const CLASS: ClassId = 1; +const IRI_KEY: &str = "predicate_iri"; + +// ---------------------------------------------------------------- generator + +struct SplitMix64(u64); +impl SplitMix64 { + fn next(&mut self) -> u64 { + self.0 = self.0.wrapping_add(0x9E37_79B9_7F4A_7C15); + let mut z = self.0; + z = (z ^ (z >> 30)).wrapping_mul(0xBF58_476D_1CE4_E5B9); + z = (z ^ (z >> 27)).wrapping_mul(0x94D0_49BB_1331_11EB); + z ^ (z >> 31) + } +} + +/// The value of property `j` on row `id`; the oracle's only source of truth. +fn gen_val(id: u32, j: usize) -> i32 { + let x = (u64::from(id) * 2_654_435_761 + (j as u64) * 40_503) % 2_147_483_648; + ((x * 1_103_515_245 % 2_147_483_648) % 1000) as i32 +} + +// ------------------------------------------------------------------ fixture + +/// Physical ordinal (0-based, after `id`) of logical property `j`: a permutation. +fn phys_ordinal(j: usize) -> usize { + (j * 5 + 3) % NPROPS +} +/// Logical property stored at physical ordinal `ord` (inverse: 5 * 13 = 65 = 1 mod 16). +fn logical_at(ord: usize) -> usize { + ((ord + 13) * 13) % NPROPS +} +/// Physical column name: deliberately unlike `p{j}` and "Property {j}". +fn phys_name(ord: usize) -> String { + format!("m{:03}", 417 + ord * 37) +} + +#[derive(Clone, Copy, PartialEq, Eq)] +enum Layout { + Permuted, + /// Permuted plus `p_dup`, a second column carrying the iri of `p15`. + Ambiguous, +} + +fn fixture_schema(layout: Layout) -> Arc { + let mut fields = vec![Field::new("id", DataType::UInt32, false)]; + for ord in 0..NPROPS { + let j = logical_at(ord); + let md = HashMap::from([(IRI_KEY.to_string(), format!("lab:prop/p{j}"))]); + fields.push(Field::new(phys_name(ord), DataType::Int32, false).with_metadata(md)); + } + if layout == Layout::Ambiguous { + let md = HashMap::from([(IRI_KEY.to_string(), "lab:prop/p15".to_string())]); + fields.push(Field::new("p_dup", DataType::Int32, false).with_metadata(md)); + } + Arc::new(Schema::new(fields)) +} + +fn make_batch(layout: Layout, ids: std::ops::Range) -> RecordBatch { + let schema = fixture_schema(layout); + let mut cols: Vec = + vec![Arc::new(UInt32Array::from_iter_values(ids.clone()))]; + for ord in 0..NPROPS { + let j = logical_at(ord); + cols.push(Arc::new(Int32Array::from_iter_values( + ids.clone().map(|id| gen_val(id, j)), + ))); + } + if layout == Layout::Ambiguous { + cols.push(Arc::new(Int32Array::from_iter_values( + ids.clone().map(|id| gen_val(id, 15)), + ))); + } + RecordBatch::try_new(schema, cols).expect("fixture batch") +} + +fn reader( + layout: Layout, + range: std::ops::Range, +) -> RecordBatchIterator>> { + const CHUNK: u32 = 1_000_000; + let (lo, hi) = (range.start, range.end); + let chunks = (hi - lo).div_ceil(CHUNK); + let it = (0..chunks).map(move |c| { + let a = lo + c * CHUNK; + let b = (a + CHUNK).min(hi); + Ok(make_batch(layout, a..b)) + }); + RecordBatchIterator::new(it, fixture_schema(layout)) +} + +/// Written in ascending id order as ONE fragment (`max_rows_per_file` lifted). +/// +/// The limit must fit in `u32`: Lance 13 compares it as +/// `params.max_rows_per_file as u32` (`dataset/write.rs:1077`), so `1 << 40` +/// truncates to 0 and closes a file after EVERY batch. At 1M rows the reader +/// emitted one batch and hid that; at 10M it produced many fragments and the +/// attest guard refused the table as `NotContiguous`. +async fn write_fixture(uri: &str, layout: Layout, n: u32) -> Result { + let params = WriteParams { + max_rows_per_file: u32::MAX as usize, + ..Default::default() + }; + Dataset::write(reader(layout, 0..n), uri, Some(params)) + .await + .map_err(|e| e.to_string()) +} + +// ----------------------------------------------------------------- ClassView + +struct LabClass { + fields: Vec, +} +impl LabClass { + fn with_fields(n: usize) -> Self { + Self { + fields: (0..n) + .map(|j| FieldRef::new(format!("lab:prop/p{j}"), format!("Property {j}"))) + .collect(), + } + } +} +impl ClassView for LabClass { + fn fields(&self, _class: ClassId) -> &[FieldRef] { + &self.fields + } + fn template(&self, _class: ClassId) -> DisplayTemplate { + DisplayTemplate::Detail + } + fn dolce_category_id(&self, _class: ClassId) -> u8 { + 0 + } +} + +#[derive(Debug, Clone, PartialEq, Eq)] +enum Refusal { + AbsentField { + pos: u8, + }, + Unauthorized { + pos: u8, + }, + UnknownProperty { + pos: u8, + iri: String, + }, + AmbiguousProperty { + pos: u8, + iri: String, + columns: Vec, + }, + StaleSchema { + resolved: u64, + current: u64, + }, +} + +/// The only path from demanded positions to a column list. On any refusal there +/// is no column list at all: the caller cannot widen to "all columns". +struct Resolution { + version: u64, + positions: Vec, + columns: Vec, +} + +impl Resolution { + fn check_current(&self, latest: u64) -> Result<(), Refusal> { + if latest != self.version { + return Err(Refusal::StaleSchema { + resolved: self.version, + current: latest, + }); + } + Ok(()) + } +} + +/// position -> `FieldRef.predicate_iri` -> exactly one schema field whose +/// metadata `predicate_iri` matches. `at` is the handle pinned at the attested +/// version; its schema is the one read. +fn resolve( + view: &dyn ClassView, + class: ClassId, + demanded: &WideFieldMask, + presence: FieldMask, + permit: &WideFieldMask, + at: &Dataset, +) -> Result { + let fields = view.fields(class); + let schema = at.schema(); + let mut out = Resolution { + version: at.version().version, + positions: Vec::new(), + columns: Vec::new(), + }; + for p in 0..=255u8 { + if !demanded.has(p) { + continue; + } + if usize::from(p) >= fields.len() || !presence.has(p) { + return Err(Refusal::AbsentField { pos: p }); + } + if !permit.has(p) { + return Err(Refusal::Unauthorized { pos: p }); + } + let iri = &fields[usize::from(p)].predicate_iri; + let hits: Vec<&str> = schema + .fields + .iter() + .filter(|f| f.metadata.get(IRI_KEY).map(String::as_str) == Some(iri.as_str())) + .map(|f| f.name.as_str()) + .collect(); + match hits.len() { + 0 => { + return Err(Refusal::UnknownProperty { + pos: p, + iri: iri.clone(), + }); + } + 1 => { + out.positions.push(p); + out.columns.push(hits[0].to_string()); + } + _ => { + return Err(Refusal::AmbiguousProperty { + pos: p, + iri: iri.clone(), + columns: hits.iter().map(|s| s.to_string()).collect(), + }); + } + } + } + Ok(out) +} + +// ----------------------------------------------------------------- scanning + +#[derive(Debug, Clone, Copy, Default)] +struct Io { + rows_scanned: usize, + bytes_read: usize, +} + +/// Same callback pattern as `lance_store.rs:162-175`. +fn stats_sink() -> (Arc>, ExecutionStatsCallback) { + let stats: Arc> = Arc::default(); + let sink = stats.clone(); + let cb: ExecutionStatsCallback = Arc::new(move |c| { + let mut io = sink.lock().expect("stats lock"); + io.bytes_read = c.bytes_read; + io.rows_scanned = c.all_counts.get("rows_scanned").copied().unwrap_or(0); + }); + (stats, cb) +} + +async fn scan( + ds: &Dataset, + cols: &[String], + with_addr: bool, + mask: Option, +) -> Result<(RecordBatch, Io), String> { + let (stats, cb) = stats_sink(); + let mut s = ds.scan(); + s.project(cols).map_err(|e| e.to_string())?; + if with_addr { + s.with_row_address(); + } + if let Some(m) = mask { + s.with_row_addr_prefilter(m); + } + s.scan_stats_callback(cb); + let b = s.try_into_batch().await.map_err(|e| e.to_string())?; + let io = *stats.lock().expect("stats lock"); + Ok((b, io)) +} + +fn i32col<'a>(b: &'a RecordBatch, name: &str) -> &'a [i32] { + let a = b + .column_by_name(name) + .and_then(|c| c.as_any().downcast_ref::()) + .unwrap_or_else(|| panic!("i32 column {name}")); + &a.values()[..] +} +fn u32col<'a>(b: &'a RecordBatch, name: &str) -> &'a [u32] { + let a = b + .column_by_name(name) + .and_then(|c| c.as_any().downcast_ref::()) + .unwrap_or_else(|| panic!("u32 column {name}")); + &a.values()[..] +} +fn addrcol(b: &RecordBatch) -> &[u64] { + let a = b + .column_by_name("_rowaddr") + .and_then(|c| c.as_any().downcast_ref::()) + .expect("_rowaddr"); + &a.values()[..] +} + +fn push_unique(v: &mut Vec, s: &str) { + if !v.iter().any(|x| x == s) { + v.push(s.to_string()); + } +} + +// -------------------------------------------------------------------- WHERE + +#[derive(Clone, Copy)] +enum Where { + /// `p0 < t AND p1 < u` (scattered). + Scatter { t: i32, u: i32 }, + /// `id in [lo, hi)` (clustered). + Range { lo: u32, hi: u32 }, +} +impl Where { + fn positions(&self) -> WideFieldMask { + match self { + Where::Scatter { .. } => WideFieldMask::from_positions(&[0, 1]), + Where::Range { .. } => WideFieldMask::EMPTY, + } + } +} + +fn eval_scatter(p0: &[i32], p1: &[i32], t: i32, u: i32) -> Vec { + let mut out = vec![0u64; p0.len().div_ceil(64)]; + for (wi, (a, b)) in p0.chunks(64).zip(p1.chunks(64)).enumerate() { + let mut word = 0u64; + for (bit, (&x, &y)) in a.iter().zip(b).enumerate() { + word |= u64::from((x < t) & (y < u)) << bit; + } + out[wi] = word; + } + out +} +fn eval_range(id: &[u32], lo: u32, hi: u32) -> Vec { + let mut out = vec![0u64; id.len().div_ceil(64)]; + for (wi, a) in id.chunks(64).enumerate() { + let mut word = 0u64; + for (bit, &x) in a.iter().enumerate() { + word |= u64::from((x >= lo) & (x < hi)) << bit; + } + out[wi] = word; + } + out +} +fn eval_where(w: &Where, b: &RecordBatch, res_w: &Resolution) -> Vec { + match *w { + Where::Scatter { t, u } => eval_scatter( + i32col(b, &res_w.columns[0]), + i32col(b, &res_w.columns[1]), + t, + u, + ), + Where::Range { lo, hi } => eval_range(u32col(b, "id"), lo, hi), + } +} +fn where_cols(w: &Where, res_w: &Resolution) -> Vec { + match w { + Where::Scatter { .. } => res_w.columns.clone(), + Where::Range { .. } => vec!["id".to_string()], + } +} + +// --------------------------------------------------------------------- fold + +fn fold_mask(bits: &[u64], n: usize, cols: &[&[i32]]) -> (u64, Vec) { + let count = bits.iter().map(|w| u64::from(w.count_ones())).sum(); + let sums = cols + .iter() + .map(|c| { + let mut s = 0i64; + for_each_set(bits, n, |i| s += i64::from(c[i])); + s + }) + .collect(); + (count, sums) +} +fn fold_all(cols: &[&[i32]]) -> Vec { + cols.iter() + .map(|c| c.iter().map(|&x| i64::from(x)).sum()) + .collect() +} + +// ------------------------------------------------------------------- oracle + +struct Oracle { + bits: Vec, + count: u64, + sums: Vec, + ids: Vec, +} + +/// Scalar loop over the generator formula; never reads Lance. +fn oracle(n: u32, w: &Where, dem: &[u8]) -> Oracle { + let mut bits = vec![0u64; (n as usize).div_ceil(64)]; + let mut count = 0u64; + let mut sums = vec![0i64; dem.len()]; + for id in 0..n { + let sel = match *w { + Where::Scatter { t, u } => gen_val(id, 0) < t && gen_val(id, 1) < u, + Where::Range { lo, hi } => id >= lo && id < hi, + }; + if sel { + bits[(id / 64) as usize] |= 1u64 << (id % 64); + count += 1; + for (s, &j) in sums.iter_mut().zip(dem) { + *s += i64::from(gen_val(id, usize::from(j))); + } + } + } + let mut ids = Vec::with_capacity(count as usize); + for_each_set(&bits, n as usize, |i| ids.push(i as u32)); + Oracle { + bits, + count, + sums, + ids, + } +} + +/// Smallest-error `T` (with `U = T`) per target fraction, from the exact 2-D +/// histogram of (p0, p1) over all rows. +fn calibrate(n: u32, targets: &[f64]) -> Vec<(i32, f64)> { + const W: usize = 1001; + let mut pre = vec![0u64; W * W]; + for id in 0..n { + let (a, b) = (gen_val(id, 0) as usize, gen_val(id, 1) as usize); + pre[(a + 1) * W + (b + 1)] += 1; + } + for a in 1..W { + for b in 1..W { + pre[a * W + b] += pre[(a - 1) * W + b] + pre[a * W + b - 1] - pre[(a - 1) * W + b - 1]; + } + } + targets + .iter() + .map(|&f| { + let best = (1..=1000usize) + .min_by(|&x, &y| { + let ex = (pre[x * W + x] as f64 / f64::from(n) - f).abs(); + let ey = (pre[y * W + y] as f64 / f64::from(n) - f).abs(); + ex.partial_cmp(&ey).expect("finite") + }) + .expect("range"); + (best as i32, pre[best * W + best] as f64 / f64::from(n)) + }) + .collect() +} + +// ------------------------------------------------------------ attested lane + +/// Reads the keys once (the cold cost) and attests their order, as +/// `addressed::attest` does (`addressed.rs:101-143`), over the `id` column. +async fn attest(ds: &Dataset) -> Result<(SealedFacetLane, OrderedLaneWitness), String> { + let (b, _) = scan(ds, &["id".to_string()], true, None).await?; + let id = u32col(&b, "id"); + let addr = addrcol(&b); + if !(0..b.num_rows()).all(|i| addr[i] == i as u64) { + return Err("NotContiguous: table is not one contiguous fragment".into()); + } + // The prefilter is keyed in `_rowid` space; the lane positions are + // `_rowaddr`. They coincide only without stable row ids. Assert it. + let mut sc = ds.scan(); + sc.project::<&str>(&[]).map_err(|e| e.to_string())?; + sc.with_row_id(); + sc.with_row_address(); + let rb = sc.try_into_batch().await.map_err(|e| e.to_string())?; + let rid = rb + .column_by_name("_rowid") + .and_then(|c| c.as_any().downcast_ref::()) + .ok_or("no _rowid column")?; + let rad = rb + .column_by_name("_rowaddr") + .and_then(|c| c.as_any().downcast_ref::()) + .ok_or("no _rowaddr column")?; + if rid.values() != rad.values() { + return Err( + "RowIdSpace: _rowid != _rowaddr (stable row ids?); the lane cannot key the prefilter" + .into(), + ); + } + let keys = id.iter().map(|&v| key(v, 0)).collect(); + let lane = SealedFacetLane::attest_sorted(keys, ds.version().version) + .map_err(|e| format!("Witness({e:?})"))?; + let w = lane.witness(); + Ok((lane, w)) +} + +/// Bits over the returned rows (by position) whose address lies in `exact`. +fn keep_by_ranges(addr: &[u64], exact: &[(u32, u32)]) -> Result, String> { + let mut bits = vec![0u64; addr.len().div_ceil(64)]; + let mut r = 0usize; + let mut prev = 0u64; + for (i, &a) in addr.iter().enumerate() { + if i > 0 && a < prev { + return Err("returned rows are not in ascending address order".into()); + } + prev = a; + while r < exact.len() && u64::from(exact[r].1) <= a { + r += 1; + } + if r < exact.len() && a >= u64::from(exact[r].0) { + bits[i / 64] |= 1u64 << (i % 64); + } + } + Ok(bits) +} + +// --------------------------------------------------------------------- arms + +#[derive(Clone, Copy, PartialEq, Eq)] +enum Arm { + A, + BAllow, + BBlock, + CExact, + CDefault, +} +impl Arm { + fn name(self) -> &'static str { + match self { + Arm::A => "A-full", + Arm::BAllow => "B-allow", + Arm::BBlock => "B-block", + Arm::CExact => "C-exact", + Arm::CDefault => "C-default", + } + } +} + +// indices into the timing array (microseconds) +const T_WHERE: usize = 0; +const T_RESOLVE: usize = 1; +const T_ROUTE: usize = 2; +const T_INGRESS: usize = 3; +const T_LANCE: usize = 4; +const T_FOLD: usize = 5; +const T_TOTAL: usize = 6; + +struct Out { + t: [f64; 7], + io: Io, + count: u64, + sums: Vec, + mask_bytes: usize, + read: String, +} + +struct Env { + ds: Dataset, + /// The #1633 addressed relation: dataset uri + attested lane + witness. + rel: AddressedRelation, + attest_cold_us: f64, + view: LabClass, + presence: FieldMask, + permit_all: WideFieldMask, + /// Every column, every row, read once: the projection oracle. + full: RecordBatch, +} + +struct Case { + layout: &'static str, + frac_target: f64, + w: Where, + /// The DR-side identities (sorted): GIVEN, taken from the scalar oracle. + ids: Vec, + demanded: WideFieldMask, +} + +fn us(t: Instant) -> f64 { + t.elapsed().as_secs_f64() * 1e6 +} + +async fn run_once(env: &Env, c: &Case, arm: Arm) -> Result { + let t_all = Instant::now(); + let mut t = [0f64; 7]; + + // ClassView resolution (where positions, then demanded positions). + let s = Instant::now(); + let res_w = resolve( + &env.view, + CLASS, + &c.w.positions(), + env.presence, + &env.permit_all, + &env.ds, + ) + .map_err(|e| format!("{e:?}"))?; + let res_d = resolve( + &env.view, + CLASS, + &c.demanded, + env.presence, + &env.permit_all, + &env.ds, + ) + .map_err(|e| format!("{e:?}"))?; + t[T_RESOLVE] = us(s); + + // The attested version must still be the current one (an I/O call). + let s = Instant::now(); + let latest = env + .ds + .latest_version_id() + .await + .map_err(|e| e.to_string())?; + res_w.check_current(latest).map_err(|e| format!("{e:?}"))?; + res_d.check_current(latest).map_err(|e| format!("{e:?}"))?; + t[T_LANCE] += us(s); + + let mut io = Io::default(); + let add_io = |io: &mut Io, x: Io| { + io.rows_scanned += x.rows_scanned; + io.bytes_read += x.bytes_read; + }; + let mut mask_bytes = 0usize; + let read: String; + let (count, sums); + + match arm { + Arm::A => { + let mut cols = where_cols(&c.w, &res_w); + for d in &res_d.columns { + push_unique(&mut cols, d); + } + let s = Instant::now(); + let (b, x) = scan(&env.ds, &cols, false, None).await?; + t[T_LANCE] += us(s); + add_io(&mut io, x); + let s = Instant::now(); + let where_bits = eval_where(&c.w, &b, &res_w); + t[T_WHERE] = us(s); + let s = Instant::now(); + let dcols: Vec<&[i32]> = res_d.columns.iter().map(|d| i32col(&b, d)).collect(); + let (cn, sm) = fold_mask(&where_bits, b.num_rows(), &dcols); + t[T_FOLD] = us(s); + count = cn; + sums = sm; + read = "full".into(); + } + Arm::BAllow | Arm::BBlock => { + // Phase 1: the where-columns with their addresses. + let wc = where_cols(&c.w, &res_w); + let s = Instant::now(); + let (b1, x1) = scan(&env.ds, &wc, true, None).await?; + t[T_LANCE] += us(s); + add_io(&mut io, x1); + let s = Instant::now(); + let where_bits = eval_where(&c.w, &b1, &res_w); + t[T_WHERE] = us(s); + + // Phase 2a: MASK INGRESS (copies addresses into RowAddrTreeMap). + // Timed: building the one tree Lance receives. Not timed: its + // serialized size, which is reporting, not ingress. + let s = Instant::now(); + let addr = addrcol(&b1); + let (tree, allow) = if arm == Arm::BAllow { + let mut list = Vec::new(); + for_each_set(&where_bits, b1.num_rows(), |i| list.push(addr[i])); + (RowAddrTreeMap::from_iter(list), true) + } else { + let mut inv: Vec = where_bits.iter().map(|w| !w).collect(); + if let Some(last) = inv.last_mut() { + let rem = b1.num_rows() % 64; + if rem != 0 { + *last &= (1u64 << rem) - 1; + } + } + let mut list = Vec::new(); + for_each_set(&inv, b1.num_rows(), |i| list.push(addr[i])); + (RowAddrTreeMap::from_iter(list), false) + }; + t[T_INGRESS] = us(s); + mask_bytes = tree.serialized_size(); + let mask = if allow { + read = "allow".into(); + RowAddrMask::from_allowed(tree) + } else { + read = "block".into(); + RowAddrMask::from_block(tree) + }; + + // Phase 2b: prefilter scan, resolved demanded columns only. + let s = Instant::now(); + let (b2, x2) = scan(&env.ds, &res_d.columns, false, Some(mask)).await?; + t[T_LANCE] += us(s); + add_io(&mut io, x2); + let s = Instant::now(); + let dcols: Vec<&[i32]> = res_d.columns.iter().map(|d| i32col(&b2, d)).collect(); + sums = fold_all(&dcols); + count = b2.num_rows() as u64; + t[T_FOLD] = us(s); + } + Arm::CExact | Arm::CDefault => { + let policy = if arm == Arm::CExact { + ReadPolicy::EXACT + } else { + ReadPolicy::DEFAULT + }; + // The ids are GIVEN (DR -> Lance witness BLOCKED_D4): the oracle's + // identities, sorted, standing in for the DR selection. + let ids = &c.ids; + check_fresh(&env.rel, env.ds.uri(), latest).map_err(|e| format!("{e:?}"))?; + let s = Instant::now(); + let (exact, _rows) = ranges(&env.rel, ids).map_err(|e| format!("{e:?}"))?; + let (how, planned) = plan_read(&exact, env.rel.lane.n_rows(), policy); + t[T_ROUTE] = us(s); + + // Timed: building the ONE tree Lance receives (`range_tree`, which + // `addressed::range_mask` also uses). Not timed: its serialized + // size, which is reporting, not ingress. Same split as the B arms. + let s = Instant::now(); + let tree = match how { + Read::Full => None, + Read::Ranges | Read::Coalesced => Some(range_tree(&planned)), + }; + t[T_INGRESS] = us(s); + let mask = tree.map(|tree| { + mask_bytes = tree.serialized_size(); + RowAddrMask::from_allowed(tree) + }); + read = format!("{how:?}/{}r", planned.len()); + + let need_addr = how != Read::Ranges; + let s = Instant::now(); + let (b, x) = scan(&env.ds, &res_d.columns, need_addr, mask).await?; + t[T_LANCE] += us(s); + add_io(&mut io, x); + + let s = Instant::now(); + let dcols: Vec<&[i32]> = res_d.columns.iter().map(|d| i32col(&b, d)).collect(); + if need_addr { + let keep = keep_by_ranges(addrcol(&b), &exact)?; + let (cn, sm) = fold_mask(&keep, b.num_rows(), &dcols); + count = cn; + sums = sm; + } else { + sums = fold_all(&dcols); + count = b.num_rows() as u64; + } + t[T_FOLD] = us(s); + } + } + t[T_TOTAL] = us(t_all); + Ok(Out { + t, + io, + count, + sums, + mask_bytes, + read, + }) +} + +fn median(mut v: Vec) -> f64 { + v.sort_by(|a, b| a.partial_cmp(b).expect("finite")); + v[v.len() / 2] +} + +fn gate(t: &[f64; 7]) -> String { + let mut bad = Vec::new(); + for (i, nm) in [ + (T_WHERE, "where"), + (T_RESOLVE, "resolve"), + (T_ROUTE, "route"), + (T_FOLD, "fold"), + ] { + if t[i] >= 1000.0 { + bad.push(nm); + } + } + if bad.is_empty() { + "in-fold<1000us; lance/ingress/cold=COLD/IO(reported)".into() + } else { + format!( + "FAILED-GATE(reported):{}; lance/ingress/cold=COLD/IO(reported)", + bad.join(",") + ) + } +} + +struct Row { + arm: &'static str, + layout: &'static str, + frac_target: f64, + frac_actual: f64, + k: usize, + t: [f64; 7], + cold_total: f64, + io: Io, + selected: u64, + bytes_vs_a: f64, + mask_bytes: usize, + read: String, + gate: String, +} + +// ------------------------------------------------------ refusals & traps + +fn check(name: &str, ok: bool, fails: &mut usize) { + println!(" [{}] {name}", if ok { "ok " } else { "FAIL" }); + if !ok { + *fails += 1; + } +} + +/// Every refusal paired with a success control; no refusal yields a column list. +async fn refusal_tests( + small: &Dataset, + amb: &Dataset, + small_uri: &str, + fails: &mut usize, +) -> Result<(), String> { + println!("\n== ClassView refusals (each with a paired success control) =="); + let view = LabClass::with_fields(NPROPS); + let presence = (0..NPROPS as u8).fold(FieldMask::EMPTY, |m, p| m.with(p)); + let all = WideFieldMask::full_for(NPROPS); + let dem = WideFieldMask::from_positions(&[2, 15]); + let width = small.schema().fields.len(); + + // Control: the success path projects exactly the demanded columns, not all. + let ok = resolve(&view, CLASS, &dem, presence, &all, small); + let ok_cols = ok.as_ref().map(|r| r.columns.clone()).unwrap_or_default(); + check( + "control: resolve -> exactly the 2 demanded columns (schema has more)", + ok.is_ok() && ok_cols.len() == 2 && ok_cols.len() < width, + fails, + ); + check( + "control: resolved names equal the fixture truth (not label, not ordinal)", + ok_cols == vec![phys_name(phys_ordinal(2)), phys_name(phys_ordinal(15))], + fails, + ); + + // AbsentField: position not in the class's presence mask. + let no5 = (0..NPROPS as u8) + .filter(|&p| p != 5) + .fold(FieldMask::EMPTY, |m, p| m.with(p)); + let r = resolve( + &view, + CLASS, + &WideFieldMask::from_positions(&[5]), + no5, + &all, + small, + ); + check( + "AbsentField refused, no column list", + matches!(r, Err(Refusal::AbsentField { pos: 5 })), + fails, + ); + let r = resolve( + &view, + CLASS, + &WideFieldMask::from_positions(&[5]), + presence, + &all, + small, + ); + check( + "AbsentField control: present position resolves", + r.is_ok(), + fails, + ); + + // Unauthorized: position not in the caller's permit mask. + let permit_no7 = WideFieldMask::from_positions(&[0, 1, 2, 3, 4, 5, 6, 8, 9]); + let r = resolve( + &view, + CLASS, + &WideFieldMask::from_positions(&[7]), + presence, + &permit_no7, + small, + ); + check( + "Unauthorized refused, no column list", + matches!(r, Err(Refusal::Unauthorized { pos: 7 })), + fails, + ); + let r = resolve( + &view, + CLASS, + &WideFieldMask::from_positions(&[6]), + presence, + &permit_no7, + small, + ); + check( + "Unauthorized control: permitted position resolves", + r.is_ok(), + fails, + ); + + // UnknownProperty: the class names p16, no column carries its iri. + let view17 = LabClass::with_fields(NPROPS + 1); + let presence17 = (0..=NPROPS as u8).fold(FieldMask::EMPTY, |m, p| m.with(p)); + let all17 = WideFieldMask::full_for(NPROPS + 1); + let r = resolve( + &view17, + CLASS, + &WideFieldMask::from_positions(&[16]), + presence17, + &all17, + small, + ); + check( + "UnknownProperty refused, no column list", + matches!(&r, Err(Refusal::UnknownProperty { pos: 16, .. })), + fails, + ); + let r = resolve( + &view17, + CLASS, + &WideFieldMask::from_positions(&[15]), + presence17, + &all17, + small, + ); + check( + "UnknownProperty control: known position resolves", + r.is_ok(), + fails, + ); + + // AmbiguousProperty: the second dataset carries p_dup with p15's iri. + let r = resolve( + &view, + CLASS, + &WideFieldMask::from_positions(&[15]), + presence, + &all, + amb, + ); + check( + "AmbiguousProperty refused (2 columns), no column list", + matches!(&r, Err(Refusal::AmbiguousProperty { columns, .. }) if columns.len() == 2), + fails, + ); + let r = resolve( + &view, + CLASS, + &WideFieldMask::from_positions(&[14]), + presence, + &all, + amb, + ); + check( + "AmbiguousProperty control: unambiguous position resolves", + r.is_ok(), + fails, + ); + + // The trap check: ordinal / label mapping gives a DIFFERENT answer. + println!("\n== Mapping traps (the test would catch a positional or label mapping) =="); + let names: Vec = small + .schema() + .fields + .iter() + .map(|f| f.name.clone()) + .collect(); + let (b, _) = scan(small, &names, false, None).await?; + let n_small = b.num_rows() as u32; + let mut ordinal_wrong = 0usize; + let mut label_misses = 0usize; + for j in 0..NPROPS { + let truth: i64 = (0..n_small).map(|id| i64::from(gen_val(id, j))).sum(); + // position == physical column ordinal (after id): the forbidden shortcut. + let by_ordinal: i64 = i32col(&b, &names[1 + j]) + .iter() + .map(|&x| i64::from(x)) + .sum(); + if by_ordinal != truth { + ordinal_wrong += 1; + } + // label == column name: the other forbidden shortcut. + let label = view.field_label(CLASS, j as u8).expect("label"); + if !names.iter().any(|nm| nm == label) { + label_misses += 1; + } + // the resolver's answer + let r = resolve( + &view, + CLASS, + &WideFieldMask::from_positions(&[j as u8]), + presence, + &all, + small, + ) + .map_err(|e| format!("{e:?}"))?; + let via: i64 = i32col(&b, &r.columns[0]) + .iter() + .map(|&x| i64::from(x)) + .sum(); + if via != truth { + return Err(format!( + "resolver answer differs from the generator at p{j}" + )); + } + } + println!( + " ordinal mapping wrong on {ordinal_wrong}/{NPROPS} positions; label mapping finds no column on {label_misses}/{NPROPS}" + ); + check( + "ordinal mapping gives a different answer (trap is live)", + ordinal_wrong > 0, + fails, + ); + check( + "label mapping finds no physical column", + label_misses == NPROPS, + fails, + ); + + // StaleSchema and the stale lane, on the small dataset (mutated last). + println!("\n== Staleness (dataset appended after resolution / attest) =="); + let res = resolve(&view, CLASS, &dem, presence, &all, small).map_err(|e| format!("{e:?}"))?; + let (lane, w) = attest(small).await?; + let rel = AddressedRelation { + rel: RelId(0), + uri: small.uri().to_string(), + lane, + witness: w, + }; + let (lane, w) = (&rel.lane, &rel.witness); + let latest = small.latest_version_id().await.map_err(|e| e.to_string())?; + check( + "control: schema resolution current before append", + res.check_current(latest).is_ok(), + fails, + ); + check( + "control: lane fresh before append", + check_fresh(&rel, small.uri(), latest).is_ok(), + fails, + ); + check( + "control: seek under the real witness runs", + lane.seek_from(w, 0, &leading(3).lo_key()).is_ok(), + fails, + ); + let mut grown = small.clone(); + grown + .append(reader_range_fixture(n_small, n_small + 16), None) + .await + .map_err(|e| e.to_string())?; + let latest2 = grown.latest_version_id().await.map_err(|e| e.to_string())?; + let r = res.check_current(latest2); + check( + "StaleSchema refused after append", + matches!(r, Err(Refusal::StaleSchema { resolved, current }) if current == resolved + 1), + fails, + ); + check( + "stale lane refused after append", + check_fresh(&rel, grown.uri(), latest2).is_err(), + fails, + ); + check( + "other dataset refused", + check_fresh(&rel, "memory://elsewhere", latest).is_err(), + fails, + ); + let forged = OrderedLaneWitness::forged(latest2, lane.n_rows(), w.digest()); + check( + "forged (newer-version) witness refused by seek_from", + lane.seek_from(&forged, 0, &leading(3).lo_key()).is_err(), + fails, + ); + let _ = small_uri; + Ok(()) +} + +fn reader_range_fixture( + lo: u32, + hi: u32, +) -> RecordBatchIterator>> { + reader(Layout::Permuted, lo..hi) +} + +/// D4: no type binds a DR `Frame` to a Lance dataset version. Source-backed +/// trace (P2 Lance trace, `results/.../REPORT.md`): `lance-graph-dt64` carries +/// no `OrderedLaneWitness`/`SealedFacetLane`/Lance version, and the #1788 +/// single-authority decision (S-INGRESS) is still open. So arm C stops at the +/// boundary: its ids are GIVEN, never composed from a DR ordinal. +fn d4_boundary() { + println!( + "BLOCKED_D4: no witness binds a DR Frame to a Lance dataset version (S-INGRESS / single authority open, #1788); \ + arm C's ids are given directly; no DR ordinal is composed with _rowid" + ); +} + +// --------------------------------------------------------------------- main + +#[tokio::main(flavor = "multi_thread")] +async fn main() { + let args: Vec = std::env::args().collect(); + if args.len() < 2 { + eprintln!("usage: where_lance_arm [--n N] [--reps R]"); + std::process::exit(2); + } + let out_dir = args[1].clone(); + let mut n: u32 = 1_000_000; + let mut reps: usize = 5; + let mut i = 2; + while i < args.len() { + match args[i].as_str() { + "--n" => { + n = args[i + 1].replace('_', "").parse().expect("--n"); + i += 2; + } + "--reps" => { + reps = args[i + 1].parse().expect("--reps"); + i += 2; + } + other => { + eprintln!("unknown arg {other}"); + std::process::exit(2); + } + } + } + if let Err(e) = real_main(&out_dir, n, reps.max(1)).await { + eprintln!("ERROR: {e}"); + std::process::exit(1); + } +} + +async fn real_main(out_dir: &str, n: u32, reps: usize) -> Result<(), String> { + println!("# where_lance_arm LAB, Lance-backed, n={n}, warm reps={reps}, seed={SEED:#x}"); + println!("# mask ingress COPIES addresses into RowAddrTreeMap (S-INGRESS open, not zero-copy)"); + d4_boundary(); + + let tmp = tempfile::tempdir().map_err(|e| e.to_string())?; + let uri = |nm: &str| tmp.path().join(nm).to_str().expect("utf8 path").to_string(); + + // Fixtures. + let main_uri = uri("main.lance"); + let small_uri = uri("small.lance"); + let amb_uri = uri("amb.lance"); + write_fixture(&main_uri, Layout::Permuted, n).await?; + write_fixture(&small_uri, Layout::Permuted, 4096).await?; + write_fixture(&amb_uri, Layout::Ambiguous, 4096).await?; + let ds = Dataset::open(&main_uri).await.map_err(|e| e.to_string())?; + let small = Dataset::open(&small_uri).await.map_err(|e| e.to_string())?; + let amb = Dataset::open(&amb_uri).await.map_err(|e| e.to_string())?; + + // Refusals first: they mutate only the small dataset. + let mut fails = 0usize; + refusal_tests(&small, &amb, &small_uri, &mut fails).await?; + + // Attest the main dataset (cold) and read the projection oracle once. + let s = Instant::now(); + let (lane, witness) = attest(&ds).await?; + let attest_cold_us = us(s); + println!("\n# attest (cold, once per version): {attest_cold_us:.1} us [COLD/IO, reported]"); + let names: Vec = ds.schema().fields.iter().map(|f| f.name.clone()).collect(); + let (full, _) = scan(&ds, &names, false, None).await?; + let presence = (0..NPROPS as u8).fold(FieldMask::EMPTY, |m, p| m.with(p)); + let uri_main = ds.uri().to_string(); + let env = Env { + ds, + rel: AddressedRelation { + rel: RelId(0), + uri: uri_main, + lane, + witness, + }, + attest_cold_us, + view: LabClass::with_fields(NPROPS), + presence, + permit_all: WideFieldMask::full_for(NPROPS), + full, + }; + + // Cases. + let fracs = [0.001, 0.01, 0.1, 0.5]; + let cal = calibrate(n, &fracs); + let mut sm = SplitMix64(SEED); + let ks: [(usize, &[u8]); 3] = [ + (1, &[7u8][..]), + (4, &[0u8, 5, 10, 15][..]), + ( + 16, + &[0u8, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15][..], + ), + ]; + let mut rows: Vec = Vec::new(); + println!("\n== Measurements (us, median of {reps} warm runs; cold = first run) =="); + for (fi, &f) in fracs.iter().enumerate() { + let (t_thr, _) = cal[fi]; + let len = ((f * f64::from(n)).round() as u32).max(1); + let lo = (sm.next() % u64::from(n - len + 1)) as u32; + for (layout, w) in [ + ("scattered", Where::Scatter { t: t_thr, u: t_thr }), + ("clustered", Where::Range { lo, hi: lo + len }), + ] { + for &(k, dem) in &ks { + let orc = oracle(n, &w, dem); + if orc.count == 0 { + println!("skip {layout} f={f}: empty selection"); + continue; + } + let case = Case { + layout, + frac_target: f, + w, + ids: orc.ids.clone(), + demanded: WideFieldMask::from_positions(dem), + }; + let frac_actual = orc.count as f64 / f64::from(n); + + // Projection oracle: full read of all columns, fixture-truth names. + let proj_sums: Vec = dem + .iter() + .map(|&j| { + let col = i32col(&env.full, &phys_name(phys_ordinal(usize::from(j)))); + let mut s = 0i64; + for_each_set(&orc.bits, n as usize, |i| s += i64::from(col[i])); + s + }) + .collect(); + if proj_sums != orc.sums { + return Err(format!( + "projection oracle != generator oracle ({layout} f={f} k={k})" + )); + } + + let mut arms = vec![Arm::A, Arm::BAllow]; + if (f - 0.5).abs() < 1e-9 { + arms.push(Arm::BBlock); + } + arms.push(Arm::CExact); + arms.push(Arm::CDefault); + + let mut a_bytes = 0usize; + let mut first_line = true; + for arm in arms { + let cold = run_once(&env, &case, arm).await?; + let mut runs: Vec<[f64; 7]> = Vec::new(); + let mut last = cold; + let cold_total = last.t[T_TOTAL]; + for _ in 0..reps { + last = run_once(&env, &case, arm).await?; + runs.push(last.t); + } + // Oracle checks on the last warm run. + if last.count != orc.count || last.sums != orc.sums { + eprintln!( + "MISMATCH {} {layout} f={f} k={k}: count {} vs {}, sums {:?} vs {:?}", + arm.name(), + last.count, + orc.count, + last.sums, + orc.sums + ); + fails += 1; + } + let mut t = [0f64; 7]; + for (idx, slot) in t.iter_mut().enumerate() { + *slot = median(runs.iter().map(|r| r[idx]).collect()); + } + if arm == Arm::A { + a_bytes = last.io.bytes_read; + } + let bytes_vs_a = if a_bytes == 0 { + 0.0 + } else { + 100.0 * last.io.bytes_read as f64 / a_bytes as f64 + }; + if first_line { + println!( + "-- {layout} target {:.1}% actual {:.3}% k={k} | demanded(WideFieldMask)={} where_bits(popcount)={} dr_ids={} (ids given, BLOCKED_D4)", + f * 100.0, + frac_actual * 100.0, + case.demanded.count(), + orc.count, + orc.ids.len() + ); + first_line = false; + } + let g = gate(&t); + println!( + " {:<9} Lance-backed | where {:>9.1} resolve {:>7.1} route {:>9.1} ingress {:>10.1} lance {:>10.1} fold {:>9.1} total {:>10.1} cold {:>10.1} | scanned {:>9} sel {:>8} bytes {:>10} ({:>5.1}% of A) row_addr_mask {:>9}B read {} | {}", + arm.name(), + t[T_WHERE], + t[T_RESOLVE], + t[T_ROUTE], + t[T_INGRESS], + t[T_LANCE], + t[T_FOLD], + t[T_TOTAL], + cold_total, + last.io.rows_scanned, + last.count, + last.io.bytes_read, + bytes_vs_a, + last.mask_bytes, + last.read, + g + ); + rows.push(Row { + arm: arm.name(), + layout: case.layout, + frac_target: case.frac_target, + frac_actual, + k, + t, + cold_total, + io: last.io, + selected: last.count, + bytes_vs_a, + mask_bytes: last.mask_bytes, + read: last.read.clone(), + gate: g, + }); + } + } + } + } + + // JSON. + let dir = std::path::Path::new(out_dir) + .join("lance_arm") + .join(format!("n{n}")); + std::fs::create_dir_all(&dir).map_err(|e| e.to_string())?; + let mut js = String::new(); + js.push_str(&format!( + "{{\n \"label\": \"Lance-backed LAB\", \"n\": {n}, \"reps\": {reps}, \"attest_cold_us\": {:.1},\n \"owner_gates\": [\"S-INGRESS\", \"D4\"],\n \"rows\": [\n", + env.attest_cold_us + )); + for (i, r) in rows.iter().enumerate() { + js.push_str(&format!( + " {{\"arm\":\"{}\",\"layout\":\"{}\",\"frac_target\":{},\"frac_actual\":{:.6},\"k\":{},\"us\":{{\"where\":{:.1},\"resolve\":{:.1},\"route\":{:.1},\"ingress\":{:.1},\"lance\":{:.1},\"fold\":{:.1},\"total\":{:.1},\"cold_total\":{:.1}}},\"rows_scanned\":{},\"rows_selected\":{},\"bytes_read\":{},\"bytes_vs_A_pct\":{:.1},\"row_addr_mask_serialized_bytes\":{},\"read\":\"{}\",\"gate\":\"{}\"}}{}\n", + r.arm, r.layout, r.frac_target, r.frac_actual, r.k, + r.t[T_WHERE], r.t[T_RESOLVE], r.t[T_ROUTE], r.t[T_INGRESS], r.t[T_LANCE], r.t[T_FOLD], r.t[T_TOTAL], r.cold_total, + r.io.rows_scanned, r.selected, r.io.bytes_read, r.bytes_vs_a, r.mask_bytes, r.read, r.gate, + if i + 1 == rows.len() { "" } else { "," } + )); + } + js.push_str(" ]\n}\n"); + let path = dir.join("lance_arm.json"); + std::fs::write(&path, js).map_err(|e| e.to_string())?; + println!("\n# wrote {}", path.display()); + + if fails > 0 { + eprintln!("FAILED: {fails} check(s)/oracle mismatch(es)"); + std::process::exit(1); + } + println!("# all oracle and refusal checks passed"); + Ok(()) +} diff --git a/crates/crepe-plan-probe/src/addressed.rs b/crates/crepe-plan-probe/src/addressed.rs index 595f656c0..c8065efc8 100644 --- a/crates/crepe-plan-probe/src/addressed.rs +++ b/crates/crepe-plan-probe/src/addressed.rs @@ -234,7 +234,7 @@ pub fn plan_read(ranges: &[(u32, u32)], n_rows: u32, p: ReadPolicy) -> (Read, Ve /// The rows of every leading value in `wanted` (sorted, deduplicated) as lane /// ranges, found with galloping seeks from the previous range's end. -fn ranges( +pub fn ranges( a: &AddressedRelation, wanted: &[u32], ) -> Result<(Vec<(u32, u32)>, usize), AddressedRefusal> { @@ -262,12 +262,39 @@ fn ranges( Ok((out, rows)) } -fn range_mask(ranges: &[(u32, u32)]) -> RowAddrMask { +/// The lane ranges as Lance's row-address tree, one `insert_range` per range. +/// Lance owns the result: building it copies the ranges (seam S-INGRESS). +pub fn range_tree(ranges: &[(u32, u32)]) -> RowAddrTreeMap { let mut m = RowAddrTreeMap::new(); for &(lo, hi) in ranges { m.insert_range(u64::from(lo)..u64::from(hi)); } - RowAddrMask::from_allowed(m) + m +} + +/// [`range_tree`] as an ALLOW mask for `with_row_addr_prefilter`. +pub fn range_mask(ranges: &[(u32, u32)]) -> RowAddrMask { + RowAddrMask::from_allowed(range_tree(ranges)) +} + +/// Refuses a lane read against another dataset or a dataset that moved on +/// since the lane was attested. +pub fn check_fresh(a: &AddressedRelation, uri: &str, current: u64) -> Result<(), AddressedRefusal> { + if uri != a.uri { + return Err(AddressedRefusal::OtherDataset { + rel: a.rel, + attested: a.uri.clone(), + current: uri.to_string(), + }); + } + if current != a.lane.version() { + return Err(AddressedRefusal::Stale { + rel: a.rel, + attested: a.lane.version(), + current, + }); + } + Ok(()) } fn lane_for( @@ -290,21 +317,7 @@ async fn current_lane<'a>( let rel = plan.sides[i].key.rel; let a = lane_for(lanes, rel)?; let ds = store.dataset(rel).await?; - if ds.uri() != a.uri { - return Err(AddressedRefusal::OtherDataset { - rel, - attested: a.uri.clone(), - current: ds.uri().to_string(), - }); - } - let current = ds.version().version; - if current != a.lane.version() { - return Err(AddressedRefusal::Stale { - rel, - attested: a.lane.version(), - current, - }); - } + check_fresh(a, ds.uri(), ds.version().version)?; Ok((a, ds)) }