diff --git a/.claude/board/entries/2026-11-03-d-dr-fold-map.md b/.claude/board/entries/2026-11-03-d-dr-fold-map.md new file mode 100644 index 000000000..02911c4b9 --- /dev/null +++ b/.claude/board/entries/2026-11-03-d-dr-fold-map.md @@ -0,0 +1,14 @@ +# 2026-11-03 — D-DR-FOLD-MAP: storage, addressing and algebra kept apart + +**Status:** ARCHITECTURE MAP, read-only (`.claude/probes/2026-11-03-d-dr-fold/D-DR-FOLD-MAP.md`, tree `fb5e2d35b`). No code run; numbers quoted from earlier recorded measurements. No production change; mask-risc, EXO-140 and EXO-46 unchanged. + +- **Storage (A):** no DR → physical-location translation, readahead or I/O/fold overlap exists in production. The address-mask → Lance prefilter chain is LAB only (H14c-4, H14d). A mask cuts bytes fetched only when presence is page-local. Every recorded `bytes_read` was served from the page cache, not read from disk. The physical read-planning contract is missing and has no owner. +- **Addressing (B):** `LocalMap` / `MapTable` / `FrameStore` (dt64) are the reusable coordinate mapping. No second binding store. `lance-graph-xlog::Cube` already keeps coordinate lifetime apart from presence lifetime (`DeltaLog` under one frame); its callers are lab crates. Reuse is amortization, not a different hot algorithm. +- **Algebra (C):** the frozen IR's only in-program cross-row reads are `Gather` / `EqU32Via`. `ScatterOrU32` is terminal-only. XLOG scatter-then-AND is LAB / UNCLASSIFIED. Tile compose (`relcomp`) is LAB and returns a transient C register. A vector×tile fold and cross-tile compose do not exist. +- For the D-BAG-FOLD shape and today's lowerings, Gather is required even for the hierarchical case. That claim covers that shape only. The 83 % Gather-avoidance figure is UNMEASURED. + +**OPEN, owner decisions D1–D4:** +- D1: an in-program range-expand op; +- D2: the scatter-then-AND classification and outer-binding preservation; +- D3: binding lifetime vs presence writer; +- D4: owner of DR → Lance read planning. diff --git a/.claude/board/entries/README.md b/.claude/board/entries/README.md index a23d95764..04714380d 100644 --- a/.claude/board/entries/README.md +++ b/.claude/board/entries/README.md @@ -25,11 +25,12 @@ index row, (3) no duplicate entry id. Checks 1 and 2 are deliberately opposite directions; the stranding this convention prevents shows up in exactly one of them, never both. -563 entries, 2026-08-06 .. 2026-11-03. +564 entries, 2026-08-06 .. 2026-11-03. | date | entry id | finding | file | |---|---|---|---| | 2026-11-03 | `D-XLOG-META-WIRE-P0` | | [2026-11-03-d-xlog-meta-wire-p0.md](2026-11-03-d-xlog-meta-wire-p0.md) | +| 2026-11-03 | `D-DR-FOLD-MAP` | | [2026-11-03-d-dr-fold-map.md](2026-11-03-d-dr-fold-map.md) | | 2026-11-03 | `D-COGNITIVE-POLYFILL` | | [2026-11-03-d-cognitive-polyfill-p0.md](2026-11-03-d-cognitive-polyfill-p0.md) | | 2026-11-03 | `D-BAG-FOLD` | | [2026-11-03-d-bag-fold-witnessed-existential.md](2026-11-03-d-bag-fold-witnessed-existential.md) | | 2026-11-03 | `D-BAG-FOLD-V4` | | [2026-11-03-d-bag-fold-v4-attested-lens-gated-exec.md](2026-11-03-d-bag-fold-v4-attested-lens-gated-exec.md) | diff --git a/.claude/probes/2026-11-03-d-dr-fold/D-DR-FOLD-MAP.md b/.claude/probes/2026-11-03-d-dr-fold/D-DR-FOLD-MAP.md new file mode 100644 index 000000000..3904e26f7 --- /dev/null +++ b/.claude/probes/2026-11-03-d-dr-fold/D-DR-FOLD-MAP.md @@ -0,0 +1,182 @@ +# D-DR-FOLD: three kinds of indirection, kept apart + +**Status:** ARCHITECTURE MAP, source-grounded. Read-only pass: no code run, nothing +measured in this pass. Every number below is quoted from an earlier recorded +measurement, with its source. No production change. mask-risc unchanged. EXO-140 / +EXO-46 unchanged. + +**Source SHAs.** + +| what | SHA | +|---|---| +| tree read | lance-graph `fb5e2d35b10fc4940f5c78285bae246f90c7faad` (`origin/main` after #1940) | +| D-BAG-FOLD v3 / v4 measurements | `d651b70d4a3b2299c21f1d354ac02a3311280441` (+ #1938 / #1939) | +| H14d measurement | `802db7dfea88ecd33790a85d176910b62e2b902b` (entry `2026-10-16-d-holo-h14d-logic-mask-read.md`) | +| H14c-4 measurement | as recorded in `.claude/plans/d-holo-h14c-thought-first-v1.md` § Results (v4, 2026-10-15); the plan names no commit | +| D-DRMC-5, D-DYNBAG-1 | as recorded in their board entries (`2026-10-22-drmc-5-assoc-probe.md`, `2026-11-01-dynbag-1-n-compose.md`); no commit named | + +**Classification.** + +| label | meaning | +|---|---| +| PRODUCTION | a workspace-member crate and a caller in production | +| PRODUCTION CRATE, LAB CALLERS | a workspace member whose only callers are excluded crates | +| LAB | an excluded probe, or a recorded probe result | +| PROPOSED | named in a plan or working model, not built | +| BLOCKED | needs an owner decision first | +| UNCLASSIFIED | its legality under an existing rule is undecided | +| NOT VERIFIED | not checked in this pass | + +## Three paths + +1. **Physical indirection (Path A):** where a DR region's fragments live in storage, + and how they become resident. +2. **Logical indirection (Path B):** how a global DR id maps to a local tile position. +3. **Relational evaluation (Path C):** how resident witnesses establish a relation + without storing it. + +Prefetch and lookahead belong to A. Mapping reuse belongs to B, and it is an +amortization, not a different hot execution algorithm. A tile sweep or fold belongs to C. + +## Path A: storage, from fragments to resident tiles + +| finding | class | source | +|---|---|---| +| No DR → physical-location translation. `lance-graph-dt64` depends only on `lance-graph-contract`; "DR routes, DT interprets … Nothing here holds a value." | PRODUCTION (absence) | `crates/lance-graph-dt64/Cargo.toml`, `src/dr.rs:1-20` | +| No I/O lookahead, readahead, coalescing under DR control, or I/O/fold overlap. No crate sets `fragment_readahead`, `batch_readahead`, `io_parallelism` or uses `take_rows`. | PRODUCTION (absence) | crate-wide grep; Read of `lance-graph/src/graph/versioned.rs` (plain `Dataset::open` + scans) | +| Lance 13 `with_row_addr_prefilter`: a scattered mask cuts rows decoded, not bytes fetched; a page-local mask cuts both (1 % uniform 1.00×, 1 % clustered 0.24×, 0.1 % uniform 0.77×). | LAB, H14c-4 | `.claude/plans/d-holo-h14c-thought-first-v1.md` § Results | +| Logic → address mask R → prefilter → terminal: rows scanned = \|R\| in 96/96 cells; bytes follow page locality; a 1,024- or 8,192-address aperture reads 0.01–0.12×. OPEN in that entry: which address order puts survivors on few pages. | LAB, H14d | board entry `2026-10-16-d-holo-h14d-logic-mask-read.md` | +| Ordered keys → `seek_from` ranges → row-address prefilter. A table that is not one contiguous fragment is refused (`NotContiguous`): physical fragmentation is refused, not scheduled. | LAB | `crates/crepe-plan-probe/src/addressed.rs:1-24` | +| The only "lookahead" in the tree is `ndarray::simd::prefetch_read_t0`, a CPU-cache prefetch. It is not physical I/O prefetch. | LAB | `crates/drf-focus-probe/src/lib.rs:370-406` | +| Lance reads run on the recovery clock and folds on the live clock; read once at replay, fold resident. | PROPOSED (working model; measured in one consumer) | `.claude/knowledge/split-tunnel-sparse-append-resident-fold.md` | + +**Every `bytes_read` in this tree is what Lance requested from the local object +store, served from the operating-system page cache.** None is a physical disk read, +and none may be reported as one. No probe so far separates disk from page cache. + +**The missing contract (physical read planning):** nothing turns a DR WHERE into an +ordered list of physical fragments or page ranges. It would need to state: + +- which DR coordinates map to which Lance row addresses, under which dataset version + (H14c-5: without stable row ids, compaction moved 3,510 of 3,510 `_rowid`s); +- how row addresses group into pages and fragments; +- how far ahead future requirements are known (lookahead distance); +- the read-coalescing and concurrency policy; +- that the read plan follows the DR address plan and never changes relational + semantics. + +No owner is named for it today (decision D4 below). + +## Path B: addressing, from global DR id to a reusable local map + +| finding | class | source | +|---|---|---| +| `LocalMap::prepare`: a sorted lookup index per tile. Refuses duplicates and over-width; lookups are counted. | PRODUCTION | `crates/lance-graph-dt64/src/dr.rs:556-641` | +| `MapTable`: an append-only per-tile history. An extension keeps positions, a replacement is a new generation, a removal is an event. `map(frame, tile)` returns a borrowed `TileMap`: zero allocation, replays the layout at the frame's version. `locate` resolves across listed tiles and refuses ambiguity. | PRODUCTION | `dr.rs:653-988` | +| `FrameStore` / `Frame`: issuer identity; `check` refuses any frame that is not the current seal. | PRODUCTION | `dr.rs:376-494` | +| `Cube` builds every relation's lanes once per frame (one lookup per id, at construction). Joins and terminals reuse them, and each terminal re-checks the frame. Presence edits (`stage`/`commit` of edge Add/Retire through a per-relation `DeltaLog`) keep the frame. Only a grown tile, which must extend (`Renumbered` otherwise), rebinds. | PRODUCTION CRATE, LAB CALLERS | `crates/lance-graph-xlog/src/cube.rs:1-35, 848-1000`; callers: `xlog-relcomp-probe`, `lance-graph-codegraph` (both excluded) | +| The free functions `lance_graph_xlog::{support, bag_count}` rebuild lanes and look every tuple up per call. `Cube` is their amortized form. | PRODUCTION CRATE, LAB CALLERS | `crates/lance-graph-xlog/src/lib.rs:255-297` | +| The D-BAG-FOLD v4 pointer lens is minted per key version and retired by the frame on any presence change (A4′, conservative). | LAB | `REPORT-V4.md` | + +**`LocalMap` / `MapTable` already are the reusable coordinate mapping.** No second +binding store is needed or proposed. Reusing them is an amortization of prepare and +lookup cost; it does not change the hot execution algorithm. + +**Hazard (BLOCKED, D3):** `MapTable::map` deliberately answers old frames (replay +by version). A map read therefore cannot detect stale presence. Cube-style reuse is +correct only if the cube is the single presence writer. This is the same condition +as D-BAG-FOLD v3 case 3b. + +Foreign keys use the same contract: `MapTable::locate` plus `Cube`'s tiled mode +(`tile_ordinal * 256 + local`). Plan row MISSING-7 ("multi-tile global → (tile, +local) map", `d-dynbag-holographic-bags-v1.md:526`) looks at least partly closed by +this. Whether the plan was updated is NOT VERIFIED. Cross-tile composition is still +missing (D-DYNBAG-1 OPEN). + +DuckDB comparison (SECONDHAND, not fetched in this pass): a selection vector, or a +dictionary vector, is a per-chunk index indirection over unchanged data, which is +conceptually `$i`. It is ephemeral per chunk. `MapTable` is versioned and reused +across queries. + +## Path C: algebra, from resident tiles to a terminal + +| finding | class | source | +|---|---|---| +| `Pred::Range{lo, hi}`: a contiguous range write, with no per-row compare. On an address-ordered lane a key prefix names a subtree, so its mask is a range. | PRODUCTION (frozen IR) | `crates/lance-graph-mask-risc/src/ir.rs:159-180` | +| `And`/`Or`/`AndNot`/`Not`/`Ternlog`: co-addressed conjunction, with no index lane. | PRODUCTION | `ir.rs:230-256` | +| `MaskOp::Gather{lane, foreign, under}` and `Pred::EqU32Via` are the only in-program cross-row reads. | PRODUCTION | `ir.rs:156-158, 257-282` | +| `ScatterOrU32` is terminal only: its scattered mask "is never an intermediate" (survival condition). `CountKeyRunsU32` is a run fold on a key-ordered lane, and yields only a count. | PRODUCTION | `ir.rs:322-389` | +| XLOG support `q(Y) :- r(X,Y), s(Y,Z)`: two `ScatterOrU32` programs, then `ndarray::simd::mask_and` outside mask-risc. No Gather, but an indexed scatter, and two program results combined. The answer is over Y only: the X and Z bindings are not preserved. | **LAB / UNCLASSIFIED** until the computed-mask survival rule and outer-coordinate binding preservation are established (D2) | `crates/lance-graph-xlog/src/lib.rs:16-35, 225-270` | +| XLOG bag count: per-side `GroupReduce Count`, then a checked i64 product. This is multiplicity, kept separate as EXO-108. | PRODUCTION CRATE, LAB CALLERS | `lib.rs:273-297` | +| `relcomp::compose` over DR64 64×64 Boolean registers: "ORs one row of B per set bit of A". It indexes by bit position inside a resident tile, with no pointer lane. It returns C as a transient 512 B register, which is the composed relation. Measured 50 ns sparse, 2.4 µs dense per tile pair. | LAB | `crates/dr-math-binding-probe/src/relcomp.rs:1-60, 212`; entries D-DRMC-5, D-DYNBAG-1 | +| `ncompose` ℕ-compose via `matmul_i8_to_i32` is bag-exact. About 21 µs per tile, attributed in that entry to per-call pack/alloc (HYPOTHESIS there). | LAB | `crates/xlog-relcomp-probe/src/ncompose.rs`; entry D-DYNBAG-1 | +| No vector×tile fold, where a 64/256-bit frontier r·S·T would reach the terminal without forming C. `relcomp`'s `Step::Push` is a stack push. No cross-tile compose. | absence | as above | + +## Answers to the brief + +**1. Implementations and callers.** +- Path A: absent in production; LAB probes only. +- Path B: PRODUCTION (`dt64`). `Cube` is a production crate with lab callers. +- Path C: the operators are PRODUCTION (mask-risc); the composition kernels are LAB. + +**2. Connections between the paths.** +- A↔B: none. DR coordinates never become Lance row addresses in production. H14d's + address mask is the lab prototype. +- B↔C: only `Cube`, whose lanes hold local positions. +- A↔C: none. Resident-tile reuse is a working model, not a component. + +**3. Where a tile sweep can remove execution-time Gather, in principle:** +- every witness is on the same row axis: `And`/`Ternlog`, no lane; +- the cross-row relation is a contiguous range known at plan time: `Pred::Range`; +- both endpoints lie in one resident DR64/DR256 tile: bit-matrix compose, or a + vector×tile fold; +- hierarchical ancestor relations. In the D-BAG-FOLD fixture b(d) and c(d) are key + prefixes of d, so "d lies under some b ∈ R_in" is a union of contiguous ranges. + +**4. Where Gather is required:** +- a data-dependent, non-contiguous foreign key; +- a relation spanning tiles (no cross-tile compose exists); +- an id domain wider than one tile with no tiled binding. + +For the examined query shape (the D-BAG-FOLD existential) and the lowerings +available today, Gather is also required for the hierarchical case: the frozen IR has +no in-program op that expands a mask over ranges or reduces runs to a parent mask +(`ScatterOrU32` cannot feed another program; `CountKeyRunsU32` yields a count only). +**This statement covers that shape and those lowerings only, not query shapes in +general.** + +**5. Binding reuse that exists:** +- `MapTable` / `TileMap`: borrowed, zero allocation, version replay; +- `Cube`: lanes built once, presence through `DeltaLog` without a remint; +- `LocalMap::lookups`, to prove that no lookup happens per query. + +**6. Physical I/O:** +- present: the upstream Lance prefilter and scan statistics, used only in lab probes; +- absent: everything in the read-planning contract above; +- proposed: the split-tunnel working model; page-scale address clustering (H14d OPEN). + +**7. The 83 % hypothesis: UNMEASURED.** Four denominators must be kept apart: +queries, relationships, candidate accesses, and cost. +- The only related numbers are locality, not avoidance (LAB v3): 47.6 % (17,476 + nodes) and 70.4 % (2,340 nodes) of `$i` reads land in the reading row's own 64-row + word, and every pointer stays inside the candidate subtree. +- Falsifiers: + - F1: uniform random foreign keys over N ids. Two endpoints share a 256-wide tile + with probability ≈ 256/N, about 0.4 % at N = 65,536. + - F2: sparse candidates. A sweep pays the same whole-lane passes as Arm L plus the + expansion, so cost avoided can be about 0 even when every query avoids Gather. + - F3: dense tiles. Compose cost grows with popcount(A). + - F4: relations that span tiles have no sweep at all. + +**8. Benchmark plan.** Carried out separately as the D-DR-FOLD lab comparison; see +its report. + +**9. Owner decisions:** +- D1 (mask-risc owner): is an in-program range-expand or run-to-parent mask op + admissible? +- D2 (#1791 / EXO-140 owner): is scatter-then-AND, as XLOG support does it, a + demanded result or a forbidden intermediate, and must outer bindings be preserved? +- D3 (EXO-46 owner): may a coordinate binding outlive presence changes (the `Cube` + split), and who is the single presence writer? +- D4 (DR / storage owner): who owns DR → Lance row-address translation and + page-scale address clustering?