diff --git a/.claude/board/INTEGRATION_PLANS.md b/.claude/board/INTEGRATION_PLANS.md index 74d4b474c..b01ccc921 100644 --- a/.claude/board/INTEGRATION_PLANS.md +++ b/.claude/board/INTEGRATION_PLANS.md @@ -1,3 +1,31 @@ +## 2026-10-20 — recursive-perspective-rebinding-v1 — WORLD × ANGLE × THOUGHT over composable relational registers → `.claude/plans/recursive-perspective-rebinding-v1.md` + +**Status:** D-RPR-0 PROPOSED, LAB ONLY, documentation only; D-RPR-1 not authorized; D-RPR-2/3 BLOCKED (D-TGA-3, D-ASM-4). + +- The navigation terms map onto existing components: WORLD = sealed version with thought 0 as givens; ANGLE = focus binding + prepared relation planes (+ proposed map palettes); THOUGHT = sparse deltas over one shared frame. +- The addressing operation is the measured H14c/H14d canon `Thought(t)[T_a[i] : T_b[i]]`; the global → local binding it needs is admission A1 (D-RMAP-1 §6), still open. +- Rebind exists and is measured (D-RMAP-1A §8.4): it pays for small k only, so the open quantity is the k distribution along real question chains. Hypothesis H1: a frame's escapes seed the next angle. +- MeTTa operations sorted into four classes from the existing polyfill classification and Decision H. + +## 2026-10-20 — relstore-quack-petgraph-traversal-v1 (reframed) — compiled relational geometry and amortization; petgraph optional → `.claude/plans/relstore-quack-petgraph-traversal-v1.md` + +**Status:** D-RST-0..9 LAB EXPLORATION, not admitted to production; no production code may be rewired to it (operator, 2026-10-20); documentation only. Operator correction (2026-10-20): the goal is reusable relational geometry amortized across front ends and thought versions; petgraph is an optional consumer (D-RST-3 OPTIONAL / DEFERRED / MEASUREMENT-GATED); tile GEMM consumes prepared layouts and does not obtain them. + +- Critical path: D-RST-9 (native borrowed views, no petgraph) → D-RST-1 → 2 → 4 → 5 → 6 → 7. Off it: D-RST-3, D-RST-8 (A2), F-PG, F8 path D. +- New §5b: three costs (preparation, execution, maintenance) measured apart; break-even from measured values only; Boolean existence vs BF16 multiplicity, with the BF16 exactness bound (integers to 256). +- New F8: four execution paths on one relation, cold/warm, 1/8/64 thoughts, sparse to hub distributions. +- Prior art named: the planner's `AdjacencyStore`/`AdjacencyBatch` (pattern CSR + CSC, borrowed, no-`Copy`) is the starting point for D-RST-9; no fourth owned CSR type. +- Supersedes the entry below on framing only; its ownership decisions stand. + +## 2026-10-20 — relstore-quack-petgraph-traversal-v1 — stored relations, compiled traversal, petgraph over borrowed layouts → `.claude/plans/relstore-quack-petgraph-traversal-v1.md` + +**Status:** D-RST-0..8 PROPOSAL; documentation only. Operator decision B (2026-10-20): RelationshipMapStore owns persistence, layouts, domains, seals and invalidation; Quack compiles; mask-risc executes. D-RMAP-1A §1 and the 2026-10-05 storage-portability ruling are unchanged. + +- The store satisfies Quack's existing `Registrar` seam; derived relations stay queryable relations with multiplicity, provenance and dependencies; versions are `DurableRef` / `ThoughtCommitId`, and the coordinate domain is itself a relation, so its seal needs no counter. +- petgraph: a borrowed `RelView` implementing `GraphBase`/`IntoNeighbors`/`Visitable`/`NodeIndexable` runs unmodified `Bfs`/`Dfs`; batched cursors and a typed traversal vocabulary compile to Quack. +- Finding: multi-hop over a large domain is blocked by scatter (push) as well as gather (pull); the tile is the only admitted shape (within a focus). Admissions A1 (global→local binding), A2 (large-domain multi-hop), A3 (scalar petgraph iteration) are open. +- The existing petgraph `lance-fold-hop` experiment consumes the quarantined `OutgoingHop`; the plan does not build on it. + ## 2026-10-19 — mork-pathmap-schnellverschluss-v1 — MORK / PathMap behind one replaceable boundary → `.claude/plans/mork-pathmap-schnellverschluss-v1.md` **Status:** D-MPS-0/1 MEASURED (probe, results only); D-MPS-2..5 upstream PROPOSALS; D-MPS-6 BLOCKED; D-MPS-7 OPEN. diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index bef9f6044..526bb675e 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -25,6 +25,20 @@ These PRs merged with no board rows. Every row below is a PROPOSAL (documents an | D-RMAP-1A | separate map tables, borrowed projections, thought-selected palettes (`d-rmap-separate-tables-thought-palettes-v1.md`) | PROPOSAL; generation witnesses corrected 2026-10-19 (§6a) | refusal on map regeneration, palette-slot reassignment, focus-slot reuse, incarnation reuse, stale tile, cross-tenant change | | D-RMAP-1A-V | versioning crosswalk: map/palette/binding generations derived from D-TGA `ThoughtCommitId` + #1605 `DurableRef`; focus generation and index seal retained (`d-rmap-separate-tables-thought-palettes-v1.md` §7) | PROPOSED; no code, no type; falsifiers specified, none run | X1–X9 in §7.4 (cross-dataset, compaction, palette ABA, slot reuse, uncommitted change, rebase, authorization, squash+replay, deletion/DRed) | | D-RMAP-1A-P | focus-bound RelationshipMap probe over existing sorted/trie indexes: CSR, 256×256 tile via mask-risc, incremental rebind, thought deltas, place masks, postings (`d-rmap-separate-tables-thought-palettes-v1.md` §8; `crates/mork-pathmap-probe/src/rmap.rs`) | MEASURED, probe only; nothing adopted | 12 tests vs scan oracle; 11 guards disable-verified; break-even 76/11/1.8 reuses (sparse d2 / d8 / dense) | +| D-RST-0 | stored relations × Quack × petgraph: architecture and phased plan (`relstore-quack-petgraph-traversal-v1.md`) | LAB EXPLORATION, not admitted to production (operator, 2026-10-20); documentation only; reframed 2026-10-20 around relational geometry and amortization (petgraph optional, tile GEMM a consumer of prepared layouts) | operator decision B; admissions A1–A6, blockers B1–B3 listed in §8 | +| D-RST-1 | `DurableRef` / `ThoughtCommitId` contract types, never-reused rule (coordinated with D-CPG CPG-00) | LAB ONLY, not admitted to production; Queued | reuse test; no duplicate with CPG-00 | +| D-RST-2 | `RelationshipMapStore` skeleton: catalog, facts through the cycle seal, sorted permutation, `LayoutSeal`, domain as a relation, `impl Registrar` | LAB ONLY, not admitted to production; Queued | F4 refusals (reseal, domain ABA, compaction, scope, dirty, deletion) | +| D-RST-3 | petgraph `RelView` over a borrowed, version-pinned pattern-CSR view in a new fork experiment (brought forward; isolated probe with temporary ownership) | OPTIONAL / DEFERRED / MEASUREMENT-GATED (operator, 2026-10-20); experiment AdaWorldAPI/petgraph#4 kept as evidence, not extended | then-F1 (now F-PG) met: unmodified `Bfs`/`Dfs` equal a `DiGraph` oracle; view borrows the prepared buffers (pointer check), 0 allocations to build or walk; Quack one-hop reads the same `dst` buffer at the same version, one prepare; 7 disable runs red | +| D-RST-4 | layouts in the store: base indexes at seal time, specialized layouts on demand; pattern-only `u32` CSR/CSC on BlasGraph storage types, bitmap row blocks through mask-risc, join table; capability table | LAB ONLY, not admitted to production; Queued (production reads wait on A1; layouts from D-RST-9) | F2: four front ends (Quack, Cypher, Gremlin adapter, RelAlg/Datalog), one `RelationVersion`, identical set and bag answers | +| D-RST-5 | derived relations: `materialize`, multiplicity, provenance, dependencies, invalidation, DRed | LAB ONLY, not admitted to production; Queued | F4 deletion, F5 set/bag/path | +| D-RST-6 | typed traversal vocabulary; multi-hop within a focus through stored intermediates | LAB ONLY, not admitted to production; Queued (A1) | F3: no `Gather`/`ScatterOrU32` in any program; `NonFunctionalChain` lifted only in focus | +| D-RST-7 | triangle and amortization | LAB ONLY, not admitted to production; Queued | F6 (existence/count/enumeration, sparse/dense/skewed), F7 measured crossovers, F8 four-path amortization (A native iterator, B compiled sparse, C bit tiles, D GEMM via existing probe kernels) | +| D-RST-8 | multi-hop over a large domain | BLOCKED on admission A2 | — | +| D-RST-9 | native borrowed relation views without petgraph: n-ary sorted permutations as ranges, pattern CSR/CSC as the binary case, stable fact ids, version pin, not `Copy`; starts from the planner's `AdjacencyStore`/`AdjacencyBatch` | LAB ONLY, not admitted to production; Queued (critical path, first) | F1: oracle equality, zero copies, compaction-stable ids, stale and out-of-domain refusals | +| D-RPR-0 | recursive perspective rebinding: architectural distillation (`recursive-perspective-rebinding-v1.md`) | LAB ONLY, not admitted to production; PROPOSED; documentation only | claims graded IMPLEMENTED / MEASURED / PROBE / PROPOSED / OPEN against origin/main | +| D-RPR-1 | recursive question-chain probe over one corpus: escapes seed the next angle (H1), `rebind` vs full prepare vs no frame vs MORK oracle | LAB ONLY; not authorized (waits on admission A1 and an operator go) | exact parity; named refusal per failure case; marginal cost per angle with its crossover | +| D-RPR-2 | premise revision with selective retraction | BLOCKED on D-TGA-3 (per-derived-cell provenance) | stale-premise refusal only, until then | +| D-RPR-3 | MeTTa-facing parity of a chain's answers | BLOCKED on D-ASM-4 (not authorized) | — | | D-MPS-8 | native join over a versioned relationship table: R and S laid out by `y` slot, leapfrog intersection as one mask-risc `And`, per-side generations and reseal, per-thought filter planes (`mork-pathmap-schnellverschluss-v1.md` §7b; `crates/mork-pathmap-probe/src/join.rs`) | MEASURED, probe only; not admitted (D-RMAP-1 BIND membrane review) | 8 join tests + native arms in the differential suite vs the oracle; 7 guards disable-verified; cyclic bodies open | | D-RMAP-2 | context-declaration reuse and GraphBLAS/BF16 tile probe (`d-rmap-2-context-tile-algebra-probe-v1.md`) | PROPOSAL; not run | arms 1–5 in the plan; 297× (D-GTM-0j) is not evidence | | D-SRAM-0 | inventory Soufflé AST/ast2ram/RAM passes against D-ASM/D-RA/D-TGA owners | Queued | evidence per mapped pass; no code imported | diff --git a/.claude/board/entries/2026-10-20-recursive-perspective-rebinding.md b/.claude/board/entries/2026-10-20-recursive-perspective-rebinding.md new file mode 100644 index 000000000..a040e6eb2 --- /dev/null +++ b/.claude/board/entries/2026-10-20-recursive-perspective-rebinding.md @@ -0,0 +1,15 @@ +# Recursive perspective rebinding: distillation (2026-10-20) + +**Status:** PROPOSED, lab only (D-RPR-0). Plan: `.claude/plans/recursive-perspective-rebinding-v1.md`. Documentation only. + +## Findings (read on origin/main) + +- `WORLD × ANGLE × THOUGHT` needs no new nouns. WORLD is a sealed version with thought 0 as givens (D-TGA:6–9). ANGLE is `Focus` + `CsrPred`/tile in the `rmap.rs` probe, with `MapPalette` proposed (D-RMAP-1A). THOUGHT is `ThoughtDelta` over one shared tile (32 B per thought, per-eval flat across 1/8/64; D-RMAP-1A §8.5). +- `TableA[i] : TableB[i]` is the measured H14c/H14d expression `Thought(t)[T_a[i] : T_b[i]]`; the MeTTa council v3 header names those plans as the correct path and marks the preimage-join design a gather violation. +- `rebind` (k slots, one forward seek each) is measured: it wins for small k, loses at k = 256, and on the dense relation from k ≥ 16 (D-RMAP-1A §8.4). Recursive questioning is cheap only if real chains keep k small; that distribution is unmeasured. +- The probe already emits a missing-dependency signal: `ScopedReach.escaped` plus the `IncompleteFocus` refusal. Hypothesis H1: escapes seed the next angle. +- "Teleport" in this repo includes D-TELEPORT-0/1/2, source-restricted folds through Lance 13's row-address prefilter, which is the prefetch mechanism here. + +## Open + +A1 (pre-fold binding admission), H1, the k distribution, n-ary co-addressing and repeated variables, selective retraction provenance (D-TGA-3), a refusal for recursive dependency cycles, `ThoughtCommitId` never-reused rule. diff --git a/.claude/board/entries/2026-10-20-relstore-geometry-not-graph-api.md b/.claude/board/entries/2026-10-20-relstore-geometry-not-graph-api.md new file mode 100644 index 000000000..670d5b5a5 --- /dev/null +++ b/.claude/board/entries/2026-10-20-relstore-geometry-not-graph-api.md @@ -0,0 +1,25 @@ +# D-RST reframed: relational geometry first, petgraph optional, tile GEMM a consumer (2026-10-20) + +**Status:** PROPOSAL (D-RST-0..9). Plan: `.claude/plans/relstore-quack-petgraph-traversal-v1.md`. Documentation only. Supersedes the framing of `2026-10-20-relstore-quack-petgraph-plan.md`; its ownership decisions and findings stand. + +## Decision (operator, 2026-10-20, recorded as provenance) + +- Petgraph is an optional consumer, intermediate and usability reference, beside the Gremlin adapter. It decides no storage format, coordinate model, algebra, optimizer, execution ABI or lifecycle. D-RST-3 is OPTIONAL / DEFERRED / MEASUREMENT-GATED; petgraph parity is the optional falsifier F-PG. +- Tile GEMM accelerates arithmetic on prepared dense operands. It does not resolve identities, scatter, gather or traverse sparse data. +- Target: minimum repeated work per relational reasoning step across front ends and compatible thought versions. + +## Findings (read in source) + +- `lance-graph-planner/src/adjacency/` already holds a pattern-only CSR + CSC with edge ids and a borrowed `AdjacencyBatch` that is deliberately not `Clone`/`Copy` (ruling 2026-07-29). Gaps for D-RST-9: `u64` widths, edge id = arrival position (not a stable fact identity), no version pin, binary only. Core `CsrIndex` is a third owned CSR. No fourth is added. +- The D-RST-3 experiment's view (petgraph#4) is `Copy`, which that ruling forbids; petgraph's `GraphRef: Copy` is met instead by implementing the visit traits for `&View`. +- D-RST-3's measurements: a borrowed CSR walk costs about 2 ns per edge against 300–400 ns per edge of preparation; a Quack one-hop scans the whole lane (O(E)) because the sort order is not attested (admission A6). + +## Open + +- A1–A6, B1–B3 (plan §8). F8 (four-path amortization) is unmeasured; path D runs only through the existing D-RMAP-2 probe kernels. + +## Status ruling (operator, later 2026-10-20) + +Lab exploration only. The relationship store is one option among many for certain data shapes, unfit to promote until further testing; nothing from D-RST or D-RMAP-2 is admitted to production, and no production code may be rewired to it. + +Checked in source: none of the measured paths touched DataFusion. `mork-pathmap-probe`, `lance-graph-quack`, `lance-graph-mask-risc` and the petgraph `relstore-visit` experiment have no DataFusion dependency. BlasGraph and DataFusion are separate `ExecutionStrategy` arms in `lance-graph/src/query.rs`, and the BlasGraph arm returns "not yet wired to input datasets". The measured slowness is the probes' own: an indirect sort plus a second sort in preparation, a full-lane scan with per-query lowering in the Quack hop, and per-element allocation in the BF16 polyfill. diff --git a/.claude/board/entries/2026-10-20-relstore-quack-petgraph-plan.md b/.claude/board/entries/2026-10-20-relstore-quack-petgraph-plan.md new file mode 100644 index 000000000..65aef5629 --- /dev/null +++ b/.claude/board/entries/2026-10-20-relstore-quack-petgraph-plan.md @@ -0,0 +1,22 @@ +# Stored relations, compiled traversal, petgraph over borrowed layouts (2026-10-20) + +**Status:** PROPOSAL (D-RST-0..8). Plan: `.claude/plans/relstore-quack-petgraph-traversal-v1.md`. Documentation only. + +## Decision + +Operator decision B: `RelationshipMapStore` owns relationship persistence, identities, layouts, coordinate-domain seals and invalidation; Quack owns compilation and numeric query semantics; mask-risc on `ndarray::simd` stays the only population executor. D-RMAP-1A §1 and the 2026-10-05 storage-portability ruling stay in force. + +## Findings (read in source) + +- Multi-hop over a large domain is blocked by scatter as well as gather: pushing a frontier writes at positions taken from values, pulling reads at them. The tile (frontier broadcast ∧ a pre-transposed plane) is the only admitted shape, and it is bounded by the focus. Gremlin's `NonFunctionalChain` is therefore two problems: a phase handoff (the store's) and a hop (the ruling's). +- The petgraph fork's `experiments/lance-fold-hop` consumes the quarantined `OutgoingHop` (pinned pre-quarantine); it is not a base for this work. +- petgraph `Bfs`/`Dfs` need only `GraphRef + Visitable + IntoNeighbors`; the visit map is a `FixedBitSet` keyed by `IndexType`, so a dense-slot view reuses it unchanged. +- Quack's `Registrar` has only a test implementor; three library `Binder`s exist (sap, dir-sim, glove-parity). +- A coordinate domain stored as a relation gets its own `DurableRef`, which removes domain ABA without a counter. + +- BlasGraph's shipped CSR (read in source): 2 KiB `BitVec` value per entry, `usize` indices, `mxv` clones the matrix per call (even the CSC transpose clones), `mxv` reads the vector at stored column indices (a scalar gather), and ndarray is not on the CSR/`mxv` path. The store reuses BlasGraph's storage types with a pattern-only `u32` borrowed variant; dense row blocks in the predecessor (pull) direction become bitmap rows executed by ndarray mask ops through mask-risc (`StorageFormat::Bitmap` is already reserved), `F'[t] = any(pred(t) ∧ F)` with no indexed read; the push direction (selecting successor rows) and sparse rows need one. Structural (edge-existence) and valued (hyperdimensional) relations stay distinct. Base indexes are built at seal time; specialized layouts on demand. +- Principle (operator): write a relationship as a relationship; the seal fixes identity, version and declared base indexes, specialized layouts are prepared once on demand and retained, so RelAlg pays translation once per version and only for demanded geometry. + +## Open + +- Admissions A1 (global→local binding), A2 (large-domain multi-hop), A3 (scalar petgraph iteration), A4 (store home/backend), A5 (Registrar shape); blockers B1 (`DurableRef`/`ThoughtCommitId` types), B2 (fork pin), B3 (cyclic joins at scale). diff --git a/.claude/board/entries/README.md b/.claude/board/entries/README.md index c96592ef1..19ed0f174 100644 --- a/.claude/board/entries/README.md +++ b/.claude/board/entries/README.md @@ -25,10 +25,13 @@ index row, (3) no duplicate entry id. Checks 1 and 2 are deliberately opposite directions; the stranding this convention prevents shows up in exactly one of them, never both. -374 entries, 2026-08-06 .. 2026-10-19. +377 entries, 2026-08-06 .. 2026-10-20. | date | entry id | finding | file | |---|---|---|---| +| 2026-10-20 | `relstore-quack-petgraph-plan` | | [2026-10-20-relstore-quack-petgraph-plan.md](2026-10-20-relstore-quack-petgraph-plan.md) | +| 2026-10-20 | `D-RST` | | [2026-10-20-relstore-geometry-not-graph-api.md](2026-10-20-relstore-geometry-not-graph-api.md) | +| 2026-10-20 | `recursive-perspective-rebinding` | | [2026-10-20-recursive-perspective-rebinding.md](2026-10-20-recursive-perspective-rebinding.md) | | 2026-10-19 | `post-merge-recovery-1601-1617` | | [2026-10-19-post-merge-recovery-1601-1617.md](2026-10-19-post-merge-recovery-1601-1617.md) | | 2026-10-19 | `native-join-relationship-table` | | [2026-10-19-native-join-relationship-table.md](2026-10-19-native-join-relationship-table.md) | | 2026-10-19 | `mork-pathmap-schnellverschluss` | | [2026-10-19-mork-pathmap-schnellverschluss.md](2026-10-19-mork-pathmap-schnellverschluss.md) | diff --git a/.claude/plans/recursive-perspective-rebinding-v1.md b/.claude/plans/recursive-perspective-rebinding-v1.md new file mode 100644 index 000000000..6d10739f4 --- /dev/null +++ b/.claude/plans/recursive-perspective-rebinding-v1.md @@ -0,0 +1,223 @@ +# D-RPR: recursive perspective rebinding over composable relational registers + +**Status: LAB EXPLORATION, 2026-10-20. Architecture and experimental design only. Nothing here is admitted to production, no production code may be rewired to it, and no implementation is authorized by this document.** Every claim carries its grade: IMPLEMENTED (code on `main`), MEASURED (a recorded probe run, with its source), PROBE (probe code exists, run or not), PROPOSED (plan text only), OPEN (undecided or unknown). + +**Question.** Can globally consequential reasoning be decomposed into a sequence of bounded changes of perspective, each reusing the relationships its predecessors already resolved and resolving only what the next question adds? This is a question about addressing, memory organization and reuse. It proposes no graph engine, interpreter, world simulator or neural architecture. + +**Sources read for this distillation (origin/main, 2026-10-20):** +- `d-holo-h14c-thought-first-v1.md`, `d-holo-h14d-logic-mask-read-v1.md` (MEASURED); +- `d-rmap-implicit-relationship-register-v1.md` (D-RMAP-1), `d-rmap-separate-tables-thought-palettes-v1.md` (D-RMAP-1A), `d-rmap-2-context-tile-algebra-probe-v1.md`; +- `crates/mork-pathmap-probe/src/{rmap,join,boundary,lib}.rs` and its tests; +- `mork-pathmap-schnellverschluss-v1.md`; +- `metta-datalog-assimilation-v1.md`, its council `-v3.md`, `metta-polyfill-v1.md`, `crates/lance-graph-metta`; +- `thought-git-api-v1.md` (D-TGA), `causal-pickaxe-resident-git-lance13-temporal-v1.md` (D-CPG), `ce64-hypothesis-layers-v1.md` (givens layer); +- D-TELEPORT-0/1/2 board entries; `relstore-quack-petgraph-traversal-v1.md` (D-RST). + +## 1. Experience and navigation are different things + +`Idea(WHERE, WHAT, HOW, WHY)` names kinds of question. It is not a schema, and nothing here stores it as four columns or four registers. A question's kind determines which relationships must become addressable next; it does not determine how. + +`WORLD × ANGLE × THOUGHT` names the navigation, and each of the three already has a home: + +| Navigation term | Existing component | Grade | +|---|---|---| +| WORLD | canonical tables at a sealed version; within a thought table, thought 0 is the givens of that version ("Thoughts `1..=63` branch off main (thought 0, the givens of the current sealed version)", D-TGA:6–9) | givens layer IMPLEMENTED as a probe (`cognitive-shader-driver/examples/givens_layer_probe.rs`, refusal `WriteError::GivensLayer`); D-TGA PROPOSED | +| ANGLE | a selection of relationship maps plus a focus binding: `MapPalette` (0..255 local map selectors → `MapId` + generation), `FocusBinding` (global ids → focus slots), the prepared relation planes | `Focus`, `Csr`, tile IMPLEMENTED in the probe `rmap.rs`; `MapPalette`/`ThoughtMapPaletteBinding` PROPOSED (D-RMAP-1A §1) | +| THOUGHT | one of 64 slots (0 givens, 1..63 hypotheses) holding sparse typed deltas over the shared frame | `ThoughtDelta` PROBE in `rmap.rs`; D-TGA thought table PROPOSED | + +So the brief does not need new nouns. What it adds is a **relation between successive angles** (§4), which nothing on `main` names yet. + +## 2. The addressing operation is the H14c/H14d canon + +The brief's `TableA[i] : TableB[i]` is the measured canonical expression `Thought(t)[T_a[i] : T_b[i]]` (H14c, MEASURED 2026-10-15; H14d, MEASURED 2026-10-16). `i` is a shared coordinate, presence says whether a value is there and never where, and "the join is the layout". The council v3 header of the MeTTa assimilation plan names these two plans as the correct path and marks the preimage-join design (D-ASM-2, D-ASM-11/11A/11B) as a gather violation. + +H14d runs four stages, each shrinking what the next touches: logic reduced symbolically → address mask over presence planes → masked read at the source (Lance 13 `with_row_addr_prefilter`, the D-TELEPORT-2 probe) → terminal. Only addresses are materialized before a terminal. + +**Where translation happens.** Turning a relationship between two independently keyed tables into a shared `i` requires resolving one side's identities in the other's coordinate space. That is the global → local binding. D-RMAP-1 §6: "A pre-fold global->local binding is only a candidate new admissible *membrane* and must be independently reviewed. Until accepted, such operations are a typed refusal." The `rmap.rs` probe performs it (`Focus::bind`, `slot_of` by binary search) and states that it measures it without settling its legality. **Admission A1 is OPEN and is the single precondition for everything below.** + +**Prefetch** is two existing, measured mechanisms, not a new one: the address mask restricting the Lance read (D-TELEPORT-2, H14d stage 3), and the prepared relation planes of a focus (D-RMAP-1A §8). Temporary working copies are admitted by this brief; authoritative ownership stays with the canonical tables. + +## 3. Composable registers: what exists, at what size + +| Register | Concrete form in the probe | Size measured (D-RMAP-1A §8.3) | +|---|---|---| +| WORLD, 0..255 local slots | `struct Focus { ids: Vec, by_id: Vec<(u32, u8)>, pub generation: u64 }`, `SLOTS = 256` (`rmap.rs:58–65, 201–208`) | binding 3 072 B | +| ANGLE, per predicate | `CsrPred { p, offsets: [u32; SLOTS+1], dst: Vec, row: Vec, escaped: Vec<(u8, u32, u32)>, has_escape: SlotSet }`; the 256 × 256 tile built from it | CSR 18–66 KB; tile 8 KiB per predicate | +| THOUGHT, 0..63 | `ThoughtDelta` sparse edits over the shared tile | 32 B per thought as edits, 16 KiB lowered to executor planes; per-eval 2.61 / 2.66 / 2.79 µs at 1 / 8 / 64 thoughts (sparse d2), "the shared tile is stored once" (§8.5) | + +Nothing is a three-dimensional tensor. The world's values are never copied into an angle, and an angle is never copied into a thought. + +**Arity.** The probe's facts are `[u32; 3]` (subject, predicate, object), so its angles are predicate-indexed binary relations; `join.rs` co-addresses one shared variable. General n-ary binding over several shared variables, with repeated variables inside one pattern, has **no design on `main`** (OPEN). The triangle needs two variables co-addressed at once; only the 256 × 256 tile gives that, inside one focus (Schnellverschluss §7b). + +## 4. Recursive rebinding, built from existing pieces + +The brief's `F_{k+1} = Rebind(W, A_{k+1}, F_k)` already has a measured kernel. `rmap.rs` `rebind` replaces k slots of a focus with one forward seek each, re-resolves the old escapes against the new focus, remaps or escapes edges from unchanged members, and rebuilds the tile. The test `incremental_rebind_equals_full_preparation` pins it to a full preparation. D-RMAP-1A §8.4 measured it (µs, incremental / full): + +| k replaced | sparse d2 | sparse d8 | dense | +|---|---|---|---| +| 1 | 41.8 / 357.2 | 223.9 / 1657.7 | 308.2 / 892.9 | +| 16 | 60.4 / 433.1 | 298.1 / 1985.2 | 752.6 / 744.9 | +| 128 | 156.5 / 310.7 | 850.4 / 937.7 | 970.3 / 833.2 | +| 256 | 344.3 / 305.3 | 1238.4 / 971.3 | 1011.0 / 644.9 | + +Rebind wins for small k and loses at k = 256, and on the dense relation from k ≥ 16. **Whether recursive questioning is cheap is therefore a question about the distribution of k along a real chain of questions.** No probe has measured that distribution. + +**The three operations stay distinct, and each already has its mechanism:** + +| Change | Mechanism | What it costs or refuses | +|---|---|---| +| THOUGHT (another hypothesis, same angle) | a new sparse delta; nothing rebinds | measured flat across 1 / 8 / 64 thoughts (§8.5) | +| ANGLE (other relationships, other slots) | `rebind` for slots, a new `CsrPred` for each new predicate | measured above; a new predicate costs its own prepare | +| WORLD (an admitted canonical change) | a new sealed version | every frame built from the old one refuses: `Witness { facts, seal, focus, scope }`, `Refusal::{FactsChanged, IndexResealed, FocusChanged, ScopeChanged, ThoughtMismatch, OrderingMismatch, IncompleteFocus}` (IMPLEMENTED in the probe) | + +**What produces the next angle.** The brief leaves "identify missing dependency" open. The probe already emits the signal. A scoped reach records the global ids its edges left the focus for (`ScopedReach { reached, escaped: BTreeSet }`, exact and never truncated). `global()` refuses with `IncompleteFocus` while any escape exists. **Hypothesis H1:** the escapes of angle `k` are the seeds of angle `k+1`, so "the explanation lies outside the focus" is detected, not assumed, and `rebind` brings exactly those identities in. H1 is PROPOSED and measured nowhere. + +**Ownership along the chain.** A frame has one owner at a time. The mailbox moves to the frame (rotating ownership), so a child angle borrows within the mailbox that holds the frame. Hand-over is a move, which the type system checks (E-CE64-MB-4). The number of hand-overs per chain is a cost to count, not a blocker. + +**Reconsidering the parent.** A deeper angle may contradict an assumption of an earlier one: +- **Contradiction.** A contradiction of a given is recorded and never merged: "A hypothesis write that offers a different value for a given's lane is refused and recorded as a contradiction" (ce64-hypothesis-layers givens rule, IMPLEMENTED in the probe). +- **Retracting what depended on it.** That is delete-and-rederive (D-TGA Rule 2, PROPOSED). It needs, per derived cell, the cells it was derived from, and that provenance is undecided (D-TGA-3, OPEN). +- **Making a revision canonical.** That is `land` through the revision gate G1–G7 (D-TGA-4, PROPOSED). + +Until D-TGA-3 is decided, a parent can be **refused** on a changed premise, but its dependants cannot be **retracted** selectively. + +## 5. PathMap, MORK, Hyperon/MeTTa: what each contributes + +**PathMap (navigation to the next angle).** Measured in D-RMAP-1A §8.6: +- Prefix seek beats a sorted run on sparse relations (7.0 vs 8.5 µs; 100.9 vs 114.7 µs) and loses on dense (1 184 vs 566 µs). +- The trie costs 4–6× the run to build, and its keys take at least 3× the bytes. +- On postings, a zipper merge takes 2.32 µs, a sorted merge 0.41 µs, and a mask-risc `And` + `Count` 0.082 µs. + +So a trie is a candidate *navigation* index for choosing seeds, not a store and not the executor. A Lance-owned zipper over sorted relation orders is D-MPS-6, BLOCKED. + +**MORK (matching).** MORK and PathMap store sets (`PathMap<()>`), so bags are refused (`Refusal::SetOnly`). Its leapfrog route was read in source, never observed running. D-ASM-10 keeps MORK as an oracle outside the workspace. The native shared-variable join (MEASURED, Schnellverschluss §7b, `cargo run --release --example join_cost`, medians of 5) took the following at n = 50 000: + +| Step | Native | PathMap | MORK | +|---|---|---|---| +| Prepare / index build | 2.75 ms | 14.65 ms | 22.87 ms | +| Bag count | 35.5 µs | — | — | +| Answers streamed | 0.33 ms | — | — | +| Answers into a map | 7.39 ms | — | — | + +Only the intersection runs on mask-risc; enumeration and counting are plain loops. One join shape was measured, on synthetic data, warm, with no reuse across different queries. + +**Hyperon/MeTTa.** The existing classification (`metta-polyfill-v1.md:128–132`, Decision H) sorted into the brief's four classes: + +| Class | Operations | Source | +|---|---|---| +| 1. Exactly representable | facts in a space; `match` over one pattern; conjunctions with shared variables; no new terms | polyfill (PROPOSED) | +| 2. Representable after binding or materialization | recursive `=` rules that are Datalog (no function symbols in heads, no grounded calls), through the external driver d′, one fold per iteration, bounded, "refusing an unbounded closure rather than truncating" | Decision D, gated (PROPOSED) | +| 3. Requires functionality not yet built | repeated variables inside one pattern (no plan text addresses it); bags whose multiplicity lives on a foreign table (the keyed count undercounts, council N1c); nested expressions (refused in W1, N2); n-ary co-addressing (§3) | OPEN | +| 4. Kept external | term construction, `superpose`/`collapse`, grounded operations, `new-state`/`change-state!`, dependent types, `=` rules with a function symbol in the head: answered by Hyperon's interpreter, refused or routed on our side, never approximated | Decision H, council F6 | + +`crates/lance-graph-metta` is a stub: "Nothing is shipped here yet". No `Space` implementation exists (D-ASM-4/D-MTP-2 queued, not authorized). Oracles: Hyperon, PeTTa, datafrog, and MORK as a fourth, compared per category (bag / set / count), never averaged (Decision G, R-8). + +**Assimilation rule** (unchanged from those plans): take the algorithm and its semantic contract into RelAlg / Quack / R2IL / mask-risc only where exactness is shown against the oracle. At the boundary, reconstruct MeTTa-compatible bindings only on request. No foreign runtime objects, S-expressions or JSON on a hot path, and no Hyperon interpreter as a hidden second evaluator. + +## 6. Reuse is permitted only under a witness + +A prepared frame is reused only while its `Witness` equals the current one. Otherwise it refuses and names the stale component (IMPLEMENTED in the probe). D-RMAP-1A §7.4 specifies nine stale-state cases, none of them run. Among them: +- X1: datasets that collide at the same version; +- X2: compaction without stable row ids; +- X3: palette ABA; +- X4: a reused thought slot; +- X5: a dirty thought; +- X6: a rebase; +- X7: a revoked scope; +- X8: lineage after a squash; +- X9: a deletion on main. + +Durable versions are `DurableRef { dataset_id, branch, version }` and speculative ones `ThoughtCommitId`. Neither type exists in `crates/` (PROPOSED, D-CPG). Their never-reused rule is OPEN (D-TGA gaps). + +## 7. Three kinds of plasticity, three admission levels + +| Kind | What changes | Admission | +|---|---|---| +| Epistemic | conclusions within one angle: thought deltas, never the givens | none beyond the thought's own write rules | +| Perspective | which relationships and slots are addressable: rebind, palette selection | no canonical write; frames refuse on stale witnesses | +| Ontological | canonical facts, classifications or relationships | `land` through G1–G7 with retraction (D-TGA-3/-4), authorized, and the strongest evidence; PROPOSED, nothing built | + +A bounded frame does not establish global consistency. The only guard on `main` is `IncompleteFocus`: a frame with escapes cannot answer globally. Any experiment below must show that guard firing, not just that answers match. + +## 8. Minimal experiments (pre-registration sketch; none authorized, none run) + +**D-RPR-1: recursive question chain over one corpus.** Probe in `mork-pathmap-probe`, lab only. It reuses `Focus`, `discover`, `Csr`, `rebind`, `ThoughtDelta`, `Witness` and `join.rs`, and adds no new types. + +**Corpus.** The KJV SPO triples exported by `deepnsm-v2/examples/bible_wave.rs --export spo.tsv`. They need Gutenberg #10 and the `v0.1.0-cam96-data` release assets, which are fetched, not committed. A committed synthetic world is a fallback and must be labelled as such. The figures "31 327 triples" and "606 subjects" were not found in the example's source and must be re-measured before they are cited. + +**Chain.** Angle `A_0` is seeded by one question's entities and predicates. Each later angle is seeded by the previous one's escapes (H1), plus the predicates the next question kind adds. + +**Arms:** +- (a) MORK / PathMap repeated matching: set semantics only, as an external oracle; +- (b) native direct lookup with no retained frame (the probe's "direct/run" arm); +- (c) native, with a full prepare per angle; +- (d) native with `rebind` and retained predicate CSRs. + +**Measured per angle, kept apart:** +1. the initial binding cost; +2. the marginal cost of the next angle, given what is retained; +3. relationships reused; +4. new identities and predicates resolved; +5. indexed reads and bytes prefetched; +6. fold time; +7. 1 / 8 / 64 thoughts; +8. refusal on a changed premise; +9. resident bytes and the reuse crossover; +10. exact parity: bags against a nested-loop oracle, sets against MORK. + +Also measured: the distribution of k per rebind and the number of ownership hand-overs. + +**Reuse must fail, each case with a named refusal:** + +| Case | Refusal | +|---|---| +| no shared slots with the parent | full prepare, reported as k = 256 | +| a changed predicate order | `OrderingMismatch` | +| a new canonical version | `FactsChanged` / `IndexResealed` | +| a scope change | `ScopeChanged` | +| a rejected parent hypothesis | `ThoughtMismatch` | +| an answer outside the focus | `IncompleteFocus` | +| a recursive dependency cycle | **no refusal exists yet: OPEN** | + +**Acceptance.** Exact parity on every arm. Every case above refuses rather than answers. Arm (d)'s chain total is reported against (b) and (c) with its crossover. + +**Falsified if** arm (d)'s marginal cost per angle is not below arm (c)'s for the measured k distribution, or if any narrow frame returns an answer without `IncompleteFocus` when its oracle needs out-of-focus facts. + +**D-RPR-2: premise revision and selective retraction.** BLOCKED on D-TGA-3, which leaves the per-derived-cell provenance undecided. Until then only refusal is testable, not retraction. + +**D-RPR-3: MeTTa-facing parity of a chain's answers.** BLOCKED on D-ASM-4 (the `LanceSpace` harness), which is not authorized. + +## 9. Reuse before creating anything + +| Need | Existing | Grade | +|---|---|---| +| shared-coordinate operation | `Thought(t)[T_a[i] : T_b[i]]` | MEASURED (H14c/H14d) | +| focus, binding, escapes, refusals, witness | `rmap.rs` | PROBE | +| incremental rebind | `rmap.rs` `rebind`, §8.4 table | MEASURED | +| shared-variable join with bags | `join.rs`, Schnellverschluss §7b | MEASURED | +| source-restricted reads | Lance 13 row-address prefilter, D-TELEPORT-2 | MEASURED (probe) | +| thoughts over one frame | `ThoughtDelta`, §8.5 | MEASURED | +| givens immutability and contradiction records | `givens_layer_probe.rs` | PROBE | +| versions, branches, land, retraction | D-TGA, D-CPG | PROPOSED | +| map selection per thought | `MapPalette`, `ThoughtMapPaletteBinding` (D-RMAP-1A) | PROPOSED | +| MeTTa boundary and oracles | `metta-polyfill-v1.md`, D-ASM-4/-10 | PROPOSED; crate stub | +| relationship store ownership | D-RMAP-1A §1, D-RST (lab) | PROPOSED | + +**Nothing new is needed for D-RPR-1** except the chain driver and the measurements. + +## 10. Open, in order of consequence + +1. **A1:** is the pre-fold global → local binding an admitted membrane? Everything here presupposes it. +2. **H1:** do escapes predict the next angle's seeds in a real question chain? +3. **The k distribution** along real chains; it decides whether rebind pays (§4). +4. **n-ary co-addressing and repeated variables** (§3, §5 class 3). +5. **Selective retraction provenance** (D-TGA-3). +6. **A refusal for recursive dependency cycles.** +7. **`ThoughtCommitId`'s never-reused rule.** + +| D-id | Content | Status | +|---|---|---| +| D-RPR-0 | this distillation | PROPOSED, lab only | +| D-RPR-1 | recursive chain probe | not authorized; waits on A1 and an operator go | +| D-RPR-2 | premise revision with retraction | BLOCKED on D-TGA-3 | +| D-RPR-3 | MeTTa-facing parity | BLOCKED on D-ASM-4 | diff --git a/.claude/plans/relstore-quack-petgraph-traversal-v1.md b/.claude/plans/relstore-quack-petgraph-traversal-v1.md new file mode 100644 index 000000000..da009e2df --- /dev/null +++ b/.claude/plans/relstore-quack-petgraph-traversal-v1.md @@ -0,0 +1,345 @@ +# D-RST: compiled relational geometry and its amortization — RelationshipMapStore × RelAlg/Datalog × Quack + +(File name kept for existing links; the plan was first written as "RelationshipMapStore × Quack × petgraph". See the correction below.) + +**Status: LAB EXPLORATION, 2026-10-20 (operator). Nothing here is admitted to production, and no production code may be rewired to it.** The relationship store tested here is one option among many, meant for certain data shapes. It is unfit to promote until further testing: the measured preparation is slow (300–400 ns per edge, §7), and nothing has been compared with the alternatives on real workloads. Every "critical path", "PR" or "production" below describes a lab sequence that could only become a production proposal after its own review. Documentation only. No crate, type, opcode, ABI, dependency or ruling change lands with this plan. Every capability below is labelled IMPLEMENTED, PROBE, PROPOSAL or RULING from its own source. + +**Decision taken (operator, 2026-10-20), recorded as provenance, not as validation:** option B. `RelationshipMapStore` owns relationship persistence, identities, index layouts, coordinate-domain seals and invalidation. Quack owns compilation and numeric query semantics. mask-risc on `ndarray::simd` stays the only population executor. This keeps the D-RMAP-1A §1 ownership table and the 2026-10-05 Quack storage-portability ruling as they are. Neither is superseded here. + +**Correction (operator, 2026-10-20, later the same day), recorded as provenance, not as validation: the center of gravity is reusable relational geometry, not a graph API.** +1. **Petgraph is an optional consumer, an optional intermediate and a usability reference**, on the same footing as the Gremlin adapter. It does not decide the storage format, coordinate model, algebra, optimizer, execution ABI or relationship lifecycle. The first version of this plan made it a pillar ("petgraph makes it usable") and brought D-RST-3 forward. Both are withdrawn: D-RST-3 is OPTIONAL / DEFERRED / MEASUREMENT-GATED (§7), and petgraph parity is an optional interoperability falsifier (F-PG), not an acceptance criterion. +2. **Tile GEMM consumes prepared relationships; it does not obtain them.** It speeds up arithmetic once the operands are in a compatible dense layout. It does not resolve identities, scatter, gather or traverse sparse data (§5b). + +**Goal.** Write relationships once. Normalize irregular access into reusable relational geometry: identities, coordinate domains, index permutations, prepared layouts and version dependencies. Amortize that preparation across joins, projections, recursion and traversal, whichever front end asks (SQL-shaped Quack, Cypher, Datalog, RelAlg, Gremlin, petgraph) and across compatible thought versions. **The optimization target is the minimum repeated work per relational reasoning step, not maximum arithmetic throughput.** + +**Success criterion.** Datalog, RelAlg and Quack workloads run efficiently over retained relational layouts without importing petgraph or Gremlin. Petgraph-shaped iterators or selected petgraph algorithms may later be exposed over the same layouts, which would show usability, not decide the architecture. + +**Scope.** Relations are n-ary: Datalog predicates, variable permutations, join domains, derived relations, recursion and provenance. Graph traversal is one special case of relational composition (a binary relation composed with itself). Neither a graph API nor a GEMM kernel may change the representation chosen for a relation. + +Read with: +- [D-RMAP-1A](d-rmap-separate-tables-thought-palettes-v1.md) §§1, 3, 6–8: ownership, borrowed views, falsifiers, versioning, focus probe. +- [Schnellverschluss](mork-pathmap-schnellverschluss-v1.md) §7b: the native join over a versioned relationship table. +- [D-CPG](causal-pickaxe-resident-git-lance13-temporal-v1.md) §1: `DurableRef`, `ThoughtCommitId`. +- [D-TGA](thought-git-api-v1.md). +- [H14c](d-holo-h14c-thought-first-v1.md) §1: the address is the register. +- Board entry `2026-10-05-quack-storage-portability.md`. +- `.claude/knowledge/fold-execution-laws.md`. + +**Short form:** the store remembers the geometry, BIND resolves once, RelAlg/Datalog define the bindings, Quack compiles, mask-risc executes. Graph APIs are optional views over the same layouts. + +**The amortization principle (operator, 2026-10-20): write a relationship as a relationship.** A relation is written once, as relation facts, and the seal fixes its identity, its version and the base indexes its declaration names (a sorted permutation, and CSR/CSC where declared). Specialized execution geometry (bitmap blocks, focus tiles, a co-addressed join domain on one shared variable) is prepared on the first demand that needs it, once, and retained under a dependency seal on that relation version. Nothing is rebuilt per query or per front end, and nothing is built for a query that never comes: a relation with several joinable attributes has many candidate join domains, and only the demanded ones exist. The seal establishes validity; preparation establishes geometry, and F7 measures whether it paid for itself. RelAlg's semijoins, joins and projections then read prepared relationship layouts instead of translating identities on every evaluation. That is where RelAlg gets faster: the translation is paid once per sealed version, the evaluation is a fold. §5a checks what the shipped BlasGraph CSR contributes to this. + +## 0. Findings that shape this plan + +1. **Multi-hop under the no-gather ruling is blocked by scatter as much as by gather.** One hop from a frontier mask has three spellings: + - **Push:** for each set source `s`, write `next[t]` for every `t` in `out(s)`. This writes at positions taken from values: scatter. + - **Pull:** `next[t] = OR over s in in(t) of frontier[s]`. This reads at positions taken from values: gather. + - **Tile:** frontier broadcast `∧` a pre-transposed `(target, source)` bit plane, then any per target. Every read and write is at its own coordinate. It costs `|focus|²` bits, so it is bounded by the focus. + + So the only admitted multi-hop shape today is the tile. Over a domain larger than a focus, multi-hop needs an admission decision (§8, A2). It is not an implementation gap a store can fill. Gremlin's `NonFunctionalChain` (`lance-graph-quack/tests/gremlin_parity.rs`, the `Refusal` enum: "needs a barrier and a workspace inside one computation. Gap: no typed phase handoff exists") is two problems. The handoff is the store's to give, as a stored intermediate. The hop is the ruling's to admit. +2. **The existing petgraph integration rests on the quarantined gather.** `AdaWorldAPI/petgraph` `experiments/lance-fold-hop` (fork PR #3) consumes `lance_graph_mask_risc::OutgoingHop`. That is two `Gather` ops plus `ScatterOrU32`, pinned to lance-graph `66636da`, from before #1616's quarantine. Its README already says it is one demanded hop, and that iterative BFS "needs a reviewed phase-handoff / continuation primitive". ISSUES.md records that bumping the pin must enable `quarantined-gather-hop` explicitly. This plan does not build on it. +3. **(Optional consumer; evidence for D-RST-3.) petgraph's traversal needs only two traits.** `Bfs::new` / `Dfs::new` need `GraphRef + Visitable`, and `next` needs `IntoNeighbors` (`crates/petgraph/src/visit/traversal.rs`). The visit map is a `FixedBitSet` keyed by `IndexType::index()` (`visit/mod.rs`, `impl VisitMap for FixedBitSet`), so a view whose `NodeId` is a dense `u32` slot gets petgraph's own visit map. `Csr` (`crates/petgraph/src/csr.rs`) shows the shape: `IntoNeighbors for &Csr` returns a slice iterator over `neighbors_slice`. It owns its `Vec`s, so the view must borrow instead. +4. **The Registrar seam is real, minimal and unowned.** `lance-graph-quack/src/bind.rs`: `trait Registrar { fn register(&mut self, name: &str, width: u32) -> Result; }`. The doc says "No crate owns that today; this trait names the seam, and only tests implement it". The only implementor is a `#[cfg(test)] Mem`. Three `Binder`s exist in library code: `lance-graph-sap` `CatsBinder`, `lance-graph-dir-sim` `UserBinder`, `lance-graph-glove-parity` `CanonicalBinder`. +5. **A coordinate domain is itself a relation.** The join probe needs a seal for its shared domain (Schnellverschluss §7b: a new `y` re-addresses every side). Stored as a relation (the dictionary of global ids, in slot order), the domain gets a `DurableRef` like any other table. No counter is added. A domain change is a new version, and versions are not reused, so domain ABA cannot happen. + +## 1. Current capability inventory + +| Capability | Where | Grade | +|---|---|---| +| Quack query surface: scan/filter/project/aggregate lowered to one `Program`; "may build a `Program` and must never evaluate one" | `lance-graph-quack/src/lib.rs` module doc | IMPLEMENTED | +| `Binder` (catalog + codebook, resolved once), `Registrar` (catalog + allocation seam), `create_table` → `ResolvedTable` | `lance-graph-quack/src/bind.rs` | IMPLEMENTED (trait); Registrar has no production implementor | +| Gremlin parity adapter: `V, Has, Out, In, Where, Dedup, Values, Count, Sum, GroupCountBy, Emit`; refusals incl. `NonFunctionalChain`, `ComposedFunctionalHop`, `Repeat`, `PathMultiplicity`, `BagOfElementsNotAMask` | `lance-graph-quack/tests/gremlin_parity.rs` | IMPLEMENTED as test code ("adds no operator to Quack and no op to mask-risc") | +| Cypher → Quack: single-label aggregates, one-hop walk count, `count(DISTINCT b)` under `dst` order; refuses a second hop, variable length ("a hop chain over a computed frontier"), WHERE with a hop | `lance-graph-cypher-quack/src/lib.rs` (`Refusal::Gap` doc) | IMPLEMENTED | +| RelAlg: set algebra over one row universe (`Rel`, `Measure`), lowered via R2IL `FoldDialect`; "recursion (`μ`) and the hop (preimage) are not built" | `lance-graph-relalg/src/{ir,lower}.rs`, `lib.rs` | IMPLEMENTED (prototype for D-CMR) | +| α-acyclicity test (GYO) for a join body; routes acyclic → semijoin chain, cyclic → intersection | `lance-graph-relalg/src/join_body.rs` | IMPLEMENTED (test only; no execution) | +| R2IL fold dialect: `VIA, RANGE, SUM, MIN, MAX, GROUP_SUM, KEEP` (+ refused `KEY_RUNS, ANY, ALL, SCATTER_OR, BLEND`) | `lance-graph-r2il`, opcodes from `ogar-r2il` | IMPLEMENTED; no JOIN or HOP op | +| mask-risc: `MaskOp::{Pred, And, Or, Xor, AndNot, Not, Ternlog, Gather}`; terminals incl. `Keep`, `Count`, `GroupReduce`, `ScatterOrU32` | `lance-graph-mask-risc/src/ir.rs` | IMPLEMENTED; no frontier-continuation primitive | +| `OutgoingHop` (two `Gather` + `ScatterOrU32`) | `lance-graph-mask-risc/src/graph_hop.rs`, feature `quarantined-gather-hop` (off) | QUARANTINED (`ISS-GATHER-HOP-1616`) | +| Focus-bound RelationshipMap: sorted run, PathMap trie, slot-space CSR, 256×256 tile via mask-risc, incremental rebind, thought deltas | `mork-pathmap-probe/src/rmap.rs` | PROBE (D-RMAP-1A §8) | +| Versioned relationship table: shared-variable domain, per-side generations, `And` intersection, bag answers, star joins, filters per thought | `mork-pathmap-probe/src/join.rs` | PROBE (Schnellverschluss §7b, D-MPS-8) | +| `DurableRef{dataset, branch, version}`, `ThoughtCommitId` | D-CPG §1 | PROPOSAL; no type in `crates/` | +| Durable write boundary: cycle seal → `LanceCycleWriter`, one `DatasetVersion` per cycle | `cycle_sink.rs`; storage-portability entry | IMPLEMENTED | +| Ownership: store owns facts/orders/maps/tiles; Quack "must not own semantic maps"; RelAlg never owns maps | D-RMAP-1A §1 | PROPOSAL (controlling contract) | +| Quack owns read semantics, the backend supplies capabilities, the seal owns durable writes; "no universal `Storage` trait" | storage-portability entry, 2026-10-05 | RULING | +| No gather; "any address translation is explicit, counted, and outside the direct-position path" | H14c §1 rule 6; 2026-10-16 ruling | RULING | +| A pre-fold global→local binding is a candidate membrane that needs its own review; until then, a typed refusal | D-RMAP-1 §6 | PROPOSAL (refusal stands) | +| petgraph visit traits, `Bfs`/`Dfs`/`Walker`, `Csr` | `AdaWorldAPI/petgraph` `crates/petgraph/src/{visit,csr.rs}` (0.8.3, MSRV 1.85, dev toolchain 1.99) | IMPLEMENTED upstream | +| BlasGraph sparse storage: `CsrStorage`, `CscStorage`, `HyperCsrStorage`, COO→CSR, `GrBMatrix` (CSR + optional CSC), `mxv`/`vxm`/`mxm` over a `dyn Semiring` incl. `HdrSemiring::Boolean`; `StorageFormat::Bitmap` reserved | `lance-graph/src/graph/blasgraph/{sparse,matrix,semiring}.rs`; planner `blasgraph_planner.rs` (Cypher `Expand` → `mxm`) | IMPLEMENTED; a second copy in `holograph/src/graphblas` | +| ndarray dense mask algebra: `mask_and`, `mask_or`, `mask_andnot`, `mask_ternlog`, popcount/any; dense BLAS levels 1–3 | `ndarray/src/simd_masking_ops.rs`, `src/hpc/blas_level{1,2,3}.rs` | IMPLEMENTED; no sparse/CSR primitive under `src/hpc` or `src/simd*` (searched `csr`, `spmv`, `sparse`) | +| Pattern CSR + CSC substrate with edge ids, one store per relationship type ("Kuzu pattern"); `adjacent`, `adjacent_incoming`, `edge_ids`, degrees, `from_edges`; a borrowed `AdjacencyBatch<'a>` (no allocation; deliberately not `Clone`/`Copy`, operator ruling 2026-07-29 "copies are forbidden, borrows are only for the same mailbox"); `intersect` returns an owned `IntersectionResult` | `lance-graph-planner/src/adjacency/{csr,batch,propagate,distance}.rs` | IMPLEMENTED; `u64` offsets/targets/ids, owned `Vec`s, no version pin, edge id = arrival position in the input (`from_edges`), so not a stable fact identity; binary only; used only inside its own module | +| `CsrIndex` (offsets + neighbours, `u64`), built from an Arrow `RecordBatch` | `lance-graph/src/csr_index.rs` | IMPLEMENTED; owned, out-edges only | +| Borrowed, version-pinned pattern-CSR view traversed by petgraph's unmodified `Bfs`/`Dfs`; Quack one-hop over the same buffers | `AdaWorldAPI/petgraph#4`, `experiments/relstore-visit` | PROBE (D-RST-3, now deferred); its view is `Copy`, which the 2026-07-29 ruling forbids (§3.3) | +| BF16 16 × 16 tile GEMM vs bit-tile composition, polyfill tier only | `mork-pathmap-probe/src/tile_gemm.rs` (#1623, D-RMAP-2) | PROBE | +| `RelationshipMapStore`, `RelationCatalog` | — | not found under `crates/` (searched `*.rs`, `*.md`) | + +**Prior art to reuse before adding a type.** Three owned CSR families already exist: the planner's `AdjacencyStore` (CSR **and** CSC, pattern-only, with a borrowed batch view that already follows the no-`Copy` ruling), core `CsrIndex`, and BlasGraph's valued `CsrStorage`/`CscStorage` (plus the `holograph` copy). The native borrowed view of §3.1 starts from `AdjacencyStore`/`AdjacencyBatch`: it already has the shape (pattern CSR + CSC + ids, borrowed batch). Its gaps are `u32` slots, a stable fact identity instead of the arrival position, a version pin, and n-ary relations as sorted permutations. No fourth owned CSR type is added; the probe CSRs (`rmap.rs`, `relstore-visit`) stay probes. Masked adjacency is already a presence plane in mask-risc (`Planes.masks`) and the probe tile (`rmap.rs`, 256 × 256); no new tile type is needed for the Boolean path. + +## 2. Ownership and seams + +```text + Quack query Cypher Datalog / RelAlg optional consumers + \ | / (petgraph, Gremlin adapter) + \______ PARSE ______/ : + | : + Quack COMPILE (V4: what to compute) : + | : + BIND: identities, coordinate domain, authorization, version, : + eligible prepared layouts — resolved once : + | : + BUNDLE: choose a prepared layout (may not change the answer) : + | : + mask-risc / ndarray::simd EXECUTE over borrowed planes : + | : + RelationshipMapStore: catalog · facts · derived relations · : + permutations · layouts · domains · seals · invalidation ◄········ borrowed views of the + | same layouts (no copy, + cycle seal → Lance (DurableRef) ThoughtCommitId no second graph) + | + SoA: canonical identity-addressed state (separate admission) +``` + +| Owner | Owns | Must not own | +|---|---|---| +| RelationshipMapStore | relation catalog (arity, argument sorts, set/bag, functional constraints); facts; derived relations with multiplicity, provenance and dependencies; sorted permutations; physical layouts (CSR, tiles, join tables); coordinate domains; layout seals; invalidation | Datalog/Gremlin syntax, a query evaluator, a version counter of its own, CE64 fields | +| BIND | resolving names, identities, domains, authorization and versions into a witness; attesting which layouts are eligible | layouts, storage, execution | +| Quack | compiling relational and graph operations into `Program`s; numeric query semantics; the `Registrar`/`Binder` seams | storage, catalog state, evaluation | +| RelAlg / Datalog | variable bindings, join bodies, set vs bag, recursion and fixpoint meaning | map tables, index builds | +| mask-risc / `ndarray::simd` | population execution | anything else | +| petgraph, Gremlin adapter (optional consumers) | their algorithm and iterator vocabulary, where a measured workload adopts it | storage format, coordinate model, algebra, optimizer, execution ABI, relationship lifecycle, a second graph copy | +| SoA | canonical identity-addressed state; admission of stable semantic relationships | derived relations (admission is optional and separate) | +| cycle seal / Lance | durable writes, `DurableRef` | relational semantics | + +### 2.1 Seams (proposed shapes, not code) + +```rust +// The store satisfies Quack's existing seam for its tables. No new catalog trait. +impl Registrar for RelationshipMapStore { /* register(name, width) -> TableId */ } + +// Relations carry more than a width. Declared beside, not inside, Registrar, +// so the Quack trait does not change in the first PRs. +pub struct RelationDecl { name, arity, sorts: [SortId; N], semantics: SetOrBag, functional: Option } +pub struct RelationId(u32); // resolved once; never a version +pub struct LayoutId(u32); // one physical layout of one relation version + +/// Every read is pinned to this. Nothing here is a counter the store mints. +pub struct RelationVersion { + relation: RelationId, + at: Revision, // Durable(DurableRef) | Speculative(ThoughtCommitId) +} +pub struct LayoutSeal { + layout: LayoutId, + built_from: RelationVersion, + domain: RelationVersion, // the coordinate domain is a relation too (finding 5) + addresses: Option, // row-address map, dropped with stable row ids (D-RMAP-1A §7.3) +} + +/// What BIND hands Quack and mask-risc. Borrowed, ephemeral, never stored. +pub struct BoundRelation<'s> { version: RelationVersion, layout: LayoutSeal, planes: Planes<'s>, scope: AuthorizationReceipt } + +// Derived relations are relations. +store.materialize(compiled_result, deps: &[RelationVersion]) -> RelationVersion +``` + +Durable versions come from the seal (`DurableRef`); speculative ones from a thought commit (`ThoughtCommitId`). The store records which versions a layout or derived relation was built from and refuses a read when any of them differs. It persists no counter of its own (D-RMAP-1A §7.7, option B). + +## 3. Relational access: borrowed views and compiled cursors + +### 3.1 Native borrowed relation views (critical path) + +A relation version is read through a **view**: a borrow of the store's prepared buffers, pinned to the `RelationVersion` and `LayoutSeal` they were built from, refused for any other. A view owns nothing and copies nothing. Following the 2026-07-29 ruling, a view is **not** `Clone` or `Copy` (the shape `AdjacencyBatch` already has); it is passed by reference, so its reach is bounded by the borrow it came from. + +- **n-ary first.** A relation's facts are columns, one per argument, plus a stable fact identity. Its base indexes are sorted permutations in the variable orders its declaration names (for `r(x, y, z)`: `(x, y, z)`, `(y, x, z)`, …), read as ranges. +- **CSR and CSC are the binary specialization**: offsets over a sorted permutation. CSR serves push and neighbour iteration, CSC serves pull (§5a); both may be declared. +- **The fact identity is the edge id.** A physical row offset is an address that compaction moves (D-RMAP-1A §7.3); a view reports it separately and never as an identity. +- **Start from what exists.** The planner's `AdjacencyStore`/`AdjacencyBatch` already has CSR + CSC + ids and a borrowed batch view (§1). The gaps are `u32` slots, a stable fact identity, a version pin and n-ary permutations, plus BlasGraph's pattern-only reading (§5a). Whether that becomes a change to `AdjacencyStore` or a view beside it is decided by D-RST-9, not here. + +### 3.2 Compiled relational cursors + +The same `RelationVersion` and `LayoutSeal`, consumed as whole spans and planes, compiled by Quack: +- index ranges over sorted permutations (a range read needs the seal to attest the order; §8 A6); +- CSR/CSC spans per slot range; +- semijoins, joins and projections over a co-addressed join domain (join-table presence planes, intersection = `And`; Schnellverschluss §7b); +- tile planes and frontier masks (the only admitted multi-hop shape, finding 1). + +**Typed traversal vocabulary.** Arbitrary `Iterator::filter` / `flat_map` closures cannot be inspected and are not compiled or fused. The plan instead proposes a small typed vocabulary that compiles (terminal `.compile(&quack)`) and can also run as an ordinary iterator (terminal `.iter()`): + +| Typed step | Meaning | Compiles to | +|---|---|---| +| `from(mask \| slots)` | start population | resident plane | +| `out(rel)` / `in_(rel)` | one hop | tile hop (in focus); refused over a large domain until A2 | +| `has(field, Cmp)` | filter | `Pred` under the population | +| `dedup()` | set semantics | mask | +| `count()` / `group_count(field)` | terminals | `Count` / `GroupReduce` | +| `repeat_bounded(k)` / `until_fixpoint()` | iteration | k tile hops / hop until the plane stops changing, within focus | +| `emit()` | materialize | explicit terminal; the only exit producing ids | + +Each step carries set/bag/path semantics in its type. Mixing them without an explicit `dedup` or `paths` step is a compile error, never a silent collapse (Gremlin's `BagOfElementsNotAMask` and `PathMultiplicity` stay real refusals). A chain that cannot compile refuses at `.compile`. It never falls back to `.iter()` on its own (no silent fallback). The traversal is a value that is compiled, not an object stepped per traverser, so no traverser VM is needed. + +### 3.3 Optional: petgraph over the same views (deferred, measurement-gated) + +Petgraph's `Bfs`/`Dfs` need `GraphRef + Visitable` and `IntoNeighbors` (finding 3). Over a §3.1 view they can run unmodified, with the iterator a slice of the stored layout and `EdgeId` the fact identity. AdaWorldAPI/petgraph#4 showed this works (§7, D-RST-3 result). It is not on the critical path, and nothing in §§3.1–3.2 depends on it. + +Two constraints for any later adoption: +- **`GraphRef: Copy` conflicts with the no-`Copy` view.** petgraph#4's view is `Copy`, which the 2026-07-29 ruling forbids. The compatible shape implements the visit traits for `&View` (petgraph's blanket `impl GraphRef for &G`), so the copyable thing is a shared reference and the view itself stays unique. +- **`IntoNeighborsDirected`** only where the store holds CSC; never computed on demand. + +Admission A3 (whether host iteration sits outside the no-gather ruling) only needs an answer if petgraph or another scalar iterator is adopted. + +## 4. Execution flow + +1. **Declare / store.** `RelationDecl` → store catalog. Facts arrive through the cycle seal. The store records the resulting `DurableRef` against the relation. +2. **Seal, then prepare on demand.** At seal time the store builds only the base indexes the declaration names (sorted permutation; CSR/CSC where declared) and the coordinate domain as a relation. A specialized layout (bitmap blocks, a focus tile, a join table for one shared variable) is built when a compiled demand first needs it and is retained under its `LayoutSeal`, so the next demand on the same relation version reuses it. Each preparation is timed and counted (normalization, index build, translation), and an unused layout is never built. +3. **Bind.** BIND resolves names → ids, picks the domain, checks authorization, pins versions, and lists the layouts whose seals match. The global→local translation runs here or in step 2, never inside a fold (H14c rule 6), and it is subject to admission A1. +4. **Compile.** Quack turns the query (Quack query, Cypher, Gremlin, RelAlg/Datalog, typed traversal) into a demand plus a `Program` over the eligible layouts' planes. +5. **Bundle.** Choose a layout using the capability table (§5). By fold law 3, the choice may not change the answer; F2 is the test. +6. **Execute.** mask-risc over borrowed planes. Results come back as masks, counts or group reductions; tuples only through an explicit `emit`/`materialize`. +7. **Remember.** A result that later steps reuse is materialized as a derived relation: its rows plus multiplicity, provenance (source fact rows or binding tuples) and dependencies. It can be queried, laid out and bound like any relation. A bare mask is never stored as a relation; it carries neither bindings nor multiplicity. + +## 5. Layout reuse and invalidation + +**Measured selection inputs (probes, warm, one machine; to be re-measured cold in F7):** + +| Layout | Fits | Measured | Source | +|---|---|---|---| +| sorted permutation | sparse, any domain; seeks | build 29–483 ms for 0.4–2.9 M facts | D-RMAP-1A §8.3 | +| CSR (slot space) | sparse in focus | wins at degree 2 (3.3 vs 5.1 µs) | §8.3 | +| 256×256 tile | dense in focus; the only no-gather multi-hop | wins dense (4.5 vs 12.4 µs) | §8.3 | +| join table on a shared variable | star / acyclic joins on one variable | prepare 2.7 ms vs PathMap 14.7 ms (n = 50 000); count 36 µs | Schnellverschluss §7b | +| PathMap trie | prefix seek on sparse | faster than the run on sparse, 4–6× slower to build | §8.6 | + +**Invalidation rule:** a layout or derived relation is valid only while every `RelationVersion` in its seal or dependency list equals the current one. Otherwise it is refused, never silently reused. +- **Maintenance:** incremental where measured to win (one side of a join table, or k slots of a focus at small k); a rebuild otherwise. The probe crossovers are the starting policy, to be re-measured. +- **Deletions:** a derived relation over a deleted fact is retracted before re-derivation (DRed, D-TGA Rule 2; D-RMAP-1A X9). +- **Thoughts:** a thought reads layouts at its pinned versions and adds sparse deltas. Layouts are not copied per thought (D-RMAP-1A §8.5 measured 32 B per thought as sparse edits). + +## 5a. BlasGraph CSR and ndarray: what to reuse (read in source, 2026-10-20) + +BlasGraph already ships the relationship-matrix vocabulary this plan needs: CSR, CSC, hypersparse CSR, COO→CSR, a Boolean semiring, and a planner that lowers Cypher `Expand` to `mxm`. The store should reuse that vocabulary for its CSR/CSC layouts instead of carrying the probe's CSR into production. As shipped it cannot serve as a relationship layout, for five reasons, each read in source: + +1. **Every entry is a 2 KiB value.** `CsrStorage { row_ptrs: Vec, col_indices: Vec, values: Vec }`, and `BitVec` is `[u64; 256]` (16 384 bits). A structural relationship has no value, so 400 000 edges would cost about 800 MB of values. The store needs a pattern-only CSR/CSC (values absent, or a `u32` canonical fact row for provenance). +2. **Indices are `usize`.** Eight bytes per index, where a sealed domain needs `u32` slots (and a focus `u8`). +3. **`mxv` copies the matrix per call.** `let a = if desc.transpose_input0() { self.transpose() } else { self.clone() };`. Even the "zero-copy" `transpose` with a CSC present clones `col_ptrs`, `row_indices` and `values`. A stored layout must be borrowed by every evaluation. +4. **`mxv` is a pull with an indexed read.** For each row `i`, for each `(j, a_val)` in the row: `vec.get(j)`. That reads the input vector at positions taken from stored column indices, the gather shape, run as scalar host code with a virtual `Semiring` call and a `BitVec` clone per product. +5. **ndarray is not on this path.** BlasGraph routes Hamming and popcount through `ndarray_bridge::dispatch_hamming`, but CSR construction and `mxv` use plain loops. ndarray has dense mask algebra and dense BLAS, and no sparse primitive. + +**The BF16 16 × 16 tile GEMM is measured separately** (D-RMAP-2 probe, #1623). On a host without AMX or AVX512-BF16 only ndarray's F32x16 polyfill tier runs; there the bit arm wins every cell, and that result says nothing about tile-hardware performance. The GEMM stays a candidate execution path for tile hosts and weighted counts until one is measured; §5b sets the terms on which it can win. + +**What follows.** +- **Structural and valued relations are different things, and both stay.** BlasGraph's `HdrSemiring::Boolean` is bitwise AND/OR over 16 384-bit hyperdimensional values, not edge existence. A structural relation's value is the edge's existence (plus an optional fact identity), with no 2 KiB payload. A valued relation keeps BlasGraph's hyperdimensional vectors, or another typed edge value, read only when the computation needs it. The lean CSR is a pattern-only interpretation inside the same sparse-storage family; it does not replace BlasGraph's hyperdimensional algebra. +- **Reuse means a representation change, not a constructor.** `CsrStorage` owns its `Vec`s, so borrowing buffers and dropping values is a new representation. The first step is a borrowed, version-pinned pattern-CSR **view** defined alongside the existing storage types; whether to refactor `CsrStorage` itself is decided after it has been measured. The probe's CSR is retired in favour of that view; no third owned CSR type is added. +- **Dense rows go to ndarray, and the direction decides whether that is gather-free.** BlasGraph's reserved `StorageFormat::Bitmap` is the bridge: a bitmap row is an ndarray mask. But the representation does not settle the question; the operation does. With `A[s,t]` an edge from source `s` to target `t` and `F` the frontier: + + | Representation | Operation | Indexed access | + |---|---|---| + | lean CSR | iterate neighbours | row-offset lookup, then a contiguous scan | + | sparse bitmap blocks | intersect frontier words | block lookup (an indexed read per 64 columns), then a word-wise AND | + | dense **predecessor** bitmap (per target, a row over sources) | `F'[t] = any(pred(t) ∧ F)` | none: co-addressed AND plus a reduction | + | dense **successor** bitmap (per source, a row over targets) | push: OR the rows of the selected sources | source-row selection is indexed by the frontier's set bits, then OR accumulation | + + Only the predecessor form is the admitted mask-risc case, and it is the tile of D-RMAP-1A §8 generalized from 256 slots to any row block. Expanding a whole frontier with it still needs one AND-and-reduce per target, so it costs `|targets| × |sources| / 64` words whatever the frontier's size; it is chosen per row block by density (BUNDLE), never globally. The compiler therefore picks the relation's direction to match the algebra (CSC for pull, CSR for push and neighbour iteration), which is a reason to keep both CSR and CSC rather than one universal layout. +- **Sparse rows still need an indexed read.** A sparse row lists column indices; any hop over it reads the frontier at those indices. Hypersparse bitmap blocks (a list of `(block, 64-bit word)`) make that read word-granular instead of bit-granular: one indexed read per 64 columns instead of one per column, and every read counted. That is the concrete content of admission A2 option (a)/(b); this plan does not admit it. +- **Duplication:** `holograph/src/graphblas` carries a second GraphBLAS. The store binds to one (the `lance-graph` BlasGraph types) and records the other as debt, not a second consumer. + +## 5b. Tile GEMM consumes prepared geometry (operator correction, 2026-10-20) + +```text +global relationship facts sparse · irregular identities · CSR / CSC / trie + │ + gather / resolve / pack select coordinates, build dense tiles, align shared domains + │ ← possibly expensive preparation + tile GEMM / mask-risc repeated, regular arithmetic on prepared operands + │ + relational results counts · existence · frontiers · derived relations +``` + +A GEMM or AMX kernel loads its operands from memory, so they need not be in registers already, but they must already be laid out as tiles. Every gather, pack or coordinate normalization happens before the kernel, and it is paid there. If a traversal gathers and packs each time it runs, even a very fast tile kernel can lose to an ordinary CSR iterator. That is the case for retaining prepared tiles under a versioned seal in the store (§4 step 2). + +**Three costs, measured separately for every tile accelerator:** +1. **Preparation**: identity resolution, coordinate alignment, index traversal, gathering, packing, tile construction, sealing. +2. **Execution**: mask-risc bit operations, SIMD, or BF16/AMX GEMM on prepared operands. +3. **Maintenance**: version checks, invalidation, incremental updates, repacking, rebuilds. + +For `N` uses and `M` invalidations of one prepared layout: + +```text +T_tile(N, M) = T_prepare + N · T_exec + M · T_maintain +T_sparse(N) = N · T_traverse (CSR/CSC or a native iterator, no tile) +``` + +The tile wins only where `T_exec < T_traverse`, and then only beyond the reuse count where the two lines cross. Break-even counts are reported from measured `T_prepare`, `T_exec`, `T_maintain` and `T_traverse`, never from an assumed kernel speedup. A warm tile kernel compared with a cold sparse traversal is not an end-to-end result. A faster isolated GEMM kernel does not establish a faster graph operation. + +**Representations keep their roles:** + +| Technique | Best role | +|---|---| +| borrowed pattern CSR/CSC, sorted permutations, native cursors | sparse, irregular navigation and range reads | +| sparse bitmap blocks | indexed word-level access, every indexed read counted | +| dense predecessor bitmaps (CSC direction) | gather-free, co-addressed frontier tests | +| mask-risc / ndarray bit tiles | dense Boolean relationship composition | +| BF16 / AMX tile GEMM | dense arithmetic composition, where density, reuse and precision justify it | +| RelationshipMapStore | retaining prepared layouts so their cost is paid across reuse | + +Sparse data is not forced into dense tiles in order to use GEMM. Structural Boolean adjacency stays distinct from BlasGraph's hyperdimensional `BitVec` values (§5a). + +**Boolean and arithmetic are different questions.** Reachability and existence need only `And`/`Or`; a BF16 product computes path multiplicities. Thresholding a count to existence is an explicit derivation step, never an implicit reading. BF16 operands carry an 8-bit significand, so they represent integers exactly only up to 256. The F32 accumulator is exact up to 2^24. Any arithmetic path through BF16 checks its counts against those bounds, and any count it cannot represent exactly is refused. Polyfill (F32x16) measurements and native AMX/AVX512-BF16 measurements are reported apart; this host has neither instruction set (#1623). + +Evidence so far: on the polyfill tier the bit arm wins every D-RMAP-2 cell (#1623). D-RST-3 measured a borrowed CSR walk at about 2 ns per edge against a preparation of 300–400 ns per edge (§7), which is exactly the cost the store exists to amortize. + +## 6. Falsifiers (exact; each guard gets a disable run) + +| # | Test | Pass condition | Disable that must turn it red | +|---|---|---|---| +| F1 | native borrowed views (§3.1), no petgraph: range reads over sorted permutations of an n-ary relation, CSR and CSC spans of its binary projection | equal a nested-loop oracle built from the facts; zero copies (each returned slice lies inside the layout buffer); building a view allocates nothing; fact ids survive compaction while rows move; a stale version and an out-of-domain id are refused | build the view from a copied `Vec`; drop a row; use the row offset as the id; skip the version check | +| F2 | one stored relation read by a Quack query, Cypher, the Gremlin adapter and a RelAlg/Datalog rule | identical ids and exact counts in set and bag semantics, against a nested-loop oracle; same `RelationVersion` cited by all four | change one layout so it answers set where bag was asked | +| F2b | hypothesis: a Quack query, a Gremlin `out()` expansion and a Datalog recursive step over the same relation compile to compatible demands and reuse one prepared layout, without sharing an execution model | the second and third front ends find the layout prepared by the first (prepare count stays 1, asserted), answers equal their oracles | give each front end its own layout key (prepare count goes to 3) | +| F3 | `out(r).in(r)` and `repeat_bounded(3)` within a focus, over a stored intermediate | equals the oracle; no `Gather`, no `ScatterOrU32` in any `Program` (asserted on the op list); `NonFunctionalChain` lifted only for the tiled case; over a large domain it still refuses | pass the hop-1 result as a raw mask instead of a stored intermediate with its seal (version check must refuse) | +| F4 | reseal; domain ABA (same ids, new domain version); thought slot reuse; compaction without stable row ids; authorization revoked; dirty speculative state; deletion | each is a typed refusal naming the stale component, never a plausible answer (D-RMAP-1A X1–X9, §6a) | key any check on a bare integer or on content | +| F5 | distinct neighbours vs parallel edges; repeated traversers (bag); path multiplicity; recursive reachability | set, bag and path answers stay distinct where the oracle says they differ; a bag→mask step refuses | collapse bag to set inside a step | +| F6 | triangle `edge(X,Y), edge(Y,Z), edge(Z,X)`: existence, count, enumeration; sparse, dense, skewed (hub) graphs | equals an independent oracle; MORK/PathMap where supported (set only); within focus via tiles; larger domains report the refusal instead of an answer | enumerate from the existence mask alone (must miss multiplicity) | +| F7 | cold and warm; 1/8/64 thoughts; domain churn; incremental reseal; shipped BlasGraph `mxv` vs pattern-only borrowed CSR vs bitmap rows through mask-risc, on one relation | measured crossover points, each reported with its command and repetition count; nothing derived by arithmetic is labelled measured | n/a (measurement), but every timed path asserts its answer first | +| F8 | **amortization**, one fixed relation, four paths: (A) native iterator over borrowed CSR; (B) compiled sparse traversal over CSR/CSC; (C) prepared Boolean bit tiles through mask-risc; (D) prepared arithmetic tiles through GEMM. Each cold and warm, then over repeated queries and 1/8/64 compatible thought versions; sparse, medium, dense and skewed (hub) distributions | reported per path: preparation time and bytes allocated, execution latency, index reads and copies, layout reuse count, invalidation/rebuild cost, the end-to-end crossover from §5b's two lines; exact semantic parity across A–D where the semantics coincide (existence for A–C, counts for D checked against an exact integer oracle and the BF16 bounds); BF16 polyfill and hardware numbers in separate columns | n/a (measurement); every timed path asserts its answer first, and a path that skips its preparation in the cold run is a defect | +| F-PG | **optional interoperability, only if petgraph is adopted (§3.3)**: unmodified `Bfs`, `Dfs` over a §3.1 view | visited order and set equal petgraph over an independently built `Graph`; zero copies; fact ids as edge ids; the view is not `Copy` | as F1 | + +**F6 is not a gate for F1–F5, F7 or F8. F-PG gates nothing.** The store and traversal capabilities stand on their own. + +## 7. Sequence: independently mergeable PRs + +| D-id | PR | Content | Depends on | +|---|---|---|---| +| D-RST-0 | this plan | documentation; board rows | — | +| D-RST-1 | identity types | `DurableRef`, `ThoughtCommitId` as contract types (D-CPG CPG-00 owns them; coordinate, don't duplicate) with a never-reused rule test | D-CPG agreement (B1) | +| D-RST-2 | store skeleton | `RelationshipMapStore` crate: catalog, `RelationDecl`, facts through the cycle seal, sorted permutation layout, `LayoutSeal`, coordinate domain as a relation; `impl Registrar`; F4 version and domain refusals | D-RST-1 | +| D-RST-9 | native borrowed relation views (critical path, first) | §3.1 views without petgraph: n-ary sorted permutations read as ranges, pattern CSR/CSC as the binary case, stable fact ids, version pin, not `Copy`; starting from the planner's `AdjacencyStore`/`AdjacencyBatch` and BlasGraph's pattern reading, not a new owned CSR; F1 | nothing (an isolated probe can run before D-RST-1/2) | +| D-RST-3 | **OPTIONAL / DEFERRED / MEASUREMENT-GATED**: petgraph over a borrowed view | built as an experiment before this correction (AdaWorldAPI/petgraph#4); kept as evidence, not extended; adoption only after D-RST-7 measures direct relational methods against petgraph-style iteration, and only for the workloads where it wins (F-PG) | D-RST-9, D-RST-7 | +| D-RST-4 | layouts | pattern-only `u32` CSR/CSC from D-RST-9 (§§3.1, 5a), bitmap row blocks through mask-risc (`StorageFormat::Bitmap`), and the join table from the probe, behind the store's capability table; base indexes at seal time, specialized layouts prepared on demand and retained under their seal; F2, F2b | D-RST-2, D-RST-9 | +| D-RST-5 | derived relations | `materialize` with multiplicity, provenance, dependencies; invalidation; DRed retraction; F4 deletion cases, F5 | D-RST-4 | +| D-RST-6 | compiled relational cursors | semijoins, joins and projections over retained layouts (§3.2); the typed step vocabulary; tile multi-hop through stored intermediates; Gremlin adapter `out().in()` within focus; F3 | D-RST-5, A1 | +| D-RST-7 | triangle and amortization | F6, F7 and F8, cold and warm; F8's path D only through the existing D-RMAP-2 probe kernels, with no new production kernel | D-RST-6 | +| D-RST-8 | large-domain multi-hop | only after admission A2 | A2 | + +**Lab sequence (not a production path):** D-RST-9 → D-RST-1 → D-RST-2 → D-RST-4 → D-RST-5 → D-RST-6 → D-RST-7. **Off it:** D-RST-3 (optional petgraph), D-RST-8 (waits on A2), F-PG, and F8's path D (a measurement through existing probe kernels, never a reason to add one). + +**D-RST-3 result (2026-10-20, AdaWorldAPI/petgraph#4, `experiments/relstore-visit`), recorded as evidence for a deferred optional item.** The view met what was then F1 (now F-PG): petgraph's unmodified `Bfs`/`Dfs` match a `DiGraph` oracle built from the facts on fixtures covering parallel edges, self-loops, isolated nodes and empty relations. The edge id is the fact id, and it survives compaction while rows move. The view borrows the prepared buffers, and building it or walking every neighbour list allocates nothing. The F2b-shaped falsifier passes: a Quack one-hop reads the same `dst` buffer at the same version, and the store prepares once. + +Measured at 1M edges: the neighbour walk and BFS cost the same as petgraph `Csr` (2.0 vs 1.8–1.9 ns per edge). Prepare takes 308–396 ns per edge and is not optimized. The Quack one-hop is O(E) per query (14–18 ms for 64 hops vs 0.3 µs through the view), because Quack cannot know that the `src` lane is sorted; that question belongs to the critical path and is §8 A6. The experiment's view is `Copy`, which the 2026-07-29 ruling forbids; §3.3 names the compatible shape. It is not extended further. + +Each step stays a probe or isolated experiment (`mork-pathmap-probe`, the petgraph fork's `experiments/`). None of it is production code, none is wired into an existing crate, and none may be promoted without a separate review after further testing. + +## 8. Open admission decisions and blockers + +- **A1, the global→local binding.** Translating global ids to slots during preparation, by dictionary search, is the "candidate membrane" of D-RMAP-1 §6. Both probes use it. It needs an explicit admission: either accepted as the counted translation H14c rule 6 permits outside the fold, or refused. Until then D-RST-4's layouts are buildable but not admitted for production reads. +- **A2, large-domain multi-hop.** Push is scatter and pull is gather (finding 1). Options: + - (a) blocked tiles: `|domain|²` bits in 256×256 blocks, with only non-empty blocks stored, i.e. a sparse block matrix; + - (b) admit a counted translation step between hops (a phase handoff through a stored intermediate); + - (c) stay refused. + + This decides whether `NonFunctionalChain` can be lifted beyond a focus. §5a narrows the choice: dense row blocks in the predecessor (pull) direction need no indexed read (`any(pred(t) ∧ F)` through mask-risc); the push direction and sparse rows do, and hypersparse bitmap blocks bring that read down to one per 64 columns. +- **A3, scalar host iteration (only if petgraph or another scalar iterator is adopted).** Whether host iteration over a borrowed layout sits outside the no-gather ruling, which governs mask-risc folds. This plan reads it that way and asks for confirmation when D-RST-3 is taken up. +- **A6, attested order.** A sorted permutation turns a one-key read into a range only if Quack may rely on the order. D-RST-3 measured the cost of not relying on it (a full-lane scan per query). Whether the `LayoutSeal` carries an ordering witness that Quack's existing `Cmp::Range` lowering (D-DIAMOND) can validate is open. +- **A4, store backend and home.** Which crate, and whether facts live in their own Lance datasets written through the cycle seal. This is consistent with "the seal owns durable writes"; the storage-portability ruling forbids a universal `Storage` trait, so the store is one owner, not an abstraction. +- **A5, Registrar shape.** `register(name, width)` cannot describe a relation's arity or sorts. Either the store keeps `RelationDecl` beside it (this plan) or Quack's seam grows. The latter changes a Quack trait and needs its own review. +- **B1, identity types.** `DurableRef` and `ThoughtCommitId` do not exist in code, and `ThoughtCommitId`'s never-reused rule is unspecified (D-RMAP-1A §7.6 gap 1). D-RST-2 cannot ship real seals before D-RST-1. +- **B2, petgraph fork pin (optional work only).** `lance-fold-hop` pins a pre-quarantine rev of lance-graph; `relstore-visit` pins a current rev and does not enable `quarantined-gather-hop`. +- **B3, cyclic joins at scale.** The triangle over a large domain has no admitted shape (F6 reports a refusal there). Not a blocker for D-RST-1..6. + +**Stop rule:** any step that copies canonical facts per thought, adds a second evaluator or traverser VM, persists a version counter beside Lance and Thought Git, treats a pre-fold translation as already approved, adds a production interface or dependency only to satisfy petgraph compatibility, or forces a representation in order to use GEMM has chosen the wrong boundary.