From 9172a80f6b518d3b81f43a068a358fb186dbe0ff Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 20 Oct 2026 00:46:54 +0000 Subject: [PATCH 01/10] plan: stored relations, compiled traversal, petgraph over borrowed layouts MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Documentation only (D-RST-0..8). Records operator decision B: RelationshipMapStore owns relationship persistence, identities, layouts, coordinate-domain seals and invalidation; Quack compiles; mask- risc on ndarray::simd stays the only population executor. D-RMAP-1A §1 and the 2026-10-05 storage-portability ruling are unchanged. The plan inventories what exists (Registrar has only a test implementor; Gremlin multi-hop refuses as NonFunctionalChain; RelAlg has no recursion or hop; mask-risc has no frontier continuation), defines the store's seams over Quack's existing Registrar, a borrowed petgraph RelView that runs unmodified Bfs/Dfs, a typed traversal vocabulary that compiles to Quack, layout reuse and invalidation, falsifiers F1-F7, and nine mergeable PRs. Two findings set its limits. Multi-hop over a large domain is blocked by scatter as well as gather, so the tile within a focus is the only admitted multi-hop shape until admission A2. The petgraph fork's existing lance-fold-hop experiment consumes the quarantined OutgoingHop and is not a base for this work. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VyWZYKycy6MoYDxHRVnV5K --- .claude/board/INTEGRATION_PLANS.md | 9 + .claude/board/STATUS_BOARD.md | 9 + ...2026-10-20-relstore-quack-petgraph-plan.md | 19 ++ .claude/board/entries/README.md | 3 +- .../relstore-quack-petgraph-traversal-v1.md | 241 ++++++++++++++++++ 5 files changed, 280 insertions(+), 1 deletion(-) create mode 100644 .claude/board/entries/2026-10-20-relstore-quack-petgraph-plan.md create mode 100644 .claude/plans/relstore-quack-petgraph-traversal-v1.md diff --git a/.claude/board/INTEGRATION_PLANS.md b/.claude/board/INTEGRATION_PLANS.md index 74d4b474c..14cd55958 100644 --- a/.claude/board/INTEGRATION_PLANS.md +++ b/.claude/board/INTEGRATION_PLANS.md @@ -1,3 +1,12 @@ +## 2026-10-20 — relstore-quack-petgraph-traversal-v1 — stored relations, compiled traversal, petgraph over borrowed layouts → `.claude/plans/relstore-quack-petgraph-traversal-v1.md` + +**Status:** D-RST-0..8 PROPOSAL; documentation only. Operator decision B (2026-10-20): RelationshipMapStore owns persistence, layouts, domains, seals and invalidation; Quack compiles; mask-risc executes. D-RMAP-1A §1 and the 2026-10-05 storage-portability ruling are unchanged. + +- The store satisfies Quack's existing `Registrar` seam; derived relations stay queryable relations with multiplicity, provenance and dependencies; versions are `DurableRef` / `ThoughtCommitId`, and the coordinate domain is itself a relation, so its seal needs no counter. +- petgraph: a borrowed `RelView` implementing `GraphBase`/`IntoNeighbors`/`Visitable`/`NodeIndexable` runs unmodified `Bfs`/`Dfs`; batched cursors and a typed traversal vocabulary compile to Quack. +- Finding: multi-hop over a large domain is blocked by scatter (push) as well as gather (pull); the tile is the only admitted shape (within a focus). Admissions A1 (global→local binding), A2 (large-domain multi-hop), A3 (scalar petgraph iteration) are open. +- The existing petgraph `lance-fold-hop` experiment consumes the quarantined `OutgoingHop`; the plan does not build on it. + ## 2026-10-19 — mork-pathmap-schnellverschluss-v1 — MORK / PathMap behind one replaceable boundary → `.claude/plans/mork-pathmap-schnellverschluss-v1.md` **Status:** D-MPS-0/1 MEASURED (probe, results only); D-MPS-2..5 upstream PROPOSALS; D-MPS-6 BLOCKED; D-MPS-7 OPEN. diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index bef9f6044..5ab0426e9 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -25,6 +25,15 @@ These PRs merged with no board rows. Every row below is a PROPOSAL (documents an | D-RMAP-1A | separate map tables, borrowed projections, thought-selected palettes (`d-rmap-separate-tables-thought-palettes-v1.md`) | PROPOSAL; generation witnesses corrected 2026-10-19 (§6a) | refusal on map regeneration, palette-slot reassignment, focus-slot reuse, incarnation reuse, stale tile, cross-tenant change | | D-RMAP-1A-V | versioning crosswalk: map/palette/binding generations derived from D-TGA `ThoughtCommitId` + #1605 `DurableRef`; focus generation and index seal retained (`d-rmap-separate-tables-thought-palettes-v1.md` §7) | PROPOSED; no code, no type; falsifiers specified, none run | X1–X9 in §7.4 (cross-dataset, compaction, palette ABA, slot reuse, uncommitted change, rebase, authorization, squash+replay, deletion/DRed) | | D-RMAP-1A-P | focus-bound RelationshipMap probe over existing sorted/trie indexes: CSR, 256×256 tile via mask-risc, incremental rebind, thought deltas, place masks, postings (`d-rmap-separate-tables-thought-palettes-v1.md` §8; `crates/mork-pathmap-probe/src/rmap.rs`) | MEASURED, probe only; nothing adopted | 12 tests vs scan oracle; 11 guards disable-verified; break-even 76/11/1.8 reuses (sparse d2 / d8 / dense) | +| D-RST-0 | stored relations × Quack × petgraph: architecture and phased plan (`relstore-quack-petgraph-traversal-v1.md`) | PROPOSAL; documentation only | operator decision B; admissions A1–A5, blockers B1–B3 listed in §8 | +| D-RST-1 | `DurableRef` / `ThoughtCommitId` contract types, never-reused rule (coordinated with D-CPG CPG-00) | Queued | reuse test; no duplicate with CPG-00 | +| D-RST-2 | `RelationshipMapStore` skeleton: catalog, facts through the cycle seal, sorted permutation, `LayoutSeal`, domain as a relation, `impl Registrar` | Queued | F4 refusals (reseal, domain ABA, compaction, scope, dirty, deletion) | +| D-RST-3 | petgraph `RelView` over a sealed CSR in a new fork experiment | Queued | F1: unmodified `Bfs`/`Dfs` equal the oracle; no edge copy (pointer-in-buffer check) | +| D-RST-4 | CSR / tile / join-table layouts in the store, capability table | Queued (production reads wait on A1) | F2: five front ends, one `RelationVersion`, identical set and bag answers | +| D-RST-5 | derived relations: `materialize`, multiplicity, provenance, dependencies, invalidation, DRed | Queued | F4 deletion, F5 set/bag/path | +| D-RST-6 | typed traversal vocabulary; multi-hop within a focus through stored intermediates | Queued (A1) | F3: no `Gather`/`ScatterOrU32` in any program; `NonFunctionalChain` lifted only in focus | +| D-RST-7 | triangle and amortization | Queued | F6 (existence/count/enumeration, sparse/dense/skewed), F7 measured crossovers | +| D-RST-8 | multi-hop over a large domain | BLOCKED on admission A2 | — | | D-MPS-8 | native join over a versioned relationship table: R and S laid out by `y` slot, leapfrog intersection as one mask-risc `And`, per-side generations and reseal, per-thought filter planes (`mork-pathmap-schnellverschluss-v1.md` §7b; `crates/mork-pathmap-probe/src/join.rs`) | MEASURED, probe only; not admitted (D-RMAP-1 BIND membrane review) | 8 join tests + native arms in the differential suite vs the oracle; 7 guards disable-verified; cyclic bodies open | | D-RMAP-2 | context-declaration reuse and GraphBLAS/BF16 tile probe (`d-rmap-2-context-tile-algebra-probe-v1.md`) | PROPOSAL; not run | arms 1–5 in the plan; 297× (D-GTM-0j) is not evidence | | D-SRAM-0 | inventory Soufflé AST/ast2ram/RAM passes against D-ASM/D-RA/D-TGA owners | Queued | evidence per mapped pass; no code imported | diff --git a/.claude/board/entries/2026-10-20-relstore-quack-petgraph-plan.md b/.claude/board/entries/2026-10-20-relstore-quack-petgraph-plan.md new file mode 100644 index 000000000..6f27587a3 --- /dev/null +++ b/.claude/board/entries/2026-10-20-relstore-quack-petgraph-plan.md @@ -0,0 +1,19 @@ +# Stored relations, compiled traversal, petgraph over borrowed layouts (2026-10-20) + +**Status:** PROPOSAL (D-RST-0..8). Plan: `.claude/plans/relstore-quack-petgraph-traversal-v1.md`. Documentation only. + +## Decision + +Operator decision B: `RelationshipMapStore` owns relationship persistence, identities, layouts, coordinate-domain seals and invalidation; Quack owns compilation and numeric query semantics; mask-risc on `ndarray::simd` stays the only population executor. D-RMAP-1A §1 and the 2026-10-05 storage-portability ruling stay in force. + +## Findings (read in source) + +- Multi-hop over a large domain is blocked by scatter as well as gather: pushing a frontier writes at positions taken from values, pulling reads at them. The tile (frontier broadcast ∧ a pre-transposed plane) is the only admitted shape, and it is bounded by the focus. Gremlin's `NonFunctionalChain` is therefore two problems: a phase handoff (the store's) and a hop (the ruling's). +- The petgraph fork's `experiments/lance-fold-hop` consumes the quarantined `OutgoingHop` (pinned pre-quarantine); it is not a base for this work. +- petgraph `Bfs`/`Dfs` need only `GraphRef + Visitable + IntoNeighbors`; the visit map is a `FixedBitSet` keyed by `IndexType`, so a dense-slot view reuses it unchanged. +- Quack's `Registrar` has only a test implementor; three library `Binder`s exist (sap, dir-sim, glove-parity). +- A coordinate domain stored as a relation gets its own `DurableRef`, which removes domain ABA without a counter. + +## Open + +- Admissions A1 (global→local binding), A2 (large-domain multi-hop), A3 (scalar petgraph iteration), A4 (store home/backend), A5 (Registrar shape); blockers B1 (`DurableRef`/`ThoughtCommitId` types), B2 (fork pin), B3 (cyclic joins at scale). diff --git a/.claude/board/entries/README.md b/.claude/board/entries/README.md index c96592ef1..2686a7a2b 100644 --- a/.claude/board/entries/README.md +++ b/.claude/board/entries/README.md @@ -25,10 +25,11 @@ index row, (3) no duplicate entry id. Checks 1 and 2 are deliberately opposite directions; the stranding this convention prevents shows up in exactly one of them, never both. -374 entries, 2026-08-06 .. 2026-10-19. +375 entries, 2026-08-06 .. 2026-10-20. | date | entry id | finding | file | |---|---|---|---| +| 2026-10-20 | `relstore-quack-petgraph-plan` | | [2026-10-20-relstore-quack-petgraph-plan.md](2026-10-20-relstore-quack-petgraph-plan.md) | | 2026-10-19 | `post-merge-recovery-1601-1617` | | [2026-10-19-post-merge-recovery-1601-1617.md](2026-10-19-post-merge-recovery-1601-1617.md) | | 2026-10-19 | `native-join-relationship-table` | | [2026-10-19-native-join-relationship-table.md](2026-10-19-native-join-relationship-table.md) | | 2026-10-19 | `mork-pathmap-schnellverschluss` | | [2026-10-19-mork-pathmap-schnellverschluss.md](2026-10-19-mork-pathmap-schnellverschluss.md) | diff --git a/.claude/plans/relstore-quack-petgraph-traversal-v1.md b/.claude/plans/relstore-quack-petgraph-traversal-v1.md new file mode 100644 index 000000000..d309b8bc9 --- /dev/null +++ b/.claude/plans/relstore-quack-petgraph-traversal-v1.md @@ -0,0 +1,241 @@ +# D-RST: RelationshipMapStore × Quack × petgraph — stored relations, compiled traversal + +**Status: PROPOSAL, 2026-10-20. Documentation only.** No crate, type, opcode, ABI, dependency or ruling change lands with this plan. Every capability below is labelled IMPLEMENTED, PROBE, PROPOSAL or RULING from its own source. + +**Decision taken (operator, 2026-10-20), recorded as provenance, not as validation:** option B. `RelationshipMapStore` owns relationship persistence, identities, index layouts, coordinate-domain seals and invalidation. Quack owns compilation and numeric query semantics. mask-risc on `ndarray::simd` stays the only population executor. This keeps the D-RMAP-1A §1 ownership table and the 2026-10-05 Quack storage-portability ruling as they are. Neither is superseded here. + +Read with: +- [D-RMAP-1A](d-rmap-separate-tables-thought-palettes-v1.md) §§1, 3, 6–8: ownership, borrowed views, falsifiers, versioning, focus probe. +- [Schnellverschluss](mork-pathmap-schnellverschluss-v1.md) §7b: the native join over a versioned relationship table. +- [D-CPG](causal-pickaxe-resident-git-lance13-temporal-v1.md) §1: `DurableRef`, `ThoughtCommitId`. +- [D-TGA](thought-git-api-v1.md). +- [H14c](d-holo-h14c-thought-first-v1.md) §1: the address is the register. +- Board entry `2026-10-05-quack-storage-portability.md`. +- `.claude/knowledge/fold-execution-laws.md`. + +**Short form:** the store remembers, BIND resolves once, Quack compiles, mask-risc executes, petgraph makes it usable. + +## 0. Findings that shape this plan + +1. **Multi-hop under the no-gather ruling is blocked by scatter as much as by gather.** One hop from a frontier mask has three spellings: + - **Push:** for each set source `s`, write `next[t]` for every `t` in `out(s)`. This writes at positions taken from values: scatter. + - **Pull:** `next[t] = OR over s in in(t) of frontier[s]`. This reads at positions taken from values: gather. + - **Tile:** frontier broadcast `∧` a pre-transposed `(target, source)` bit plane, then any per target. Every read and write is at its own coordinate. It costs `|focus|²` bits, so it is bounded by the focus. + + So the only admitted multi-hop shape today is the tile. Over a domain larger than a focus, multi-hop needs an admission decision (§8, A2). It is not an implementation gap a store can fill. Gremlin's `NonFunctionalChain` (`lance-graph-quack/tests/gremlin_parity.rs`, the `Refusal` enum: "needs a barrier and a workspace inside one computation. Gap: no typed phase handoff exists") is two problems. The handoff is the store's to give, as a stored intermediate. The hop is the ruling's to admit. +2. **The existing petgraph integration rests on the quarantined gather.** `AdaWorldAPI/petgraph` `experiments/lance-fold-hop` (fork PR #3) consumes `lance_graph_mask_risc::OutgoingHop`. That is two `Gather` ops plus `ScatterOrU32`, pinned to lance-graph `66636da`, from before #1616's quarantine. Its README already says it is one demanded hop, and that iterative BFS "needs a reviewed phase-handoff / continuation primitive". ISSUES.md records that bumping the pin must enable `quarantined-gather-hop` explicitly. This plan does not build on it. +3. **petgraph's traversal needs only two traits.** `Bfs::new` / `Dfs::new` need `GraphRef + Visitable`, and `next` needs `IntoNeighbors` (`crates/petgraph/src/visit/traversal.rs`). The visit map is a `FixedBitSet` keyed by `IndexType::index()` (`visit/mod.rs`, `impl VisitMap for FixedBitSet`), so a view whose `NodeId` is a dense `u32` slot gets petgraph's own visit map. `Csr` (`crates/petgraph/src/csr.rs`) shows the shape: `IntoNeighbors for &Csr` returns a slice iterator over `neighbors_slice`. It owns its `Vec`s, so the view must borrow instead. +4. **The Registrar seam is real, minimal and unowned.** `lance-graph-quack/src/bind.rs`: `trait Registrar { fn register(&mut self, name: &str, width: u32) -> Result; }`. The doc says "No crate owns that today; this trait names the seam, and only tests implement it". The only implementor is a `#[cfg(test)] Mem`. Three `Binder`s exist in library code: `lance-graph-sap` `CatsBinder`, `lance-graph-dir-sim` `UserBinder`, `lance-graph-glove-parity` `CanonicalBinder`. +5. **A coordinate domain is itself a relation.** The join probe needs a seal for its shared domain (Schnellverschluss §7b: a new `y` re-addresses every side). Stored as a relation (the dictionary of global ids, in slot order), the domain gets a `DurableRef` like any other table. No counter is added. A domain change is a new version, and versions are not reused, so domain ABA cannot happen. + +## 1. Current capability inventory + +| Capability | Where | Grade | +|---|---|---| +| Quack query surface: scan/filter/project/aggregate lowered to one `Program`; "may build a `Program` and must never evaluate one" | `lance-graph-quack/src/lib.rs` module doc | IMPLEMENTED | +| `Binder` (catalog + codebook, resolved once), `Registrar` (catalog + allocation seam), `create_table` → `ResolvedTable` | `lance-graph-quack/src/bind.rs` | IMPLEMENTED (trait); Registrar has no production implementor | +| Gremlin parity adapter: `V, Has, Out, In, Where, Dedup, Values, Count, Sum, GroupCountBy, Emit`; refusals incl. `NonFunctionalChain`, `ComposedFunctionalHop`, `Repeat`, `PathMultiplicity`, `BagOfElementsNotAMask` | `lance-graph-quack/tests/gremlin_parity.rs` | IMPLEMENTED as test code ("adds no operator to Quack and no op to mask-risc") | +| Cypher → Quack: single-label aggregates, one-hop walk count, `count(DISTINCT b)` under `dst` order; refuses a second hop, variable length ("a hop chain over a computed frontier"), WHERE with a hop | `lance-graph-cypher-quack/src/lib.rs` (`Refusal::Gap` doc) | IMPLEMENTED | +| RelAlg: set algebra over one row universe (`Rel`, `Measure`), lowered via R2IL `FoldDialect`; "recursion (`μ`) and the hop (preimage) are not built" | `lance-graph-relalg/src/{ir,lower}.rs`, `lib.rs` | IMPLEMENTED (prototype for D-CMR) | +| α-acyclicity test (GYO) for a join body; routes acyclic → semijoin chain, cyclic → intersection | `lance-graph-relalg/src/join_body.rs` | IMPLEMENTED (test only; no execution) | +| R2IL fold dialect: `VIA, RANGE, SUM, MIN, MAX, GROUP_SUM, KEEP` (+ refused `KEY_RUNS, ANY, ALL, SCATTER_OR, BLEND`) | `lance-graph-r2il`, opcodes from `ogar-r2il` | IMPLEMENTED; no JOIN or HOP op | +| mask-risc: `MaskOp::{Pred, And, Or, Xor, AndNot, Not, Ternlog, Gather}`; terminals incl. `Keep`, `Count`, `GroupReduce`, `ScatterOrU32` | `lance-graph-mask-risc/src/ir.rs` | IMPLEMENTED; no frontier-continuation primitive | +| `OutgoingHop` (two `Gather` + `ScatterOrU32`) | `lance-graph-mask-risc/src/graph_hop.rs`, feature `quarantined-gather-hop` (off) | QUARANTINED (`ISS-GATHER-HOP-1616`) | +| Focus-bound RelationshipMap: sorted run, PathMap trie, slot-space CSR, 256×256 tile via mask-risc, incremental rebind, thought deltas | `mork-pathmap-probe/src/rmap.rs` | PROBE (D-RMAP-1A §8) | +| Versioned relationship table: shared-variable domain, per-side generations, `And` intersection, bag answers, star joins, filters per thought | `mork-pathmap-probe/src/join.rs` | PROBE (Schnellverschluss §7b, D-MPS-8) | +| `DurableRef{dataset, branch, version}`, `ThoughtCommitId` | D-CPG §1 | PROPOSAL; no type in `crates/` | +| Durable write boundary: cycle seal → `LanceCycleWriter`, one `DatasetVersion` per cycle | `cycle_sink.rs`; storage-portability entry | IMPLEMENTED | +| Ownership: store owns facts/orders/maps/tiles; Quack "must not own semantic maps"; RelAlg never owns maps | D-RMAP-1A §1 | PROPOSAL (controlling contract) | +| Quack owns read semantics, the backend supplies capabilities, the seal owns durable writes; "no universal `Storage` trait" | storage-portability entry, 2026-10-05 | RULING | +| No gather; "any address translation is explicit, counted, and outside the direct-position path" | H14c §1 rule 6; 2026-10-16 ruling | RULING | +| A pre-fold global→local binding is a candidate membrane that needs its own review; until then, a typed refusal | D-RMAP-1 §6 | PROPOSAL (refusal stands) | +| petgraph visit traits, `Bfs`/`Dfs`/`Walker`, `Csr` | `AdaWorldAPI/petgraph` `crates/petgraph/src/{visit,csr.rs}` (0.8.3, MSRV 1.85, dev toolchain 1.99) | IMPLEMENTED upstream | +| `RelationshipMapStore`, `RelationCatalog` | — | not found under `crates/` (searched `*.rs`, `*.md`) | + +## 2. Ownership and seams + +```text + SQL-shaped quack::Query Cypher Gremlin Datalog/RelAlg petgraph algorithms + \ | | / | + \______________ PARSE ___/____________/ | + | | + Quack COMPILE (V4: what to compute) | + | | + BIND: identities, coordinate domain, authorization, version, | + layout capabilities — resolved once ─────────────────────┤ + | | + BUNDLE: choose layout (may not change the answer) | + | | + mask-risc / ndarray::simd EXECUTE over borrowed planes | + | | + RelationshipMapStore: catalog · facts · derived relations · | + orders · layouts · domains · seals · invalidation ◄── borrowed views ─┘ + | + cycle seal → Lance (DurableRef) ThoughtCommitId (speculative) + | + SoA: canonical identity-addressed state (separate admission) +``` + +| Owner | Owns | Must not own | +|---|---|---| +| RelationshipMapStore | relation catalog (arity, argument sorts, set/bag, functional constraints); facts; derived relations with multiplicity, provenance and dependencies; sorted permutations; physical layouts (CSR, tiles, join tables); coordinate domains; layout seals; invalidation | Datalog/Gremlin syntax, a query evaluator, a version counter of its own, CE64 fields | +| BIND | resolving names, identities, domains, authorization and versions into a witness; attesting which layouts are eligible | layouts, storage, execution | +| Quack | compiling relational and graph operations into `Program`s; numeric query semantics; the `Registrar`/`Binder` seams | storage, catalog state, evaluation | +| RelAlg / Datalog | variable bindings, join bodies, set vs bag, recursion and fixpoint meaning | map tables, index builds | +| mask-risc / `ndarray::simd` | population execution | anything else | +| petgraph | algorithm and iterator vocabulary | storage, a second graph copy | +| SoA | canonical identity-addressed state; admission of stable semantic relationships | derived relations (admission is optional and separate) | +| cycle seal / Lance | durable writes, `DurableRef` | relational semantics | + +### 2.1 Seams (proposed shapes, not code) + +```rust +// The store satisfies Quack's existing seam for its tables. No new catalog trait. +impl Registrar for RelationshipMapStore { /* register(name, width) -> TableId */ } + +// Relations carry more than a width. Declared beside, not inside, Registrar, +// so the Quack trait does not change in the first PRs. +pub struct RelationDecl { name, arity, sorts: [SortId; N], semantics: SetOrBag, functional: Option } +pub struct RelationId(u32); // resolved once; never a version +pub struct LayoutId(u32); // one physical layout of one relation version + +/// Every read is pinned to this. Nothing here is a counter the store mints. +pub struct RelationVersion { + relation: RelationId, + at: Revision, // Durable(DurableRef) | Speculative(ThoughtCommitId) +} +pub struct LayoutSeal { + layout: LayoutId, + built_from: RelationVersion, + domain: RelationVersion, // the coordinate domain is a relation too (finding 5) + addresses: Option, // row-address map, dropped with stable row ids (D-RMAP-1A §7.3) +} + +/// What BIND hands Quack and mask-risc. Borrowed, ephemeral, never stored. +pub struct BoundRelation<'s> { version: RelationVersion, layout: LayoutSeal, planes: Planes<'s>, scope: AuthorizationReceipt } + +// Derived relations are relations. +store.materialize(compiled_result, deps: &[RelationVersion]) -> RelationVersion +``` + +Durable versions come from the seal (`DurableRef`); speculative ones from a thought commit (`ThoughtCommitId`). The store records which versions a layout or derived relation was built from and refuses a read when any of them differs. It persists no counter of its own (D-RMAP-1A §7.7, option B). + +## 3. The petgraph contract: one identity, two consumption shapes + +**Shape 1, scalar, unmodified petgraph.** A borrowed view type over one sealed CSR layout: + +```rust +#[derive(Clone, Copy)] +pub struct RelView<'s> { seal: &'s LayoutSeal, off: &'s [u32], dst: &'s [u32], n: usize } + +impl GraphBase for RelView<'_> { type NodeId = Slot; type EdgeId = EdgeRow; } // Slot: IndexType over u32 +impl<'s> IntoNeighbors for RelView<'s> { type Neighbors = SlotIter<'s>; /* slice of dst[off[a]..off[a+1]] */ } +impl Visitable for RelView<'_> { type Map = FixedBitSet; /* capacity n */ } +impl NodeIndexable / NodeCount / NodeCompactIndexable // the slot domain is dense +impl IntoEdgeReferences / IntoEdges / Data // edge id = canonical row: provenance survives +impl IntoNeighborsDirected // only when the store holds the reverse permutation; + // otherwise unimplemented, never computed on demand +``` + +- Petgraph's `Bfs`, `Dfs`, `DfsPostOrder` and `Topo` then run unmodified. Nothing is copied: the iterator is a slice of the stored layout. +- `EdgeId` is the canonical fact row, so a path can be traced back to facts. +- Out-of-focus or out-of-domain neighbours are a property of the layout. A view over a full-domain CSR has none; a view over a focus layout must report escapes and refuse `global()` reads (D-RMAP-1A §8). + +**Classification (admission A3).** Shape 1 is ordinary host iteration inside petgraph's algorithm. It is not population execution and does not go through mask-risc, so the no-gather ruling, which governs fold execution, does not decide it. This plan records that reading as an open admission item rather than assuming it. + +**Shape 2, batched relational cursors.** The same `RelationVersion` and `LayoutSeal`, consumed as whole spans and planes: +- CSR spans per slot range; +- tile planes and frontier masks (the only admitted multi-hop shape, finding 1); +- join-table presence planes (intersection = `And`); +- index intersections over sorted permutations. + +These are Quack inputs, compiled to `Program`s. + +**Compiling iterator chains.** Arbitrary `Iterator::filter` / `flat_map` closures cannot be inspected and will not be compiled or fused. The plan instead proposes a small typed traversal vocabulary that is both an ordinary `Iterator` (terminal `.iter()`) and compilable (terminal `.compile(&quack)`): + +| Typed step | Meaning | Compiles to | +|---|---|---| +| `from(mask \| slots)` | start population | resident plane | +| `out(rel)` / `in_(rel)` | one hop | tile hop (in focus); refused over a large domain until A2 | +| `has(field, Cmp)` | filter | `Pred` under the population | +| `dedup()` | set semantics | mask | +| `count()` / `group_count(field)` | terminals | `Count` / `GroupReduce` | +| `repeat_bounded(k)` / `until_fixpoint()` | iteration | k tile hops / hop until the plane stops changing, within focus | +| `emit()` | materialize | explicit terminal; the only exit producing ids | + +Each step carries set/bag/path semantics in its type. Mixing them without an explicit `dedup` or `paths` step is a compile error, never a silent collapse (Gremlin's `BagOfElementsNotAMask` and `PathMultiplicity` stay real refusals). A chain that cannot compile refuses at `.compile`. It never falls back to `.iter()` on its own (no silent fallback). This keeps Gremlin's step vocabulary without a traverser VM: the traversal is a value that is compiled, not an object stepped per traverser. + +## 4. Execution flow + +1. **Declare / store.** `RelationDecl` → store catalog. Facts arrive through the cycle seal. The store records the resulting `DurableRef` against the relation. +2. **Prepare (normalize once).** The store builds layouts from a sealed version: a sorted permutation, CSR, tiles for a declared focus, join tables for a declared shared variable, and the coordinate domain as a relation. Each layout gets a `LayoutSeal`, and each preparation is timed and counted (normalization, index build, translation). +3. **Bind.** BIND resolves names → ids, picks the domain, checks authorization, pins versions, and lists the layouts whose seals match. The global→local translation runs here or in step 2, never inside a fold (H14c rule 6), and it is subject to admission A1. +4. **Compile.** Quack turns the query (Quack query, Cypher, Gremlin, RelAlg/Datalog, typed traversal) into a demand plus a `Program` over the eligible layouts' planes. +5. **Bundle.** Choose a layout using the capability table (§5). By fold law 3, the choice may not change the answer; F2 is the test. +6. **Execute.** mask-risc over borrowed planes. Results come back as masks, counts or group reductions; tuples only through an explicit `emit`/`materialize`. +7. **Remember.** A result that later steps reuse is materialized as a derived relation: its rows plus multiplicity, provenance (source fact rows or binding tuples) and dependencies. It can be queried, laid out and bound like any relation. A bare mask is never stored as a relation; it carries neither bindings nor multiplicity. + +## 5. Layout reuse and invalidation + +**Measured selection inputs (probes, warm, one machine; to be re-measured cold in F7):** + +| Layout | Fits | Measured | Source | +|---|---|---|---| +| sorted permutation | sparse, any domain; seeks | build 29–483 ms for 0.4–2.9 M facts | D-RMAP-1A §8.3 | +| CSR (slot space) | sparse in focus | wins at degree 2 (3.3 vs 5.1 µs) | §8.3 | +| 256×256 tile | dense in focus; the only no-gather multi-hop | wins dense (4.5 vs 12.4 µs) | §8.3 | +| join table on a shared variable | star / acyclic joins on one variable | prepare 2.7 ms vs PathMap 14.7 ms (n = 50 000); count 36 µs | Schnellverschluss §7b | +| PathMap trie | prefix seek on sparse | faster than the run on sparse, 4–6× slower to build | §8.6 | + +**Invalidation rule:** a layout or derived relation is valid only while every `RelationVersion` in its seal or dependency list equals the current one. Otherwise it is refused, never silently reused. +- **Maintenance:** incremental where measured to win (one side of a join table, or k slots of a focus at small k); a rebuild otherwise. The probe crossovers are the starting policy, to be re-measured. +- **Deletions:** a derived relation over a deleted fact is retracted before re-derivation (DRed, D-TGA Rule 2; D-RMAP-1A X9). +- **Thoughts:** a thought reads layouts at its pinned versions and adds sparse deltas. Layouts are not copied per thought (D-RMAP-1A §8.5 measured 32 B per thought as sparse edits). + +## 6. Falsifiers (exact; each guard gets a disable run) + +| # | Test | Pass condition | Disable that must turn it red | +|---|---|---|---| +| F1 | unmodified `Bfs`, `Dfs`, `DfsPostOrder` over `RelView` on one sealed CSR | visited order and set equal petgraph over an independently built `Graph` (test-only oracle); zero edge copies, checked by asserting the iterator's slice pointer lies inside the layout buffer | build the view from a copied `Vec` (the pointer check fails); drop a neighbour | +| F2 | one stored relation read by a Quack query, Cypher, the Gremlin adapter, a RelAlg/Datalog rule and petgraph | identical ids and exact counts in set and bag semantics, against a nested-loop oracle; same `RelationVersion` cited by all five | change one layout so it answers set where bag was asked | +| F3 | `out(r).in(r)` and `repeat_bounded(3)` within a focus, over a stored intermediate | equals the oracle; no `Gather`, no `ScatterOrU32` in any `Program` (asserted on the op list); `NonFunctionalChain` lifted only for the tiled case; over a large domain it still refuses | pass the hop-1 result as a raw mask instead of a stored intermediate with its seal (version check must refuse) | +| F4 | reseal; domain ABA (same ids, new domain version); thought slot reuse; compaction without stable row ids; authorization revoked; dirty speculative state; deletion | each is a typed refusal naming the stale component, never a plausible answer (D-RMAP-1A X1–X9, §6a) | key any check on a bare integer or on content | +| F5 | distinct neighbours vs parallel edges; repeated traversers (bag); path multiplicity; recursive reachability | set, bag and path answers stay distinct where the oracle says they differ; a bag→mask step refuses | collapse bag to set inside a step | +| F6 | triangle `edge(X,Y), edge(Y,Z), edge(Z,X)`: existence, count, enumeration; sparse, dense, skewed (hub) graphs | equals an independent oracle; MORK/PathMap where supported (set only); within focus via tiles; larger domains report the refusal instead of an answer | enumerate from the existence mask alone (must miss multiplicity) | +| F7 | cold and warm; 1/8/64 thoughts; domain churn; incremental reseal; scalar `RelView` BFS vs batched tile frontier | measured crossover points, each reported with its command and repetition count; nothing derived by arithmetic is labelled measured | n/a (measurement), but every timed path asserts its answer first | + +**F6 is not a gate for F1–F5 or F7.** The store and traversal capabilities stand on their own. + +## 7. Sequence: independently mergeable PRs + +| D-id | PR | Content | Depends on | +|---|---|---|---| +| D-RST-0 | this plan | documentation; board rows | — | +| D-RST-1 | identity types | `DurableRef`, `ThoughtCommitId` as contract types (D-CPG CPG-00 owns them; coordinate, don't duplicate) with a never-reused rule test | D-CPG agreement (B1) | +| D-RST-2 | store skeleton | `RelationshipMapStore` crate: catalog, `RelationDecl`, facts through the cycle seal, sorted permutation layout, `LayoutSeal`, coordinate domain as a relation; `impl Registrar`; F4 version and domain refusals | D-RST-1 | +| D-RST-3 | petgraph shape 1 | `RelView` in a new fork experiment (`experiments/relstore-visit`, isolated workspace, pinned rev), replacing nothing in `lance-fold-hop`; F1 | D-RST-2 | +| D-RST-4 | layouts | CSR, tile and join-table layouts ported from the probes into the store, behind its capability table; F2 (Quack query, Cypher, petgraph first) | D-RST-2 | +| D-RST-5 | derived relations | `materialize` with multiplicity, provenance, dependencies; invalidation; DRed retraction; F4 deletion cases, F5 | D-RST-4 | +| D-RST-6 | typed traversal + multi-hop in focus | the typed step vocabulary; tile multi-hop through stored intermediates; Gremlin adapter `out().in()` within focus; F3 | D-RST-5, A1 | +| D-RST-7 | triangle and amortization | F6 and F7, cold and warm | D-RST-6 | +| D-RST-8 | large-domain multi-hop | only after admission A2 | A2 | + +Each PR keeps the probes in `mork-pathmap-probe` as probes. Production code is new code in the store crate with its own tests; it does not import the probe crate. + +## 8. Open admission decisions and blockers + +- **A1, the global→local binding.** Translating global ids to slots during preparation, by dictionary search, is the "candidate membrane" of D-RMAP-1 §6. Both probes use it. It needs an explicit admission: either accepted as the counted translation H14c rule 6 permits outside the fold, or refused. Until then D-RST-4's layouts are buildable but not admitted for production reads. +- **A2, large-domain multi-hop.** Push is scatter and pull is gather (finding 1). Options: + - (a) blocked tiles: `|domain|²` bits in 256×256 blocks, with only non-empty blocks stored, i.e. a sparse block matrix; + - (b) admit a counted translation step between hops (a phase handoff through a stored intermediate); + - (c) stay refused. + + This decides whether `NonFunctionalChain` can be lifted beyond a focus. +- **A3, scalar petgraph iteration.** Whether shape-1 host iteration over a borrowed layout sits outside the no-gather ruling, which governs mask-risc folds. This plan reads it that way and asks for confirmation. +- **A4, store backend and home.** Which crate, and whether facts live in their own Lance datasets written through the cycle seal. This is consistent with "the seal owns durable writes"; the storage-portability ruling forbids a universal `Storage` trait, so the store is one owner, not an abstraction. +- **A5, Registrar shape.** `register(name, width)` cannot describe a relation's arity or sorts. Either the store keeps `RelationDecl` beside it (this plan) or Quack's seam grows. The latter changes a Quack trait and needs its own review. +- **B1, identity types.** `DurableRef` and `ThoughtCommitId` do not exist in code, and `ThoughtCommitId`'s never-reused rule is unspecified (D-RMAP-1A §7.6 gap 1). D-RST-2 cannot ship real seals before D-RST-1. +- **B2, petgraph fork pin.** The existing experiment pins a pre-quarantine rev of lance-graph. The new experiment pins a current rev and does not enable `quarantined-gather-hop`. +- **B3, cyclic joins at scale.** The triangle over a large domain has no admitted shape (F6 reports a refusal there). Not a blocker for D-RST-1..6. + +**Stop rule:** any step that copies canonical facts per thought, adds a second evaluator or traverser VM, persists a version counter beside Lance and Thought Git, or treats a pre-fold translation as already approved has chosen the wrong boundary. From aeacb14fb1f1b56858b19aa7042f04b7840552db Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 20 Oct 2026 00:50:45 +0000 Subject: [PATCH 02/10] plan D-RST: write relationships as relationships; BlasGraph CSR via ndarray MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds the amortization principle (a relation is written once as facts, and its relationship-shaped layouts are built at seal time, so RelAlg pays identity translation once per version) and §5a, a read-in-source check of the shipped BlasGraph CSR as the store's layout. BlasGraph's storage types (CSR, CSC, hypersparse CSR, COO->CSR, Boolean semiring) are the vocabulary to reuse, but as shipped they carry a 2 KiB BitVec value per entry, usize indices, an mxv that clones the matrix per call (the CSC transpose clones too), an indexed read of the input vector per entry, and no ndarray on the CSR/mxv path. The plan reuses the types with a pattern-only u32 borrowed variant, sends dense row blocks to ndarray mask ops through mask-risc via the reserved StorageFormat::Bitmap, and narrows admission A2 to sparse rows. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VyWZYKycy6MoYDxHRVnV5K --- .claude/board/STATUS_BOARD.md | 2 +- ...2026-10-20-relstore-quack-petgraph-plan.md | 3 +++ .../relstore-quack-petgraph-traversal-v1.md | 26 ++++++++++++++++--- 3 files changed, 27 insertions(+), 4 deletions(-) diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index 5ab0426e9..fbe94e3cf 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -29,7 +29,7 @@ These PRs merged with no board rows. Every row below is a PROPOSAL (documents an | D-RST-1 | `DurableRef` / `ThoughtCommitId` contract types, never-reused rule (coordinated with D-CPG CPG-00) | Queued | reuse test; no duplicate with CPG-00 | | D-RST-2 | `RelationshipMapStore` skeleton: catalog, facts through the cycle seal, sorted permutation, `LayoutSeal`, domain as a relation, `impl Registrar` | Queued | F4 refusals (reseal, domain ABA, compaction, scope, dirty, deletion) | | D-RST-3 | petgraph `RelView` over a sealed CSR in a new fork experiment | Queued | F1: unmodified `Bfs`/`Dfs` equal the oracle; no edge copy (pointer-in-buffer check) | -| D-RST-4 | CSR / tile / join-table layouts in the store, capability table | Queued (production reads wait on A1) | F2: five front ends, one `RelationVersion`, identical set and bag answers | +| D-RST-4 | layouts in the store, built at seal time: pattern-only `u32` CSR/CSC on BlasGraph storage types, bitmap row blocks through mask-risc, join table; capability table | Queued (production reads wait on A1) | F2: five front ends, one `RelationVersion`, identical set and bag answers | | D-RST-5 | derived relations: `materialize`, multiplicity, provenance, dependencies, invalidation, DRed | Queued | F4 deletion, F5 set/bag/path | | D-RST-6 | typed traversal vocabulary; multi-hop within a focus through stored intermediates | Queued (A1) | F3: no `Gather`/`ScatterOrU32` in any program; `NonFunctionalChain` lifted only in focus | | D-RST-7 | triangle and amortization | Queued | F6 (existence/count/enumeration, sparse/dense/skewed), F7 measured crossovers | diff --git a/.claude/board/entries/2026-10-20-relstore-quack-petgraph-plan.md b/.claude/board/entries/2026-10-20-relstore-quack-petgraph-plan.md index 6f27587a3..4b4237936 100644 --- a/.claude/board/entries/2026-10-20-relstore-quack-petgraph-plan.md +++ b/.claude/board/entries/2026-10-20-relstore-quack-petgraph-plan.md @@ -14,6 +14,9 @@ Operator decision B: `RelationshipMapStore` owns relationship persistence, ident - Quack's `Registrar` has only a test implementor; three library `Binder`s exist (sap, dir-sim, glove-parity). - A coordinate domain stored as a relation gets its own `DurableRef`, which removes domain ABA without a counter. +- BlasGraph's shipped CSR (read in source): 2 KiB `BitVec` value per entry, `usize` indices, `mxv` clones the matrix per call (even the CSC transpose clones), `mxv` reads the vector at stored column indices (a scalar gather), and ndarray is not on the CSR/`mxv` path. The store reuses BlasGraph's storage types with a pattern-only `u32` borrowed variant; dense row blocks become bitmap rows executed by ndarray mask ops through mask-risc (`StorageFormat::Bitmap` is already reserved); only sparse rows need an indexed read. +- Principle (operator): write a relationship as a relationship; layouts are built at seal time from relation facts, so RelAlg pays translation once per version. + ## Open - Admissions A1 (global→local binding), A2 (large-domain multi-hop), A3 (scalar petgraph iteration), A4 (store home/backend), A5 (Registrar shape); blockers B1 (`DurableRef`/`ThoughtCommitId` types), B2 (fork pin), B3 (cyclic joins at scale). diff --git a/.claude/plans/relstore-quack-petgraph-traversal-v1.md b/.claude/plans/relstore-quack-petgraph-traversal-v1.md index d309b8bc9..d7ffeaa9f 100644 --- a/.claude/plans/relstore-quack-petgraph-traversal-v1.md +++ b/.claude/plans/relstore-quack-petgraph-traversal-v1.md @@ -15,6 +15,8 @@ Read with: **Short form:** the store remembers, BIND resolves once, Quack compiles, mask-risc executes, petgraph makes it usable. +**The amortization principle (operator, 2026-10-20): write a relationship as a relationship.** A relation is written once, as relation facts, and its relationship-shaped layouts (adjacency by subject and by object, join tables on a declared shared variable) are built at seal time from those facts. They are not rebuilt per query or per front end. RelAlg's semijoins, joins and projections then read prepared relationship layouts instead of translating identities on every evaluation. That is where RelAlg gets faster: the translation is paid once per sealed version, the evaluation is a fold. §5a checks what the shipped BlasGraph CSR contributes to this. + ## 0. Findings that shape this plan 1. **Multi-hop under the no-gather ruling is blocked by scatter as much as by gather.** One hop from a frontier mask has three spellings: @@ -50,6 +52,8 @@ Read with: | No gather; "any address translation is explicit, counted, and outside the direct-position path" | H14c §1 rule 6; 2026-10-16 ruling | RULING | | A pre-fold global→local binding is a candidate membrane that needs its own review; until then, a typed refusal | D-RMAP-1 §6 | PROPOSAL (refusal stands) | | petgraph visit traits, `Bfs`/`Dfs`/`Walker`, `Csr` | `AdaWorldAPI/petgraph` `crates/petgraph/src/{visit,csr.rs}` (0.8.3, MSRV 1.85, dev toolchain 1.99) | IMPLEMENTED upstream | +| BlasGraph sparse storage: `CsrStorage`, `CscStorage`, `HyperCsrStorage`, COO→CSR, `GrBMatrix` (CSR + optional CSC), `mxv`/`vxm`/`mxm` over a `dyn Semiring` incl. `HdrSemiring::Boolean`; `StorageFormat::Bitmap` reserved | `lance-graph/src/graph/blasgraph/{sparse,matrix,semiring}.rs`; planner `blasgraph_planner.rs` (Cypher `Expand` → `mxm`) | IMPLEMENTED; a second copy in `holograph/src/graphblas` | +| ndarray dense mask algebra: `mask_and`, `mask_or`, `mask_andnot`, `mask_ternlog`, popcount/any; dense BLAS levels 1–3 | `ndarray/src/simd_masking_ops.rs`, `src/hpc/blas_level{1,2,3}.rs` | IMPLEMENTED; no sparse/CSR primitive under `src/hpc` or `src/simd*` (searched `csr`, `spmv`, `sparse`) | | `RelationshipMapStore`, `RelationCatalog` | — | not found under `crates/` (searched `*.rs`, `*.md`) | ## 2. Ownership and seams @@ -192,6 +196,22 @@ Each step carries set/bag/path semantics in its type. Mixing them without an exp - **Deletions:** a derived relation over a deleted fact is retracted before re-derivation (DRed, D-TGA Rule 2; D-RMAP-1A X9). - **Thoughts:** a thought reads layouts at its pinned versions and adds sparse deltas. Layouts are not copied per thought (D-RMAP-1A §8.5 measured 32 B per thought as sparse edits). +## 5a. BlasGraph CSR and ndarray: what to reuse (read in source, 2026-10-20) + +BlasGraph already ships the relationship-matrix vocabulary this plan needs: CSR, CSC, hypersparse CSR, COO→CSR, a Boolean semiring, and a planner that lowers Cypher `Expand` to `mxm`. The store should reuse that vocabulary for its CSR/CSC layouts instead of carrying the probe's CSR into production. As shipped it cannot serve as a relationship layout, for five reasons, each read in source: + +1. **Every entry is a 2 KiB value.** `CsrStorage { row_ptrs: Vec, col_indices: Vec, values: Vec }`, and `BitVec` is `[u64; 256]` (16 384 bits). A structural relationship has no value, so 400 000 edges would cost about 800 MB of values. The store needs a pattern-only CSR/CSC (values absent, or a `u32` canonical fact row for provenance). +2. **Indices are `usize`.** Eight bytes per index, where a sealed domain needs `u32` slots (and a focus `u8`). +3. **`mxv` copies the matrix per call.** `let a = if desc.transpose_input0() { self.transpose() } else { self.clone() };`. Even the "zero-copy" `transpose` with a CSC present clones `col_ptrs`, `row_indices` and `values`. A stored layout must be borrowed by every evaluation. +4. **`mxv` is a pull with an indexed read.** For each row `i`, for each `(j, a_val)` in the row: `vec.get(j)`. That reads the input vector at positions taken from stored column indices, the gather shape, run as scalar host code with a virtual `Semiring` call and a `BitVec` clone per product. +5. **ndarray is not on this path.** BlasGraph routes Hamming and popcount through `ndarray_bridge::dispatch_hamming`, but CSR construction and `mxv` use plain loops. ndarray has dense mask algebra and dense BLAS, and no sparse primitive. + +**What follows.** +- **Reuse:** the store's CSR/CSC layouts are BlasGraph's storage types, extended with a pattern-only, `u32`-indexed variant that borrows its buffers. No third CSR type is added beside BlasGraph's and the probe's; the probe's CSR is retired in favour of it. +- **Dense rows go to ndarray.** BlasGraph's reserved `StorageFormat::Bitmap` is the bridge: a row stored as a bitmap over its column domain is exactly an ndarray mask, and a Boolean hop over bitmap rows is `mask_and` plus any, through mask-risc, with no indexed read. That is the tile of D-RMAP-1A §8, generalized from 256 slots to any bitmap row. It costs `|rows| × |cols|` bits, so it is chosen per row block by density (BUNDLE), never globally. +- **Sparse rows still need an indexed read.** A sparse row lists column indices; any hop over it reads the frontier at those indices. Hypersparse bitmap blocks (a list of `(block, 64-bit word)`) make that read word-granular instead of bit-granular: one indexed read per 64 columns instead of one per column, and every read counted. That is the concrete content of admission A2 option (a)/(b); this plan does not admit it. +- **Duplication:** `holograph/src/graphblas` carries a second GraphBLAS. The store binds to one (the `lance-graph` BlasGraph types) and records the other as debt, not a second consumer. + ## 6. Falsifiers (exact; each guard gets a disable run) | # | Test | Pass condition | Disable that must turn it red | @@ -202,7 +222,7 @@ Each step carries set/bag/path semantics in its type. Mixing them without an exp | F4 | reseal; domain ABA (same ids, new domain version); thought slot reuse; compaction without stable row ids; authorization revoked; dirty speculative state; deletion | each is a typed refusal naming the stale component, never a plausible answer (D-RMAP-1A X1–X9, §6a) | key any check on a bare integer or on content | | F5 | distinct neighbours vs parallel edges; repeated traversers (bag); path multiplicity; recursive reachability | set, bag and path answers stay distinct where the oracle says they differ; a bag→mask step refuses | collapse bag to set inside a step | | F6 | triangle `edge(X,Y), edge(Y,Z), edge(Z,X)`: existence, count, enumeration; sparse, dense, skewed (hub) graphs | equals an independent oracle; MORK/PathMap where supported (set only); within focus via tiles; larger domains report the refusal instead of an answer | enumerate from the existence mask alone (must miss multiplicity) | -| F7 | cold and warm; 1/8/64 thoughts; domain churn; incremental reseal; scalar `RelView` BFS vs batched tile frontier | measured crossover points, each reported with its command and repetition count; nothing derived by arithmetic is labelled measured | n/a (measurement), but every timed path asserts its answer first | +| F7 | cold and warm; 1/8/64 thoughts; domain churn; incremental reseal; scalar `RelView` BFS vs batched tile frontier; shipped BlasGraph `mxv` vs pattern-only borrowed CSR vs bitmap rows through mask-risc, on one relation | measured crossover points, each reported with its command and repetition count; nothing derived by arithmetic is labelled measured | n/a (measurement), but every timed path asserts its answer first | **F6 is not a gate for F1–F5 or F7.** The store and traversal capabilities stand on their own. @@ -214,7 +234,7 @@ Each step carries set/bag/path semantics in its type. Mixing them without an exp | D-RST-1 | identity types | `DurableRef`, `ThoughtCommitId` as contract types (D-CPG CPG-00 owns them; coordinate, don't duplicate) with a never-reused rule test | D-CPG agreement (B1) | | D-RST-2 | store skeleton | `RelationshipMapStore` crate: catalog, `RelationDecl`, facts through the cycle seal, sorted permutation layout, `LayoutSeal`, coordinate domain as a relation; `impl Registrar`; F4 version and domain refusals | D-RST-1 | | D-RST-3 | petgraph shape 1 | `RelView` in a new fork experiment (`experiments/relstore-visit`, isolated workspace, pinned rev), replacing nothing in `lance-fold-hop`; F1 | D-RST-2 | -| D-RST-4 | layouts | CSR, tile and join-table layouts ported from the probes into the store, behind its capability table; F2 (Quack query, Cypher, petgraph first) | D-RST-2 | +| D-RST-4 | layouts | pattern-only `u32` CSR/CSC on BlasGraph's storage types (§5a), bitmap row blocks through mask-risc (`StorageFormat::Bitmap`), and the join table from the probe, behind the store's capability table; layouts built at seal time; F2 (Quack query, Cypher, petgraph first) | D-RST-2 | | D-RST-5 | derived relations | `materialize` with multiplicity, provenance, dependencies; invalidation; DRed retraction; F4 deletion cases, F5 | D-RST-4 | | D-RST-6 | typed traversal + multi-hop in focus | the typed step vocabulary; tile multi-hop through stored intermediates; Gremlin adapter `out().in()` within focus; F3 | D-RST-5, A1 | | D-RST-7 | triangle and amortization | F6 and F7, cold and warm | D-RST-6 | @@ -230,7 +250,7 @@ Each PR keeps the probes in `mork-pathmap-probe` as probes. Production code is n - (b) admit a counted translation step between hops (a phase handoff through a stored intermediate); - (c) stay refused. - This decides whether `NonFunctionalChain` can be lifted beyond a focus. + This decides whether `NonFunctionalChain` can be lifted beyond a focus. §5a narrows the choice: dense row blocks need no indexed read at all (bitmap rows through mask-risc); only sparse rows do, and hypersparse bitmap blocks bring that read down to one per 64 columns. - **A3, scalar petgraph iteration.** Whether shape-1 host iteration over a borrowed layout sits outside the no-gather ruling, which governs mask-risc folds. This plan reads it that way and asks for confirmation. - **A4, store backend and home.** Which crate, and whether facts live in their own Lance datasets written through the cycle seal. This is consistent with "the seal owns durable writes"; the storage-portability ruling forbids a universal `Storage` trait, so the store is one owner, not an abstraction. - **A5, Registrar shape.** `register(name, width)` cannot describe a relation's arity or sorts. Either the store keeps `RelationDecl` beside it (this plan) or Quack's seam grows. The latter changes a Quack trait and needs its own review. From 3329eeb6a5a4af607ef5beace856eeb1df1d9049 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 20 Oct 2026 00:58:33 +0000 Subject: [PATCH 03/10] D-RMAP-2: first measured cell, BF16 16x16 tile GEMM vs bit tile MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Boolean composition, walk counts and triangle counts on one 256 x 256 focus, through ndarray::simd::bf16_tile_gemm_16x16_packed (the facade, not a new kernel) and through a bit matrix, both against a triple-loop oracle. 4 tests; 5 guards disable-verified. D-RMAP-2 §3's exactness claim holds: BF16 walk counts equal the oracle exactly and count > 0 equals Boolean composition at 1 %, 10 %, 50 % and with skewed hubs. On this host only the F32x16 polyfill tier runs (no AMX, no avx512_bf16): A*A takes about 19 ms, while the bit arm composes in 2.6-25 us and gives the same walk counts by popcount in 165-192 us. The polyfill allocates per output element, recorded as ISS-BF16-POLYFILL-PER-ELEMENT-ALLOC; tile-tier hosts are unmeasured. Recorded in D-RMAP-2 §5a, the D-RST plan §5a, STATUS_BOARD, ISSUES and a board entry. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VyWZYKycy6MoYDxHRVnV5K --- .claude/board/ISSUES.md | 9 + .claude/board/STATUS_BOARD.md | 2 +- .../2026-10-20-d-rmap-2-tile-gemm-cell.md | 13 + .claude/board/entries/README.md | 3 +- .../d-rmap-2-context-tile-algebra-probe-v1.md | 18 + .../relstore-quack-petgraph-traversal-v1.md | 2 + crates/mork-pathmap-probe/Cargo.toml | 4 + .../examples/tile_gemm_cost.rs | 98 ++++++ crates/mork-pathmap-probe/src/lib.rs | 1 + crates/mork-pathmap-probe/src/tile_gemm.rs | 315 ++++++++++++++++++ crates/mork-pathmap-probe/tests/tile_gemm.rs | 87 +++++ 11 files changed, 550 insertions(+), 2 deletions(-) create mode 100644 .claude/board/entries/2026-10-20-d-rmap-2-tile-gemm-cell.md create mode 100644 crates/mork-pathmap-probe/examples/tile_gemm_cost.rs create mode 100644 crates/mork-pathmap-probe/src/tile_gemm.rs create mode 100644 crates/mork-pathmap-probe/tests/tile_gemm.rs diff --git a/.claude/board/ISSUES.md b/.claude/board/ISSUES.md index e371fcf4e..1934981fc 100644 --- a/.claude/board/ISSUES.md +++ b/.claude/board/ISSUES.md @@ -1,3 +1,12 @@ +## ISS-BF16-POLYFILL-PER-ELEMENT-ALLOC — ndarray's BF16 16×16 polyfill allocates per output element (2026-10-20) + +**Status:** OPEN. **Basis:** VERIFIED-IN-CODE (`ndarray/src/simd_ops.rs`, `bf16_tile_gemm_16x16`) and MEASURED (D-RMAP-2 §5a: a 256 × 256 `A·A` takes about 19 ms, about 0.9 GMAC/s, on an AVX-512 host without AMX or `avx512_bf16`). + +- Per 16 × 16 call: two decode buffers (`vec!` for A and B), then for each of the 256 output elements a fresh `vec![0.0; k]` filled by a strided read of a B column, an FMA chain and a horizontal `reduce_sum`. About 257 heap allocations per call. +- `bf16_tile_gemm_16x16_packed` on the same tier unpacks the VNNI buffer back to row-major (another allocation) before calling it. +- Consequence: on hosts without the tile tiers, the BF16 tile path cannot be compared with the bit arm on cost; it measures allocation, not arithmetic. +- **What closes it:** an outer-product polyfill in ndarray (broadcast `A[i, kk]`, FMA into a 16-lane accumulator per row, no per-element buffer), with the existing exactness tests, then the D-RMAP-2 §5a probe re-run. + ## ISS-MORK-LIBRARY-GLOBAL-ALLOCATOR — linking MORK replaces the binary's allocator (2026-10-19) **Status:** OPEN. **Basis:** MEASURED (`crates/mork-pathmap-probe`, a `#[global_allocator]` in the probe fails to link: "conflicts with global allocator in: pathmap"). diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index fbe94e3cf..b9c8cf361 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -35,7 +35,7 @@ These PRs merged with no board rows. Every row below is a PROPOSAL (documents an | D-RST-7 | triangle and amortization | Queued | F6 (existence/count/enumeration, sparse/dense/skewed), F7 measured crossovers | | D-RST-8 | multi-hop over a large domain | BLOCKED on admission A2 | — | | D-MPS-8 | native join over a versioned relationship table: R and S laid out by `y` slot, leapfrog intersection as one mask-risc `And`, per-side generations and reseal, per-thought filter planes (`mork-pathmap-schnellverschluss-v1.md` §7b; `crates/mork-pathmap-probe/src/join.rs`) | MEASURED, probe only; not admitted (D-RMAP-1 BIND membrane review) | 8 join tests + native arms in the differential suite vs the oracle; 7 guards disable-verified; cyclic bodies open | -| D-RMAP-2 | context-declaration reuse and GraphBLAS/BF16 tile probe (`d-rmap-2-context-tile-algebra-probe-v1.md`) | PROPOSAL; not run | arms 1–5 in the plan; 297× (D-GTM-0j) is not evidence | +| D-RMAP-2 | context-declaration reuse and GraphBLAS/BF16 tile probe (`d-rmap-2-context-tile-algebra-probe-v1.md`) | PARTIAL: one cell MEASURED (§5a, polyfill tier only); tile tiers and GraphBLAS arm not run | arms 1–5 in the plan; 297× (D-GTM-0j) is not evidence | | D-SRAM-0 | inventory Soufflé AST/ast2ram/RAM passes against D-ASM/D-RA/D-TGA owners | Queued | evidence per mapped pass; no code imported | | D-SRAM-1 | bound AST → RAM-like schedule for pure conjunctive selection | Queued | same answers and read sets on hand-built fixtures; unsupported ops typed-refused | | D-SRAM-2 | cold optimizer rewrites (SCC/strata, magic sets, ordering, dedup) | Queued (research until D-ASM AST placement and D-RA-2) | optimized == unoptimized == oracle; rewrite off removes only the plan shape; disabling a semantic guard fails | diff --git a/.claude/board/entries/2026-10-20-d-rmap-2-tile-gemm-cell.md b/.claude/board/entries/2026-10-20-d-rmap-2-tile-gemm-cell.md new file mode 100644 index 000000000..c45e2a9eb --- /dev/null +++ b/.claude/board/entries/2026-10-20-d-rmap-2-tile-gemm-cell.md @@ -0,0 +1,13 @@ +# D-RMAP-2 tile arm: Boolean composition and triangles on one focus (2026-10-20) + +**Status:** MEASURED (one cell of D-RMAP-2), probe only, one host. Record: `.claude/plans/d-rmap-2-context-tile-algebra-probe-v1.md` §5a. Code: `crates/mork-pathmap-probe/src/tile_gemm.rs`. + +## Measured + +- On a 256 × 256 focus, `ndarray::simd::bf16_tile_gemm_16x16_packed` gives walk counts equal to a triple-loop oracle exactly, and `count > 0` equals Boolean composition (1 %, 10 %, 50 %, skewed hubs). 5 guards disable-verified. +- The tier that ran is the F32x16 FMA polyfill (no AMX, no `avx512_bf16` on this host): `A·A` about 19 ms. The bit arm: composition 2.6–25 µs, walk counts by popcount 165–192 µs, triangles 1.9–72 µs. Staging bits to BF16 (16 KiB to 256 KiB) costs 150–400 µs, more than the whole bit composition. +- The polyfill allocates per output element (`ISS-BF16-POLYFILL-PER-ELEMENT-ALLOC`), so this host measures allocation, not GEMM. + +## Open + +- AMX and `VDPBF16PS` hosts; blocked tiles above one focus; the GraphBLAS arm. diff --git a/.claude/board/entries/README.md b/.claude/board/entries/README.md index 2686a7a2b..f9479a191 100644 --- a/.claude/board/entries/README.md +++ b/.claude/board/entries/README.md @@ -25,11 +25,12 @@ index row, (3) no duplicate entry id. Checks 1 and 2 are deliberately opposite directions; the stranding this convention prevents shows up in exactly one of them, never both. -375 entries, 2026-08-06 .. 2026-10-20. +376 entries, 2026-08-06 .. 2026-10-20. | date | entry id | finding | file | |---|---|---|---| | 2026-10-20 | `relstore-quack-petgraph-plan` | | [2026-10-20-relstore-quack-petgraph-plan.md](2026-10-20-relstore-quack-petgraph-plan.md) | +| 2026-10-20 | `D-RMAP-2` | | [2026-10-20-d-rmap-2-tile-gemm-cell.md](2026-10-20-d-rmap-2-tile-gemm-cell.md) | | 2026-10-19 | `post-merge-recovery-1601-1617` | | [2026-10-19-post-merge-recovery-1601-1617.md](2026-10-19-post-merge-recovery-1601-1617.md) | | 2026-10-19 | `native-join-relationship-table` | | [2026-10-19-native-join-relationship-table.md](2026-10-19-native-join-relationship-table.md) | | 2026-10-19 | `mork-pathmap-schnellverschluss` | | [2026-10-19-mork-pathmap-schnellverschluss.md](2026-10-19-mork-pathmap-schnellverschluss.md) | diff --git a/.claude/plans/d-rmap-2-context-tile-algebra-probe-v1.md b/.claude/plans/d-rmap-2-context-tile-algebra-probe-v1.md index ab4fe4721..070e13854 100644 --- a/.claude/plans/d-rmap-2-context-tile-algebra-probe-v1.md +++ b/.claude/plans/d-rmap-2-context-tile-algebra-probe-v1.md @@ -92,6 +92,24 @@ Report: total bound cost, sorted-index creation/maintenance, native scan time, r - For sparse 1-hop, bitset or direct indexed seeks may defeat GEMM. For repeated dense 2-hop/all-pairs, tile GEMM may have a valid economic region. A loss in the full-cost test remains a loss. - Do not add an MKL/GraphBLAS C dependency, hot JSON representation, new runtime/AST, unauthorized gather or hidden cross-coordinate index hop. +## 5a. First measured cell: Boolean composition and triangles on one 256 × 256 focus (2026-10-20) + +> **Status: MEASURED, probe only, one host.** Code: `crates/mork-pathmap-probe/src/tile_gemm.rs`, `tests/tile_gemm.rs` (4 tests; 5 guards disable-verified), `examples/tile_gemm_cost.rs` (`cargo run --release --example tile_gemm_cost`). Release, warm, medians of 7, every timed answer checked against a triple-loop oracle. The GEMM arm is `ndarray::simd::bf16_tile_gemm_16x16_packed` through the facade, not a new kernel. + +**Tier that ran: the F32x16 FMA polyfill** (`bf16_tile_gemm_tier()`). This host has `avx512_vnni` and `avx512bw` but neither AMX nor `avx512_bf16`. No tile-tier number exists here; the AMX and `VDPBF16PS` cells of §4 remain unmeasured. + +| Map | edges | triangles | bit compose | bit walk counts (popcount) | bit triangles | BF16 stage | BF16 `A·A` (packed / row-major) | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| 1 % | 658 | 23 | 3.6 µs | 175 µs | 1.9 µs | 155 µs | 19.6 / 19.4 ms | +| 10 % | 6 529 | 16 617 | 4.5 µs | 190 µs | 13.5 µs | 256 µs | 19.4 / 22.3 ms | +| 50 % | 32 776 | 2 099 676 | 25.2 µs | 192 µs | 71.8 µs | 400 µs | 19.5 / 18.9 ms | +| 1 % + 4 hubs | 2 706 | 20 273 | 2.6 µs | 165 µs | 5.8 µs | 150 µs | 19.7 / 19.4 ms | + +- **§3's exactness claim holds as measured.** BF16 walk counts equal the oracle exactly (`assert_eq!`, not a tolerance), and `count > 0` equals Boolean composition, on all four maps. +- **On this host the bit arm wins every cell**, including walk counts. Popcount of `row_i(A) ∧ row_j(Aᵀ)` gives the same counts as the GEMM in about 170–190 µs, against about 19 ms. Staging alone (bits to dense BF16, 16 KiB to 256 KiB) costs more than the whole bit composition. +- **The 19 ms is the polyfill's shape, not GEMM's.** `ndarray::simd_ops::bf16_tile_gemm_16x16` allocates two decode buffers per call, then one strided B-column buffer and one horizontal reduce per output element: about 257 heap allocations per 16 × 16 call, 65 000 per 256 × 256 product. The packed entry on this tier also unpacks and allocates per call. Recorded as `ISS-BF16-POLYFILL-PER-ELEMENT-ALLOC`; the row-major entry is no faster, so the unpack is not the cost. +- **Not measured:** AMX and `VDPBF16PS` hosts; dense relations above one focus (blocked 256 × 256 tiles); weighted or evidence-bearing relations, which §3 excludes anyway. + ## 6. Recommendation Use **lance-context GenericStore/SchemaSpec only as a candidate cold durable declaration precedent**, **petgraph::visit as an optional read-only compatibility/oracle facade**, **GraphBLAS as the Boolean algebra oracle**, and **MKL BF16 GEMM as an explicitly falsifiable physical tile accelerator**. The map bank remains in independent typed tables; OGAR provides vocabulary; BIND selects and borrows; RelAlg describes; existing Quack/R2IL/Mask-RISC remains the execution family; CE64 is opaque. diff --git a/.claude/plans/relstore-quack-petgraph-traversal-v1.md b/.claude/plans/relstore-quack-petgraph-traversal-v1.md index d7ffeaa9f..2853286b5 100644 --- a/.claude/plans/relstore-quack-petgraph-traversal-v1.md +++ b/.claude/plans/relstore-quack-petgraph-traversal-v1.md @@ -206,6 +206,8 @@ BlasGraph already ships the relationship-matrix vocabulary this plan needs: CSR, 4. **`mxv` is a pull with an indexed read.** For each row `i`, for each `(j, a_val)` in the row: `vec.get(j)`. That reads the input vector at positions taken from stored column indices, the gather shape, run as scalar host code with a virtual `Semiring` call and a `BitVec` clone per product. 5. **ndarray is not on this path.** BlasGraph routes Hamming and popcount through `ndarray_bridge::dispatch_hamming`, but CSR construction and `mxv` use plain loops. ndarray has dense mask algebra and dense BLAS, and no sparse primitive. +**The BF16 16 × 16 tile GEMM, measured on one focus (D-RMAP-2 §5a).** Exact for 0/1 operands, as D-RMAP-2 §3 says. On this host only the polyfill tier runs, and the bit arm wins every cell, walk counts included (popcount of row ∧ column: about 180 µs vs 19 ms for `A·A`). The GEMM stays a candidate layout for tile hosts and weighted counts, unmeasured until an AMX or `avx512_bf16` host runs the same probe. + **What follows.** - **Reuse:** the store's CSR/CSC layouts are BlasGraph's storage types, extended with a pattern-only, `u32`-indexed variant that borrows its buffers. No third CSR type is added beside BlasGraph's and the probe's; the probe's CSR is retired in favour of it. - **Dense rows go to ndarray.** BlasGraph's reserved `StorageFormat::Bitmap` is the bridge: a row stored as a bitmap over its column domain is exactly an ndarray mask, and a Boolean hop over bitmap rows is `mask_and` plus any, through mask-risc, with no indexed read. That is the tile of D-RMAP-1A §8, generalized from 256 slots to any bitmap row. It costs `|rows| × |cols|` bits, so it is chosen per row block by density (BUNDLE), never globally. diff --git a/crates/mork-pathmap-probe/Cargo.toml b/crates/mork-pathmap-probe/Cargo.toml index 6e446813b..7498f61d3 100644 --- a/crates/mork-pathmap-probe/Cargo.toml +++ b/crates/mork-pathmap-probe/Cargo.toml @@ -34,6 +34,10 @@ pathmap = { git = "https://github.com/AdaWorldAPI/PathMap", rev = "ef02591" } # The native executor for the D-RMAP focus probe (`src/rmap.rs`): the prepared # relationship operands are run as its programs, never re-implemented here. lance-graph-mask-risc = { path = "../lance-graph-mask-risc" } +# The BF16 16x16 tile GEMM for the D-RMAP-2 tile arm (`src/tile_gemm.rs`), +# through the `ndarray::simd` facade. Same coordinate as mask-risc's, so the +# graph keeps one ndarray package. +ndarray = { git = "https://github.com/AdaWorldAPI/ndarray", branch = "master", default-features = false, features = ["std"] } # MORK's workspace names PathMap as `path = "../PathMap/"`, outside its own # repository, so a git dependency on MORK cannot resolve it. This patch points diff --git a/crates/mork-pathmap-probe/examples/tile_gemm_cost.rs b/crates/mork-pathmap-probe/examples/tile_gemm_cost.rs new file mode 100644 index 000000000..2aacb39b8 --- /dev/null +++ b/crates/mork-pathmap-probe/examples/tile_gemm_cost.rs @@ -0,0 +1,98 @@ +//! D-RMAP-2 tile arm: Boolean composition and triangle counts on one +//! 256 x 256 focus map, bit matrix vs BF16 16x16 tile GEMM. +//! +//! Run: `cd crates/mork-pathmap-probe && cargo run --release --example tile_gemm_cost`. +//! Release, warm caches, medians of `REPS`. Every timed answer is checked +//! against the triple-loop oracle first. + +use std::hint::black_box; +use std::time::{Duration, Instant}; + +use mork_pathmap_probe::tile_gemm::{ + Bf16Staged, BitMat, N, fixture, oracle_square, oracle_triangles, threshold, tier, + triangles_from_square, +}; + +const REPS: usize = 7; + +fn time(mut f: impl FnMut() -> T) -> (Duration, T) { + let mut ts = Vec::with_capacity(REPS); + let mut last = None; + for _ in 0..REPS { + let t = Instant::now(); + let v = black_box(f()); + ts.push(t.elapsed()); + last = Some(v); + } + ts.sort(); + (ts[REPS / 2], last.unwrap()) +} + +fn us(d: Duration) -> f64 { + d.as_secs_f64() * 1e6 +} + +fn main() { + println!("bf16 tier: {} | release, medians of {REPS}", tier()); + println!( + "case, edges, triangles, transpose_us, bit_compose_us, bit_counts_us, bit_triangles_us, bf16_stage_us, bf16_square_packed_us, bf16_square_rowmajor_us, threshold_us, bf16_triangles_us, bit_bytes, bf16_bytes" + ); + for (label, p, hubs, seed) in [ + ("1%", 10u32, 0usize, 1u64), + ("10%", 100, 0, 2), + ("50%", 500, 0, 3), + ("1% + 4 hubs", 10, 4, 4), + ] { + let a = fixture(p, hubs, seed); + let want_sq = oracle_square(&a); + let want_t = oracle_triangles(&a); + + let (t_tr, at) = time(|| black_box(&a).transpose()); + let (t_bc, c) = time(|| black_box(&a).compose(black_box(&at), black_box(&a))); + let mut want_bits = BitMat::empty(); + for i in 0..N { + for j in 0..N { + if want_sq[i * N + j] > 0 { + want_bits.set(i, j); + } + } + } + assert_eq!(c, want_bits); + let (t_bq, bq) = time(|| black_box(&a).square_counts(black_box(&at))); + assert_eq!(bq, want_sq); + let (t_bt, tri) = time(|| black_box(&a).triangles(black_box(&at))); + assert_eq!(tri, want_t); + + let (t_st, staged) = time(|| Bf16Staged::stage(black_box(&a))); + let mut sq = vec![0f32; N * N]; + let (t_sq, _) = time(|| { + black_box(&staged).square(black_box(&mut sq)); + }); + assert!(sq.iter().zip(&want_sq).all(|(g, &w)| *g == w as f32)); + let mut sq2 = vec![0f32; N * N]; + let (t_sq2, _) = time(|| { + black_box(&staged).square_rowmajor(black_box(&mut sq2)); + }); + assert_eq!(sq2, sq); + let (t_th, cb) = time(|| threshold(black_box(&sq))); + assert_eq!(cb, want_bits); + let (t_gt, tri2) = time(|| triangles_from_square(black_box(&a), black_box(&sq))); + assert_eq!(tri2, want_t); + + println!( + "{label}, {}, {want_t}, {:.1}, {:.1}, {:.1}, {:.1}, {:.1}, {:.1}, {:.1}, {:.1}, {:.1}, {}, {}", + a.edges(), + us(t_tr), + us(t_bc), + us(t_bq), + us(t_bt), + us(t_st), + us(t_sq), + us(t_sq2), + us(t_th), + us(t_gt), + 2 * N * N / 8, + staged.bytes() + ); + } +} diff --git a/crates/mork-pathmap-probe/src/lib.rs b/crates/mork-pathmap-probe/src/lib.rs index 14e593970..4101bd14f 100644 --- a/crates/mork-pathmap-probe/src/lib.rs +++ b/crates/mork-pathmap-probe/src/lib.rs @@ -26,6 +26,7 @@ pub mod boundary; pub mod join; pub mod reference; pub mod rmap; +pub mod tile_gemm; pub use boundary::{Backend, BoundIndex, Refusal, bind, capability}; diff --git a/crates/mork-pathmap-probe/src/tile_gemm.rs b/crates/mork-pathmap-probe/src/tile_gemm.rs new file mode 100644 index 000000000..63535f4bf --- /dev/null +++ b/crates/mork-pathmap-probe/src/tile_gemm.rs @@ -0,0 +1,315 @@ +//! D-RMAP-2 tile arm: Boolean relationship composition and triangle counting +//! over one 256 × 256 focus map, as a bit matrix and as a BF16 16 × 16 tile +//! GEMM. +//! +//! STATUS: PROBE. Results only. +//! +//! - **Bit arm.** `A` is 256 rows of 256 bits (8 KiB). Composition +//! `C = A ∘ B` is the outer-product form: for every coordinate `k`, each row +//! `i` whose bit `k` is set ORs in row `k` of `B`. Every row is addressed by +//! its own coordinate (`i`, `k`); no position is taken from a stored value. +//! - **GEMM arm.** `A` staged as dense BF16 (0 or 1.0, 128 KiB), multiplied by +//! `ndarray::simd::bf16_tile_gemm_16x16_packed` in 16 × 16 output blocks with +//! `K = 256`, then thresholded. For 0/1 operands the products are exact, and +//! every count is at most 256, far below 2^24, so `count > 0` equals Boolean +//! composition (D-RMAP-2 §3). The counts themselves are walk counts, which +//! the bit arm cannot give without popcounts. +//! +//! Triangles: `T = Σ_{i,j,k} A[i,j]·A[j,k]·A[k,i]`, the directed 3-cycles +//! counted once per rotation. Both arms compute it, against a triple-loop +//! oracle. Nothing here applies to weighted, negated or evidence-bearing +//! relations; that is the boundary D-RMAP-2 §3 states. + +use ndarray::simd::{ + PackedBf16B, bf16_tile_gemm_16x16, bf16_tile_gemm_16x16_packed, bf16_tile_gemm_tier, +}; + +/// Focus side. +pub const N: usize = 256; +const W: usize = N / 64; +const BF16_ONE: u16 = 0x3F80; + +/// A Boolean relation over one focus: row `i`, bit `j` set when `A[i, j]`. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct BitMat { + rows: Vec<[u64; W]>, +} + +impl BitMat { + pub fn empty() -> Self { + BitMat { + rows: vec![[0; W]; N], + } + } + + pub fn from_edges(edges: impl IntoIterator) -> Self { + let mut m = Self::empty(); + for (i, j) in edges { + m.set(i as usize, j as usize); + } + m + } + + pub fn set(&mut self, i: usize, j: usize) { + self.rows[i][j >> 6] |= 1 << (j & 63); + } + + pub fn get(&self, i: usize, j: usize) -> bool { + self.rows[i][j >> 6] >> (j & 63) & 1 == 1 + } + + pub fn row(&self, i: usize) -> &[u64; W] { + &self.rows[i] + } + + pub fn edges(&self) -> usize { + self.rows + .iter() + .flatten() + .map(|w| w.count_ones() as usize) + .sum() + } + + pub fn transpose(&self) -> BitMat { + let mut t = Self::empty(); + for i in 0..N { + for j in 0..N { + if self.get(i, j) { + t.set(j, i); + } + } + } + t + } + + /// `C = A ∘ B` (Boolean), outer-product form over the transpose of `A`: + /// for each `k`, the rows `i` with `A[i, k]` OR in `B[k]`. + pub fn compose(&self, at: &BitMat, b: &BitMat) -> BitMat { + let mut c = Self::empty(); + for k in 0..N { + let bk = b.rows[k]; + for (wi, &word) in at.rows[k].iter().enumerate() { + let mut w = word; + while w != 0 { + let i = wi * 64 + w.trailing_zeros() as usize; + let ci = &mut c.rows[i]; + for x in 0..W { + ci[x] |= bk[x]; + } + w &= w - 1; + } + } + } + c + } + + /// Walk counts `(A · B)[i, j] = popcount(row_i(A) ∧ row_j(Bᵀ))`, the bit + /// arm's answer to the counts GEMM gives. Row-major 256 × 256. + pub fn square_counts(&self, bt: &BitMat) -> Vec { + let mut c = vec![0u32; N * N]; + for i in 0..N { + let ri = self.rows[i]; + for j in 0..N { + let cj = bt.rows[j]; + let mut n = 0u32; + for x in 0..W { + n += (ri[x] & cj[x]).count_ones(); + } + c[i * N + j] = n; + } + } + c + } + + /// Directed 3-cycles: for every edge `(i, j)`, the `k` with `A[j, k]` and + /// `A[k, i]` is `row_j(A) ∧ row_i(Aᵀ)`; popcount and sum. + pub fn triangles(&self, at: &BitMat) -> u64 { + let mut t = 0u64; + for i in 0..N { + let ti = at.rows[i]; + for (wj, &word) in self.rows[i].iter().enumerate() { + let mut w = word; + while w != 0 { + let j = wj * 64 + w.trailing_zeros() as usize; + let rj = self.rows[j]; + for x in 0..W { + t += u64::from((rj[x] & ti[x]).count_ones()); + } + w &= w - 1; + } + } + } + t + } +} + +/// `A` staged for the GEMM: row-major BF16, plus each 16-column block of `A` +/// as a packed `B` operand (`K = 256`). +pub struct Bf16Staged { + a: Vec, + b_blocks: Vec, + /// The same blocks row-major, for the polyfill entry. On a host without + /// AMX or `avx512bf16`, the packed entry unpacks (and allocates) per call + /// before taking the polyfill, so the row-major entry is that tier's + /// honest cost. + b_rows: Vec>, +} + +impl Bf16Staged { + pub fn stage(m: &BitMat) -> Self { + let mut a = vec![0u16; N * N]; + for i in 0..N { + for j in 0..N { + if m.get(i, j) { + a[i * N + j] = BF16_ONE; + } + } + } + let b_rows: Vec> = (0..N / 16) + .map(|bj| { + let mut blk = vec![0u16; N * 16]; + for k in 0..N { + blk[k * 16..k * 16 + 16] + .copy_from_slice(&a[k * N + bj * 16..k * N + bj * 16 + 16]); + } + blk + }) + .collect(); + let b_blocks = b_rows.iter().map(|b| PackedBf16B::pack(b, N)).collect(); + Bf16Staged { + a, + b_blocks, + b_rows, + } + } + + /// Bytes held: the BF16 matrix and the packed blocks. + pub fn bytes(&self) -> usize { + self.a.len() * 2 + + self + .b_blocks + .iter() + .map(|b| b.data().len() * 2) + .sum::() + } + + /// Walk counts through the row-major polyfill entry (no pack, no unpack). + pub fn square_rowmajor(&self, c: &mut [f32]) { + assert_eq!(c.len(), N * N); + let mut blk = [0f32; 256]; + for bi in 0..N / 16 { + let a_rows = &self.a[bi * 16 * N..(bi + 1) * 16 * N]; + for (bj, b) in self.b_rows.iter().enumerate() { + blk.fill(0.0); + bf16_tile_gemm_16x16(a_rows, b, &mut blk, N); + for r in 0..16 { + let at = (bi * 16 + r) * N + bj * 16; + c[at..at + 16].copy_from_slice(&blk[r * 16..r * 16 + 16]); + } + } + } + } + + /// Walk counts `C = A · A` in f32, row-major 256 × 256, through the + /// tier-dispatched packed entry. + pub fn square(&self, c: &mut [f32]) { + assert_eq!(c.len(), N * N); + let mut blk = [0f32; 256]; + for bi in 0..N / 16 { + let a_rows = &self.a[bi * 16 * N..(bi + 1) * 16 * N]; + for (bj, b) in self.b_blocks.iter().enumerate() { + blk.fill(0.0); + bf16_tile_gemm_16x16_packed(a_rows, b, &mut blk); + for r in 0..16 { + let at = (bi * 16 + r) * N + bj * 16; + c[at..at + 16].copy_from_slice(&blk[r * 16..r * 16 + 16]); + } + } + } + } +} + +/// The tier ndarray will take on this host, for the report. +pub fn tier() -> &'static str { + bf16_tile_gemm_tier() +} + +/// Boolean composition from walk counts: `C[i, j] = counts[i, j] > 0`. +pub fn threshold(counts: &[f32]) -> BitMat { + let mut m = BitMat::empty(); + for i in 0..N { + for j in 0..N { + if counts[i * N + j] > 0.0 { + m.set(i, j); + } + } + } + m +} + +/// Triangles from walk counts: `Σ_{i,j} (A²)[j, i] · A[i, j]`. +pub fn triangles_from_square(m: &BitMat, sq: &[f32]) -> u64 { + let mut t = 0u64; + for i in 0..N { + for j in 0..N { + if m.get(i, j) { + t += sq[j * N + i] as u64; + } + } + } + t +} + +/// Independent oracle: a triple loop over the bits. +pub fn oracle_square(m: &BitMat) -> Vec { + let mut c = vec![0u32; N * N]; + for i in 0..N { + for k in 0..N { + if m.get(i, k) { + for j in 0..N { + if m.get(k, j) { + c[i * N + j] += 1; + } + } + } + } + } + c +} + +pub fn oracle_triangles(m: &BitMat) -> u64 { + let sq = oracle_square(m); + let mut t = 0u64; + for i in 0..N { + for j in 0..N { + if m.get(i, j) { + t += u64::from(sq[j * N + i]); + } + } + } + t +} + +/// A deterministic fixture: each edge with probability `p_permille / 1000`; +/// `hubs` rows (and columns) are dense, which gives a skewed degree +/// distribution. +pub fn fixture(p_permille: u32, hubs: usize, seed: u64) -> BitMat { + let mut s = seed; + let mut next = || { + s = s.wrapping_add(0x9E37_79B9_7F4A_7C15); + let mut z = s; + z = (z ^ (z >> 30)).wrapping_mul(0xBF58_476D_1CE4_E5B9); + z = (z ^ (z >> 27)).wrapping_mul(0x94D0_49BB_1331_11EB); + z ^ (z >> 31) + }; + let mut m = BitMat::empty(); + for i in 0..N { + for j in 0..N { + let hub = i < hubs || j < hubs; + if hub || (next() % 1000) < u64::from(p_permille) { + m.set(i, j); + } + } + } + m +} diff --git a/crates/mork-pathmap-probe/tests/tile_gemm.rs b/crates/mork-pathmap-probe/tests/tile_gemm.rs new file mode 100644 index 000000000..6e874d3ad --- /dev/null +++ b/crates/mork-pathmap-probe/tests/tile_gemm.rs @@ -0,0 +1,87 @@ +//! The D-RMAP-2 tile arm against a triple-loop oracle. + +use mork_pathmap_probe::tile_gemm::{ + Bf16Staged, BitMat, N, fixture, oracle_square, oracle_triangles, threshold, + triangles_from_square, +}; + +const CASES: [(u32, usize, u64); 4] = [(10, 0, 1), (100, 0, 2), (500, 0, 3), (10, 4, 4)]; + +#[test] +fn bit_and_bf16_composition_equal_the_oracle() { + for (p, hubs, seed) in CASES { + let a = fixture(p, hubs, seed); + let at = a.transpose(); + let want = oracle_square(&a); + let mut want_bits = BitMat::empty(); + for i in 0..N { + for j in 0..N { + if want[i * N + j] > 0 { + want_bits.set(i, j); + } + } + } + assert_eq!(a.compose(&at, &a), want_bits, "bit arm p={p} hubs={hubs}"); + assert_eq!(a.square_counts(&at), want, "bit counts p={p} hubs={hubs}"); + let staged = Bf16Staged::stage(&a); + let mut sq = vec![0f32; N * N]; + staged.square(&mut sq); + // Exact walk counts, not only their support. + for (got, &w) in sq.iter().zip(&want) { + assert_eq!(*got, w as f32, "bf16 count p={p} hubs={hubs}"); + } + assert_eq!( + threshold(&sq), + want_bits, + "bf16 threshold p={p} hubs={hubs}" + ); + // Not degenerate: two hops reach something one hop does not. + assert_ne!(want_bits, a, "p={p}"); + } +} + +#[test] +fn triangle_counts_agree_across_arms() { + for (p, hubs, seed) in CASES { + let a = fixture(p, hubs, seed); + let at = a.transpose(); + let want = oracle_triangles(&a); + assert_eq!(a.triangles(&at), want, "bit arm p={p} hubs={hubs}"); + let mut sq = vec![0f32; N * N]; + Bf16Staged::stage(&a).square(&mut sq); + assert_eq!( + triangles_from_square(&a, &sq), + want, + "bf16 arm p={p} hubs={hubs}" + ); + } + // Silence twin: a DAG (edges only i < j) has no directed cycle. + let dag = BitMat::from_edges( + (0..N).flat_map(|i| (i + 1..(i + 4).min(N)).map(move |j| (i as u8, j as u8))), + ); + assert!(dag.edges() > 0); + assert_eq!(dag.triangles(&dag.transpose()), 0); + assert_eq!(oracle_triangles(&dag), 0); +} + +/// The counts carry multiplicity the Boolean result drops: some pair is +/// reached by more than one walk, so a threshold is a real loss of +/// information, not a no-op. +#[test] +fn walk_counts_exceed_one_where_support_does_not_show_it() { + let a = fixture(100, 0, 2); + let mut sq = vec![0f32; N * N]; + Bf16Staged::stage(&a).square(&mut sq); + assert!(sq.iter().any(|&c| c > 1.0)); +} + +/// An asymmetric relation, so a composition that forgets the transpose is +/// caught. +#[test] +fn composition_respects_direction() { + let a = BitMat::from_edges([(0, 1), (1, 2)]); + let c = a.compose(&a.transpose(), &a); + assert!(c.get(0, 2)); + assert!(!c.get(2, 0)); + assert_eq!(c.edges(), 1); +} From b5672a5c6d5f32fe9c344a0a2fa461a66eaefadc Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 20 Oct 2026 01:00:33 +0000 Subject: [PATCH 04/10] =?UTF-8?q?D-RMAP-2=20=C2=A75a:=20record=20that=20th?= =?UTF-8?q?is=20host=20lacks=20AMX=20at=20CPUID=20level?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CPUID, XCR0 and arch_prctl(ARCH_REQ_XCOMP_PERM) on syscall 158 checked directly, per ndarray AMX_GOTCHAS Gotcha 4: no AMX, no AVX512-BF16; the permission request returns EOPNOTSUPP. The polyfill tier is the host's, not a detection bug. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VyWZYKycy6MoYDxHRVnV5K --- .claude/plans/d-rmap-2-context-tile-algebra-probe-v1.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.claude/plans/d-rmap-2-context-tile-algebra-probe-v1.md b/.claude/plans/d-rmap-2-context-tile-algebra-probe-v1.md index 070e13854..08cd5d512 100644 --- a/.claude/plans/d-rmap-2-context-tile-algebra-probe-v1.md +++ b/.claude/plans/d-rmap-2-context-tile-algebra-probe-v1.md @@ -96,7 +96,7 @@ Report: total bound cost, sorted-index creation/maintenance, native scan time, r > **Status: MEASURED, probe only, one host.** Code: `crates/mork-pathmap-probe/src/tile_gemm.rs`, `tests/tile_gemm.rs` (4 tests; 5 guards disable-verified), `examples/tile_gemm_cost.rs` (`cargo run --release --example tile_gemm_cost`). Release, warm, medians of 7, every timed answer checked against a triple-loop oracle. The GEMM arm is `ndarray::simd::bf16_tile_gemm_16x16_packed` through the facade, not a new kernel. -**Tier that ran: the F32x16 FMA polyfill** (`bf16_tile_gemm_tier()`). This host has `avx512_vnni` and `avx512bw` but neither AMX nor `avx512_bf16`. No tile-tier number exists here; the AMX and `VDPBF16PS` cells of §4 remain unmeasured. +**Tier that ran: the F32x16 FMA polyfill** (`bf16_tile_gemm_tier()`). This host has `avx512_vnni` and `avx512bw` but neither AMX nor `avx512_bf16`. No tile-tier number exists here; the AMX and `VDPBF16PS` cells of §4 remain unmeasured. This is the hardware, not detection (ndarray `.claude/AMX_GOTCHAS.md` Gotcha 4: a wrong `arch_prctl` syscall number once made `amx_available()` false on real AMX hosts). Checked directly on this host: CPUID.7.0 EDX bits 22/24/25 (AMX-BF16/TILE/INT8) = 0, CPUID.7.1 EAX bit 5 (AVX512-BF16) = 0, XCR0 = `0xe7` (XTILECFG/XTILEDATA bits 17/18 clear), and `arch_prctl(ARCH_REQ_XCOMP_PERM, XFEATURE_XTILEDATA)` on syscall 158 returns `-95` (EOPNOTSUPP). The AMX status in the gotcha file was measured on an Emerald Rapids VM, which this one is not. | Map | edges | triangles | bit compose | bit walk counts (popcount) | bit triangles | BF16 stage | BF16 `A·A` (packed / row-major) | | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | From 7cccbd119203b1184af120edabbd5c245bb402c7 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 20 Oct 2026 01:02:32 +0000 Subject: [PATCH 05/10] plan D-RST: bitmap direction, structural vs valued, seal vs demand MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review refinements, documentation only. - A bitmap row is gather-free only in the predecessor (pull) direction: F'[t] = any(pred(t) AND F). Pushing through successor rows selects rows by the frontier's set bits, an indexed access. The table of representations and their indexed access is in §5a; CSR and CSC both stay, so the compiler can match direction to algebra. - BlasGraph's Boolean semiring is bitwise AND/OR over 16 384-bit hyperdimensional values, not edge existence. Structural and valued relations are kept as distinct interpretations; the lean CSR starts as a borrowed, version-pinned pattern-CSR view beside the existing owned types, not a refactor of them. - The seal fixes identity, version and declared base indexes; specialized layouts (bitmap blocks, tiles, join domains) are prepared on first demand and retained under their seal. - Edge ids are stable fact identities; a row offset is one only under a stable-row-id contract (F1 now tests compaction). - New F2b: the cross-front-end layout-reuse hypothesis, falsifiable by the prepare count. D-RST-3 is brought forward as an isolated probe. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VyWZYKycy6MoYDxHRVnV5K --- .claude/board/STATUS_BOARD.md | 4 +-- ...2026-10-20-relstore-quack-petgraph-plan.md | 4 +-- .../relstore-quack-petgraph-traversal-v1.md | 31 +++++++++++++------ 3 files changed, 25 insertions(+), 14 deletions(-) diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index b9c8cf361..37e9b13c9 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -28,8 +28,8 @@ These PRs merged with no board rows. Every row below is a PROPOSAL (documents an | D-RST-0 | stored relations × Quack × petgraph: architecture and phased plan (`relstore-quack-petgraph-traversal-v1.md`) | PROPOSAL; documentation only | operator decision B; admissions A1–A5, blockers B1–B3 listed in §8 | | D-RST-1 | `DurableRef` / `ThoughtCommitId` contract types, never-reused rule (coordinated with D-CPG CPG-00) | Queued | reuse test; no duplicate with CPG-00 | | D-RST-2 | `RelationshipMapStore` skeleton: catalog, facts through the cycle seal, sorted permutation, `LayoutSeal`, domain as a relation, `impl Registrar` | Queued | F4 refusals (reseal, domain ABA, compaction, scope, dirty, deletion) | -| D-RST-3 | petgraph `RelView` over a sealed CSR in a new fork experiment | Queued | F1: unmodified `Bfs`/`Dfs` equal the oracle; no edge copy (pointer-in-buffer check) | -| D-RST-4 | layouts in the store, built at seal time: pattern-only `u32` CSR/CSC on BlasGraph storage types, bitmap row blocks through mask-risc, join table; capability table | Queued (production reads wait on A1) | F2: five front ends, one `RelationVersion`, identical set and bag answers | +| D-RST-3 | petgraph `RelView` over a borrowed, version-pinned pattern-CSR view in a new fork experiment (brought forward; isolated probe with temporary ownership) | Queued (no longer waits on D-RST-1/2) | F1: unmodified `Bfs`/`Dfs` equal the oracle; no edge copy (pointer-in-buffer check) | +| D-RST-4 | layouts in the store: base indexes at seal time, specialized layouts on demand; pattern-only `u32` CSR/CSC on BlasGraph storage types, bitmap row blocks through mask-risc, join table; capability table | Queued (production reads wait on A1) | F2: five front ends, one `RelationVersion`, identical set and bag answers | | D-RST-5 | derived relations: `materialize`, multiplicity, provenance, dependencies, invalidation, DRed | Queued | F4 deletion, F5 set/bag/path | | D-RST-6 | typed traversal vocabulary; multi-hop within a focus through stored intermediates | Queued (A1) | F3: no `Gather`/`ScatterOrU32` in any program; `NonFunctionalChain` lifted only in focus | | D-RST-7 | triangle and amortization | Queued | F6 (existence/count/enumeration, sparse/dense/skewed), F7 measured crossovers | diff --git a/.claude/board/entries/2026-10-20-relstore-quack-petgraph-plan.md b/.claude/board/entries/2026-10-20-relstore-quack-petgraph-plan.md index 4b4237936..65aef5629 100644 --- a/.claude/board/entries/2026-10-20-relstore-quack-petgraph-plan.md +++ b/.claude/board/entries/2026-10-20-relstore-quack-petgraph-plan.md @@ -14,8 +14,8 @@ Operator decision B: `RelationshipMapStore` owns relationship persistence, ident - Quack's `Registrar` has only a test implementor; three library `Binder`s exist (sap, dir-sim, glove-parity). - A coordinate domain stored as a relation gets its own `DurableRef`, which removes domain ABA without a counter. -- BlasGraph's shipped CSR (read in source): 2 KiB `BitVec` value per entry, `usize` indices, `mxv` clones the matrix per call (even the CSC transpose clones), `mxv` reads the vector at stored column indices (a scalar gather), and ndarray is not on the CSR/`mxv` path. The store reuses BlasGraph's storage types with a pattern-only `u32` borrowed variant; dense row blocks become bitmap rows executed by ndarray mask ops through mask-risc (`StorageFormat::Bitmap` is already reserved); only sparse rows need an indexed read. -- Principle (operator): write a relationship as a relationship; layouts are built at seal time from relation facts, so RelAlg pays translation once per version. +- BlasGraph's shipped CSR (read in source): 2 KiB `BitVec` value per entry, `usize` indices, `mxv` clones the matrix per call (even the CSC transpose clones), `mxv` reads the vector at stored column indices (a scalar gather), and ndarray is not on the CSR/`mxv` path. The store reuses BlasGraph's storage types with a pattern-only `u32` borrowed variant; dense row blocks in the predecessor (pull) direction become bitmap rows executed by ndarray mask ops through mask-risc (`StorageFormat::Bitmap` is already reserved), `F'[t] = any(pred(t) ∧ F)` with no indexed read; the push direction (selecting successor rows) and sparse rows need one. Structural (edge-existence) and valued (hyperdimensional) relations stay distinct. Base indexes are built at seal time; specialized layouts on demand. +- Principle (operator): write a relationship as a relationship; the seal fixes identity, version and declared base indexes, specialized layouts are prepared once on demand and retained, so RelAlg pays translation once per version and only for demanded geometry. ## Open diff --git a/.claude/plans/relstore-quack-petgraph-traversal-v1.md b/.claude/plans/relstore-quack-petgraph-traversal-v1.md index 2853286b5..7f3b2c2af 100644 --- a/.claude/plans/relstore-quack-petgraph-traversal-v1.md +++ b/.claude/plans/relstore-quack-petgraph-traversal-v1.md @@ -15,7 +15,7 @@ Read with: **Short form:** the store remembers, BIND resolves once, Quack compiles, mask-risc executes, petgraph makes it usable. -**The amortization principle (operator, 2026-10-20): write a relationship as a relationship.** A relation is written once, as relation facts, and its relationship-shaped layouts (adjacency by subject and by object, join tables on a declared shared variable) are built at seal time from those facts. They are not rebuilt per query or per front end. RelAlg's semijoins, joins and projections then read prepared relationship layouts instead of translating identities on every evaluation. That is where RelAlg gets faster: the translation is paid once per sealed version, the evaluation is a fold. §5a checks what the shipped BlasGraph CSR contributes to this. +**The amortization principle (operator, 2026-10-20): write a relationship as a relationship.** A relation is written once, as relation facts, and the seal fixes its identity, its version and the base indexes its declaration names (a sorted permutation, and CSR/CSC where declared). Specialized execution geometry (bitmap blocks, focus tiles, a co-addressed join domain on one shared variable) is prepared on the first demand that needs it, once, and retained under a dependency seal on that relation version. Nothing is rebuilt per query or per front end, and nothing is built for a query that never comes: a relation with several joinable attributes has many candidate join domains, and only the demanded ones exist. The seal establishes validity; preparation establishes geometry, and F7 measures whether it paid for itself. RelAlg's semijoins, joins and projections then read prepared relationship layouts instead of translating identities on every evaluation. That is where RelAlg gets faster: the translation is paid once per sealed version, the evaluation is a fold. §5a checks what the shipped BlasGraph CSR contributes to this. ## 0. Findings that shape this plan @@ -136,13 +136,13 @@ impl GraphBase for RelView<'_> { type NodeId = Slot; type EdgeId = EdgeRow; } impl<'s> IntoNeighbors for RelView<'s> { type Neighbors = SlotIter<'s>; /* slice of dst[off[a]..off[a+1]] */ } impl Visitable for RelView<'_> { type Map = FixedBitSet; /* capacity n */ } impl NodeIndexable / NodeCount / NodeCompactIndexable // the slot domain is dense -impl IntoEdgeReferences / IntoEdges / Data // edge id = canonical row: provenance survives +impl IntoEdgeReferences / IntoEdges / Data // edge id = stable fact identity (see below) impl IntoNeighborsDirected // only when the store holds the reverse permutation; // otherwise unimplemented, never computed on demand ``` - Petgraph's `Bfs`, `Dfs`, `DfsPostOrder` and `Topo` then run unmodified. Nothing is copied: the iterator is a slice of the stored layout. -- `EdgeId` is the canonical fact row, so a path can be traced back to facts. +- `EdgeId` is the stable **fact identity**, so a path can be traced back to facts. A canonical row offset is a stable edge id only where the relation's row-address contract guarantees it (Lance stable row ids, opt-in per dataset; D-RMAP-1A §7.3 measured 3 510 of 3 510 row addresses moved by compaction without them). Otherwise the row offset is only the physical address the iterator reads at, and the edge id is the fact identity stored beside it. - Out-of-focus or out-of-domain neighbours are a property of the layout. A view over a full-domain CSR has none; a view over a focus layout must report escapes and refuse `global()` reads (D-RMAP-1A §8). **Classification (admission A3).** Shape 1 is ordinary host iteration inside petgraph's algorithm. It is not population execution and does not go through mask-risc, so the no-gather ruling, which governs fold execution, does not decide it. This plan records that reading as an open admission item rather than assuming it. @@ -172,7 +172,7 @@ Each step carries set/bag/path semantics in its type. Mixing them without an exp ## 4. Execution flow 1. **Declare / store.** `RelationDecl` → store catalog. Facts arrive through the cycle seal. The store records the resulting `DurableRef` against the relation. -2. **Prepare (normalize once).** The store builds layouts from a sealed version: a sorted permutation, CSR, tiles for a declared focus, join tables for a declared shared variable, and the coordinate domain as a relation. Each layout gets a `LayoutSeal`, and each preparation is timed and counted (normalization, index build, translation). +2. **Seal, then prepare on demand.** At seal time the store builds only the base indexes the declaration names (sorted permutation; CSR/CSC where declared) and the coordinate domain as a relation. A specialized layout (bitmap blocks, a focus tile, a join table for one shared variable) is built when a compiled demand first needs it and is retained under its `LayoutSeal`, so the next demand on the same relation version reuses it. Each preparation is timed and counted (normalization, index build, translation), and an unused layout is never built. 3. **Bind.** BIND resolves names → ids, picks the domain, checks authorization, pins versions, and lists the layouts whose seals match. The global→local translation runs here or in step 2, never inside a fold (H14c rule 6), and it is subject to admission A1. 4. **Compile.** Quack turns the query (Quack query, Cypher, Gremlin, RelAlg/Datalog, typed traversal) into a demand plus a `Program` over the eligible layouts' planes. 5. **Bundle.** Choose a layout using the capability table (§5). By fold law 3, the choice may not change the answer; F2 is the test. @@ -209,8 +209,18 @@ BlasGraph already ships the relationship-matrix vocabulary this plan needs: CSR, **The BF16 16 × 16 tile GEMM, measured on one focus (D-RMAP-2 §5a).** Exact for 0/1 operands, as D-RMAP-2 §3 says. On this host only the polyfill tier runs, and the bit arm wins every cell, walk counts included (popcount of row ∧ column: about 180 µs vs 19 ms for `A·A`). The GEMM stays a candidate layout for tile hosts and weighted counts, unmeasured until an AMX or `avx512_bf16` host runs the same probe. **What follows.** -- **Reuse:** the store's CSR/CSC layouts are BlasGraph's storage types, extended with a pattern-only, `u32`-indexed variant that borrows its buffers. No third CSR type is added beside BlasGraph's and the probe's; the probe's CSR is retired in favour of it. -- **Dense rows go to ndarray.** BlasGraph's reserved `StorageFormat::Bitmap` is the bridge: a row stored as a bitmap over its column domain is exactly an ndarray mask, and a Boolean hop over bitmap rows is `mask_and` plus any, through mask-risc, with no indexed read. That is the tile of D-RMAP-1A §8, generalized from 256 slots to any bitmap row. It costs `|rows| × |cols|` bits, so it is chosen per row block by density (BUNDLE), never globally. +- **Structural and valued relations are different things, and both stay.** BlasGraph's `HdrSemiring::Boolean` is bitwise AND/OR over 16 384-bit hyperdimensional values, not edge existence. A structural relation's value is the edge's existence (plus an optional fact identity), with no 2 KiB payload. A valued relation keeps BlasGraph's hyperdimensional vectors, or another typed edge value, read only when the computation needs it. The lean CSR is a pattern-only interpretation inside the same sparse-storage family; it does not replace BlasGraph's hyperdimensional algebra. +- **Reuse means a representation change, not a constructor.** `CsrStorage` owns its `Vec`s, so borrowing buffers and dropping values is a new representation. The first step is a borrowed, version-pinned pattern-CSR **view** defined alongside the existing storage types; whether to refactor `CsrStorage` itself is decided after it has been measured. The probe's CSR is retired in favour of that view; no third owned CSR type is added. +- **Dense rows go to ndarray, and the direction decides whether that is gather-free.** BlasGraph's reserved `StorageFormat::Bitmap` is the bridge: a bitmap row is an ndarray mask. But the representation does not settle the question; the operation does. With `A[s,t]` an edge from source `s` to target `t` and `F` the frontier: + + | Representation | Operation | Indexed access | + |---|---|---| + | lean CSR | iterate neighbours | row-offset lookup, then a contiguous scan | + | sparse bitmap blocks | intersect frontier words | block lookup (an indexed read per 64 columns), then a word-wise AND | + | dense **predecessor** bitmap (per target, a row over sources) | `F'[t] = any(pred(t) ∧ F)` | none: co-addressed AND plus a reduction | + | dense **successor** bitmap (per source, a row over targets) | push: OR the rows of the selected sources | source-row selection is indexed by the frontier's set bits, then OR accumulation | + + Only the predecessor form is the admitted mask-risc case, and it is the tile of D-RMAP-1A §8 generalized from 256 slots to any row block. Expanding a whole frontier with it still needs one AND-and-reduce per target, so it costs `|targets| × |sources| / 64` words whatever the frontier's size; it is chosen per row block by density (BUNDLE), never globally. The compiler therefore picks the relation's direction to match the algebra (CSC for pull, CSR for push and neighbour iteration), which is a reason to keep both CSR and CSC rather than one universal layout. - **Sparse rows still need an indexed read.** A sparse row lists column indices; any hop over it reads the frontier at those indices. Hypersparse bitmap blocks (a list of `(block, 64-bit word)`) make that read word-granular instead of bit-granular: one indexed read per 64 columns instead of one per column, and every read counted. That is the concrete content of admission A2 option (a)/(b); this plan does not admit it. - **Duplication:** `holograph/src/graphblas` carries a second GraphBLAS. The store binds to one (the `lance-graph` BlasGraph types) and records the other as debt, not a second consumer. @@ -218,8 +228,9 @@ BlasGraph already ships the relationship-matrix vocabulary this plan needs: CSR, | # | Test | Pass condition | Disable that must turn it red | |---|---|---|---| -| F1 | unmodified `Bfs`, `Dfs`, `DfsPostOrder` over `RelView` on one sealed CSR | visited order and set equal petgraph over an independently built `Graph` (test-only oracle); zero edge copies, checked by asserting the iterator's slice pointer lies inside the layout buffer | build the view from a copied `Vec` (the pointer check fails); drop a neighbour | +| F1 | unmodified `Bfs`, `Dfs`, `DfsPostOrder` over `RelView` on one sealed CSR | visited order and set equal petgraph over an independently built `Graph` (test-only oracle); zero edge copies, checked by asserting the iterator's slice pointer lies inside the layout buffer; edge ids survive a compaction of the fact rows (stable fact identity, not the row offset) | build the view from a copied `Vec` (the pointer check fails); drop a neighbour; use the row offset as the edge id (compaction changes it) | | F2 | one stored relation read by a Quack query, Cypher, the Gremlin adapter, a RelAlg/Datalog rule and petgraph | identical ids and exact counts in set and bag semantics, against a nested-loop oracle; same `RelationVersion` cited by all five | change one layout so it answers set where bag was asked | +| F2b | hypothesis: a petgraph BFS frontier, a Gremlin `out()` expansion and a Datalog recursive step over the same relation compile to compatible demands and reuse one prepared layout, without sharing an execution model | the second and third front ends find the layout prepared by the first (prepare count stays 1 across the three, asserted), answers equal their oracles | give each front end its own layout key (prepare count goes to 3) | | F3 | `out(r).in(r)` and `repeat_bounded(3)` within a focus, over a stored intermediate | equals the oracle; no `Gather`, no `ScatterOrU32` in any `Program` (asserted on the op list); `NonFunctionalChain` lifted only for the tiled case; over a large domain it still refuses | pass the hop-1 result as a raw mask instead of a stored intermediate with its seal (version check must refuse) | | F4 | reseal; domain ABA (same ids, new domain version); thought slot reuse; compaction without stable row ids; authorization revoked; dirty speculative state; deletion | each is a typed refusal naming the stale component, never a plausible answer (D-RMAP-1A X1–X9, §6a) | key any check on a bare integer or on content | | F5 | distinct neighbours vs parallel edges; repeated traversers (bag); path multiplicity; recursive reachability | set, bag and path answers stay distinct where the oracle says they differ; a bag→mask step refuses | collapse bag to set inside a step | @@ -235,8 +246,8 @@ BlasGraph already ships the relationship-matrix vocabulary this plan needs: CSR, | D-RST-0 | this plan | documentation; board rows | — | | D-RST-1 | identity types | `DurableRef`, `ThoughtCommitId` as contract types (D-CPG CPG-00 owns them; coordinate, don't duplicate) with a never-reused rule test | D-CPG agreement (B1) | | D-RST-2 | store skeleton | `RelationshipMapStore` crate: catalog, `RelationDecl`, facts through the cycle seal, sorted permutation layout, `LayoutSeal`, coordinate domain as a relation; `impl Registrar`; F4 version and domain refusals | D-RST-1 | -| D-RST-3 | petgraph shape 1 | `RelView` in a new fork experiment (`experiments/relstore-visit`, isolated workspace, pinned rev), replacing nothing in `lance-fold-hop`; F1 | D-RST-2 | -| D-RST-4 | layouts | pattern-only `u32` CSR/CSC on BlasGraph's storage types (§5a), bitmap row blocks through mask-risc (`StorageFormat::Bitmap`), and the join table from the probe, behind the store's capability table; layouts built at seal time; F2 (Quack query, Cypher, petgraph first) | D-RST-2 | +| D-RST-3 | petgraph shape 1, brought forward | `RelView` over a minimal borrowed, version-pinned pattern-CSR view in a new fork experiment (`experiments/relstore-visit`, isolated workspace, pinned rev), with explicit temporary ownership of its buffers (an isolated probe, not the store); replaces nothing in `lance-fold-hop`; F1 | the pattern-CSR view only (can run before D-RST-1/2) | +| D-RST-4 | layouts | pattern-only `u32` CSR/CSC on BlasGraph's storage types (§5a), bitmap row blocks through mask-risc (`StorageFormat::Bitmap`), and the join table from the probe, behind the store's capability table; base indexes at seal time, specialized layouts prepared on demand and retained under their seal; F2, F2b (Quack query, Cypher, petgraph first) | D-RST-2 | | D-RST-5 | derived relations | `materialize` with multiplicity, provenance, dependencies; invalidation; DRed retraction; F4 deletion cases, F5 | D-RST-4 | | D-RST-6 | typed traversal + multi-hop in focus | the typed step vocabulary; tile multi-hop through stored intermediates; Gremlin adapter `out().in()` within focus; F3 | D-RST-5, A1 | | D-RST-7 | triangle and amortization | F6 and F7, cold and warm | D-RST-6 | @@ -252,7 +263,7 @@ Each PR keeps the probes in `mork-pathmap-probe` as probes. Production code is n - (b) admit a counted translation step between hops (a phase handoff through a stored intermediate); - (c) stay refused. - This decides whether `NonFunctionalChain` can be lifted beyond a focus. §5a narrows the choice: dense row blocks need no indexed read at all (bitmap rows through mask-risc); only sparse rows do, and hypersparse bitmap blocks bring that read down to one per 64 columns. + This decides whether `NonFunctionalChain` can be lifted beyond a focus. §5a narrows the choice: dense row blocks in the predecessor (pull) direction need no indexed read (`any(pred(t) ∧ F)` through mask-risc); the push direction and sparse rows do, and hypersparse bitmap blocks bring that read down to one per 64 columns. - **A3, scalar petgraph iteration.** Whether shape-1 host iteration over a borrowed layout sits outside the no-gather ruling, which governs mask-risc folds. This plan reads it that way and asks for confirmation. - **A4, store backend and home.** Which crate, and whether facts live in their own Lance datasets written through the cycle seal. This is consistent with "the seal owns durable writes"; the storage-portability ruling forbids a universal `Storage` trait, so the store is one owner, not an abstraction. - **A5, Registrar shape.** `register(name, width)` cannot describe a relation's arity or sorts. Either the store keeps `RelationDecl` beside it (this plan) or Quack's seam grows. The latter changes a Quack trait and needs its own review. From cd6b45750e92522477f5067a7001bec71cae8442 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 20 Oct 2026 01:10:41 +0000 Subject: [PATCH 06/10] Move the D-RMAP-2 tile-GEMM probe to #1623 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Reverts b5672a5c and 3329eeb6 on this branch. Their content now lives in #1623 as cherry-picks, so its measurements, disable-verified guards and provenance are kept there. This PR returns to D-RST-0, the plan, and gains one pointer to #1623 in §5a. No protected history is rewritten: the append-only gate compares against the merge-base with main, and the reverted ISSUES.md and board-entry additions were never on main. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VyWZYKycy6MoYDxHRVnV5K --- .claude/board/ISSUES.md | 9 - .claude/board/STATUS_BOARD.md | 2 +- .../2026-10-20-d-rmap-2-tile-gemm-cell.md | 13 - .claude/board/entries/README.md | 3 +- .../d-rmap-2-context-tile-algebra-probe-v1.md | 18 - .../relstore-quack-petgraph-traversal-v1.md | 2 +- crates/mork-pathmap-probe/Cargo.toml | 4 - .../examples/tile_gemm_cost.rs | 98 ------ crates/mork-pathmap-probe/src/lib.rs | 1 - crates/mork-pathmap-probe/src/tile_gemm.rs | 315 ------------------ crates/mork-pathmap-probe/tests/tile_gemm.rs | 87 ----- 11 files changed, 3 insertions(+), 549 deletions(-) delete mode 100644 .claude/board/entries/2026-10-20-d-rmap-2-tile-gemm-cell.md delete mode 100644 crates/mork-pathmap-probe/examples/tile_gemm_cost.rs delete mode 100644 crates/mork-pathmap-probe/src/tile_gemm.rs delete mode 100644 crates/mork-pathmap-probe/tests/tile_gemm.rs diff --git a/.claude/board/ISSUES.md b/.claude/board/ISSUES.md index 1934981fc..e371fcf4e 100644 --- a/.claude/board/ISSUES.md +++ b/.claude/board/ISSUES.md @@ -1,12 +1,3 @@ -## ISS-BF16-POLYFILL-PER-ELEMENT-ALLOC — ndarray's BF16 16×16 polyfill allocates per output element (2026-10-20) - -**Status:** OPEN. **Basis:** VERIFIED-IN-CODE (`ndarray/src/simd_ops.rs`, `bf16_tile_gemm_16x16`) and MEASURED (D-RMAP-2 §5a: a 256 × 256 `A·A` takes about 19 ms, about 0.9 GMAC/s, on an AVX-512 host without AMX or `avx512_bf16`). - -- Per 16 × 16 call: two decode buffers (`vec!` for A and B), then for each of the 256 output elements a fresh `vec![0.0; k]` filled by a strided read of a B column, an FMA chain and a horizontal `reduce_sum`. About 257 heap allocations per call. -- `bf16_tile_gemm_16x16_packed` on the same tier unpacks the VNNI buffer back to row-major (another allocation) before calling it. -- Consequence: on hosts without the tile tiers, the BF16 tile path cannot be compared with the bit arm on cost; it measures allocation, not arithmetic. -- **What closes it:** an outer-product polyfill in ndarray (broadcast `A[i, kk]`, FMA into a 16-lane accumulator per row, no per-element buffer), with the existing exactness tests, then the D-RMAP-2 §5a probe re-run. - ## ISS-MORK-LIBRARY-GLOBAL-ALLOCATOR — linking MORK replaces the binary's allocator (2026-10-19) **Status:** OPEN. **Basis:** MEASURED (`crates/mork-pathmap-probe`, a `#[global_allocator]` in the probe fails to link: "conflicts with global allocator in: pathmap"). diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index 37e9b13c9..0c8846c38 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -35,7 +35,7 @@ These PRs merged with no board rows. Every row below is a PROPOSAL (documents an | D-RST-7 | triangle and amortization | Queued | F6 (existence/count/enumeration, sparse/dense/skewed), F7 measured crossovers | | D-RST-8 | multi-hop over a large domain | BLOCKED on admission A2 | — | | D-MPS-8 | native join over a versioned relationship table: R and S laid out by `y` slot, leapfrog intersection as one mask-risc `And`, per-side generations and reseal, per-thought filter planes (`mork-pathmap-schnellverschluss-v1.md` §7b; `crates/mork-pathmap-probe/src/join.rs`) | MEASURED, probe only; not admitted (D-RMAP-1 BIND membrane review) | 8 join tests + native arms in the differential suite vs the oracle; 7 guards disable-verified; cyclic bodies open | -| D-RMAP-2 | context-declaration reuse and GraphBLAS/BF16 tile probe (`d-rmap-2-context-tile-algebra-probe-v1.md`) | PARTIAL: one cell MEASURED (§5a, polyfill tier only); tile tiers and GraphBLAS arm not run | arms 1–5 in the plan; 297× (D-GTM-0j) is not evidence | +| D-RMAP-2 | context-declaration reuse and GraphBLAS/BF16 tile probe (`d-rmap-2-context-tile-algebra-probe-v1.md`) | PROPOSAL; not run | arms 1–5 in the plan; 297× (D-GTM-0j) is not evidence | | D-SRAM-0 | inventory Soufflé AST/ast2ram/RAM passes against D-ASM/D-RA/D-TGA owners | Queued | evidence per mapped pass; no code imported | | D-SRAM-1 | bound AST → RAM-like schedule for pure conjunctive selection | Queued | same answers and read sets on hand-built fixtures; unsupported ops typed-refused | | D-SRAM-2 | cold optimizer rewrites (SCC/strata, magic sets, ordering, dedup) | Queued (research until D-ASM AST placement and D-RA-2) | optimized == unoptimized == oracle; rewrite off removes only the plan shape; disabling a semantic guard fails | diff --git a/.claude/board/entries/2026-10-20-d-rmap-2-tile-gemm-cell.md b/.claude/board/entries/2026-10-20-d-rmap-2-tile-gemm-cell.md deleted file mode 100644 index c45e2a9eb..000000000 --- a/.claude/board/entries/2026-10-20-d-rmap-2-tile-gemm-cell.md +++ /dev/null @@ -1,13 +0,0 @@ -# D-RMAP-2 tile arm: Boolean composition and triangles on one focus (2026-10-20) - -**Status:** MEASURED (one cell of D-RMAP-2), probe only, one host. Record: `.claude/plans/d-rmap-2-context-tile-algebra-probe-v1.md` §5a. Code: `crates/mork-pathmap-probe/src/tile_gemm.rs`. - -## Measured - -- On a 256 × 256 focus, `ndarray::simd::bf16_tile_gemm_16x16_packed` gives walk counts equal to a triple-loop oracle exactly, and `count > 0` equals Boolean composition (1 %, 10 %, 50 %, skewed hubs). 5 guards disable-verified. -- The tier that ran is the F32x16 FMA polyfill (no AMX, no `avx512_bf16` on this host): `A·A` about 19 ms. The bit arm: composition 2.6–25 µs, walk counts by popcount 165–192 µs, triangles 1.9–72 µs. Staging bits to BF16 (16 KiB to 256 KiB) costs 150–400 µs, more than the whole bit composition. -- The polyfill allocates per output element (`ISS-BF16-POLYFILL-PER-ELEMENT-ALLOC`), so this host measures allocation, not GEMM. - -## Open - -- AMX and `VDPBF16PS` hosts; blocked tiles above one focus; the GraphBLAS arm. diff --git a/.claude/board/entries/README.md b/.claude/board/entries/README.md index f9479a191..2686a7a2b 100644 --- a/.claude/board/entries/README.md +++ b/.claude/board/entries/README.md @@ -25,12 +25,11 @@ index row, (3) no duplicate entry id. Checks 1 and 2 are deliberately opposite directions; the stranding this convention prevents shows up in exactly one of them, never both. -376 entries, 2026-08-06 .. 2026-10-20. +375 entries, 2026-08-06 .. 2026-10-20. | date | entry id | finding | file | |---|---|---|---| | 2026-10-20 | `relstore-quack-petgraph-plan` | | [2026-10-20-relstore-quack-petgraph-plan.md](2026-10-20-relstore-quack-petgraph-plan.md) | -| 2026-10-20 | `D-RMAP-2` | | [2026-10-20-d-rmap-2-tile-gemm-cell.md](2026-10-20-d-rmap-2-tile-gemm-cell.md) | | 2026-10-19 | `post-merge-recovery-1601-1617` | | [2026-10-19-post-merge-recovery-1601-1617.md](2026-10-19-post-merge-recovery-1601-1617.md) | | 2026-10-19 | `native-join-relationship-table` | | [2026-10-19-native-join-relationship-table.md](2026-10-19-native-join-relationship-table.md) | | 2026-10-19 | `mork-pathmap-schnellverschluss` | | [2026-10-19-mork-pathmap-schnellverschluss.md](2026-10-19-mork-pathmap-schnellverschluss.md) | diff --git a/.claude/plans/d-rmap-2-context-tile-algebra-probe-v1.md b/.claude/plans/d-rmap-2-context-tile-algebra-probe-v1.md index 08cd5d512..ab4fe4721 100644 --- a/.claude/plans/d-rmap-2-context-tile-algebra-probe-v1.md +++ b/.claude/plans/d-rmap-2-context-tile-algebra-probe-v1.md @@ -92,24 +92,6 @@ Report: total bound cost, sorted-index creation/maintenance, native scan time, r - For sparse 1-hop, bitset or direct indexed seeks may defeat GEMM. For repeated dense 2-hop/all-pairs, tile GEMM may have a valid economic region. A loss in the full-cost test remains a loss. - Do not add an MKL/GraphBLAS C dependency, hot JSON representation, new runtime/AST, unauthorized gather or hidden cross-coordinate index hop. -## 5a. First measured cell: Boolean composition and triangles on one 256 × 256 focus (2026-10-20) - -> **Status: MEASURED, probe only, one host.** Code: `crates/mork-pathmap-probe/src/tile_gemm.rs`, `tests/tile_gemm.rs` (4 tests; 5 guards disable-verified), `examples/tile_gemm_cost.rs` (`cargo run --release --example tile_gemm_cost`). Release, warm, medians of 7, every timed answer checked against a triple-loop oracle. The GEMM arm is `ndarray::simd::bf16_tile_gemm_16x16_packed` through the facade, not a new kernel. - -**Tier that ran: the F32x16 FMA polyfill** (`bf16_tile_gemm_tier()`). This host has `avx512_vnni` and `avx512bw` but neither AMX nor `avx512_bf16`. No tile-tier number exists here; the AMX and `VDPBF16PS` cells of §4 remain unmeasured. This is the hardware, not detection (ndarray `.claude/AMX_GOTCHAS.md` Gotcha 4: a wrong `arch_prctl` syscall number once made `amx_available()` false on real AMX hosts). Checked directly on this host: CPUID.7.0 EDX bits 22/24/25 (AMX-BF16/TILE/INT8) = 0, CPUID.7.1 EAX bit 5 (AVX512-BF16) = 0, XCR0 = `0xe7` (XTILECFG/XTILEDATA bits 17/18 clear), and `arch_prctl(ARCH_REQ_XCOMP_PERM, XFEATURE_XTILEDATA)` on syscall 158 returns `-95` (EOPNOTSUPP). The AMX status in the gotcha file was measured on an Emerald Rapids VM, which this one is not. - -| Map | edges | triangles | bit compose | bit walk counts (popcount) | bit triangles | BF16 stage | BF16 `A·A` (packed / row-major) | -| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | -| 1 % | 658 | 23 | 3.6 µs | 175 µs | 1.9 µs | 155 µs | 19.6 / 19.4 ms | -| 10 % | 6 529 | 16 617 | 4.5 µs | 190 µs | 13.5 µs | 256 µs | 19.4 / 22.3 ms | -| 50 % | 32 776 | 2 099 676 | 25.2 µs | 192 µs | 71.8 µs | 400 µs | 19.5 / 18.9 ms | -| 1 % + 4 hubs | 2 706 | 20 273 | 2.6 µs | 165 µs | 5.8 µs | 150 µs | 19.7 / 19.4 ms | - -- **§3's exactness claim holds as measured.** BF16 walk counts equal the oracle exactly (`assert_eq!`, not a tolerance), and `count > 0` equals Boolean composition, on all four maps. -- **On this host the bit arm wins every cell**, including walk counts. Popcount of `row_i(A) ∧ row_j(Aᵀ)` gives the same counts as the GEMM in about 170–190 µs, against about 19 ms. Staging alone (bits to dense BF16, 16 KiB to 256 KiB) costs more than the whole bit composition. -- **The 19 ms is the polyfill's shape, not GEMM's.** `ndarray::simd_ops::bf16_tile_gemm_16x16` allocates two decode buffers per call, then one strided B-column buffer and one horizontal reduce per output element: about 257 heap allocations per 16 × 16 call, 65 000 per 256 × 256 product. The packed entry on this tier also unpacks and allocates per call. Recorded as `ISS-BF16-POLYFILL-PER-ELEMENT-ALLOC`; the row-major entry is no faster, so the unpack is not the cost. -- **Not measured:** AMX and `VDPBF16PS` hosts; dense relations above one focus (blocked 256 × 256 tiles); weighted or evidence-bearing relations, which §3 excludes anyway. - ## 6. Recommendation Use **lance-context GenericStore/SchemaSpec only as a candidate cold durable declaration precedent**, **petgraph::visit as an optional read-only compatibility/oracle facade**, **GraphBLAS as the Boolean algebra oracle**, and **MKL BF16 GEMM as an explicitly falsifiable physical tile accelerator**. The map bank remains in independent typed tables; OGAR provides vocabulary; BIND selects and borrows; RelAlg describes; existing Quack/R2IL/Mask-RISC remains the execution family; CE64 is opaque. diff --git a/.claude/plans/relstore-quack-petgraph-traversal-v1.md b/.claude/plans/relstore-quack-petgraph-traversal-v1.md index 7f3b2c2af..ac9179a75 100644 --- a/.claude/plans/relstore-quack-petgraph-traversal-v1.md +++ b/.claude/plans/relstore-quack-petgraph-traversal-v1.md @@ -206,7 +206,7 @@ BlasGraph already ships the relationship-matrix vocabulary this plan needs: CSR, 4. **`mxv` is a pull with an indexed read.** For each row `i`, for each `(j, a_val)` in the row: `vec.get(j)`. That reads the input vector at positions taken from stored column indices, the gather shape, run as scalar host code with a virtual `Semiring` call and a `BitVec` clone per product. 5. **ndarray is not on this path.** BlasGraph routes Hamming and popcount through `ndarray_bridge::dispatch_hamming`, but CSR construction and `mxv` use plain loops. ndarray has dense mask algebra and dense BLAS, and no sparse primitive. -**The BF16 16 × 16 tile GEMM, measured on one focus (D-RMAP-2 §5a).** Exact for 0/1 operands, as D-RMAP-2 §3 says. On this host only the polyfill tier runs, and the bit arm wins every cell, walk counts included (popcount of row ∧ column: about 180 µs vs 19 ms for `A·A`). The GEMM stays a candidate layout for tile hosts and weighted counts, unmeasured until an AMX or `avx512_bf16` host runs the same probe. +**The BF16 16 × 16 tile GEMM is measured separately** (D-RMAP-2 probe, #1623). On a host without AMX or AVX512-BF16 only ndarray's F32x16 polyfill tier runs; there the bit arm wins every cell, and that result says nothing about tile-hardware performance. The GEMM stays a candidate layout for tile hosts and weighted counts until one is measured. **What follows.** - **Structural and valued relations are different things, and both stay.** BlasGraph's `HdrSemiring::Boolean` is bitwise AND/OR over 16 384-bit hyperdimensional values, not edge existence. A structural relation's value is the edge's existence (plus an optional fact identity), with no 2 KiB payload. A valued relation keeps BlasGraph's hyperdimensional vectors, or another typed edge value, read only when the computation needs it. The lean CSR is a pattern-only interpretation inside the same sparse-storage family; it does not replace BlasGraph's hyperdimensional algebra. diff --git a/crates/mork-pathmap-probe/Cargo.toml b/crates/mork-pathmap-probe/Cargo.toml index 7498f61d3..6e446813b 100644 --- a/crates/mork-pathmap-probe/Cargo.toml +++ b/crates/mork-pathmap-probe/Cargo.toml @@ -34,10 +34,6 @@ pathmap = { git = "https://github.com/AdaWorldAPI/PathMap", rev = "ef02591" } # The native executor for the D-RMAP focus probe (`src/rmap.rs`): the prepared # relationship operands are run as its programs, never re-implemented here. lance-graph-mask-risc = { path = "../lance-graph-mask-risc" } -# The BF16 16x16 tile GEMM for the D-RMAP-2 tile arm (`src/tile_gemm.rs`), -# through the `ndarray::simd` facade. Same coordinate as mask-risc's, so the -# graph keeps one ndarray package. -ndarray = { git = "https://github.com/AdaWorldAPI/ndarray", branch = "master", default-features = false, features = ["std"] } # MORK's workspace names PathMap as `path = "../PathMap/"`, outside its own # repository, so a git dependency on MORK cannot resolve it. This patch points diff --git a/crates/mork-pathmap-probe/examples/tile_gemm_cost.rs b/crates/mork-pathmap-probe/examples/tile_gemm_cost.rs deleted file mode 100644 index 2aacb39b8..000000000 --- a/crates/mork-pathmap-probe/examples/tile_gemm_cost.rs +++ /dev/null @@ -1,98 +0,0 @@ -//! D-RMAP-2 tile arm: Boolean composition and triangle counts on one -//! 256 x 256 focus map, bit matrix vs BF16 16x16 tile GEMM. -//! -//! Run: `cd crates/mork-pathmap-probe && cargo run --release --example tile_gemm_cost`. -//! Release, warm caches, medians of `REPS`. Every timed answer is checked -//! against the triple-loop oracle first. - -use std::hint::black_box; -use std::time::{Duration, Instant}; - -use mork_pathmap_probe::tile_gemm::{ - Bf16Staged, BitMat, N, fixture, oracle_square, oracle_triangles, threshold, tier, - triangles_from_square, -}; - -const REPS: usize = 7; - -fn time(mut f: impl FnMut() -> T) -> (Duration, T) { - let mut ts = Vec::with_capacity(REPS); - let mut last = None; - for _ in 0..REPS { - let t = Instant::now(); - let v = black_box(f()); - ts.push(t.elapsed()); - last = Some(v); - } - ts.sort(); - (ts[REPS / 2], last.unwrap()) -} - -fn us(d: Duration) -> f64 { - d.as_secs_f64() * 1e6 -} - -fn main() { - println!("bf16 tier: {} | release, medians of {REPS}", tier()); - println!( - "case, edges, triangles, transpose_us, bit_compose_us, bit_counts_us, bit_triangles_us, bf16_stage_us, bf16_square_packed_us, bf16_square_rowmajor_us, threshold_us, bf16_triangles_us, bit_bytes, bf16_bytes" - ); - for (label, p, hubs, seed) in [ - ("1%", 10u32, 0usize, 1u64), - ("10%", 100, 0, 2), - ("50%", 500, 0, 3), - ("1% + 4 hubs", 10, 4, 4), - ] { - let a = fixture(p, hubs, seed); - let want_sq = oracle_square(&a); - let want_t = oracle_triangles(&a); - - let (t_tr, at) = time(|| black_box(&a).transpose()); - let (t_bc, c) = time(|| black_box(&a).compose(black_box(&at), black_box(&a))); - let mut want_bits = BitMat::empty(); - for i in 0..N { - for j in 0..N { - if want_sq[i * N + j] > 0 { - want_bits.set(i, j); - } - } - } - assert_eq!(c, want_bits); - let (t_bq, bq) = time(|| black_box(&a).square_counts(black_box(&at))); - assert_eq!(bq, want_sq); - let (t_bt, tri) = time(|| black_box(&a).triangles(black_box(&at))); - assert_eq!(tri, want_t); - - let (t_st, staged) = time(|| Bf16Staged::stage(black_box(&a))); - let mut sq = vec![0f32; N * N]; - let (t_sq, _) = time(|| { - black_box(&staged).square(black_box(&mut sq)); - }); - assert!(sq.iter().zip(&want_sq).all(|(g, &w)| *g == w as f32)); - let mut sq2 = vec![0f32; N * N]; - let (t_sq2, _) = time(|| { - black_box(&staged).square_rowmajor(black_box(&mut sq2)); - }); - assert_eq!(sq2, sq); - let (t_th, cb) = time(|| threshold(black_box(&sq))); - assert_eq!(cb, want_bits); - let (t_gt, tri2) = time(|| triangles_from_square(black_box(&a), black_box(&sq))); - assert_eq!(tri2, want_t); - - println!( - "{label}, {}, {want_t}, {:.1}, {:.1}, {:.1}, {:.1}, {:.1}, {:.1}, {:.1}, {:.1}, {:.1}, {}, {}", - a.edges(), - us(t_tr), - us(t_bc), - us(t_bq), - us(t_bt), - us(t_st), - us(t_sq), - us(t_sq2), - us(t_th), - us(t_gt), - 2 * N * N / 8, - staged.bytes() - ); - } -} diff --git a/crates/mork-pathmap-probe/src/lib.rs b/crates/mork-pathmap-probe/src/lib.rs index 4101bd14f..14e593970 100644 --- a/crates/mork-pathmap-probe/src/lib.rs +++ b/crates/mork-pathmap-probe/src/lib.rs @@ -26,7 +26,6 @@ pub mod boundary; pub mod join; pub mod reference; pub mod rmap; -pub mod tile_gemm; pub use boundary::{Backend, BoundIndex, Refusal, bind, capability}; diff --git a/crates/mork-pathmap-probe/src/tile_gemm.rs b/crates/mork-pathmap-probe/src/tile_gemm.rs deleted file mode 100644 index 63535f4bf..000000000 --- a/crates/mork-pathmap-probe/src/tile_gemm.rs +++ /dev/null @@ -1,315 +0,0 @@ -//! D-RMAP-2 tile arm: Boolean relationship composition and triangle counting -//! over one 256 × 256 focus map, as a bit matrix and as a BF16 16 × 16 tile -//! GEMM. -//! -//! STATUS: PROBE. Results only. -//! -//! - **Bit arm.** `A` is 256 rows of 256 bits (8 KiB). Composition -//! `C = A ∘ B` is the outer-product form: for every coordinate `k`, each row -//! `i` whose bit `k` is set ORs in row `k` of `B`. Every row is addressed by -//! its own coordinate (`i`, `k`); no position is taken from a stored value. -//! - **GEMM arm.** `A` staged as dense BF16 (0 or 1.0, 128 KiB), multiplied by -//! `ndarray::simd::bf16_tile_gemm_16x16_packed` in 16 × 16 output blocks with -//! `K = 256`, then thresholded. For 0/1 operands the products are exact, and -//! every count is at most 256, far below 2^24, so `count > 0` equals Boolean -//! composition (D-RMAP-2 §3). The counts themselves are walk counts, which -//! the bit arm cannot give without popcounts. -//! -//! Triangles: `T = Σ_{i,j,k} A[i,j]·A[j,k]·A[k,i]`, the directed 3-cycles -//! counted once per rotation. Both arms compute it, against a triple-loop -//! oracle. Nothing here applies to weighted, negated or evidence-bearing -//! relations; that is the boundary D-RMAP-2 §3 states. - -use ndarray::simd::{ - PackedBf16B, bf16_tile_gemm_16x16, bf16_tile_gemm_16x16_packed, bf16_tile_gemm_tier, -}; - -/// Focus side. -pub const N: usize = 256; -const W: usize = N / 64; -const BF16_ONE: u16 = 0x3F80; - -/// A Boolean relation over one focus: row `i`, bit `j` set when `A[i, j]`. -#[derive(Clone, Debug, PartialEq, Eq)] -pub struct BitMat { - rows: Vec<[u64; W]>, -} - -impl BitMat { - pub fn empty() -> Self { - BitMat { - rows: vec![[0; W]; N], - } - } - - pub fn from_edges(edges: impl IntoIterator) -> Self { - let mut m = Self::empty(); - for (i, j) in edges { - m.set(i as usize, j as usize); - } - m - } - - pub fn set(&mut self, i: usize, j: usize) { - self.rows[i][j >> 6] |= 1 << (j & 63); - } - - pub fn get(&self, i: usize, j: usize) -> bool { - self.rows[i][j >> 6] >> (j & 63) & 1 == 1 - } - - pub fn row(&self, i: usize) -> &[u64; W] { - &self.rows[i] - } - - pub fn edges(&self) -> usize { - self.rows - .iter() - .flatten() - .map(|w| w.count_ones() as usize) - .sum() - } - - pub fn transpose(&self) -> BitMat { - let mut t = Self::empty(); - for i in 0..N { - for j in 0..N { - if self.get(i, j) { - t.set(j, i); - } - } - } - t - } - - /// `C = A ∘ B` (Boolean), outer-product form over the transpose of `A`: - /// for each `k`, the rows `i` with `A[i, k]` OR in `B[k]`. - pub fn compose(&self, at: &BitMat, b: &BitMat) -> BitMat { - let mut c = Self::empty(); - for k in 0..N { - let bk = b.rows[k]; - for (wi, &word) in at.rows[k].iter().enumerate() { - let mut w = word; - while w != 0 { - let i = wi * 64 + w.trailing_zeros() as usize; - let ci = &mut c.rows[i]; - for x in 0..W { - ci[x] |= bk[x]; - } - w &= w - 1; - } - } - } - c - } - - /// Walk counts `(A · B)[i, j] = popcount(row_i(A) ∧ row_j(Bᵀ))`, the bit - /// arm's answer to the counts GEMM gives. Row-major 256 × 256. - pub fn square_counts(&self, bt: &BitMat) -> Vec { - let mut c = vec![0u32; N * N]; - for i in 0..N { - let ri = self.rows[i]; - for j in 0..N { - let cj = bt.rows[j]; - let mut n = 0u32; - for x in 0..W { - n += (ri[x] & cj[x]).count_ones(); - } - c[i * N + j] = n; - } - } - c - } - - /// Directed 3-cycles: for every edge `(i, j)`, the `k` with `A[j, k]` and - /// `A[k, i]` is `row_j(A) ∧ row_i(Aᵀ)`; popcount and sum. - pub fn triangles(&self, at: &BitMat) -> u64 { - let mut t = 0u64; - for i in 0..N { - let ti = at.rows[i]; - for (wj, &word) in self.rows[i].iter().enumerate() { - let mut w = word; - while w != 0 { - let j = wj * 64 + w.trailing_zeros() as usize; - let rj = self.rows[j]; - for x in 0..W { - t += u64::from((rj[x] & ti[x]).count_ones()); - } - w &= w - 1; - } - } - } - t - } -} - -/// `A` staged for the GEMM: row-major BF16, plus each 16-column block of `A` -/// as a packed `B` operand (`K = 256`). -pub struct Bf16Staged { - a: Vec, - b_blocks: Vec, - /// The same blocks row-major, for the polyfill entry. On a host without - /// AMX or `avx512bf16`, the packed entry unpacks (and allocates) per call - /// before taking the polyfill, so the row-major entry is that tier's - /// honest cost. - b_rows: Vec>, -} - -impl Bf16Staged { - pub fn stage(m: &BitMat) -> Self { - let mut a = vec![0u16; N * N]; - for i in 0..N { - for j in 0..N { - if m.get(i, j) { - a[i * N + j] = BF16_ONE; - } - } - } - let b_rows: Vec> = (0..N / 16) - .map(|bj| { - let mut blk = vec![0u16; N * 16]; - for k in 0..N { - blk[k * 16..k * 16 + 16] - .copy_from_slice(&a[k * N + bj * 16..k * N + bj * 16 + 16]); - } - blk - }) - .collect(); - let b_blocks = b_rows.iter().map(|b| PackedBf16B::pack(b, N)).collect(); - Bf16Staged { - a, - b_blocks, - b_rows, - } - } - - /// Bytes held: the BF16 matrix and the packed blocks. - pub fn bytes(&self) -> usize { - self.a.len() * 2 - + self - .b_blocks - .iter() - .map(|b| b.data().len() * 2) - .sum::() - } - - /// Walk counts through the row-major polyfill entry (no pack, no unpack). - pub fn square_rowmajor(&self, c: &mut [f32]) { - assert_eq!(c.len(), N * N); - let mut blk = [0f32; 256]; - for bi in 0..N / 16 { - let a_rows = &self.a[bi * 16 * N..(bi + 1) * 16 * N]; - for (bj, b) in self.b_rows.iter().enumerate() { - blk.fill(0.0); - bf16_tile_gemm_16x16(a_rows, b, &mut blk, N); - for r in 0..16 { - let at = (bi * 16 + r) * N + bj * 16; - c[at..at + 16].copy_from_slice(&blk[r * 16..r * 16 + 16]); - } - } - } - } - - /// Walk counts `C = A · A` in f32, row-major 256 × 256, through the - /// tier-dispatched packed entry. - pub fn square(&self, c: &mut [f32]) { - assert_eq!(c.len(), N * N); - let mut blk = [0f32; 256]; - for bi in 0..N / 16 { - let a_rows = &self.a[bi * 16 * N..(bi + 1) * 16 * N]; - for (bj, b) in self.b_blocks.iter().enumerate() { - blk.fill(0.0); - bf16_tile_gemm_16x16_packed(a_rows, b, &mut blk); - for r in 0..16 { - let at = (bi * 16 + r) * N + bj * 16; - c[at..at + 16].copy_from_slice(&blk[r * 16..r * 16 + 16]); - } - } - } - } -} - -/// The tier ndarray will take on this host, for the report. -pub fn tier() -> &'static str { - bf16_tile_gemm_tier() -} - -/// Boolean composition from walk counts: `C[i, j] = counts[i, j] > 0`. -pub fn threshold(counts: &[f32]) -> BitMat { - let mut m = BitMat::empty(); - for i in 0..N { - for j in 0..N { - if counts[i * N + j] > 0.0 { - m.set(i, j); - } - } - } - m -} - -/// Triangles from walk counts: `Σ_{i,j} (A²)[j, i] · A[i, j]`. -pub fn triangles_from_square(m: &BitMat, sq: &[f32]) -> u64 { - let mut t = 0u64; - for i in 0..N { - for j in 0..N { - if m.get(i, j) { - t += sq[j * N + i] as u64; - } - } - } - t -} - -/// Independent oracle: a triple loop over the bits. -pub fn oracle_square(m: &BitMat) -> Vec { - let mut c = vec![0u32; N * N]; - for i in 0..N { - for k in 0..N { - if m.get(i, k) { - for j in 0..N { - if m.get(k, j) { - c[i * N + j] += 1; - } - } - } - } - } - c -} - -pub fn oracle_triangles(m: &BitMat) -> u64 { - let sq = oracle_square(m); - let mut t = 0u64; - for i in 0..N { - for j in 0..N { - if m.get(i, j) { - t += u64::from(sq[j * N + i]); - } - } - } - t -} - -/// A deterministic fixture: each edge with probability `p_permille / 1000`; -/// `hubs` rows (and columns) are dense, which gives a skewed degree -/// distribution. -pub fn fixture(p_permille: u32, hubs: usize, seed: u64) -> BitMat { - let mut s = seed; - let mut next = || { - s = s.wrapping_add(0x9E37_79B9_7F4A_7C15); - let mut z = s; - z = (z ^ (z >> 30)).wrapping_mul(0xBF58_476D_1CE4_E5B9); - z = (z ^ (z >> 27)).wrapping_mul(0x94D0_49BB_1331_11EB); - z ^ (z >> 31) - }; - let mut m = BitMat::empty(); - for i in 0..N { - for j in 0..N { - let hub = i < hubs || j < hubs; - if hub || (next() % 1000) < u64::from(p_permille) { - m.set(i, j); - } - } - } - m -} diff --git a/crates/mork-pathmap-probe/tests/tile_gemm.rs b/crates/mork-pathmap-probe/tests/tile_gemm.rs deleted file mode 100644 index 6e874d3ad..000000000 --- a/crates/mork-pathmap-probe/tests/tile_gemm.rs +++ /dev/null @@ -1,87 +0,0 @@ -//! The D-RMAP-2 tile arm against a triple-loop oracle. - -use mork_pathmap_probe::tile_gemm::{ - Bf16Staged, BitMat, N, fixture, oracle_square, oracle_triangles, threshold, - triangles_from_square, -}; - -const CASES: [(u32, usize, u64); 4] = [(10, 0, 1), (100, 0, 2), (500, 0, 3), (10, 4, 4)]; - -#[test] -fn bit_and_bf16_composition_equal_the_oracle() { - for (p, hubs, seed) in CASES { - let a = fixture(p, hubs, seed); - let at = a.transpose(); - let want = oracle_square(&a); - let mut want_bits = BitMat::empty(); - for i in 0..N { - for j in 0..N { - if want[i * N + j] > 0 { - want_bits.set(i, j); - } - } - } - assert_eq!(a.compose(&at, &a), want_bits, "bit arm p={p} hubs={hubs}"); - assert_eq!(a.square_counts(&at), want, "bit counts p={p} hubs={hubs}"); - let staged = Bf16Staged::stage(&a); - let mut sq = vec![0f32; N * N]; - staged.square(&mut sq); - // Exact walk counts, not only their support. - for (got, &w) in sq.iter().zip(&want) { - assert_eq!(*got, w as f32, "bf16 count p={p} hubs={hubs}"); - } - assert_eq!( - threshold(&sq), - want_bits, - "bf16 threshold p={p} hubs={hubs}" - ); - // Not degenerate: two hops reach something one hop does not. - assert_ne!(want_bits, a, "p={p}"); - } -} - -#[test] -fn triangle_counts_agree_across_arms() { - for (p, hubs, seed) in CASES { - let a = fixture(p, hubs, seed); - let at = a.transpose(); - let want = oracle_triangles(&a); - assert_eq!(a.triangles(&at), want, "bit arm p={p} hubs={hubs}"); - let mut sq = vec![0f32; N * N]; - Bf16Staged::stage(&a).square(&mut sq); - assert_eq!( - triangles_from_square(&a, &sq), - want, - "bf16 arm p={p} hubs={hubs}" - ); - } - // Silence twin: a DAG (edges only i < j) has no directed cycle. - let dag = BitMat::from_edges( - (0..N).flat_map(|i| (i + 1..(i + 4).min(N)).map(move |j| (i as u8, j as u8))), - ); - assert!(dag.edges() > 0); - assert_eq!(dag.triangles(&dag.transpose()), 0); - assert_eq!(oracle_triangles(&dag), 0); -} - -/// The counts carry multiplicity the Boolean result drops: some pair is -/// reached by more than one walk, so a threshold is a real loss of -/// information, not a no-op. -#[test] -fn walk_counts_exceed_one_where_support_does_not_show_it() { - let a = fixture(100, 0, 2); - let mut sq = vec![0f32; N * N]; - Bf16Staged::stage(&a).square(&mut sq); - assert!(sq.iter().any(|&c| c > 1.0)); -} - -/// An asymmetric relation, so a composition that forgets the transpose is -/// caught. -#[test] -fn composition_respects_direction() { - let a = BitMat::from_edges([(0, 1), (1, 2)]); - let c = a.compose(&a.transpose(), &a); - assert!(c.get(0, 2)); - assert!(!c.get(2, 0)); - assert_eq!(c.edges(), 1); -} From f29168110f5d9658b357d5e21250fca1c88910d9 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 20 Oct 2026 01:26:09 +0000 Subject: [PATCH 07/10] D-RST-3: record the petgraph#4 result Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VyWZYKycy6MoYDxHRVnV5K --- .claude/board/STATUS_BOARD.md | 2 +- .claude/plans/relstore-quack-petgraph-traversal-v1.md | 4 ++++ 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index 0c8846c38..7b4fa18eb 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -28,7 +28,7 @@ These PRs merged with no board rows. Every row below is a PROPOSAL (documents an | D-RST-0 | stored relations × Quack × petgraph: architecture and phased plan (`relstore-quack-petgraph-traversal-v1.md`) | PROPOSAL; documentation only | operator decision B; admissions A1–A5, blockers B1–B3 listed in §8 | | D-RST-1 | `DurableRef` / `ThoughtCommitId` contract types, never-reused rule (coordinated with D-CPG CPG-00) | Queued | reuse test; no duplicate with CPG-00 | | D-RST-2 | `RelationshipMapStore` skeleton: catalog, facts through the cycle seal, sorted permutation, `LayoutSeal`, domain as a relation, `impl Registrar` | Queued | F4 refusals (reseal, domain ABA, compaction, scope, dirty, deletion) | -| D-RST-3 | petgraph `RelView` over a borrowed, version-pinned pattern-CSR view in a new fork experiment (brought forward; isolated probe with temporary ownership) | Queued (no longer waits on D-RST-1/2) | F1: unmodified `Bfs`/`Dfs` equal the oracle; no edge copy (pointer-in-buffer check) | +| D-RST-3 | petgraph `RelView` over a borrowed, version-pinned pattern-CSR view in a new fork experiment (brought forward; isolated probe with temporary ownership) | In PR (AdaWorldAPI/petgraph#4) | F1 met: unmodified `Bfs`/`Dfs` equal a `DiGraph` oracle; view borrows the prepared buffers (pointer check), 0 allocations to build or walk; Quack one-hop reads the same `dst` buffer at the same version, one prepare; 7 disable runs red | | D-RST-4 | layouts in the store: base indexes at seal time, specialized layouts on demand; pattern-only `u32` CSR/CSC on BlasGraph storage types, bitmap row blocks through mask-risc, join table; capability table | Queued (production reads wait on A1) | F2: five front ends, one `RelationVersion`, identical set and bag answers | | D-RST-5 | derived relations: `materialize`, multiplicity, provenance, dependencies, invalidation, DRed | Queued | F4 deletion, F5 set/bag/path | | D-RST-6 | typed traversal vocabulary; multi-hop within a focus through stored intermediates | Queued (A1) | F3: no `Gather`/`ScatterOrU32` in any program; `NonFunctionalChain` lifted only in focus | diff --git a/.claude/plans/relstore-quack-petgraph-traversal-v1.md b/.claude/plans/relstore-quack-petgraph-traversal-v1.md index ac9179a75..9678f8037 100644 --- a/.claude/plans/relstore-quack-petgraph-traversal-v1.md +++ b/.claude/plans/relstore-quack-petgraph-traversal-v1.md @@ -253,6 +253,10 @@ BlasGraph already ships the relationship-matrix vocabulary this plan needs: CSR, | D-RST-7 | triangle and amortization | F6 and F7, cold and warm | D-RST-6 | | D-RST-8 | large-domain multi-hop | only after admission A2 | A2 | +**D-RST-3 result (2026-10-20, AdaWorldAPI/petgraph#4, `experiments/relstore-visit`).** The view met F1: petgraph's unmodified `Bfs`/`Dfs` match a `DiGraph` oracle built from the facts on fixtures covering parallel edges, self-loops, isolated nodes and empty relations. The edge id is the fact id, and it survives compaction while rows move. The view borrows the prepared buffers, and building it or walking every neighbour list allocates nothing. The F2b-shaped falsifier passes: a Quack one-hop reads the same `dst` buffer at the same version, and the store prepares once. + +Measured at 1M edges: the neighbour walk and BFS cost the same as petgraph `Csr` (2.0 vs 1.8–1.9 ns per edge). Prepare takes 308–396 ns per edge and is not optimized. Admission question for D-RST-1/2: the Quack hop is O(E) per query (14–18 ms for 64 hops vs 0.3 µs through the view), because Quack cannot know that the `src` lane is sorted. A sealed ordering witness (`Cmp::Range`) would make it a range read; whether the seal carries one is open. + Each PR keeps the probes in `mork-pathmap-probe` as probes. Production code is new code in the store crate with its own tests; it does not import the probe crate. ## 8. Open admission decisions and blockers From 6d44a4c3f396892b81e838256d110e3c164336f9 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 20 Oct 2026 01:31:49 +0000 Subject: [PATCH 08/10] D-RST: reframe around relational geometry; petgraph optional MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Operator correction (2026-10-20). The plan's center of gravity is reusable relational geometry amortized across front ends and thought versions, not a graph API: - petgraph becomes an optional consumer beside the Gremlin adapter; it leaves the execution path of the diagram, and its parity check becomes the optional falsifier F-PG. D-RST-3 is OPTIONAL / DEFERRED / MEASUREMENT-GATED; petgraph#4 is kept as evidence. - New D-RST-9 (critical path, first): native borrowed relation views with no petgraph, starting from the planner's existing AdjacencyStore / AdjacencyBatch rather than a fourth owned CSR type. - New §5b: tile GEMM consumes prepared geometry. Preparation, execution and maintenance are measured apart; break-even comes from measured values only; Boolean existence and BF16 multiplicity are kept apart, with the BF16 exactness bound. - New F8: four execution paths on one relation, cold and warm, 1/8/64 thought versions, sparse to hub distributions. - New admission A6: an attested sort order, without which a Quack one-hop scans the whole lane (measured in D-RST-3). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VyWZYKycy6MoYDxHRVnV5K --- .claude/board/INTEGRATION_PLANS.md | 10 + .claude/board/STATUS_BOARD.md | 9 +- ...6-10-20-relstore-geometry-not-graph-api.md | 19 ++ .claude/board/entries/README.md | 3 +- .../relstore-quack-petgraph-traversal-v1.md | 197 ++++++++++++------ 5 files changed, 168 insertions(+), 70 deletions(-) create mode 100644 .claude/board/entries/2026-10-20-relstore-geometry-not-graph-api.md diff --git a/.claude/board/INTEGRATION_PLANS.md b/.claude/board/INTEGRATION_PLANS.md index 14cd55958..69a61239c 100644 --- a/.claude/board/INTEGRATION_PLANS.md +++ b/.claude/board/INTEGRATION_PLANS.md @@ -1,3 +1,13 @@ +## 2026-10-20 — relstore-quack-petgraph-traversal-v1 (reframed) — compiled relational geometry and amortization; petgraph optional → `.claude/plans/relstore-quack-petgraph-traversal-v1.md` + +**Status:** D-RST-0..9 PROPOSAL; documentation only. Operator correction (2026-10-20): the goal is reusable relational geometry amortized across front ends and thought versions; petgraph is an optional consumer (D-RST-3 OPTIONAL / DEFERRED / MEASUREMENT-GATED); tile GEMM consumes prepared layouts and does not obtain them. + +- Critical path: D-RST-9 (native borrowed views, no petgraph) → D-RST-1 → 2 → 4 → 5 → 6 → 7. Off it: D-RST-3, D-RST-8 (A2), F-PG, F8 path D. +- New §5b: three costs (preparation, execution, maintenance) measured apart; break-even from measured values only; Boolean existence vs BF16 multiplicity, with the BF16 exactness bound (integers to 256). +- New F8: four execution paths on one relation, cold/warm, 1/8/64 thoughts, sparse to hub distributions. +- Prior art named: the planner's `AdjacencyStore`/`AdjacencyBatch` (pattern CSR + CSC, borrowed, no-`Copy`) is the starting point for D-RST-9; no fourth owned CSR type. +- Supersedes the entry below on framing only; its ownership decisions stand. + ## 2026-10-20 — relstore-quack-petgraph-traversal-v1 — stored relations, compiled traversal, petgraph over borrowed layouts → `.claude/plans/relstore-quack-petgraph-traversal-v1.md` **Status:** D-RST-0..8 PROPOSAL; documentation only. Operator decision B (2026-10-20): RelationshipMapStore owns persistence, layouts, domains, seals and invalidation; Quack compiles; mask-risc executes. D-RMAP-1A §1 and the 2026-10-05 storage-portability ruling are unchanged. diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index 7b4fa18eb..5b6cacbe9 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -25,15 +25,16 @@ These PRs merged with no board rows. Every row below is a PROPOSAL (documents an | D-RMAP-1A | separate map tables, borrowed projections, thought-selected palettes (`d-rmap-separate-tables-thought-palettes-v1.md`) | PROPOSAL; generation witnesses corrected 2026-10-19 (§6a) | refusal on map regeneration, palette-slot reassignment, focus-slot reuse, incarnation reuse, stale tile, cross-tenant change | | D-RMAP-1A-V | versioning crosswalk: map/palette/binding generations derived from D-TGA `ThoughtCommitId` + #1605 `DurableRef`; focus generation and index seal retained (`d-rmap-separate-tables-thought-palettes-v1.md` §7) | PROPOSED; no code, no type; falsifiers specified, none run | X1–X9 in §7.4 (cross-dataset, compaction, palette ABA, slot reuse, uncommitted change, rebase, authorization, squash+replay, deletion/DRed) | | D-RMAP-1A-P | focus-bound RelationshipMap probe over existing sorted/trie indexes: CSR, 256×256 tile via mask-risc, incremental rebind, thought deltas, place masks, postings (`d-rmap-separate-tables-thought-palettes-v1.md` §8; `crates/mork-pathmap-probe/src/rmap.rs`) | MEASURED, probe only; nothing adopted | 12 tests vs scan oracle; 11 guards disable-verified; break-even 76/11/1.8 reuses (sparse d2 / d8 / dense) | -| D-RST-0 | stored relations × Quack × petgraph: architecture and phased plan (`relstore-quack-petgraph-traversal-v1.md`) | PROPOSAL; documentation only | operator decision B; admissions A1–A5, blockers B1–B3 listed in §8 | +| D-RST-0 | stored relations × Quack × petgraph: architecture and phased plan (`relstore-quack-petgraph-traversal-v1.md`) | PROPOSAL; documentation only; reframed 2026-10-20 around relational geometry and amortization (petgraph optional, tile GEMM a consumer of prepared layouts) | operator decision B; admissions A1–A6, blockers B1–B3 listed in §8 | | D-RST-1 | `DurableRef` / `ThoughtCommitId` contract types, never-reused rule (coordinated with D-CPG CPG-00) | Queued | reuse test; no duplicate with CPG-00 | | D-RST-2 | `RelationshipMapStore` skeleton: catalog, facts through the cycle seal, sorted permutation, `LayoutSeal`, domain as a relation, `impl Registrar` | Queued | F4 refusals (reseal, domain ABA, compaction, scope, dirty, deletion) | -| D-RST-3 | petgraph `RelView` over a borrowed, version-pinned pattern-CSR view in a new fork experiment (brought forward; isolated probe with temporary ownership) | In PR (AdaWorldAPI/petgraph#4) | F1 met: unmodified `Bfs`/`Dfs` equal a `DiGraph` oracle; view borrows the prepared buffers (pointer check), 0 allocations to build or walk; Quack one-hop reads the same `dst` buffer at the same version, one prepare; 7 disable runs red | -| D-RST-4 | layouts in the store: base indexes at seal time, specialized layouts on demand; pattern-only `u32` CSR/CSC on BlasGraph storage types, bitmap row blocks through mask-risc, join table; capability table | Queued (production reads wait on A1) | F2: five front ends, one `RelationVersion`, identical set and bag answers | +| D-RST-3 | petgraph `RelView` over a borrowed, version-pinned pattern-CSR view in a new fork experiment (brought forward; isolated probe with temporary ownership) | OPTIONAL / DEFERRED / MEASUREMENT-GATED (operator, 2026-10-20); experiment AdaWorldAPI/petgraph#4 kept as evidence, not extended | then-F1 (now F-PG) met: unmodified `Bfs`/`Dfs` equal a `DiGraph` oracle; view borrows the prepared buffers (pointer check), 0 allocations to build or walk; Quack one-hop reads the same `dst` buffer at the same version, one prepare; 7 disable runs red | +| D-RST-4 | layouts in the store: base indexes at seal time, specialized layouts on demand; pattern-only `u32` CSR/CSC on BlasGraph storage types, bitmap row blocks through mask-risc, join table; capability table | Queued (production reads wait on A1; layouts from D-RST-9) | F2: four front ends (Quack, Cypher, Gremlin adapter, RelAlg/Datalog), one `RelationVersion`, identical set and bag answers | | D-RST-5 | derived relations: `materialize`, multiplicity, provenance, dependencies, invalidation, DRed | Queued | F4 deletion, F5 set/bag/path | | D-RST-6 | typed traversal vocabulary; multi-hop within a focus through stored intermediates | Queued (A1) | F3: no `Gather`/`ScatterOrU32` in any program; `NonFunctionalChain` lifted only in focus | -| D-RST-7 | triangle and amortization | Queued | F6 (existence/count/enumeration, sparse/dense/skewed), F7 measured crossovers | +| D-RST-7 | triangle and amortization | Queued | F6 (existence/count/enumeration, sparse/dense/skewed), F7 measured crossovers, F8 four-path amortization (A native iterator, B compiled sparse, C bit tiles, D GEMM via existing probe kernels) | | D-RST-8 | multi-hop over a large domain | BLOCKED on admission A2 | — | +| D-RST-9 | native borrowed relation views without petgraph: n-ary sorted permutations as ranges, pattern CSR/CSC as the binary case, stable fact ids, version pin, not `Copy`; starts from the planner's `AdjacencyStore`/`AdjacencyBatch` | Queued (critical path, first) | F1: oracle equality, zero copies, compaction-stable ids, stale and out-of-domain refusals | | D-MPS-8 | native join over a versioned relationship table: R and S laid out by `y` slot, leapfrog intersection as one mask-risc `And`, per-side generations and reseal, per-thought filter planes (`mork-pathmap-schnellverschluss-v1.md` §7b; `crates/mork-pathmap-probe/src/join.rs`) | MEASURED, probe only; not admitted (D-RMAP-1 BIND membrane review) | 8 join tests + native arms in the differential suite vs the oracle; 7 guards disable-verified; cyclic bodies open | | D-RMAP-2 | context-declaration reuse and GraphBLAS/BF16 tile probe (`d-rmap-2-context-tile-algebra-probe-v1.md`) | PROPOSAL; not run | arms 1–5 in the plan; 297× (D-GTM-0j) is not evidence | | D-SRAM-0 | inventory Soufflé AST/ast2ram/RAM passes against D-ASM/D-RA/D-TGA owners | Queued | evidence per mapped pass; no code imported | diff --git a/.claude/board/entries/2026-10-20-relstore-geometry-not-graph-api.md b/.claude/board/entries/2026-10-20-relstore-geometry-not-graph-api.md new file mode 100644 index 000000000..dd0fdd92c --- /dev/null +++ b/.claude/board/entries/2026-10-20-relstore-geometry-not-graph-api.md @@ -0,0 +1,19 @@ +# D-RST reframed: relational geometry first, petgraph optional, tile GEMM a consumer (2026-10-20) + +**Status:** PROPOSAL (D-RST-0..9). Plan: `.claude/plans/relstore-quack-petgraph-traversal-v1.md`. Documentation only. Supersedes the framing of `2026-10-20-relstore-quack-petgraph-plan.md`; its ownership decisions and findings stand. + +## Decision (operator, 2026-10-20, recorded as provenance) + +- Petgraph is an optional consumer, intermediate and usability reference, beside the Gremlin adapter. It decides no storage format, coordinate model, algebra, optimizer, execution ABI or lifecycle. D-RST-3 is OPTIONAL / DEFERRED / MEASUREMENT-GATED; petgraph parity is the optional falsifier F-PG. +- Tile GEMM accelerates arithmetic on prepared dense operands. It does not resolve identities, scatter, gather or traverse sparse data. +- Target: minimum repeated work per relational reasoning step across front ends and compatible thought versions. + +## Findings (read in source) + +- `lance-graph-planner/src/adjacency/` already holds a pattern-only CSR + CSC with edge ids and a borrowed `AdjacencyBatch` that is deliberately not `Clone`/`Copy` (ruling 2026-07-29). Gaps for D-RST-9: `u64` widths, edge id = arrival position (not a stable fact identity), no version pin, binary only. Core `CsrIndex` is a third owned CSR. No fourth is added. +- The D-RST-3 experiment's view (petgraph#4) is `Copy`, which that ruling forbids; petgraph's `GraphRef: Copy` is met instead by implementing the visit traits for `&View`. +- D-RST-3's measurements: a borrowed CSR walk costs about 2 ns per edge against 300–400 ns per edge of preparation; a Quack one-hop scans the whole lane (O(E)) because the sort order is not attested (admission A6). + +## Open + +- A1–A6, B1–B3 (plan §8). F8 (four-path amortization) is unmeasured; path D runs only through the existing D-RMAP-2 probe kernels. diff --git a/.claude/board/entries/README.md b/.claude/board/entries/README.md index 2686a7a2b..69387f64a 100644 --- a/.claude/board/entries/README.md +++ b/.claude/board/entries/README.md @@ -25,11 +25,12 @@ index row, (3) no duplicate entry id. Checks 1 and 2 are deliberately opposite directions; the stranding this convention prevents shows up in exactly one of them, never both. -375 entries, 2026-08-06 .. 2026-10-20. +376 entries, 2026-08-06 .. 2026-10-20. | date | entry id | finding | file | |---|---|---|---| | 2026-10-20 | `relstore-quack-petgraph-plan` | | [2026-10-20-relstore-quack-petgraph-plan.md](2026-10-20-relstore-quack-petgraph-plan.md) | +| 2026-10-20 | `D-RST` | | [2026-10-20-relstore-geometry-not-graph-api.md](2026-10-20-relstore-geometry-not-graph-api.md) | | 2026-10-19 | `post-merge-recovery-1601-1617` | | [2026-10-19-post-merge-recovery-1601-1617.md](2026-10-19-post-merge-recovery-1601-1617.md) | | 2026-10-19 | `native-join-relationship-table` | | [2026-10-19-native-join-relationship-table.md](2026-10-19-native-join-relationship-table.md) | | 2026-10-19 | `mork-pathmap-schnellverschluss` | | [2026-10-19-mork-pathmap-schnellverschluss.md](2026-10-19-mork-pathmap-schnellverschluss.md) | diff --git a/.claude/plans/relstore-quack-petgraph-traversal-v1.md b/.claude/plans/relstore-quack-petgraph-traversal-v1.md index 9678f8037..de3abd07d 100644 --- a/.claude/plans/relstore-quack-petgraph-traversal-v1.md +++ b/.claude/plans/relstore-quack-petgraph-traversal-v1.md @@ -1,9 +1,21 @@ -# D-RST: RelationshipMapStore × Quack × petgraph — stored relations, compiled traversal +# D-RST: compiled relational geometry and its amortization — RelationshipMapStore × RelAlg/Datalog × Quack + +(File name kept for existing links; the plan was first written as "RelationshipMapStore × Quack × petgraph". See the correction below.) **Status: PROPOSAL, 2026-10-20. Documentation only.** No crate, type, opcode, ABI, dependency or ruling change lands with this plan. Every capability below is labelled IMPLEMENTED, PROBE, PROPOSAL or RULING from its own source. **Decision taken (operator, 2026-10-20), recorded as provenance, not as validation:** option B. `RelationshipMapStore` owns relationship persistence, identities, index layouts, coordinate-domain seals and invalidation. Quack owns compilation and numeric query semantics. mask-risc on `ndarray::simd` stays the only population executor. This keeps the D-RMAP-1A §1 ownership table and the 2026-10-05 Quack storage-portability ruling as they are. Neither is superseded here. +**Correction (operator, 2026-10-20, later the same day), recorded as provenance, not as validation: the center of gravity is reusable relational geometry, not a graph API.** +1. **Petgraph is an optional consumer, an optional intermediate and a usability reference**, on the same footing as the Gremlin adapter. It does not decide the storage format, coordinate model, algebra, optimizer, execution ABI or relationship lifecycle. The first version of this plan made it a pillar ("petgraph makes it usable") and brought D-RST-3 forward. Both are withdrawn: D-RST-3 is OPTIONAL / DEFERRED / MEASUREMENT-GATED (§7), and petgraph parity is an optional interoperability falsifier (F-PG), not an acceptance criterion. +2. **Tile GEMM consumes prepared relationships; it does not obtain them.** It speeds up arithmetic once the operands are in a compatible dense layout. It does not resolve identities, scatter, gather or traverse sparse data (§5b). + +**Goal.** Write relationships once. Normalize irregular access into reusable relational geometry: identities, coordinate domains, index permutations, prepared layouts and version dependencies. Amortize that preparation across joins, projections, recursion and traversal, whichever front end asks (SQL-shaped Quack, Cypher, Datalog, RelAlg, Gremlin, petgraph) and across compatible thought versions. **The optimization target is the minimum repeated work per relational reasoning step, not maximum arithmetic throughput.** + +**Success criterion.** Datalog, RelAlg and Quack workloads run efficiently over retained relational layouts without importing petgraph or Gremlin. Petgraph-shaped iterators or selected petgraph algorithms may later be exposed over the same layouts, which would show usability, not decide the architecture. + +**Scope.** Relations are n-ary: Datalog predicates, variable permutations, join domains, derived relations, recursion and provenance. Graph traversal is one special case of relational composition (a binary relation composed with itself). Neither a graph API nor a GEMM kernel may change the representation chosen for a relation. + Read with: - [D-RMAP-1A](d-rmap-separate-tables-thought-palettes-v1.md) §§1, 3, 6–8: ownership, borrowed views, falsifiers, versioning, focus probe. - [Schnellverschluss](mork-pathmap-schnellverschluss-v1.md) §7b: the native join over a versioned relationship table. @@ -13,7 +25,7 @@ Read with: - Board entry `2026-10-05-quack-storage-portability.md`. - `.claude/knowledge/fold-execution-laws.md`. -**Short form:** the store remembers, BIND resolves once, Quack compiles, mask-risc executes, petgraph makes it usable. +**Short form:** the store remembers the geometry, BIND resolves once, RelAlg/Datalog define the bindings, Quack compiles, mask-risc executes. Graph APIs are optional views over the same layouts. **The amortization principle (operator, 2026-10-20): write a relationship as a relationship.** A relation is written once, as relation facts, and the seal fixes its identity, its version and the base indexes its declaration names (a sorted permutation, and CSR/CSC where declared). Specialized execution geometry (bitmap blocks, focus tiles, a co-addressed join domain on one shared variable) is prepared on the first demand that needs it, once, and retained under a dependency seal on that relation version. Nothing is rebuilt per query or per front end, and nothing is built for a query that never comes: a relation with several joinable attributes has many candidate join domains, and only the demanded ones exist. The seal establishes validity; preparation establishes geometry, and F7 measures whether it paid for itself. RelAlg's semijoins, joins and projections then read prepared relationship layouts instead of translating identities on every evaluation. That is where RelAlg gets faster: the translation is paid once per sealed version, the evaluation is a fold. §5a checks what the shipped BlasGraph CSR contributes to this. @@ -26,7 +38,7 @@ Read with: So the only admitted multi-hop shape today is the tile. Over a domain larger than a focus, multi-hop needs an admission decision (§8, A2). It is not an implementation gap a store can fill. Gremlin's `NonFunctionalChain` (`lance-graph-quack/tests/gremlin_parity.rs`, the `Refusal` enum: "needs a barrier and a workspace inside one computation. Gap: no typed phase handoff exists") is two problems. The handoff is the store's to give, as a stored intermediate. The hop is the ruling's to admit. 2. **The existing petgraph integration rests on the quarantined gather.** `AdaWorldAPI/petgraph` `experiments/lance-fold-hop` (fork PR #3) consumes `lance_graph_mask_risc::OutgoingHop`. That is two `Gather` ops plus `ScatterOrU32`, pinned to lance-graph `66636da`, from before #1616's quarantine. Its README already says it is one demanded hop, and that iterative BFS "needs a reviewed phase-handoff / continuation primitive". ISSUES.md records that bumping the pin must enable `quarantined-gather-hop` explicitly. This plan does not build on it. -3. **petgraph's traversal needs only two traits.** `Bfs::new` / `Dfs::new` need `GraphRef + Visitable`, and `next` needs `IntoNeighbors` (`crates/petgraph/src/visit/traversal.rs`). The visit map is a `FixedBitSet` keyed by `IndexType::index()` (`visit/mod.rs`, `impl VisitMap for FixedBitSet`), so a view whose `NodeId` is a dense `u32` slot gets petgraph's own visit map. `Csr` (`crates/petgraph/src/csr.rs`) shows the shape: `IntoNeighbors for &Csr` returns a slice iterator over `neighbors_slice`. It owns its `Vec`s, so the view must borrow instead. +3. **(Optional consumer; evidence for D-RST-3.) petgraph's traversal needs only two traits.** `Bfs::new` / `Dfs::new` need `GraphRef + Visitable`, and `next` needs `IntoNeighbors` (`crates/petgraph/src/visit/traversal.rs`). The visit map is a `FixedBitSet` keyed by `IndexType::index()` (`visit/mod.rs`, `impl VisitMap for FixedBitSet`), so a view whose `NodeId` is a dense `u32` slot gets petgraph's own visit map. `Csr` (`crates/petgraph/src/csr.rs`) shows the shape: `IntoNeighbors for &Csr` returns a slice iterator over `neighbors_slice`. It owns its `Vec`s, so the view must borrow instead. 4. **The Registrar seam is real, minimal and unowned.** `lance-graph-quack/src/bind.rs`: `trait Registrar { fn register(&mut self, name: &str, width: u32) -> Result; }`. The doc says "No crate owns that today; this trait names the seam, and only tests implement it". The only implementor is a `#[cfg(test)] Mem`. Three `Binder`s exist in library code: `lance-graph-sap` `CatsBinder`, `lance-graph-dir-sim` `UserBinder`, `lance-graph-glove-parity` `CanonicalBinder`. 5. **A coordinate domain is itself a relation.** The join probe needs a seal for its shared domain (Schnellverschluss §7b: a new `y` re-addresses every side). Stored as a relation (the dictionary of global ids, in slot order), the domain gets a `DurableRef` like any other table. No counter is added. A domain change is a new version, and versions are not reused, so domain ABA cannot happen. @@ -54,30 +66,36 @@ Read with: | petgraph visit traits, `Bfs`/`Dfs`/`Walker`, `Csr` | `AdaWorldAPI/petgraph` `crates/petgraph/src/{visit,csr.rs}` (0.8.3, MSRV 1.85, dev toolchain 1.99) | IMPLEMENTED upstream | | BlasGraph sparse storage: `CsrStorage`, `CscStorage`, `HyperCsrStorage`, COO→CSR, `GrBMatrix` (CSR + optional CSC), `mxv`/`vxm`/`mxm` over a `dyn Semiring` incl. `HdrSemiring::Boolean`; `StorageFormat::Bitmap` reserved | `lance-graph/src/graph/blasgraph/{sparse,matrix,semiring}.rs`; planner `blasgraph_planner.rs` (Cypher `Expand` → `mxm`) | IMPLEMENTED; a second copy in `holograph/src/graphblas` | | ndarray dense mask algebra: `mask_and`, `mask_or`, `mask_andnot`, `mask_ternlog`, popcount/any; dense BLAS levels 1–3 | `ndarray/src/simd_masking_ops.rs`, `src/hpc/blas_level{1,2,3}.rs` | IMPLEMENTED; no sparse/CSR primitive under `src/hpc` or `src/simd*` (searched `csr`, `spmv`, `sparse`) | +| Pattern CSR + CSC substrate with edge ids, one store per relationship type ("Kuzu pattern"); `adjacent`, `adjacent_incoming`, `edge_ids`, degrees, `from_edges`; a borrowed `AdjacencyBatch<'a>` (no allocation; deliberately not `Clone`/`Copy`, operator ruling 2026-07-29 "copies are forbidden, borrows are only for the same mailbox"); `intersect` returns an owned `IntersectionResult` | `lance-graph-planner/src/adjacency/{csr,batch,propagate,distance}.rs` | IMPLEMENTED; `u64` offsets/targets/ids, owned `Vec`s, no version pin, edge id = arrival position in the input (`from_edges`), so not a stable fact identity; binary only; used only inside its own module | +| `CsrIndex` (offsets + neighbours, `u64`), built from an Arrow `RecordBatch` | `lance-graph/src/csr_index.rs` | IMPLEMENTED; owned, out-edges only | +| Borrowed, version-pinned pattern-CSR view traversed by petgraph's unmodified `Bfs`/`Dfs`; Quack one-hop over the same buffers | `AdaWorldAPI/petgraph#4`, `experiments/relstore-visit` | PROBE (D-RST-3, now deferred); its view is `Copy`, which the 2026-07-29 ruling forbids (§3.3) | +| BF16 16 × 16 tile GEMM vs bit-tile composition, polyfill tier only | `mork-pathmap-probe/src/tile_gemm.rs` (#1623, D-RMAP-2) | PROBE | | `RelationshipMapStore`, `RelationCatalog` | — | not found under `crates/` (searched `*.rs`, `*.md`) | +**Prior art to reuse before adding a type.** Three owned CSR families already exist: the planner's `AdjacencyStore` (CSR **and** CSC, pattern-only, with a borrowed batch view that already follows the no-`Copy` ruling), core `CsrIndex`, and BlasGraph's valued `CsrStorage`/`CscStorage` (plus the `holograph` copy). The native borrowed view of §3.1 starts from `AdjacencyStore`/`AdjacencyBatch`: it already has the shape (pattern CSR + CSC + ids, borrowed batch). Its gaps are `u32` slots, a stable fact identity instead of the arrival position, a version pin, and n-ary relations as sorted permutations. No fourth owned CSR type is added; the probe CSRs (`rmap.rs`, `relstore-visit`) stay probes. Masked adjacency is already a presence plane in mask-risc (`Planes.masks`) and the probe tile (`rmap.rs`, 256 × 256); no new tile type is needed for the Boolean path. + ## 2. Ownership and seams ```text - SQL-shaped quack::Query Cypher Gremlin Datalog/RelAlg petgraph algorithms - \ | | / | - \______________ PARSE ___/____________/ | - | | - Quack COMPILE (V4: what to compute) | - | | - BIND: identities, coordinate domain, authorization, version, | - layout capabilities — resolved once ─────────────────────┤ - | | - BUNDLE: choose layout (may not change the answer) | - | | - mask-risc / ndarray::simd EXECUTE over borrowed planes | - | | - RelationshipMapStore: catalog · facts · derived relations · | - orders · layouts · domains · seals · invalidation ◄── borrowed views ─┘ - | - cycle seal → Lance (DurableRef) ThoughtCommitId (speculative) - | - SoA: canonical identity-addressed state (separate admission) + Quack query Cypher Datalog / RelAlg optional consumers + \ | / (petgraph, Gremlin adapter) + \______ PARSE ______/ : + | : + Quack COMPILE (V4: what to compute) : + | : + BIND: identities, coordinate domain, authorization, version, : + eligible prepared layouts — resolved once : + | : + BUNDLE: choose a prepared layout (may not change the answer) : + | : + mask-risc / ndarray::simd EXECUTE over borrowed planes : + | : + RelationshipMapStore: catalog · facts · derived relations · : + permutations · layouts · domains · seals · invalidation ◄········ borrowed views of the + | same layouts (no copy, + cycle seal → Lance (DurableRef) ThoughtCommitId no second graph) + | + SoA: canonical identity-addressed state (separate admission) ``` | Owner | Owns | Must not own | @@ -87,7 +105,7 @@ Read with: | Quack | compiling relational and graph operations into `Program`s; numeric query semantics; the `Registrar`/`Binder` seams | storage, catalog state, evaluation | | RelAlg / Datalog | variable bindings, join bodies, set vs bag, recursion and fixpoint meaning | map tables, index builds | | mask-risc / `ndarray::simd` | population execution | anything else | -| petgraph | algorithm and iterator vocabulary | storage, a second graph copy | +| petgraph, Gremlin adapter (optional consumers) | their algorithm and iterator vocabulary, where a measured workload adopts it | storage format, coordinate model, algebra, optimizer, execution ABI, relationship lifecycle, a second graph copy | | SoA | canonical identity-addressed state; admission of stable semantic relationships | derived relations (admission is optional and separate) | | cycle seal / Lance | durable writes, `DurableRef` | relational semantics | @@ -124,38 +142,26 @@ store.materialize(compiled_result, deps: &[RelationVersion]) -> RelationVersion Durable versions come from the seal (`DurableRef`); speculative ones from a thought commit (`ThoughtCommitId`). The store records which versions a layout or derived relation was built from and refuses a read when any of them differs. It persists no counter of its own (D-RMAP-1A §7.7, option B). -## 3. The petgraph contract: one identity, two consumption shapes +## 3. Relational access: borrowed views and compiled cursors -**Shape 1, scalar, unmodified petgraph.** A borrowed view type over one sealed CSR layout: +### 3.1 Native borrowed relation views (critical path) -```rust -#[derive(Clone, Copy)] -pub struct RelView<'s> { seal: &'s LayoutSeal, off: &'s [u32], dst: &'s [u32], n: usize } - -impl GraphBase for RelView<'_> { type NodeId = Slot; type EdgeId = EdgeRow; } // Slot: IndexType over u32 -impl<'s> IntoNeighbors for RelView<'s> { type Neighbors = SlotIter<'s>; /* slice of dst[off[a]..off[a+1]] */ } -impl Visitable for RelView<'_> { type Map = FixedBitSet; /* capacity n */ } -impl NodeIndexable / NodeCount / NodeCompactIndexable // the slot domain is dense -impl IntoEdgeReferences / IntoEdges / Data // edge id = stable fact identity (see below) -impl IntoNeighborsDirected // only when the store holds the reverse permutation; - // otherwise unimplemented, never computed on demand -``` +A relation version is read through a **view**: a borrow of the store's prepared buffers, pinned to the `RelationVersion` and `LayoutSeal` they were built from, refused for any other. A view owns nothing and copies nothing. Following the 2026-07-29 ruling, a view is **not** `Clone` or `Copy` (the shape `AdjacencyBatch` already has); it is passed by reference, so its reach is bounded by the borrow it came from. -- Petgraph's `Bfs`, `Dfs`, `DfsPostOrder` and `Topo` then run unmodified. Nothing is copied: the iterator is a slice of the stored layout. -- `EdgeId` is the stable **fact identity**, so a path can be traced back to facts. A canonical row offset is a stable edge id only where the relation's row-address contract guarantees it (Lance stable row ids, opt-in per dataset; D-RMAP-1A §7.3 measured 3 510 of 3 510 row addresses moved by compaction without them). Otherwise the row offset is only the physical address the iterator reads at, and the edge id is the fact identity stored beside it. -- Out-of-focus or out-of-domain neighbours are a property of the layout. A view over a full-domain CSR has none; a view over a focus layout must report escapes and refuse `global()` reads (D-RMAP-1A §8). +- **n-ary first.** A relation's facts are columns, one per argument, plus a stable fact identity. Its base indexes are sorted permutations in the variable orders its declaration names (for `r(x, y, z)`: `(x, y, z)`, `(y, x, z)`, …), read as ranges. +- **CSR and CSC are the binary specialization**: offsets over a sorted permutation. CSR serves push and neighbour iteration, CSC serves pull (§5a); both may be declared. +- **The fact identity is the edge id.** A physical row offset is an address that compaction moves (D-RMAP-1A §7.3); a view reports it separately and never as an identity. +- **Start from what exists.** The planner's `AdjacencyStore`/`AdjacencyBatch` already has CSR + CSC + ids and a borrowed batch view (§1). The gaps are `u32` slots, a stable fact identity, a version pin and n-ary permutations, plus BlasGraph's pattern-only reading (§5a). Whether that becomes a change to `AdjacencyStore` or a view beside it is decided by D-RST-9, not here. -**Classification (admission A3).** Shape 1 is ordinary host iteration inside petgraph's algorithm. It is not population execution and does not go through mask-risc, so the no-gather ruling, which governs fold execution, does not decide it. This plan records that reading as an open admission item rather than assuming it. +### 3.2 Compiled relational cursors -**Shape 2, batched relational cursors.** The same `RelationVersion` and `LayoutSeal`, consumed as whole spans and planes: -- CSR spans per slot range; -- tile planes and frontier masks (the only admitted multi-hop shape, finding 1); -- join-table presence planes (intersection = `And`); -- index intersections over sorted permutations. +The same `RelationVersion` and `LayoutSeal`, consumed as whole spans and planes, compiled by Quack: +- index ranges over sorted permutations (a range read needs the seal to attest the order; §8 A6); +- CSR/CSC spans per slot range; +- semijoins, joins and projections over a co-addressed join domain (join-table presence planes, intersection = `And`; Schnellverschluss §7b); +- tile planes and frontier masks (the only admitted multi-hop shape, finding 1). -These are Quack inputs, compiled to `Program`s. - -**Compiling iterator chains.** Arbitrary `Iterator::filter` / `flat_map` closures cannot be inspected and will not be compiled or fused. The plan instead proposes a small typed traversal vocabulary that is both an ordinary `Iterator` (terminal `.iter()`) and compilable (terminal `.compile(&quack)`): +**Typed traversal vocabulary.** Arbitrary `Iterator::filter` / `flat_map` closures cannot be inspected and are not compiled or fused. The plan instead proposes a small typed vocabulary that compiles (terminal `.compile(&quack)`) and can also run as an ordinary iterator (terminal `.iter()`): | Typed step | Meaning | Compiles to | |---|---|---| @@ -167,7 +173,17 @@ These are Quack inputs, compiled to `Program`s. | `repeat_bounded(k)` / `until_fixpoint()` | iteration | k tile hops / hop until the plane stops changing, within focus | | `emit()` | materialize | explicit terminal; the only exit producing ids | -Each step carries set/bag/path semantics in its type. Mixing them without an explicit `dedup` or `paths` step is a compile error, never a silent collapse (Gremlin's `BagOfElementsNotAMask` and `PathMultiplicity` stay real refusals). A chain that cannot compile refuses at `.compile`. It never falls back to `.iter()` on its own (no silent fallback). This keeps Gremlin's step vocabulary without a traverser VM: the traversal is a value that is compiled, not an object stepped per traverser. +Each step carries set/bag/path semantics in its type. Mixing them without an explicit `dedup` or `paths` step is a compile error, never a silent collapse (Gremlin's `BagOfElementsNotAMask` and `PathMultiplicity` stay real refusals). A chain that cannot compile refuses at `.compile`. It never falls back to `.iter()` on its own (no silent fallback). The traversal is a value that is compiled, not an object stepped per traverser, so no traverser VM is needed. + +### 3.3 Optional: petgraph over the same views (deferred, measurement-gated) + +Petgraph's `Bfs`/`Dfs` need `GraphRef + Visitable` and `IntoNeighbors` (finding 3). Over a §3.1 view they can run unmodified, with the iterator a slice of the stored layout and `EdgeId` the fact identity. AdaWorldAPI/petgraph#4 showed this works (§7, D-RST-3 result). It is not on the critical path, and nothing in §§3.1–3.2 depends on it. + +Two constraints for any later adoption: +- **`GraphRef: Copy` conflicts with the no-`Copy` view.** petgraph#4's view is `Copy`, which the 2026-07-29 ruling forbids. The compatible shape implements the visit traits for `&View` (petgraph's blanket `impl GraphRef for &G`), so the copyable thing is a shared reference and the view itself stays unique. +- **`IntoNeighborsDirected`** only where the store holds CSC; never computed on demand. + +Admission A3 (whether host iteration sits outside the no-gather ruling) only needs an answer if petgraph or another scalar iterator is adopted. ## 4. Execution flow @@ -206,7 +222,7 @@ BlasGraph already ships the relationship-matrix vocabulary this plan needs: CSR, 4. **`mxv` is a pull with an indexed read.** For each row `i`, for each `(j, a_val)` in the row: `vec.get(j)`. That reads the input vector at positions taken from stored column indices, the gather shape, run as scalar host code with a virtual `Semiring` call and a `BitVec` clone per product. 5. **ndarray is not on this path.** BlasGraph routes Hamming and popcount through `ndarray_bridge::dispatch_hamming`, but CSR construction and `mxv` use plain loops. ndarray has dense mask algebra and dense BLAS, and no sparse primitive. -**The BF16 16 × 16 tile GEMM is measured separately** (D-RMAP-2 probe, #1623). On a host without AMX or AVX512-BF16 only ndarray's F32x16 polyfill tier runs; there the bit arm wins every cell, and that result says nothing about tile-hardware performance. The GEMM stays a candidate layout for tile hosts and weighted counts until one is measured. +**The BF16 16 × 16 tile GEMM is measured separately** (D-RMAP-2 probe, #1623). On a host without AMX or AVX512-BF16 only ndarray's F32x16 polyfill tier runs; there the bit arm wins every cell, and that result says nothing about tile-hardware performance. The GEMM stays a candidate execution path for tile hosts and weighted counts until one is measured; §5b sets the terms on which it can win. **What follows.** - **Structural and valued relations are different things, and both stay.** BlasGraph's `HdrSemiring::Boolean` is bitwise AND/OR over 16 384-bit hyperdimensional values, not edge existence. A structural relation's value is the edge's existence (plus an optional fact identity), with no 2 KiB payload. A valued relation keeps BlasGraph's hyperdimensional vectors, or another typed edge value, read only when the computation needs it. The lean CSR is a pattern-only interpretation inside the same sparse-storage family; it does not replace BlasGraph's hyperdimensional algebra. @@ -224,20 +240,67 @@ BlasGraph already ships the relationship-matrix vocabulary this plan needs: CSR, - **Sparse rows still need an indexed read.** A sparse row lists column indices; any hop over it reads the frontier at those indices. Hypersparse bitmap blocks (a list of `(block, 64-bit word)`) make that read word-granular instead of bit-granular: one indexed read per 64 columns instead of one per column, and every read counted. That is the concrete content of admission A2 option (a)/(b); this plan does not admit it. - **Duplication:** `holograph/src/graphblas` carries a second GraphBLAS. The store binds to one (the `lance-graph` BlasGraph types) and records the other as debt, not a second consumer. +## 5b. Tile GEMM consumes prepared geometry (operator correction, 2026-10-20) + +```text +global relationship facts sparse · irregular identities · CSR / CSC / trie + │ + gather / resolve / pack select coordinates, build dense tiles, align shared domains + │ ← possibly expensive preparation + tile GEMM / mask-risc repeated, regular arithmetic on prepared operands + │ + relational results counts · existence · frontiers · derived relations +``` + +A GEMM or AMX kernel loads its operands from memory, so they need not be in registers already, but they must already be laid out as tiles. Every gather, pack or coordinate normalization happens before the kernel, and it is paid there. If a traversal gathers and packs each time it runs, even a very fast tile kernel can lose to an ordinary CSR iterator. That is the case for retaining prepared tiles under a versioned seal in the store (§4 step 2). + +**Three costs, measured separately for every tile accelerator:** +1. **Preparation**: identity resolution, coordinate alignment, index traversal, gathering, packing, tile construction, sealing. +2. **Execution**: mask-risc bit operations, SIMD, or BF16/AMX GEMM on prepared operands. +3. **Maintenance**: version checks, invalidation, incremental updates, repacking, rebuilds. + +For `N` uses and `M` invalidations of one prepared layout: + +```text +T_tile(N, M) = T_prepare + N · T_exec + M · T_maintain +T_sparse(N) = N · T_traverse (CSR/CSC or a native iterator, no tile) +``` + +The tile wins only where `T_exec < T_traverse`, and then only beyond the reuse count where the two lines cross. Break-even counts are reported from measured `T_prepare`, `T_exec`, `T_maintain` and `T_traverse`, never from an assumed kernel speedup. A warm tile kernel compared with a cold sparse traversal is not an end-to-end result. A faster isolated GEMM kernel does not establish a faster graph operation. + +**Representations keep their roles:** + +| Technique | Best role | +|---|---| +| borrowed pattern CSR/CSC, sorted permutations, native cursors | sparse, irregular navigation and range reads | +| sparse bitmap blocks | indexed word-level access, every indexed read counted | +| dense predecessor bitmaps (CSC direction) | gather-free, co-addressed frontier tests | +| mask-risc / ndarray bit tiles | dense Boolean relationship composition | +| BF16 / AMX tile GEMM | dense arithmetic composition, where density, reuse and precision justify it | +| RelationshipMapStore | retaining prepared layouts so their cost is paid across reuse | + +Sparse data is not forced into dense tiles in order to use GEMM. Structural Boolean adjacency stays distinct from BlasGraph's hyperdimensional `BitVec` values (§5a). + +**Boolean and arithmetic are different questions.** Reachability and existence need only `And`/`Or`; a BF16 product computes path multiplicities. Thresholding a count to existence is an explicit derivation step, never an implicit reading. BF16 operands carry an 8-bit significand, so they represent integers exactly only up to 256. The F32 accumulator is exact up to 2^24. Any arithmetic path through BF16 checks its counts against those bounds, and any count it cannot represent exactly is refused. Polyfill (F32x16) measurements and native AMX/AVX512-BF16 measurements are reported apart; this host has neither instruction set (#1623). + +Evidence so far: on the polyfill tier the bit arm wins every D-RMAP-2 cell (#1623). D-RST-3 measured a borrowed CSR walk at about 2 ns per edge against a preparation of 300–400 ns per edge (§7), which is exactly the cost the store exists to amortize. + ## 6. Falsifiers (exact; each guard gets a disable run) | # | Test | Pass condition | Disable that must turn it red | |---|---|---|---| -| F1 | unmodified `Bfs`, `Dfs`, `DfsPostOrder` over `RelView` on one sealed CSR | visited order and set equal petgraph over an independently built `Graph` (test-only oracle); zero edge copies, checked by asserting the iterator's slice pointer lies inside the layout buffer; edge ids survive a compaction of the fact rows (stable fact identity, not the row offset) | build the view from a copied `Vec` (the pointer check fails); drop a neighbour; use the row offset as the edge id (compaction changes it) | -| F2 | one stored relation read by a Quack query, Cypher, the Gremlin adapter, a RelAlg/Datalog rule and petgraph | identical ids and exact counts in set and bag semantics, against a nested-loop oracle; same `RelationVersion` cited by all five | change one layout so it answers set where bag was asked | -| F2b | hypothesis: a petgraph BFS frontier, a Gremlin `out()` expansion and a Datalog recursive step over the same relation compile to compatible demands and reuse one prepared layout, without sharing an execution model | the second and third front ends find the layout prepared by the first (prepare count stays 1 across the three, asserted), answers equal their oracles | give each front end its own layout key (prepare count goes to 3) | +| F1 | native borrowed views (§3.1), no petgraph: range reads over sorted permutations of an n-ary relation, CSR and CSC spans of its binary projection | equal a nested-loop oracle built from the facts; zero copies (each returned slice lies inside the layout buffer); building a view allocates nothing; fact ids survive compaction while rows move; a stale version and an out-of-domain id are refused | build the view from a copied `Vec`; drop a row; use the row offset as the id; skip the version check | +| F2 | one stored relation read by a Quack query, Cypher, the Gremlin adapter and a RelAlg/Datalog rule | identical ids and exact counts in set and bag semantics, against a nested-loop oracle; same `RelationVersion` cited by all four | change one layout so it answers set where bag was asked | +| F2b | hypothesis: a Quack query, a Gremlin `out()` expansion and a Datalog recursive step over the same relation compile to compatible demands and reuse one prepared layout, without sharing an execution model | the second and third front ends find the layout prepared by the first (prepare count stays 1, asserted), answers equal their oracles | give each front end its own layout key (prepare count goes to 3) | | F3 | `out(r).in(r)` and `repeat_bounded(3)` within a focus, over a stored intermediate | equals the oracle; no `Gather`, no `ScatterOrU32` in any `Program` (asserted on the op list); `NonFunctionalChain` lifted only for the tiled case; over a large domain it still refuses | pass the hop-1 result as a raw mask instead of a stored intermediate with its seal (version check must refuse) | | F4 | reseal; domain ABA (same ids, new domain version); thought slot reuse; compaction without stable row ids; authorization revoked; dirty speculative state; deletion | each is a typed refusal naming the stale component, never a plausible answer (D-RMAP-1A X1–X9, §6a) | key any check on a bare integer or on content | | F5 | distinct neighbours vs parallel edges; repeated traversers (bag); path multiplicity; recursive reachability | set, bag and path answers stay distinct where the oracle says they differ; a bag→mask step refuses | collapse bag to set inside a step | | F6 | triangle `edge(X,Y), edge(Y,Z), edge(Z,X)`: existence, count, enumeration; sparse, dense, skewed (hub) graphs | equals an independent oracle; MORK/PathMap where supported (set only); within focus via tiles; larger domains report the refusal instead of an answer | enumerate from the existence mask alone (must miss multiplicity) | -| F7 | cold and warm; 1/8/64 thoughts; domain churn; incremental reseal; scalar `RelView` BFS vs batched tile frontier; shipped BlasGraph `mxv` vs pattern-only borrowed CSR vs bitmap rows through mask-risc, on one relation | measured crossover points, each reported with its command and repetition count; nothing derived by arithmetic is labelled measured | n/a (measurement), but every timed path asserts its answer first | +| F7 | cold and warm; 1/8/64 thoughts; domain churn; incremental reseal; shipped BlasGraph `mxv` vs pattern-only borrowed CSR vs bitmap rows through mask-risc, on one relation | measured crossover points, each reported with its command and repetition count; nothing derived by arithmetic is labelled measured | n/a (measurement), but every timed path asserts its answer first | +| F8 | **amortization**, one fixed relation, four paths: (A) native iterator over borrowed CSR; (B) compiled sparse traversal over CSR/CSC; (C) prepared Boolean bit tiles through mask-risc; (D) prepared arithmetic tiles through GEMM. Each cold and warm, then over repeated queries and 1/8/64 compatible thought versions; sparse, medium, dense and skewed (hub) distributions | reported per path: preparation time and bytes allocated, execution latency, index reads and copies, layout reuse count, invalidation/rebuild cost, the end-to-end crossover from §5b's two lines; exact semantic parity across A–D where the semantics coincide (existence for A–C, counts for D checked against an exact integer oracle and the BF16 bounds); BF16 polyfill and hardware numbers in separate columns | n/a (measurement); every timed path asserts its answer first, and a path that skips its preparation in the cold run is a defect | +| F-PG | **optional interoperability, only if petgraph is adopted (§3.3)**: unmodified `Bfs`, `Dfs` over a §3.1 view | visited order and set equal petgraph over an independently built `Graph`; zero copies; fact ids as edge ids; the view is not `Copy` | as F1 | -**F6 is not a gate for F1–F5 or F7.** The store and traversal capabilities stand on their own. +**F6 is not a gate for F1–F5, F7 or F8. F-PG gates nothing.** The store and traversal capabilities stand on their own. ## 7. Sequence: independently mergeable PRs @@ -246,16 +309,19 @@ BlasGraph already ships the relationship-matrix vocabulary this plan needs: CSR, | D-RST-0 | this plan | documentation; board rows | — | | D-RST-1 | identity types | `DurableRef`, `ThoughtCommitId` as contract types (D-CPG CPG-00 owns them; coordinate, don't duplicate) with a never-reused rule test | D-CPG agreement (B1) | | D-RST-2 | store skeleton | `RelationshipMapStore` crate: catalog, `RelationDecl`, facts through the cycle seal, sorted permutation layout, `LayoutSeal`, coordinate domain as a relation; `impl Registrar`; F4 version and domain refusals | D-RST-1 | -| D-RST-3 | petgraph shape 1, brought forward | `RelView` over a minimal borrowed, version-pinned pattern-CSR view in a new fork experiment (`experiments/relstore-visit`, isolated workspace, pinned rev), with explicit temporary ownership of its buffers (an isolated probe, not the store); replaces nothing in `lance-fold-hop`; F1 | the pattern-CSR view only (can run before D-RST-1/2) | -| D-RST-4 | layouts | pattern-only `u32` CSR/CSC on BlasGraph's storage types (§5a), bitmap row blocks through mask-risc (`StorageFormat::Bitmap`), and the join table from the probe, behind the store's capability table; base indexes at seal time, specialized layouts prepared on demand and retained under their seal; F2, F2b (Quack query, Cypher, petgraph first) | D-RST-2 | +| D-RST-9 | native borrowed relation views (critical path, first) | §3.1 views without petgraph: n-ary sorted permutations read as ranges, pattern CSR/CSC as the binary case, stable fact ids, version pin, not `Copy`; starting from the planner's `AdjacencyStore`/`AdjacencyBatch` and BlasGraph's pattern reading, not a new owned CSR; F1 | nothing (an isolated probe can run before D-RST-1/2) | +| D-RST-3 | **OPTIONAL / DEFERRED / MEASUREMENT-GATED**: petgraph over a borrowed view | built as an experiment before this correction (AdaWorldAPI/petgraph#4); kept as evidence, not extended; adoption only after D-RST-7 measures direct relational methods against petgraph-style iteration, and only for the workloads where it wins (F-PG) | D-RST-9, D-RST-7 | +| D-RST-4 | layouts | pattern-only `u32` CSR/CSC from D-RST-9 (§§3.1, 5a), bitmap row blocks through mask-risc (`StorageFormat::Bitmap`), and the join table from the probe, behind the store's capability table; base indexes at seal time, specialized layouts prepared on demand and retained under their seal; F2, F2b | D-RST-2, D-RST-9 | | D-RST-5 | derived relations | `materialize` with multiplicity, provenance, dependencies; invalidation; DRed retraction; F4 deletion cases, F5 | D-RST-4 | -| D-RST-6 | typed traversal + multi-hop in focus | the typed step vocabulary; tile multi-hop through stored intermediates; Gremlin adapter `out().in()` within focus; F3 | D-RST-5, A1 | -| D-RST-7 | triangle and amortization | F6 and F7, cold and warm | D-RST-6 | +| D-RST-6 | compiled relational cursors | semijoins, joins and projections over retained layouts (§3.2); the typed step vocabulary; tile multi-hop through stored intermediates; Gremlin adapter `out().in()` within focus; F3 | D-RST-5, A1 | +| D-RST-7 | triangle and amortization | F6, F7 and F8, cold and warm; F8's path D only through the existing D-RMAP-2 probe kernels, with no new production kernel | D-RST-6 | | D-RST-8 | large-domain multi-hop | only after admission A2 | A2 | -**D-RST-3 result (2026-10-20, AdaWorldAPI/petgraph#4, `experiments/relstore-visit`).** The view met F1: petgraph's unmodified `Bfs`/`Dfs` match a `DiGraph` oracle built from the facts on fixtures covering parallel edges, self-loops, isolated nodes and empty relations. The edge id is the fact id, and it survives compaction while rows move. The view borrows the prepared buffers, and building it or walking every neighbour list allocates nothing. The F2b-shaped falsifier passes: a Quack one-hop reads the same `dst` buffer at the same version, and the store prepares once. +**Critical path:** D-RST-9 → D-RST-1 → D-RST-2 → D-RST-4 → D-RST-5 → D-RST-6 → D-RST-7. **Off it:** D-RST-3 (optional petgraph), D-RST-8 (waits on A2), F-PG, and F8's path D (a measurement through existing probe kernels, never a reason to add one). + +**D-RST-3 result (2026-10-20, AdaWorldAPI/petgraph#4, `experiments/relstore-visit`), recorded as evidence for a deferred optional item.** The view met what was then F1 (now F-PG): petgraph's unmodified `Bfs`/`Dfs` match a `DiGraph` oracle built from the facts on fixtures covering parallel edges, self-loops, isolated nodes and empty relations. The edge id is the fact id, and it survives compaction while rows move. The view borrows the prepared buffers, and building it or walking every neighbour list allocates nothing. The F2b-shaped falsifier passes: a Quack one-hop reads the same `dst` buffer at the same version, and the store prepares once. -Measured at 1M edges: the neighbour walk and BFS cost the same as petgraph `Csr` (2.0 vs 1.8–1.9 ns per edge). Prepare takes 308–396 ns per edge and is not optimized. Admission question for D-RST-1/2: the Quack hop is O(E) per query (14–18 ms for 64 hops vs 0.3 µs through the view), because Quack cannot know that the `src` lane is sorted. A sealed ordering witness (`Cmp::Range`) would make it a range read; whether the seal carries one is open. +Measured at 1M edges: the neighbour walk and BFS cost the same as petgraph `Csr` (2.0 vs 1.8–1.9 ns per edge). Prepare takes 308–396 ns per edge and is not optimized. The Quack one-hop is O(E) per query (14–18 ms for 64 hops vs 0.3 µs through the view), because Quack cannot know that the `src` lane is sorted; that question belongs to the critical path and is §8 A6. The experiment's view is `Copy`, which the 2026-07-29 ruling forbids; §3.3 names the compatible shape. It is not extended further. Each PR keeps the probes in `mork-pathmap-probe` as probes. Production code is new code in the store crate with its own tests; it does not import the probe crate. @@ -268,11 +334,12 @@ Each PR keeps the probes in `mork-pathmap-probe` as probes. Production code is n - (c) stay refused. This decides whether `NonFunctionalChain` can be lifted beyond a focus. §5a narrows the choice: dense row blocks in the predecessor (pull) direction need no indexed read (`any(pred(t) ∧ F)` through mask-risc); the push direction and sparse rows do, and hypersparse bitmap blocks bring that read down to one per 64 columns. -- **A3, scalar petgraph iteration.** Whether shape-1 host iteration over a borrowed layout sits outside the no-gather ruling, which governs mask-risc folds. This plan reads it that way and asks for confirmation. +- **A3, scalar host iteration (only if petgraph or another scalar iterator is adopted).** Whether host iteration over a borrowed layout sits outside the no-gather ruling, which governs mask-risc folds. This plan reads it that way and asks for confirmation when D-RST-3 is taken up. +- **A6, attested order.** A sorted permutation turns a one-key read into a range only if Quack may rely on the order. D-RST-3 measured the cost of not relying on it (a full-lane scan per query). Whether the `LayoutSeal` carries an ordering witness that Quack's existing `Cmp::Range` lowering (D-DIAMOND) can validate is open. - **A4, store backend and home.** Which crate, and whether facts live in their own Lance datasets written through the cycle seal. This is consistent with "the seal owns durable writes"; the storage-portability ruling forbids a universal `Storage` trait, so the store is one owner, not an abstraction. - **A5, Registrar shape.** `register(name, width)` cannot describe a relation's arity or sorts. Either the store keeps `RelationDecl` beside it (this plan) or Quack's seam grows. The latter changes a Quack trait and needs its own review. - **B1, identity types.** `DurableRef` and `ThoughtCommitId` do not exist in code, and `ThoughtCommitId`'s never-reused rule is unspecified (D-RMAP-1A §7.6 gap 1). D-RST-2 cannot ship real seals before D-RST-1. -- **B2, petgraph fork pin.** The existing experiment pins a pre-quarantine rev of lance-graph. The new experiment pins a current rev and does not enable `quarantined-gather-hop`. +- **B2, petgraph fork pin (optional work only).** `lance-fold-hop` pins a pre-quarantine rev of lance-graph; `relstore-visit` pins a current rev and does not enable `quarantined-gather-hop`. - **B3, cyclic joins at scale.** The triangle over a large domain has no admitted shape (F6 reports a refusal there). Not a blocker for D-RST-1..6. -**Stop rule:** any step that copies canonical facts per thought, adds a second evaluator or traverser VM, persists a version counter beside Lance and Thought Git, or treats a pre-fold translation as already approved has chosen the wrong boundary. +**Stop rule:** any step that copies canonical facts per thought, adds a second evaluator or traverser VM, persists a version counter beside Lance and Thought Git, treats a pre-fold translation as already approved, adds a production interface or dependency only to satisfy petgraph compatibility, or forces a representation in order to use GEMM has chosen the wrong boundary. From a64637f4014d3ef8da4c779c5463861de3ae420c Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 20 Oct 2026 01:34:24 +0000 Subject: [PATCH 09/10] D-RST: mark lab exploration only, not admitted to production Operator ruling (2026-10-20): the relationship store is one option among many for certain data shapes, unfit to promote until further testing; nothing may be rewired to it. Plan status, board rows and the board entry say so; the 'critical path' is renamed a lab sequence. The entry also records that none of the measured paths touched DataFusion. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VyWZYKycy6MoYDxHRVnV5K --- .claude/board/INTEGRATION_PLANS.md | 2 +- .claude/board/STATUS_BOARD.md | 16 ++++++++-------- ...2026-10-20-relstore-geometry-not-graph-api.md | 6 ++++++ .../relstore-quack-petgraph-traversal-v1.md | 6 +++--- 4 files changed, 18 insertions(+), 12 deletions(-) diff --git a/.claude/board/INTEGRATION_PLANS.md b/.claude/board/INTEGRATION_PLANS.md index 69a61239c..ee019b78e 100644 --- a/.claude/board/INTEGRATION_PLANS.md +++ b/.claude/board/INTEGRATION_PLANS.md @@ -1,6 +1,6 @@ ## 2026-10-20 — relstore-quack-petgraph-traversal-v1 (reframed) — compiled relational geometry and amortization; petgraph optional → `.claude/plans/relstore-quack-petgraph-traversal-v1.md` -**Status:** D-RST-0..9 PROPOSAL; documentation only. Operator correction (2026-10-20): the goal is reusable relational geometry amortized across front ends and thought versions; petgraph is an optional consumer (D-RST-3 OPTIONAL / DEFERRED / MEASUREMENT-GATED); tile GEMM consumes prepared layouts and does not obtain them. +**Status:** D-RST-0..9 LAB EXPLORATION, not admitted to production; no production code may be rewired to it (operator, 2026-10-20); documentation only. Operator correction (2026-10-20): the goal is reusable relational geometry amortized across front ends and thought versions; petgraph is an optional consumer (D-RST-3 OPTIONAL / DEFERRED / MEASUREMENT-GATED); tile GEMM consumes prepared layouts and does not obtain them. - Critical path: D-RST-9 (native borrowed views, no petgraph) → D-RST-1 → 2 → 4 → 5 → 6 → 7. Off it: D-RST-3, D-RST-8 (A2), F-PG, F8 path D. - New §5b: three costs (preparation, execution, maintenance) measured apart; break-even from measured values only; Boolean existence vs BF16 multiplicity, with the BF16 exactness bound (integers to 256). diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index 5b6cacbe9..876569b0f 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -25,16 +25,16 @@ These PRs merged with no board rows. Every row below is a PROPOSAL (documents an | D-RMAP-1A | separate map tables, borrowed projections, thought-selected palettes (`d-rmap-separate-tables-thought-palettes-v1.md`) | PROPOSAL; generation witnesses corrected 2026-10-19 (§6a) | refusal on map regeneration, palette-slot reassignment, focus-slot reuse, incarnation reuse, stale tile, cross-tenant change | | D-RMAP-1A-V | versioning crosswalk: map/palette/binding generations derived from D-TGA `ThoughtCommitId` + #1605 `DurableRef`; focus generation and index seal retained (`d-rmap-separate-tables-thought-palettes-v1.md` §7) | PROPOSED; no code, no type; falsifiers specified, none run | X1–X9 in §7.4 (cross-dataset, compaction, palette ABA, slot reuse, uncommitted change, rebase, authorization, squash+replay, deletion/DRed) | | D-RMAP-1A-P | focus-bound RelationshipMap probe over existing sorted/trie indexes: CSR, 256×256 tile via mask-risc, incremental rebind, thought deltas, place masks, postings (`d-rmap-separate-tables-thought-palettes-v1.md` §8; `crates/mork-pathmap-probe/src/rmap.rs`) | MEASURED, probe only; nothing adopted | 12 tests vs scan oracle; 11 guards disable-verified; break-even 76/11/1.8 reuses (sparse d2 / d8 / dense) | -| D-RST-0 | stored relations × Quack × petgraph: architecture and phased plan (`relstore-quack-petgraph-traversal-v1.md`) | PROPOSAL; documentation only; reframed 2026-10-20 around relational geometry and amortization (petgraph optional, tile GEMM a consumer of prepared layouts) | operator decision B; admissions A1–A6, blockers B1–B3 listed in §8 | -| D-RST-1 | `DurableRef` / `ThoughtCommitId` contract types, never-reused rule (coordinated with D-CPG CPG-00) | Queued | reuse test; no duplicate with CPG-00 | -| D-RST-2 | `RelationshipMapStore` skeleton: catalog, facts through the cycle seal, sorted permutation, `LayoutSeal`, domain as a relation, `impl Registrar` | Queued | F4 refusals (reseal, domain ABA, compaction, scope, dirty, deletion) | +| D-RST-0 | stored relations × Quack × petgraph: architecture and phased plan (`relstore-quack-petgraph-traversal-v1.md`) | LAB EXPLORATION, not admitted to production (operator, 2026-10-20); documentation only; reframed 2026-10-20 around relational geometry and amortization (petgraph optional, tile GEMM a consumer of prepared layouts) | operator decision B; admissions A1–A6, blockers B1–B3 listed in §8 | +| D-RST-1 | `DurableRef` / `ThoughtCommitId` contract types, never-reused rule (coordinated with D-CPG CPG-00) | LAB ONLY, not admitted to production; Queued | reuse test; no duplicate with CPG-00 | +| D-RST-2 | `RelationshipMapStore` skeleton: catalog, facts through the cycle seal, sorted permutation, `LayoutSeal`, domain as a relation, `impl Registrar` | LAB ONLY, not admitted to production; Queued | F4 refusals (reseal, domain ABA, compaction, scope, dirty, deletion) | | D-RST-3 | petgraph `RelView` over a borrowed, version-pinned pattern-CSR view in a new fork experiment (brought forward; isolated probe with temporary ownership) | OPTIONAL / DEFERRED / MEASUREMENT-GATED (operator, 2026-10-20); experiment AdaWorldAPI/petgraph#4 kept as evidence, not extended | then-F1 (now F-PG) met: unmodified `Bfs`/`Dfs` equal a `DiGraph` oracle; view borrows the prepared buffers (pointer check), 0 allocations to build or walk; Quack one-hop reads the same `dst` buffer at the same version, one prepare; 7 disable runs red | -| D-RST-4 | layouts in the store: base indexes at seal time, specialized layouts on demand; pattern-only `u32` CSR/CSC on BlasGraph storage types, bitmap row blocks through mask-risc, join table; capability table | Queued (production reads wait on A1; layouts from D-RST-9) | F2: four front ends (Quack, Cypher, Gremlin adapter, RelAlg/Datalog), one `RelationVersion`, identical set and bag answers | -| D-RST-5 | derived relations: `materialize`, multiplicity, provenance, dependencies, invalidation, DRed | Queued | F4 deletion, F5 set/bag/path | -| D-RST-6 | typed traversal vocabulary; multi-hop within a focus through stored intermediates | Queued (A1) | F3: no `Gather`/`ScatterOrU32` in any program; `NonFunctionalChain` lifted only in focus | -| D-RST-7 | triangle and amortization | Queued | F6 (existence/count/enumeration, sparse/dense/skewed), F7 measured crossovers, F8 four-path amortization (A native iterator, B compiled sparse, C bit tiles, D GEMM via existing probe kernels) | +| D-RST-4 | layouts in the store: base indexes at seal time, specialized layouts on demand; pattern-only `u32` CSR/CSC on BlasGraph storage types, bitmap row blocks through mask-risc, join table; capability table | LAB ONLY, not admitted to production; Queued (production reads wait on A1; layouts from D-RST-9) | F2: four front ends (Quack, Cypher, Gremlin adapter, RelAlg/Datalog), one `RelationVersion`, identical set and bag answers | +| D-RST-5 | derived relations: `materialize`, multiplicity, provenance, dependencies, invalidation, DRed | LAB ONLY, not admitted to production; Queued | F4 deletion, F5 set/bag/path | +| D-RST-6 | typed traversal vocabulary; multi-hop within a focus through stored intermediates | LAB ONLY, not admitted to production; Queued (A1) | F3: no `Gather`/`ScatterOrU32` in any program; `NonFunctionalChain` lifted only in focus | +| D-RST-7 | triangle and amortization | LAB ONLY, not admitted to production; Queued | F6 (existence/count/enumeration, sparse/dense/skewed), F7 measured crossovers, F8 four-path amortization (A native iterator, B compiled sparse, C bit tiles, D GEMM via existing probe kernels) | | D-RST-8 | multi-hop over a large domain | BLOCKED on admission A2 | — | -| D-RST-9 | native borrowed relation views without petgraph: n-ary sorted permutations as ranges, pattern CSR/CSC as the binary case, stable fact ids, version pin, not `Copy`; starts from the planner's `AdjacencyStore`/`AdjacencyBatch` | Queued (critical path, first) | F1: oracle equality, zero copies, compaction-stable ids, stale and out-of-domain refusals | +| D-RST-9 | native borrowed relation views without petgraph: n-ary sorted permutations as ranges, pattern CSR/CSC as the binary case, stable fact ids, version pin, not `Copy`; starts from the planner's `AdjacencyStore`/`AdjacencyBatch` | LAB ONLY, not admitted to production; Queued (critical path, first) | F1: oracle equality, zero copies, compaction-stable ids, stale and out-of-domain refusals | | D-MPS-8 | native join over a versioned relationship table: R and S laid out by `y` slot, leapfrog intersection as one mask-risc `And`, per-side generations and reseal, per-thought filter planes (`mork-pathmap-schnellverschluss-v1.md` §7b; `crates/mork-pathmap-probe/src/join.rs`) | MEASURED, probe only; not admitted (D-RMAP-1 BIND membrane review) | 8 join tests + native arms in the differential suite vs the oracle; 7 guards disable-verified; cyclic bodies open | | D-RMAP-2 | context-declaration reuse and GraphBLAS/BF16 tile probe (`d-rmap-2-context-tile-algebra-probe-v1.md`) | PROPOSAL; not run | arms 1–5 in the plan; 297× (D-GTM-0j) is not evidence | | D-SRAM-0 | inventory Soufflé AST/ast2ram/RAM passes against D-ASM/D-RA/D-TGA owners | Queued | evidence per mapped pass; no code imported | diff --git a/.claude/board/entries/2026-10-20-relstore-geometry-not-graph-api.md b/.claude/board/entries/2026-10-20-relstore-geometry-not-graph-api.md index dd0fdd92c..670d5b5a5 100644 --- a/.claude/board/entries/2026-10-20-relstore-geometry-not-graph-api.md +++ b/.claude/board/entries/2026-10-20-relstore-geometry-not-graph-api.md @@ -17,3 +17,9 @@ ## Open - A1–A6, B1–B3 (plan §8). F8 (four-path amortization) is unmeasured; path D runs only through the existing D-RMAP-2 probe kernels. + +## Status ruling (operator, later 2026-10-20) + +Lab exploration only. The relationship store is one option among many for certain data shapes, unfit to promote until further testing; nothing from D-RST or D-RMAP-2 is admitted to production, and no production code may be rewired to it. + +Checked in source: none of the measured paths touched DataFusion. `mork-pathmap-probe`, `lance-graph-quack`, `lance-graph-mask-risc` and the petgraph `relstore-visit` experiment have no DataFusion dependency. BlasGraph and DataFusion are separate `ExecutionStrategy` arms in `lance-graph/src/query.rs`, and the BlasGraph arm returns "not yet wired to input datasets". The measured slowness is the probes' own: an indirect sort plus a second sort in preparation, a full-lane scan with per-query lowering in the Quack hop, and per-element allocation in the BF16 polyfill. diff --git a/.claude/plans/relstore-quack-petgraph-traversal-v1.md b/.claude/plans/relstore-quack-petgraph-traversal-v1.md index de3abd07d..da009e2df 100644 --- a/.claude/plans/relstore-quack-petgraph-traversal-v1.md +++ b/.claude/plans/relstore-quack-petgraph-traversal-v1.md @@ -2,7 +2,7 @@ (File name kept for existing links; the plan was first written as "RelationshipMapStore × Quack × petgraph". See the correction below.) -**Status: PROPOSAL, 2026-10-20. Documentation only.** No crate, type, opcode, ABI, dependency or ruling change lands with this plan. Every capability below is labelled IMPLEMENTED, PROBE, PROPOSAL or RULING from its own source. +**Status: LAB EXPLORATION, 2026-10-20 (operator). Nothing here is admitted to production, and no production code may be rewired to it.** The relationship store tested here is one option among many, meant for certain data shapes. It is unfit to promote until further testing: the measured preparation is slow (300–400 ns per edge, §7), and nothing has been compared with the alternatives on real workloads. Every "critical path", "PR" or "production" below describes a lab sequence that could only become a production proposal after its own review. Documentation only. No crate, type, opcode, ABI, dependency or ruling change lands with this plan. Every capability below is labelled IMPLEMENTED, PROBE, PROPOSAL or RULING from its own source. **Decision taken (operator, 2026-10-20), recorded as provenance, not as validation:** option B. `RelationshipMapStore` owns relationship persistence, identities, index layouts, coordinate-domain seals and invalidation. Quack owns compilation and numeric query semantics. mask-risc on `ndarray::simd` stays the only population executor. This keeps the D-RMAP-1A §1 ownership table and the 2026-10-05 Quack storage-portability ruling as they are. Neither is superseded here. @@ -317,13 +317,13 @@ Evidence so far: on the polyfill tier the bit arm wins every D-RMAP-2 cell (#162 | D-RST-7 | triangle and amortization | F6, F7 and F8, cold and warm; F8's path D only through the existing D-RMAP-2 probe kernels, with no new production kernel | D-RST-6 | | D-RST-8 | large-domain multi-hop | only after admission A2 | A2 | -**Critical path:** D-RST-9 → D-RST-1 → D-RST-2 → D-RST-4 → D-RST-5 → D-RST-6 → D-RST-7. **Off it:** D-RST-3 (optional petgraph), D-RST-8 (waits on A2), F-PG, and F8's path D (a measurement through existing probe kernels, never a reason to add one). +**Lab sequence (not a production path):** D-RST-9 → D-RST-1 → D-RST-2 → D-RST-4 → D-RST-5 → D-RST-6 → D-RST-7. **Off it:** D-RST-3 (optional petgraph), D-RST-8 (waits on A2), F-PG, and F8's path D (a measurement through existing probe kernels, never a reason to add one). **D-RST-3 result (2026-10-20, AdaWorldAPI/petgraph#4, `experiments/relstore-visit`), recorded as evidence for a deferred optional item.** The view met what was then F1 (now F-PG): petgraph's unmodified `Bfs`/`Dfs` match a `DiGraph` oracle built from the facts on fixtures covering parallel edges, self-loops, isolated nodes and empty relations. The edge id is the fact id, and it survives compaction while rows move. The view borrows the prepared buffers, and building it or walking every neighbour list allocates nothing. The F2b-shaped falsifier passes: a Quack one-hop reads the same `dst` buffer at the same version, and the store prepares once. Measured at 1M edges: the neighbour walk and BFS cost the same as petgraph `Csr` (2.0 vs 1.8–1.9 ns per edge). Prepare takes 308–396 ns per edge and is not optimized. The Quack one-hop is O(E) per query (14–18 ms for 64 hops vs 0.3 µs through the view), because Quack cannot know that the `src` lane is sorted; that question belongs to the critical path and is §8 A6. The experiment's view is `Copy`, which the 2026-07-29 ruling forbids; §3.3 names the compatible shape. It is not extended further. -Each PR keeps the probes in `mork-pathmap-probe` as probes. Production code is new code in the store crate with its own tests; it does not import the probe crate. +Each step stays a probe or isolated experiment (`mork-pathmap-probe`, the petgraph fork's `experiments/`). None of it is production code, none is wired into an existing crate, and none may be promoted without a separate review after further testing. ## 8. Open admission decisions and blockers From d5c7925803d33a3bdb3a1838adbcf59e88748c5f Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 20 Oct 2026 07:30:44 +0000 Subject: [PATCH 10/10] plan D-RPR: recursive perspective rebinding, lab-only distillation Architecture and experimental design only, graded against origin/main: - WORLD x ANGLE x THOUGHT map onto existing components (sealed version with thought 0 as givens; focus binding plus prepared relation planes; sparse thought deltas over one shared frame). - The addressing operation is the measured H14c/H14d canon; the global to local binding it needs is admission A1, still open. - Rebind is measured (D-RMAP-1A 8.4) and pays only for small k, so the open quantity is the k distribution along real question chains. Hypothesis H1: a frame's escapes seed the next angle. - MeTTa operations sorted into four classes from the existing polyfill classification; the pre-registered chain probe D-RPR-1 is not authorized, D-RPR-2/3 are blocked on D-TGA-3 and D-ASM-4. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VyWZYKycy6MoYDxHRVnV5K --- .claude/board/INTEGRATION_PLANS.md | 9 + .claude/board/STATUS_BOARD.md | 4 + ...6-10-20-recursive-perspective-rebinding.md | 15 ++ .claude/board/entries/README.md | 3 +- .../recursive-perspective-rebinding-v1.md | 223 ++++++++++++++++++ 5 files changed, 253 insertions(+), 1 deletion(-) create mode 100644 .claude/board/entries/2026-10-20-recursive-perspective-rebinding.md create mode 100644 .claude/plans/recursive-perspective-rebinding-v1.md diff --git a/.claude/board/INTEGRATION_PLANS.md b/.claude/board/INTEGRATION_PLANS.md index ee019b78e..b01ccc921 100644 --- a/.claude/board/INTEGRATION_PLANS.md +++ b/.claude/board/INTEGRATION_PLANS.md @@ -1,3 +1,12 @@ +## 2026-10-20 — recursive-perspective-rebinding-v1 — WORLD × ANGLE × THOUGHT over composable relational registers → `.claude/plans/recursive-perspective-rebinding-v1.md` + +**Status:** D-RPR-0 PROPOSED, LAB ONLY, documentation only; D-RPR-1 not authorized; D-RPR-2/3 BLOCKED (D-TGA-3, D-ASM-4). + +- The navigation terms map onto existing components: WORLD = sealed version with thought 0 as givens; ANGLE = focus binding + prepared relation planes (+ proposed map palettes); THOUGHT = sparse deltas over one shared frame. +- The addressing operation is the measured H14c/H14d canon `Thought(t)[T_a[i] : T_b[i]]`; the global → local binding it needs is admission A1 (D-RMAP-1 §6), still open. +- Rebind exists and is measured (D-RMAP-1A §8.4): it pays for small k only, so the open quantity is the k distribution along real question chains. Hypothesis H1: a frame's escapes seed the next angle. +- MeTTa operations sorted into four classes from the existing polyfill classification and Decision H. + ## 2026-10-20 — relstore-quack-petgraph-traversal-v1 (reframed) — compiled relational geometry and amortization; petgraph optional → `.claude/plans/relstore-quack-petgraph-traversal-v1.md` **Status:** D-RST-0..9 LAB EXPLORATION, not admitted to production; no production code may be rewired to it (operator, 2026-10-20); documentation only. Operator correction (2026-10-20): the goal is reusable relational geometry amortized across front ends and thought versions; petgraph is an optional consumer (D-RST-3 OPTIONAL / DEFERRED / MEASUREMENT-GATED); tile GEMM consumes prepared layouts and does not obtain them. diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index 876569b0f..526bb675e 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -35,6 +35,10 @@ These PRs merged with no board rows. Every row below is a PROPOSAL (documents an | D-RST-7 | triangle and amortization | LAB ONLY, not admitted to production; Queued | F6 (existence/count/enumeration, sparse/dense/skewed), F7 measured crossovers, F8 four-path amortization (A native iterator, B compiled sparse, C bit tiles, D GEMM via existing probe kernels) | | D-RST-8 | multi-hop over a large domain | BLOCKED on admission A2 | — | | D-RST-9 | native borrowed relation views without petgraph: n-ary sorted permutations as ranges, pattern CSR/CSC as the binary case, stable fact ids, version pin, not `Copy`; starts from the planner's `AdjacencyStore`/`AdjacencyBatch` | LAB ONLY, not admitted to production; Queued (critical path, first) | F1: oracle equality, zero copies, compaction-stable ids, stale and out-of-domain refusals | +| D-RPR-0 | recursive perspective rebinding: architectural distillation (`recursive-perspective-rebinding-v1.md`) | LAB ONLY, not admitted to production; PROPOSED; documentation only | claims graded IMPLEMENTED / MEASURED / PROBE / PROPOSED / OPEN against origin/main | +| D-RPR-1 | recursive question-chain probe over one corpus: escapes seed the next angle (H1), `rebind` vs full prepare vs no frame vs MORK oracle | LAB ONLY; not authorized (waits on admission A1 and an operator go) | exact parity; named refusal per failure case; marginal cost per angle with its crossover | +| D-RPR-2 | premise revision with selective retraction | BLOCKED on D-TGA-3 (per-derived-cell provenance) | stale-premise refusal only, until then | +| D-RPR-3 | MeTTa-facing parity of a chain's answers | BLOCKED on D-ASM-4 (not authorized) | — | | D-MPS-8 | native join over a versioned relationship table: R and S laid out by `y` slot, leapfrog intersection as one mask-risc `And`, per-side generations and reseal, per-thought filter planes (`mork-pathmap-schnellverschluss-v1.md` §7b; `crates/mork-pathmap-probe/src/join.rs`) | MEASURED, probe only; not admitted (D-RMAP-1 BIND membrane review) | 8 join tests + native arms in the differential suite vs the oracle; 7 guards disable-verified; cyclic bodies open | | D-RMAP-2 | context-declaration reuse and GraphBLAS/BF16 tile probe (`d-rmap-2-context-tile-algebra-probe-v1.md`) | PROPOSAL; not run | arms 1–5 in the plan; 297× (D-GTM-0j) is not evidence | | D-SRAM-0 | inventory Soufflé AST/ast2ram/RAM passes against D-ASM/D-RA/D-TGA owners | Queued | evidence per mapped pass; no code imported | diff --git a/.claude/board/entries/2026-10-20-recursive-perspective-rebinding.md b/.claude/board/entries/2026-10-20-recursive-perspective-rebinding.md new file mode 100644 index 000000000..a040e6eb2 --- /dev/null +++ b/.claude/board/entries/2026-10-20-recursive-perspective-rebinding.md @@ -0,0 +1,15 @@ +# Recursive perspective rebinding: distillation (2026-10-20) + +**Status:** PROPOSED, lab only (D-RPR-0). Plan: `.claude/plans/recursive-perspective-rebinding-v1.md`. Documentation only. + +## Findings (read on origin/main) + +- `WORLD × ANGLE × THOUGHT` needs no new nouns. WORLD is a sealed version with thought 0 as givens (D-TGA:6–9). ANGLE is `Focus` + `CsrPred`/tile in the `rmap.rs` probe, with `MapPalette` proposed (D-RMAP-1A). THOUGHT is `ThoughtDelta` over one shared tile (32 B per thought, per-eval flat across 1/8/64; D-RMAP-1A §8.5). +- `TableA[i] : TableB[i]` is the measured H14c/H14d expression `Thought(t)[T_a[i] : T_b[i]]`; the MeTTa council v3 header names those plans as the correct path and marks the preimage-join design a gather violation. +- `rebind` (k slots, one forward seek each) is measured: it wins for small k, loses at k = 256, and on the dense relation from k ≥ 16 (D-RMAP-1A §8.4). Recursive questioning is cheap only if real chains keep k small; that distribution is unmeasured. +- The probe already emits a missing-dependency signal: `ScopedReach.escaped` plus the `IncompleteFocus` refusal. Hypothesis H1: escapes seed the next angle. +- "Teleport" in this repo includes D-TELEPORT-0/1/2, source-restricted folds through Lance 13's row-address prefilter, which is the prefetch mechanism here. + +## Open + +A1 (pre-fold binding admission), H1, the k distribution, n-ary co-addressing and repeated variables, selective retraction provenance (D-TGA-3), a refusal for recursive dependency cycles, `ThoughtCommitId` never-reused rule. diff --git a/.claude/board/entries/README.md b/.claude/board/entries/README.md index 69387f64a..19ed0f174 100644 --- a/.claude/board/entries/README.md +++ b/.claude/board/entries/README.md @@ -25,12 +25,13 @@ index row, (3) no duplicate entry id. Checks 1 and 2 are deliberately opposite directions; the stranding this convention prevents shows up in exactly one of them, never both. -376 entries, 2026-08-06 .. 2026-10-20. +377 entries, 2026-08-06 .. 2026-10-20. | date | entry id | finding | file | |---|---|---|---| | 2026-10-20 | `relstore-quack-petgraph-plan` | | [2026-10-20-relstore-quack-petgraph-plan.md](2026-10-20-relstore-quack-petgraph-plan.md) | | 2026-10-20 | `D-RST` | | [2026-10-20-relstore-geometry-not-graph-api.md](2026-10-20-relstore-geometry-not-graph-api.md) | +| 2026-10-20 | `recursive-perspective-rebinding` | | [2026-10-20-recursive-perspective-rebinding.md](2026-10-20-recursive-perspective-rebinding.md) | | 2026-10-19 | `post-merge-recovery-1601-1617` | | [2026-10-19-post-merge-recovery-1601-1617.md](2026-10-19-post-merge-recovery-1601-1617.md) | | 2026-10-19 | `native-join-relationship-table` | | [2026-10-19-native-join-relationship-table.md](2026-10-19-native-join-relationship-table.md) | | 2026-10-19 | `mork-pathmap-schnellverschluss` | | [2026-10-19-mork-pathmap-schnellverschluss.md](2026-10-19-mork-pathmap-schnellverschluss.md) | diff --git a/.claude/plans/recursive-perspective-rebinding-v1.md b/.claude/plans/recursive-perspective-rebinding-v1.md new file mode 100644 index 000000000..6d10739f4 --- /dev/null +++ b/.claude/plans/recursive-perspective-rebinding-v1.md @@ -0,0 +1,223 @@ +# D-RPR: recursive perspective rebinding over composable relational registers + +**Status: LAB EXPLORATION, 2026-10-20. Architecture and experimental design only. Nothing here is admitted to production, no production code may be rewired to it, and no implementation is authorized by this document.** Every claim carries its grade: IMPLEMENTED (code on `main`), MEASURED (a recorded probe run, with its source), PROBE (probe code exists, run or not), PROPOSED (plan text only), OPEN (undecided or unknown). + +**Question.** Can globally consequential reasoning be decomposed into a sequence of bounded changes of perspective, each reusing the relationships its predecessors already resolved and resolving only what the next question adds? This is a question about addressing, memory organization and reuse. It proposes no graph engine, interpreter, world simulator or neural architecture. + +**Sources read for this distillation (origin/main, 2026-10-20):** +- `d-holo-h14c-thought-first-v1.md`, `d-holo-h14d-logic-mask-read-v1.md` (MEASURED); +- `d-rmap-implicit-relationship-register-v1.md` (D-RMAP-1), `d-rmap-separate-tables-thought-palettes-v1.md` (D-RMAP-1A), `d-rmap-2-context-tile-algebra-probe-v1.md`; +- `crates/mork-pathmap-probe/src/{rmap,join,boundary,lib}.rs` and its tests; +- `mork-pathmap-schnellverschluss-v1.md`; +- `metta-datalog-assimilation-v1.md`, its council `-v3.md`, `metta-polyfill-v1.md`, `crates/lance-graph-metta`; +- `thought-git-api-v1.md` (D-TGA), `causal-pickaxe-resident-git-lance13-temporal-v1.md` (D-CPG), `ce64-hypothesis-layers-v1.md` (givens layer); +- D-TELEPORT-0/1/2 board entries; `relstore-quack-petgraph-traversal-v1.md` (D-RST). + +## 1. Experience and navigation are different things + +`Idea(WHERE, WHAT, HOW, WHY)` names kinds of question. It is not a schema, and nothing here stores it as four columns or four registers. A question's kind determines which relationships must become addressable next; it does not determine how. + +`WORLD × ANGLE × THOUGHT` names the navigation, and each of the three already has a home: + +| Navigation term | Existing component | Grade | +|---|---|---| +| WORLD | canonical tables at a sealed version; within a thought table, thought 0 is the givens of that version ("Thoughts `1..=63` branch off main (thought 0, the givens of the current sealed version)", D-TGA:6–9) | givens layer IMPLEMENTED as a probe (`cognitive-shader-driver/examples/givens_layer_probe.rs`, refusal `WriteError::GivensLayer`); D-TGA PROPOSED | +| ANGLE | a selection of relationship maps plus a focus binding: `MapPalette` (0..255 local map selectors → `MapId` + generation), `FocusBinding` (global ids → focus slots), the prepared relation planes | `Focus`, `Csr`, tile IMPLEMENTED in the probe `rmap.rs`; `MapPalette`/`ThoughtMapPaletteBinding` PROPOSED (D-RMAP-1A §1) | +| THOUGHT | one of 64 slots (0 givens, 1..63 hypotheses) holding sparse typed deltas over the shared frame | `ThoughtDelta` PROBE in `rmap.rs`; D-TGA thought table PROPOSED | + +So the brief does not need new nouns. What it adds is a **relation between successive angles** (§4), which nothing on `main` names yet. + +## 2. The addressing operation is the H14c/H14d canon + +The brief's `TableA[i] : TableB[i]` is the measured canonical expression `Thought(t)[T_a[i] : T_b[i]]` (H14c, MEASURED 2026-10-15; H14d, MEASURED 2026-10-16). `i` is a shared coordinate, presence says whether a value is there and never where, and "the join is the layout". The council v3 header of the MeTTa assimilation plan names these two plans as the correct path and marks the preimage-join design (D-ASM-2, D-ASM-11/11A/11B) as a gather violation. + +H14d runs four stages, each shrinking what the next touches: logic reduced symbolically → address mask over presence planes → masked read at the source (Lance 13 `with_row_addr_prefilter`, the D-TELEPORT-2 probe) → terminal. Only addresses are materialized before a terminal. + +**Where translation happens.** Turning a relationship between two independently keyed tables into a shared `i` requires resolving one side's identities in the other's coordinate space. That is the global → local binding. D-RMAP-1 §6: "A pre-fold global->local binding is only a candidate new admissible *membrane* and must be independently reviewed. Until accepted, such operations are a typed refusal." The `rmap.rs` probe performs it (`Focus::bind`, `slot_of` by binary search) and states that it measures it without settling its legality. **Admission A1 is OPEN and is the single precondition for everything below.** + +**Prefetch** is two existing, measured mechanisms, not a new one: the address mask restricting the Lance read (D-TELEPORT-2, H14d stage 3), and the prepared relation planes of a focus (D-RMAP-1A §8). Temporary working copies are admitted by this brief; authoritative ownership stays with the canonical tables. + +## 3. Composable registers: what exists, at what size + +| Register | Concrete form in the probe | Size measured (D-RMAP-1A §8.3) | +|---|---|---| +| WORLD, 0..255 local slots | `struct Focus { ids: Vec, by_id: Vec<(u32, u8)>, pub generation: u64 }`, `SLOTS = 256` (`rmap.rs:58–65, 201–208`) | binding 3 072 B | +| ANGLE, per predicate | `CsrPred { p, offsets: [u32; SLOTS+1], dst: Vec, row: Vec, escaped: Vec<(u8, u32, u32)>, has_escape: SlotSet }`; the 256 × 256 tile built from it | CSR 18–66 KB; tile 8 KiB per predicate | +| THOUGHT, 0..63 | `ThoughtDelta` sparse edits over the shared tile | 32 B per thought as edits, 16 KiB lowered to executor planes; per-eval 2.61 / 2.66 / 2.79 µs at 1 / 8 / 64 thoughts (sparse d2), "the shared tile is stored once" (§8.5) | + +Nothing is a three-dimensional tensor. The world's values are never copied into an angle, and an angle is never copied into a thought. + +**Arity.** The probe's facts are `[u32; 3]` (subject, predicate, object), so its angles are predicate-indexed binary relations; `join.rs` co-addresses one shared variable. General n-ary binding over several shared variables, with repeated variables inside one pattern, has **no design on `main`** (OPEN). The triangle needs two variables co-addressed at once; only the 256 × 256 tile gives that, inside one focus (Schnellverschluss §7b). + +## 4. Recursive rebinding, built from existing pieces + +The brief's `F_{k+1} = Rebind(W, A_{k+1}, F_k)` already has a measured kernel. `rmap.rs` `rebind` replaces k slots of a focus with one forward seek each, re-resolves the old escapes against the new focus, remaps or escapes edges from unchanged members, and rebuilds the tile. The test `incremental_rebind_equals_full_preparation` pins it to a full preparation. D-RMAP-1A §8.4 measured it (µs, incremental / full): + +| k replaced | sparse d2 | sparse d8 | dense | +|---|---|---|---| +| 1 | 41.8 / 357.2 | 223.9 / 1657.7 | 308.2 / 892.9 | +| 16 | 60.4 / 433.1 | 298.1 / 1985.2 | 752.6 / 744.9 | +| 128 | 156.5 / 310.7 | 850.4 / 937.7 | 970.3 / 833.2 | +| 256 | 344.3 / 305.3 | 1238.4 / 971.3 | 1011.0 / 644.9 | + +Rebind wins for small k and loses at k = 256, and on the dense relation from k ≥ 16. **Whether recursive questioning is cheap is therefore a question about the distribution of k along a real chain of questions.** No probe has measured that distribution. + +**The three operations stay distinct, and each already has its mechanism:** + +| Change | Mechanism | What it costs or refuses | +|---|---|---| +| THOUGHT (another hypothesis, same angle) | a new sparse delta; nothing rebinds | measured flat across 1 / 8 / 64 thoughts (§8.5) | +| ANGLE (other relationships, other slots) | `rebind` for slots, a new `CsrPred` for each new predicate | measured above; a new predicate costs its own prepare | +| WORLD (an admitted canonical change) | a new sealed version | every frame built from the old one refuses: `Witness { facts, seal, focus, scope }`, `Refusal::{FactsChanged, IndexResealed, FocusChanged, ScopeChanged, ThoughtMismatch, OrderingMismatch, IncompleteFocus}` (IMPLEMENTED in the probe) | + +**What produces the next angle.** The brief leaves "identify missing dependency" open. The probe already emits the signal. A scoped reach records the global ids its edges left the focus for (`ScopedReach { reached, escaped: BTreeSet }`, exact and never truncated). `global()` refuses with `IncompleteFocus` while any escape exists. **Hypothesis H1:** the escapes of angle `k` are the seeds of angle `k+1`, so "the explanation lies outside the focus" is detected, not assumed, and `rebind` brings exactly those identities in. H1 is PROPOSED and measured nowhere. + +**Ownership along the chain.** A frame has one owner at a time. The mailbox moves to the frame (rotating ownership), so a child angle borrows within the mailbox that holds the frame. Hand-over is a move, which the type system checks (E-CE64-MB-4). The number of hand-overs per chain is a cost to count, not a blocker. + +**Reconsidering the parent.** A deeper angle may contradict an assumption of an earlier one: +- **Contradiction.** A contradiction of a given is recorded and never merged: "A hypothesis write that offers a different value for a given's lane is refused and recorded as a contradiction" (ce64-hypothesis-layers givens rule, IMPLEMENTED in the probe). +- **Retracting what depended on it.** That is delete-and-rederive (D-TGA Rule 2, PROPOSED). It needs, per derived cell, the cells it was derived from, and that provenance is undecided (D-TGA-3, OPEN). +- **Making a revision canonical.** That is `land` through the revision gate G1–G7 (D-TGA-4, PROPOSED). + +Until D-TGA-3 is decided, a parent can be **refused** on a changed premise, but its dependants cannot be **retracted** selectively. + +## 5. PathMap, MORK, Hyperon/MeTTa: what each contributes + +**PathMap (navigation to the next angle).** Measured in D-RMAP-1A §8.6: +- Prefix seek beats a sorted run on sparse relations (7.0 vs 8.5 µs; 100.9 vs 114.7 µs) and loses on dense (1 184 vs 566 µs). +- The trie costs 4–6× the run to build, and its keys take at least 3× the bytes. +- On postings, a zipper merge takes 2.32 µs, a sorted merge 0.41 µs, and a mask-risc `And` + `Count` 0.082 µs. + +So a trie is a candidate *navigation* index for choosing seeds, not a store and not the executor. A Lance-owned zipper over sorted relation orders is D-MPS-6, BLOCKED. + +**MORK (matching).** MORK and PathMap store sets (`PathMap<()>`), so bags are refused (`Refusal::SetOnly`). Its leapfrog route was read in source, never observed running. D-ASM-10 keeps MORK as an oracle outside the workspace. The native shared-variable join (MEASURED, Schnellverschluss §7b, `cargo run --release --example join_cost`, medians of 5) took the following at n = 50 000: + +| Step | Native | PathMap | MORK | +|---|---|---|---| +| Prepare / index build | 2.75 ms | 14.65 ms | 22.87 ms | +| Bag count | 35.5 µs | — | — | +| Answers streamed | 0.33 ms | — | — | +| Answers into a map | 7.39 ms | — | — | + +Only the intersection runs on mask-risc; enumeration and counting are plain loops. One join shape was measured, on synthetic data, warm, with no reuse across different queries. + +**Hyperon/MeTTa.** The existing classification (`metta-polyfill-v1.md:128–132`, Decision H) sorted into the brief's four classes: + +| Class | Operations | Source | +|---|---|---| +| 1. Exactly representable | facts in a space; `match` over one pattern; conjunctions with shared variables; no new terms | polyfill (PROPOSED) | +| 2. Representable after binding or materialization | recursive `=` rules that are Datalog (no function symbols in heads, no grounded calls), through the external driver d′, one fold per iteration, bounded, "refusing an unbounded closure rather than truncating" | Decision D, gated (PROPOSED) | +| 3. Requires functionality not yet built | repeated variables inside one pattern (no plan text addresses it); bags whose multiplicity lives on a foreign table (the keyed count undercounts, council N1c); nested expressions (refused in W1, N2); n-ary co-addressing (§3) | OPEN | +| 4. Kept external | term construction, `superpose`/`collapse`, grounded operations, `new-state`/`change-state!`, dependent types, `=` rules with a function symbol in the head: answered by Hyperon's interpreter, refused or routed on our side, never approximated | Decision H, council F6 | + +`crates/lance-graph-metta` is a stub: "Nothing is shipped here yet". No `Space` implementation exists (D-ASM-4/D-MTP-2 queued, not authorized). Oracles: Hyperon, PeTTa, datafrog, and MORK as a fourth, compared per category (bag / set / count), never averaged (Decision G, R-8). + +**Assimilation rule** (unchanged from those plans): take the algorithm and its semantic contract into RelAlg / Quack / R2IL / mask-risc only where exactness is shown against the oracle. At the boundary, reconstruct MeTTa-compatible bindings only on request. No foreign runtime objects, S-expressions or JSON on a hot path, and no Hyperon interpreter as a hidden second evaluator. + +## 6. Reuse is permitted only under a witness + +A prepared frame is reused only while its `Witness` equals the current one. Otherwise it refuses and names the stale component (IMPLEMENTED in the probe). D-RMAP-1A §7.4 specifies nine stale-state cases, none of them run. Among them: +- X1: datasets that collide at the same version; +- X2: compaction without stable row ids; +- X3: palette ABA; +- X4: a reused thought slot; +- X5: a dirty thought; +- X6: a rebase; +- X7: a revoked scope; +- X8: lineage after a squash; +- X9: a deletion on main. + +Durable versions are `DurableRef { dataset_id, branch, version }` and speculative ones `ThoughtCommitId`. Neither type exists in `crates/` (PROPOSED, D-CPG). Their never-reused rule is OPEN (D-TGA gaps). + +## 7. Three kinds of plasticity, three admission levels + +| Kind | What changes | Admission | +|---|---|---| +| Epistemic | conclusions within one angle: thought deltas, never the givens | none beyond the thought's own write rules | +| Perspective | which relationships and slots are addressable: rebind, palette selection | no canonical write; frames refuse on stale witnesses | +| Ontological | canonical facts, classifications or relationships | `land` through G1–G7 with retraction (D-TGA-3/-4), authorized, and the strongest evidence; PROPOSED, nothing built | + +A bounded frame does not establish global consistency. The only guard on `main` is `IncompleteFocus`: a frame with escapes cannot answer globally. Any experiment below must show that guard firing, not just that answers match. + +## 8. Minimal experiments (pre-registration sketch; none authorized, none run) + +**D-RPR-1: recursive question chain over one corpus.** Probe in `mork-pathmap-probe`, lab only. It reuses `Focus`, `discover`, `Csr`, `rebind`, `ThoughtDelta`, `Witness` and `join.rs`, and adds no new types. + +**Corpus.** The KJV SPO triples exported by `deepnsm-v2/examples/bible_wave.rs --export spo.tsv`. They need Gutenberg #10 and the `v0.1.0-cam96-data` release assets, which are fetched, not committed. A committed synthetic world is a fallback and must be labelled as such. The figures "31 327 triples" and "606 subjects" were not found in the example's source and must be re-measured before they are cited. + +**Chain.** Angle `A_0` is seeded by one question's entities and predicates. Each later angle is seeded by the previous one's escapes (H1), plus the predicates the next question kind adds. + +**Arms:** +- (a) MORK / PathMap repeated matching: set semantics only, as an external oracle; +- (b) native direct lookup with no retained frame (the probe's "direct/run" arm); +- (c) native, with a full prepare per angle; +- (d) native with `rebind` and retained predicate CSRs. + +**Measured per angle, kept apart:** +1. the initial binding cost; +2. the marginal cost of the next angle, given what is retained; +3. relationships reused; +4. new identities and predicates resolved; +5. indexed reads and bytes prefetched; +6. fold time; +7. 1 / 8 / 64 thoughts; +8. refusal on a changed premise; +9. resident bytes and the reuse crossover; +10. exact parity: bags against a nested-loop oracle, sets against MORK. + +Also measured: the distribution of k per rebind and the number of ownership hand-overs. + +**Reuse must fail, each case with a named refusal:** + +| Case | Refusal | +|---|---| +| no shared slots with the parent | full prepare, reported as k = 256 | +| a changed predicate order | `OrderingMismatch` | +| a new canonical version | `FactsChanged` / `IndexResealed` | +| a scope change | `ScopeChanged` | +| a rejected parent hypothesis | `ThoughtMismatch` | +| an answer outside the focus | `IncompleteFocus` | +| a recursive dependency cycle | **no refusal exists yet: OPEN** | + +**Acceptance.** Exact parity on every arm. Every case above refuses rather than answers. Arm (d)'s chain total is reported against (b) and (c) with its crossover. + +**Falsified if** arm (d)'s marginal cost per angle is not below arm (c)'s for the measured k distribution, or if any narrow frame returns an answer without `IncompleteFocus` when its oracle needs out-of-focus facts. + +**D-RPR-2: premise revision and selective retraction.** BLOCKED on D-TGA-3, which leaves the per-derived-cell provenance undecided. Until then only refusal is testable, not retraction. + +**D-RPR-3: MeTTa-facing parity of a chain's answers.** BLOCKED on D-ASM-4 (the `LanceSpace` harness), which is not authorized. + +## 9. Reuse before creating anything + +| Need | Existing | Grade | +|---|---|---| +| shared-coordinate operation | `Thought(t)[T_a[i] : T_b[i]]` | MEASURED (H14c/H14d) | +| focus, binding, escapes, refusals, witness | `rmap.rs` | PROBE | +| incremental rebind | `rmap.rs` `rebind`, §8.4 table | MEASURED | +| shared-variable join with bags | `join.rs`, Schnellverschluss §7b | MEASURED | +| source-restricted reads | Lance 13 row-address prefilter, D-TELEPORT-2 | MEASURED (probe) | +| thoughts over one frame | `ThoughtDelta`, §8.5 | MEASURED | +| givens immutability and contradiction records | `givens_layer_probe.rs` | PROBE | +| versions, branches, land, retraction | D-TGA, D-CPG | PROPOSED | +| map selection per thought | `MapPalette`, `ThoughtMapPaletteBinding` (D-RMAP-1A) | PROPOSED | +| MeTTa boundary and oracles | `metta-polyfill-v1.md`, D-ASM-4/-10 | PROPOSED; crate stub | +| relationship store ownership | D-RMAP-1A §1, D-RST (lab) | PROPOSED | + +**Nothing new is needed for D-RPR-1** except the chain driver and the measurements. + +## 10. Open, in order of consequence + +1. **A1:** is the pre-fold global → local binding an admitted membrane? Everything here presupposes it. +2. **H1:** do escapes predict the next angle's seeds in a real question chain? +3. **The k distribution** along real chains; it decides whether rebind pays (§4). +4. **n-ary co-addressing and repeated variables** (§3, §5 class 3). +5. **Selective retraction provenance** (D-TGA-3). +6. **A refusal for recursive dependency cycles.** +7. **`ThoughtCommitId`'s never-reused rule.** + +| D-id | Content | Status | +|---|---|---| +| D-RPR-0 | this distillation | PROPOSED, lab only | +| D-RPR-1 | recursive chain probe | not authorized; waits on A1 and an operator go | +| D-RPR-2 | premise revision with retraction | BLOCKED on D-TGA-3 | +| D-RPR-3 | MeTTa-facing parity | BLOCKED on D-ASM-4 |