diff --git a/.claude/board/INTEGRATION_PLANS.md b/.claude/board/INTEGRATION_PLANS.md index ef5673f7b..74d4b474c 100644 --- a/.claude/board/INTEGRATION_PLANS.md +++ b/.claude/board/INTEGRATION_PLANS.md @@ -1,3 +1,11 @@ +## 2026-10-19 — mork-pathmap-schnellverschluss-v1 — MORK / PathMap behind one replaceable boundary → `.claude/plans/mork-pathmap-schnellverschluss-v1.md` + +**Status:** D-MPS-0/1 MEASURED (probe, results only); D-MPS-2..5 upstream PROPOSALS; D-MPS-6 BLOCKED; D-MPS-7 OPEN. + +- Source switch: pinned git revisions plus one `[patch]` for MORK's out-of-repo `../PathMap` path; one package identity, no upstream file edited. +- Execution switch: `Backend::{Mork, PathMapZipper, Native}` with typed refusals (`SetOnly`, `NoAdmittedNativeJoin`, `Stale`); the native arm is refused under the no-gather ruling. +- MORK stays an oracle outside the workspace (D-ASM-10); nothing is wired into Quack, R2IL or mask-risc. + ## 2026-10-19 — catch-up: seven plans from #1605 and #1612–#1617 had no index entry **Status:** INDEX CATCH-UP (post-merge recovery, entry `2026-10-19-post-merge-recovery-1601-1617.md`). All PROPOSAL; nothing implemented. diff --git a/.claude/board/ISSUES.md b/.claude/board/ISSUES.md index b86696ab0..e371fcf4e 100644 --- a/.claude/board/ISSUES.md +++ b/.claude/board/ISSUES.md @@ -1,3 +1,10 @@ +## ISS-MORK-LIBRARY-GLOBAL-ALLOCATOR — linking MORK replaces the binary's allocator (2026-10-19) + +**Status:** OPEN. **Basis:** MEASURED (`crates/mork-pathmap-probe`, a `#[global_allocator]` in the probe fails to link: "conflicts with global allocator in: pathmap"). + +- MORK's workspace enables PathMap feature `jemalloc`, and PathMap then installs a global allocator from inside the library. Any lance-graph binary that links MORK loses control of its allocator. +- **What closes it:** D-MPS-4 upstream (enable `jemalloc` from MORK's binary, not the library), or a MORK consumer that turns the feature off through Cargo feature resolution, measured. + ## ISS-GATHER-HOP-1616 — the merged `OutgoingHop` reads positions taken from another lane's values (2026-10-19) **Status:** BLOCKED (quarantined; waits on GATHER-DECISION). **Basis:** VERIFIED-IN-CODE (`crates/lance-graph-mask-risc/src/graph_hop.rs`: two `MaskOp::Gather` over `frontier[sources[i]]` and `visible[targets[i]]`, then `ScatterOrU32` by `targets[i]`). diff --git a/.claude/board/LATEST_STATE.md b/.claude/board/LATEST_STATE.md index 4335dea7d..e899b38d4 100644 --- a/.claude/board/LATEST_STATE.md +++ b/.claude/board/LATEST_STATE.md @@ -1,3 +1,8 @@ +## 2026-10-19 — MORK / PathMap Schnellverschluss probe (in PR) + +- New excluded crate `crates/mork-pathmap-probe` (own workspace, git deps on the AdaWorldAPI MORK and PathMap forks, results only) with CI job `mork-pathmap-probe.yml`; plan `mork-pathmap-schnellverschluss-v1.md`; entry `2026-10-19-mork-pathmap-schnellverschluss.md`; issue `ISS-MORK-LIBRARY-GLOBAL-ALLOCATOR`. +- Contract inventory delta: none. + ## 2026-10-19 — main after #1617 (merged `47697c77`); post-merge recovery in PR - **On main since #1597:** #1601–#1617 (17 PRs: research notes, plans, a qualia archive, the ACT-R crate, the mask-risc hop, the D-RMAP plans). None had board rows; the arc entries, STATUS_BOARD rows and plan index entries are added in this PR. diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index f16dfcc55..74903c79d 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -1,3 +1,18 @@ +## MORK / PathMap Schnellverschluss (2026-10-19) + +Plan `mork-pathmap-schnellverschluss-v1.md`; entry `entries/2026-10-19-mork-pathmap-schnellverschluss.md`. + +| D-id | scope | status | gate / falsifier | +|---|---|---|---| +| D-MPS-0 | source and type inventory; fork/upstream switch through Cargo `[patch]` | MEASURED | one `pathmap` package in the graph; 0 / 18 upstream files differ (PathMap / MORK stable branch) | +| D-MPS-1 | differential probe `crates/mork-pathmap-probe`: MORK (product-zipper, leapfrog) and PathMap zipper vs nested-loop oracle | MEASURED (probe; CI job `mork-pathmap-probe.yml`) | 8 + 1 tests in both builds; 4 disable runs red | +| D-MPS-2 | upstream MORK: PathMap as git/version dependency, not `../PathMap` | PROPOSAL | MORK git dep resolves without `[patch]` | +| D-MPS-3 | upstream MORK: `nightly` opt-in (fork `de9312e`) | PROPOSAL | upstream `main` builds on stable | +| D-MPS-4 | upstream MORK: no library-level `jemalloc` global allocator | PROPOSAL | consumer binary sets its own allocator | +| D-MPS-5 | upstream MORK: leapfrog join generic over a cursor provider + reindex hook | PROPOSAL | MORK corpus unchanged on `PathMap<()>` | +| D-MPS-6 | Lance-owned PathMap-trait cursor over a sealed `RelationOrder` | BLOCKED (`RelationOrder` not built) | no copy, or the copy stated | +| D-MPS-7 | licence: no MORK source vendored until a licence is declared (#148 unmerged) | OPEN | probe vendors nothing | + ## Post-merge recovery for #1601–#1617 (2026-10-19) These PRs merged with no board rows. Every row below is a PROPOSAL (documents and fixtures) unless it says otherwise. Entry: `entries/2026-10-19-post-merge-recovery-1601-1617.md` (review-thread reconciliation). diff --git a/.claude/board/entries/2026-10-19-mork-pathmap-schnellverschluss.md b/.claude/board/entries/2026-10-19-mork-pathmap-schnellverschluss.md new file mode 100644 index 000000000..528609a59 --- /dev/null +++ b/.claude/board/entries/2026-10-19-mork-pathmap-schnellverschluss.md @@ -0,0 +1,20 @@ +# MORK / PathMap Schnellverschluss: source switch and differential probe (2026-10-19) + +**Status:** MEASURED (D-MPS-0, D-MPS-1). Plan: `.claude/plans/mork-pathmap-schnellverschluss-v1.md`. Probe: `crates/mork-pathmap-probe`. + +## Measured + +- Forks against upstream: AdaWorldAPI/PathMap `ef02591` = upstream `master` (0 files differ). AdaWorldAPI/MORK `main` `ed57c67` = upstream `main` (0 files). The fork's stable branch `de9312e` differs in 18 files (+421 / −408) and is the only carried change. +- A git dependency on MORK does not resolve on its own: MORK names `pathmap = { path = "../PathMap/" }`, which lies outside its repository. One `[patch."https://github.com/AdaWorldAPI/MORK"] pathmap = { git = …, rev = … }` fixes it with one `pathmap` package in the graph and no edit to MORK. +- Upstream MORK `main` does not build on Rust 1.99 stable (`E0554`, nightly defaults); `de9312e` does. MORK's gxhash fork needs `+aes,+sse2`. +- MORK enables PathMap's `jemalloc` feature, which installs a `#[global_allocator]` in the library; a consumer binary cannot add its own (link error measured). +- MORK has no declared licence; upstream PR #148 (MIT) is not merged. +- `query_multi_leapfrog` takes a concrete `&PathMap<()>`; `SubtermCursor` is generic over PathMap's public `Zipper + ZipperMoving + ZipperPath + ZipperIteration` traits. +- Differential, `Q(x,y,z) :- R(x,y), S(y,z)`: MORK (product-zipper and leapfrog builds) and a PathMap read-zipper arm equal the nested-loop oracle on 8 cases (shared variable, empty, disjoint, per-atom scope, wide ids, duplicates, stale generation, ABA). Both upstream indexes are sets: bag requests are refused (`SetOnly`). Four disable runs each turned their guard's test red. +- Cost, n = 50 000, release, one run (re-run after review): PathMap index 14.8 ms (236 ns/row), query 24.1 ms; over 64 thoughts a shared index costs 1 556 ms in total vs 2 487 ms rebuilt per thought (1.6×; 24.3 vs 38.9 ms per thought). Sharing saves only the build. (The first version compared a per-thought figure with a 64-thought total and said 110×.) The MORK query time (170 ms) includes the probe's text decode and is not a measure of MORK's join. + +## Open + +- No admitted native lance-graph arm: the join reads `S` at positions chosen by `R`'s values (no-gather ruling). The native arm waits on a co-addressed producer. +- Whether a Lance-owned PathMap-trait cursor can walk a sealed `RelationOrder` without copying is unproved; `RelationOrder` is not built. +- The generation witness compares numbers, so a reused generation is not detected. diff --git a/.claude/board/entries/README.md b/.claude/board/entries/README.md index 0d86f9634..c3bc75011 100644 --- a/.claude/board/entries/README.md +++ b/.claude/board/entries/README.md @@ -25,11 +25,12 @@ index row, (3) no duplicate entry id. Checks 1 and 2 are deliberately opposite directions; the stranding this convention prevents shows up in exactly one of them, never both. -371 entries, 2026-08-06 .. 2026-10-19. +372 entries, 2026-08-06 .. 2026-10-19. | date | entry id | finding | file | |---|---|---|---| | 2026-10-19 | `post-merge-recovery-1601-1617` | | [2026-10-19-post-merge-recovery-1601-1617.md](2026-10-19-post-merge-recovery-1601-1617.md) | +| 2026-10-19 | `mork-pathmap-schnellverschluss` | | [2026-10-19-mork-pathmap-schnellverschluss.md](2026-10-19-mork-pathmap-schnellverschluss.md) | | 2026-10-19 | `D-RMAP-1A` | | [2026-10-19-d-rmap-thought-git-versioning.md](2026-10-19-d-rmap-thought-git-versioning.md) | | 2026-10-19 | `D-RMAP-1` | | [2026-10-19-d-rmap-implicit-relationship-register.md](2026-10-19-d-rmap-implicit-relationship-register.md) | | 2026-10-17 | `thinking-tables-wiring` | | [2026-10-17-thinking-tables-wiring.md](2026-10-17-thinking-tables-wiring.md) | diff --git a/.claude/plans/mork-pathmap-schnellverschluss-v1.md b/.claude/plans/mork-pathmap-schnellverschluss-v1.md new file mode 100644 index 000000000..c2aefa575 --- /dev/null +++ b/.claude/plans/mork-pathmap-schnellverschluss-v1.md @@ -0,0 +1,168 @@ +# MORK / PathMap Schnellverschluss — v1 + +**Status:** D-MPS-0 and D-MPS-1 MEASURED (probe `crates/mork-pathmap-probe`, results only). D-MPS-2..7 PROPOSAL / BLOCKED as listed in §8. No production executor, no CE64 change, no Lance storage, no upstream file edited by this work. + +**Ownership rule:** our types stay ours, upstream types stay upstream's, and the one replaceable boundary sits where they meet: `crates/mork-pathmap-probe/src/boundary.rs` plus the `[dependencies]` / `[patch]` block of the probe's `Cargo.toml`. + +Read with: `d-rmap-separate-tables-thought-palettes-v1.md` (D-RMAP-1A, ownership), `d-holo-h14c-thought-first-v1.md` and `d-holo-h14d-logic-mask-read-v1.md` (no-gather ruling), `metta-datalog-assimilation-v1-council-v3.md` and entry `2026-10-16-asm-accepted-proposals.md` (D-ASM-10: MORK is an oracle outside the workspace). + +## 1. Source inventory (verified 2026-10-19) + +| Repository | Local path | Revision used | Relation to upstream | Licence | +|---|---|---|---|---| +| AdaWorldAPI/MORK | `/home/user/mork` | `de9312e` (branch `ccr-9f3b8075-74u47l-stable`) | upstream `trueagi-io/MORK` `main` = `ed57c67`; fork `main` = `ed57c67` (0 files differ); the stable branch adds one commit, **18 files, +421 / −408** | **none declared**: no LICENSE file, no `license` field. Upstream PR #148 (`a6281fa`, adds MIT LICENSE + manifest metadata) is **not merged** into upstream `main` | +| AdaWorldAPI/PathMap | `/home/user/pathmap` | `ef02591` | upstream `Adam-Vandervorst/PathMap` `master` = `ef02591`; **0 files differ** | MIT (`Cargo.toml` + LICENSE) | +| AdaWorldAPI/PeTTa | `/home/user/petta` | `1d19a1d` | not compared (Prolog/Python, no Rust crate) | MIT (LICENSE, © Patrick Hammer 2025) | +| AdaWorldAPI/mettaclaw | `/home/user/mettaclaw` | `7b30527` | not compared (MeTTa/Python, no Rust crate) | MIT (LICENSE, © Patrick Hammer 2026) | + +All four clones are shallow. PeTTa and MeTTaClaw have no Rust surface to link; they stay oracle/reference material and are out of scope for the dependency switch. + +## 2. Type inventory (read in source, not inferred) + +**MORK** (`expr/src/lib.rs`, `interning/src/lib.rs`, `kernel/src/leapfrog.rs`, `kernel/src/space.rs` at `de9312e`): + +| Item | Definition | Consequence at the boundary | +|---|---|---| +| `Expr` | `#[derive(Clone, Copy)] #[repr(transparent)] pub struct Expr { pub ptr: *mut u8 }`, `unsafe impl Send/Sync` (lib.rs:197, 1802–1803) | raw pointer, no lifetime: nothing stops it outliving its buffer. It must never cross the boundary or be stored. | +| `ExprEnv` | `{ pub n: u8, pub v: u8, pub offset: u32, ground_skip: u16, pub base: Expr }` (lib.rs:1881) | a view into an `Expr`; same hazard | +| `Bindings` | direct-indexed slab `{ slots: Vec>, touched: Vec }`, key `ExprVar = (u8, u8)` = (namespace, introduction index) (lib.rs:1808, 1820) | variables are positional, ≤ 63 per namespace; not Lance variables | +| `Tag` | `NewVar`, `VarRef(u8)`, `SymbolSize(u8)`, `Arity(u8)` (lib.rs:90) | symbols are inline byte strings < 64 bytes, or 8-byte interned handles | +| interned symbol | `type Symbol = [u8; SYM_LEN]`, `SYM_LEN = 8` (interning/lib.rs:49–51), table `SharedMapping` behind a `NonNull` handle | an 8-byte table-relative handle. **Not** a `NodeGuid` (16 bytes, globally canonical); never map one onto the other | +| `SubtermCursor` | generic: `impl` (leapfrog.rs:88, 168) | generic over PathMap's public traits | +| `query_multi_leapfrog` | `pub fn query_multi_leapfrog, Expr) -> bool>(map: &PathMap<()>, pat_expr: Expr, effect: F) -> usize` (leapfrog.rs:1319), behind feature `leapfrog` | **concrete** `&PathMap<()>`; its `UnifyJoin` holds `SubtermCursor>`, and `build_reindex` materializes a new `PathMap<()>` for an inverted (cyclic) factor | +| `Space` | `{ pub btm: PathMap<()>, pub sm: SharedMappingHandle, … }` (space.rs:59) | MORK's store is a set of byte paths | + +**PathMap** (`src/zipper.rs`, `src/trie_map.rs` at `ef02591`): `PathMap` is a byte-keyed trie; `insert>(&mut self, k, v)`. The traits `SubtermCursor` needs have these required methods: `Zipper { path_exists, is_val, child_count, child_mask }`, `ZipperMoving { depth, focus_byte, val_count, descend_to, ascend, ascend_until, ascend_until_branch }`, `ZipperPath { path }`; every `ZipperIteration` method has a default. None of the four is sealed. Read-only and witness types exist (`ZipperReadOnlyValues`, `ZipperReadOnlyConditionalValues`, `ReadZipperWitness`, `ProductZipperWitness`, `TrieRefWitness`); `OverlayZipper` and `ProductZipper` are public. + +**Lance-graph:** `NodeGuid` and the OGAR/ClassView/envelope contracts exist (`canonical_node.rs`, `lance-graph-contract`). `RelationshipMapStore`, `RelationOrder`, `MapPalette`, `ThoughtMapPaletteBinding` and the frame/index witnesses exist **only in plans** (D-RMAP, PROPOSED); no Rust type has those names. `lance-graph-relalg` describes single-table selections and lowers them through R2IL to mask-risc; its own doc says "recursion (μ) and the hop (preimage) are not built". + +## 3. Ownership and mapping design + +``` +Lance side (owner of meaning) boundary.rs (the Schnellverschluss) upstream (owner of its types) +RelationView<'a> bind(backend, input, semantics) mork::space::Space + rows: &'a [[u32; 2]] ── encode ──▶ -> BoundIndex { generations, … } ──▶ pathmap::PathMap<()> + generation: u64 answers(&mut self, current) +Answers { [u32; 3] -> count } ◀── decode ── -> Result +``` + +- **One canonical owner of meaning.** The rows and their generations belong to the Lance side. In production that is a sealed `RelationOrder` view under D-RMAP; in the probe it is a borrowed slice. MORK and PathMap hold only a disposable index built from that view. +- **Nothing upstream-typed crosses.** Rows go in as `[u32; 2]` and answers come out as `[u32; 3]`. `Expr`, `Bindings`, `Space` and `PathMap` stay private to `boundary.rs`. +- **The upstream index is a copy and is labelled one.** `bind` encodes the rows (directly as 9-byte trie keys, or as S-expression text through MORK's parser) and the index owns those bytes. That is the translation cost in §6. A stale copy is caught by the generation witness (`Refusal::Stale`), not by a lifetime, because the copy does not borrow. +- **Typed refusals instead of silent degradation:** + - `Refusal::SetOnly(backend)`: PathMap and MORK store sets, so a request for bag multiplicities is refused rather than answered with every count 1. + - `Refusal::NoAdmittedNativeJoin`: there is no admitted lance-graph join for this input shape (§7). + - `Refusal::Stale`: the generations do not match. + - `Refusal::Upstream`: the upstream index rejected the encoded input. +- **Removal test:** deleting a backend means deleting its `Index` variant, its two helper functions, and its line in `Cargo.toml`. Nothing outside `boundary.rs` names `mork` or `pathmap`. This was checked by construction: `lib.rs` and `reference.rs` import neither. + +## 4. Source switch (fork ↔ upstream) + +**Mechanism:** Cargo's own, with no custom machinery. + +1. Git dependencies pinned by `rev` in the probe's `Cargo.toml` (`mork`, `mork-expr` → `AdaWorldAPI/MORK@de9312e3`; `pathmap` → `AdaWorldAPI/PathMap@ef02591`). +2. One `[patch."https://github.com/AdaWorldAPI/MORK"] pathmap = { git = …PathMap, rev = "ef02591" }`. + - **Why it is needed (MEASURED):** MORK's workspace declares `pathmap = { path = "../PathMap/" }`, which lies outside the MORK repository. A plain git dependency on MORK fails with `no matching package named pathmap found … Git repository https://github.com/AdaWorldAPI/MORK`. + - **What the patch does (MEASURED):** it resolves that requirement to the same PathMap revision, so the graph has exactly **one** `pathmap` package (`cargo tree -i pathmap`, and the lockfile has 1 `name = "pathmap"`). +3. Switching to upstream = changing the URL and `rev` of three dependency lines plus the patch key. **No consumer source changes.** + +**Measured constraints on the switch:** + +| Constraint | Measured | Who owns the fix | +|---|---|---| +| Upstream MORK `main` (`ed57c67`) on Rust 1.99 stable | fails: `error[E0554]: #![feature] may not be used on the stable release channel` (fast-slice-utils, via `nightly` defaults) | upstream: D-MPS-3 | +| Fork stable branch `de9312e` on 1.99 | builds (`mork-expr` + `mork`, 1 min 02 s debug) | carried fork commit until D-MPS-3 lands | +| MORK's gxhash fork | `compile_error!` without `+aes,+sse2`; the probe sets them in its own `.cargo/config.toml`, never via `RUSTFLAGS` | upstream: optional | +| PathMap `ef02591` alone on 1.99 | builds (20 s debug), no flags needed | — | +| MORK enables PathMap feature `jemalloc` | PathMap then installs a `#[global_allocator]`; any binary linking MORK has its allocator replaced and cannot add its own (the probe's allocation counter failed to link: "conflicts with global allocator in: pathmap") | upstream: D-MPS-4 | + +## 5. Entropy ledger + +| Kind | Item | Cost | Disposition | +|---|---|---|---| +| Semantic | MORK/PathMap answer sets; Lance relations are bags | duplicate derivations collapse (2 × 2 → 1) | refused by type (`SetOnly`); pinned by `duplicates_are_refused_under_bag_semantics` | +| Semantic | 8-byte MORK symbol vs 16-byte `NodeGuid` | none, because no identity is shared: ids cross as `u32` and are re-encoded per index | kept separate by design | +| Semantic | second fact store | none: both indexes are disposable copies keyed by generation | gate "storage ownership" | +| Semantic | generation witness compares numbers only | a reused generation number is not detected (ABA), pinned by `a_reused_generation_number_is_not_detected` | the producer must never reuse a generation; D-RMAP seal digests would close this | +| Maintenance | upstream files modified by us | MORK: 18 files on one carried branch (nightly → opt-in); PathMap: 0 | D-MPS-3 upstreams the 18 | +| Maintenance | out-of-repo `../PathMap` path in MORK | one `[patch]` line per consumer | D-MPS-2 upstream | +| Maintenance | library-installed global allocator | blocks any MORK linkage into a lance-graph binary | D-MPS-4 upstream | +| Maintenance | AES/SSE2 target-feature requirement | a per-crate `.cargo/config.toml`; `cargo --manifest-path` from the repo root does not read it | documented; run from the crate directory | +| Translation | encode + index build | see §6: PathMap 177–296 ns/row, MORK (text + parse) 386–938 ns/row | paid once per bound generation | +| Translation | MORK answer decode | `exec` writes `Q` facts into the space; the probe dumps text, parses it and removes the facts again | oracle-only; dominates the MORK query time in §6 | + +## 6. Measured cost (`cargo run --release --example cost`, one run per cell) + +Rows `R(i, y)` with `y` drawn from `n / 4` values, `S(y, z)` one row per `y`, answers ≈ n. Shared = one index built once and queried by k thoughts; rebuilt = an index built per thought. Both are given per thought and as a total over 64 thoughts, so the two columns compare like with like. + +| join | backend | n | build ms | ns/row | query ms | shared per thought, k=64 | rebuilt per thought | shared total, k=64 | rebuilt total, k=64 | +|---|---|---|---|---|---|---|---|---|---| +| product-zipper | PathMapZipper | 1 000 | 0.259 | 207 | 0.205 | 0.210 | 0.465 | 13.4 | 29.7 | +| | Mork | 1 000 | 0.515 | 412 | 3.006 | 3.014 | 3.521 | 192.9 | 225.4 | +| | PathMapZipper | 10 000 | 2.069 | 165 | 2.846 | 2.879 | 4.915 | 184.2 | 314.6 | +| | Mork | 10 000 | 4.510 | 361 | 28.29 | 28.36 | 32.80 | 1 815 | 2 099 | +| | PathMapZipper | 50 000 | 14.77 | 236 | 24.09 | 24.32 | 38.86 | 1 556 | 2 487 | +| | Mork | 50 000 | 25.98 | 416 | 161.7 | 162.1 | 187.7 | 10 377 | 12 013 | +| leapfrog | PathMapZipper | 50 000 | 14.75 | 236 | 21.93 | 22.16 | 36.68 | 1 418 | 2 348 | +| | Mork | 50 000 | 23.71 | 379 | 150.7 | 151.1 | 174.4 | 9 670 | 11 164 | + +Re-run 2026-10-19 after the PR #1620 review: the first version compared a per-thought shared cost with a 64-thought rebuilt total and reported about 110×, which overstated the saving. + +What it shows, and what it does not: + +- **Sharing saves the build, not the query.** Over 64 thoughts a shared PathMap index costs 1.6× less than rebuilding it per thought (n = 50 000: 1 556 ms vs 2 487 ms total; 24.3 vs 38.9 ms per thought). The query is most of the cost, so the saving is bounded by build / (build + query): 38% for PathMap at n = 50 000, 14% for MORK. +- **The MORK query column measures the probe's text decode too, not MORK's join alone.** It covers `exec`, dump, parse and removal of the `Q` facts. It is not evidence about MORK's join speed. A binding-level decode through `Space::query_multi` would be needed for that, and that path hands back `Bindings` or `&[u32]` references whose decoding is not done here. +- The leapfrog build is verified to compile MORK's `leapfrog` module (a type alias on `mork::leapfrog::SubtermCursor`). That each query took the leapfrog route is MORK's `cfg` dispatch (space.rs:1058–1064), read in source but not observed at runtime. +- No allocation counts: see the global-allocator constraint in §4. + +## 7. RelationshipMap integration and the gather ruling + +- **Can PathMap's zippers navigate a sealed `RelationOrder` without copying?** Possibly, but this is not proved. `SubtermCursor` needs only the public traits listed in §2: about 12 required methods, all byte-path navigation. A Lance-owned type implementing them over a borrowed, byte-ordered sealed column would let MORK's cursor walk Lance data in place. Nothing like that exists yet. `RelationOrder` itself is unbuilt, and its layout (fixed-width sorted keys vs a trie) decides whether `child_mask` / `descend_to` can be answered without materializing. **Status: HYPOTHESIS, D-MPS-6.** +- **Can MORK's join consume such a provider?** Not with the current API. `query_multi_leapfrog` and `UnifyJoin` are concrete over `&PathMap<()>` / `ReadZipperUntracked`, and `build_reindex` creates a new `PathMap` for cyclic factors. The generic piece already exists (`SubtermCursor`). The upstream change would be to make `UnifyJoin` generic over a cursor provider with a reindex hook. That is useful to MORK on its own, for joins over ACT arenas and mmaps (D-MPS-5). +- **Lance-side execution of this join is blocked by the 2026-10-16 ruling, not by an API.** A shared-variable join reads `S` at positions chosen by `R`'s values, and leapfrog's `seek` is that read. Inside lance-graph's admitted path the pair must already sit at one coordinate (`Thought(t)[T_a[i] : T_b[i]]`). So: + - `Backend::Native` refuses (`NoAdmittedNativeJoin`). + - The PathMap arm's seek into `S ‖ y` is oracle code and is labelled as such. + - Nothing here is wired into Quack, R2IL or mask-risc. + - The quarantined #1616 `OutgoingHop` is not used and is not a precedent. + + The native arm becomes testable only when a co-addressed producer for this input shape exists. **BLOCKED.** + +## 7a. Versioning: what `RelationView.generation` stands for (2026-10-19) + +`RelationView.generation: u64` is a probe-local stand-in. It is not a design. Inside the probe it is enough: the `Stale { bound, current }` refusal fires on any change. Outside the probe a bare `u64` fails in two ways: + +- **Across datasets.** Two datasets can both be at version 7. The code already has this ambiguity: `temporal.rs`, `temporal_pov.rs` and `scheduler.rs` all use a bare `u64`, and `versioned.rs` returns only the nodes version of a write that spans three datasets. +- **Across epistemic modes.** A version alone does not say which reader horizon applies. + +D-RMAP-1A §7 (#1619) replaces the stand-in with a binding witness built from three parts: + +- `DurableRef{dataset, branch, version}`, defined in #1605 §1; +- an index seal, which carries the row-address map unless RelationFacts uses stable row ids; +- the read coordinate `QueryReference::at(v, rung)` from `temporal.rs`. Replay uses `Strict` and never `QueryReference::default()`. + +The Schnellverschluss does not touch the witness. Switching the backend between MORK, PathMap and native changes how the join runs; the witness describes which data was bound. When the witness lands, it replaces `generation` in this crate, and the backend switch stays as it is. Today no `DurableRef` type exists in `crates/`, and CPG-00 has not run. + +## 8. Deliverables + +| D-id | Item | Status | Gate / evidence | +|---|---|---|---| +| D-MPS-0 | Source and type inventory; source switch through Cargo `[patch]`; upstream-diff count | MEASURED | §1, §2, §4; `cargo tree -i pathmap` = one package | +| D-MPS-1 | Differential probe: MORK (product-zipper and leapfrog builds) and PathMap zipper vs a nested-loop oracle; typed refusals; generation witness; cost | MEASURED | 8 integration tests + 1 unit test pass in both builds; 4 disable runs, each turning its guard's test red; clippy `-D warnings` clean both ways | +| D-MPS-2 | Upstream (MORK): name PathMap as a git/version dependency instead of `path = "../PathMap/"` | PROPOSAL | MORK as a git dependency resolves without `[patch]` | +| D-MPS-3 | Upstream (MORK): `nightly` as an opt-in feature (the fork's `de9312e`, 18 files) | PROPOSAL; the board entry of 2026-10-16 says the maintainer opens it | upstream `main` builds on stable; the fork branch can be dropped | +| D-MPS-4 | Upstream (MORK): do not enable PathMap `jemalloc` in a library dependency (leave the allocator to binaries) | PROPOSAL | a consumer binary can set its own allocator | +| D-MPS-5 | Upstream (MORK): `UnifyJoin` / `query_multi_leapfrog` generic over a cursor provider + reindex hook | PROPOSAL | MORK's own differential corpus unchanged on `PathMap<()>` | +| D-MPS-6 | Lance-owned PathMap-trait cursor over a sealed `RelationOrder` view | BLOCKED: `RelationOrder` is not built (D-RMAP PROPOSED) | must show no copy, or state the copy it needs | +| D-MPS-7 | Licence: no MORK source moved into or redistributed from lance-graph until MORK declares a licence (PR #148 unmerged) | OPEN | the probe fetches MORK and vendors nothing | +| — | Native lance-graph arm for this join | BLOCKED by the no-gather ruling | a co-addressed producer for the input shape | + +## 9. Upstream contribution proposal (short) + +Each item is useful to MORK without lance-graph, and none carries a Lance type: + +1. **Dependency path** (D-MPS-2): `pathmap = { git = "https://github.com/Adam-Vandervorst/PathMap", … }` or a crates.io version in `[workspace.dependencies]`, keeping `path` only via a local `[patch]` for developers. +2. **Stable build** (D-MPS-3): move `nightly` out of `default` in `mork-expr`, `mork` and the workspace `pathmap` features, as `de9312e` does. +3. **Allocator** (D-MPS-4): drop `jemalloc` from the library's PathMap features and enable it from MORK's own binary (`kernel/src/main.rs`) instead. +4. **Generic join input** (D-MPS-5): `UnifyJoin<'a, Z>` with `Z: Zipper + ZipperMoving + ZipperPath + ZipperIteration`, the same bound `SubtermCursor` already uses, plus a hook for the reindex of inverted factors. + +No upstream PR is opened by this work. diff --git a/.github/workflows/mork-pathmap-probe.yml b/.github/workflows/mork-pathmap-probe.yml new file mode 100644 index 000000000..ae647b672 --- /dev/null +++ b/.github/workflows/mork-pathmap-probe.yml @@ -0,0 +1,43 @@ +name: MORK / PathMap probe +# `crates/mork-pathmap-probe` is excluded from the workspace (own [workspace], +# git deps on the MORK and PathMap forks), so no workspace job builds it. It +# runs from its own directory so its `.cargo/config.toml` (AES/SSE2 for MORK's +# gxhash) applies. +on: + pull_request: + paths: + - 'crates/mork-pathmap-probe/**' + - '.github/workflows/mork-pathmap-probe.yml' + - 'rust-toolchain.toml' + push: + branches: [main] + paths: + - 'crates/mork-pathmap-probe/**' + - '.github/workflows/mork-pathmap-probe.yml' + - 'rust-toolchain.toml' + workflow_dispatch: +permissions: + contents: read +jobs: + probe: + runs-on: ubuntu-latest + timeout-minutes: 30 + defaults: + run: + working-directory: crates/mork-pathmap-probe + env: + CARGO_PROFILE_DEV_DEBUG: "0" + CARGO_INCREMENTAL: "0" + steps: + - uses: actions/checkout@v4 + - name: Setup rust toolchain (pinned by rust-toolchain.toml) + run: rustup show + - run: cargo fmt --check + - name: clippy (product-zipper join) + run: cargo clippy --all-targets -- -D warnings + - name: clippy (leapfrog join) + run: cargo clippy --all-targets --features leapfrog -- -D warnings + - name: test (product-zipper join) + run: cargo test + - name: test (leapfrog join) + run: cargo test --features leapfrog diff --git a/Cargo.toml b/Cargo.toml index 5dc8f95df..a5be97e90 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -166,6 +166,10 @@ exclude = [ # crates/onebrc-probe/README.md. Verify via # `cargo test --manifest-path crates/onebrc-probe/Cargo.toml`. "crates/onebrc-probe", + # MORK / PathMap differential probe (D-MPS-1): its own workspace, git deps + # on the AdaWorldAPI forks, results only (D-ASM-10). Run from its directory + # so its `.cargo/config.toml` applies: `cd crates/mork-pathmap-probe && cargo test`. + "crates/mork-pathmap-probe", # X-C2-1 injection harness (RP-SEAL Tier 0) — fault injection over a # synthetic cycle; anti-vacuity gate reproduces the known in-tree # false-accepts against the REAL live implementations. Verify via diff --git a/crates/mork-pathmap-probe/.cargo/config.toml b/crates/mork-pathmap-probe/.cargo/config.toml new file mode 100644 index 000000000..5e70fa31f --- /dev/null +++ b/crates/mork-pathmap-probe/.cargo/config.toml @@ -0,0 +1,4 @@ +# MORK depends on a gxhash fork that refuses to compile without AES/SSE2 +# intrinsics. Scoped to this probe; never set RUSTFLAGS in the environment. +[build] +rustflags = ["-C", "target-feature=+aes,+sse2"] diff --git a/crates/mork-pathmap-probe/Cargo.toml b/crates/mork-pathmap-probe/Cargo.toml new file mode 100644 index 000000000..d82b4e325 --- /dev/null +++ b/crates/mork-pathmap-probe/Cargo.toml @@ -0,0 +1,40 @@ +[package] +name = "mork-pathmap-probe" +version = "0.0.0" +edition = "2024" +publish = false +description = "Differential probe: MORK and PathMap as result oracles behind one Lance-owned boundary module (Schnellverschluss). Results only; no Lance storage, no production executor." + +# Its own workspace root, excluded from the lance-graph workspace (D-ASM-10: +# MORK runs as an oracle outside the workspace). Run from THIS directory, so +# `.cargo/config.toml` applies: +# cd crates/mork-pathmap-probe && cargo test +[workspace] + +[features] +default = [] +# Compiles MORK's worst-case-optimal leapfrog join and routes conjunctive +# bodies to it (MORK's own feature). +leapfrog = ["mork/leapfrog"] + +# Source switch: every MORK and PathMap source is named here and nowhere else. +# Pinned revisions. To swap fork <-> upstream, change these three URLs/revs; +# no source file changes. +# +# MORK: AdaWorldAPI fork, branch `ccr-9f3b8075-74u47l-stable` (`de9312e`) = +# upstream `trueagi-io/MORK` `ed57c67` + one commit that makes `nightly` a +# feature instead of a default, so it builds on stable Rust. Upstream `main` +# needs nightly (`#![feature]` in fast-slice-utils). +# PathMap: AdaWorldAPI fork `ef02591`, byte-identical to upstream +# `Adam-Vandervorst/PathMap` `master` at the time of writing. +[dependencies] +mork = { git = "https://github.com/AdaWorldAPI/MORK", rev = "de9312e3" } +mork-expr = { git = "https://github.com/AdaWorldAPI/MORK", rev = "de9312e3" } +pathmap = { git = "https://github.com/AdaWorldAPI/PathMap", rev = "ef02591" } + +# MORK's workspace names PathMap as `path = "../PathMap/"`, outside its own +# repository, so a git dependency on MORK cannot resolve it. This patch points +# that one requirement at the same PathMap revision as above: one `pathmap` +# package identity in the graph, no edit to MORK. +[patch."https://github.com/AdaWorldAPI/MORK"] +pathmap = { git = "https://github.com/AdaWorldAPI/PathMap", rev = "ef02591" } diff --git a/crates/mork-pathmap-probe/examples/cost.rs b/crates/mork-pathmap-probe/examples/cost.rs new file mode 100644 index 000000000..6457e0f1d --- /dev/null +++ b/crates/mork-pathmap-probe/examples/cost.rs @@ -0,0 +1,76 @@ +//! Translation and query cost, and what one shared index saves over k +//! thoughts. Shared and rebuilt costs are printed on the same basis: per thought +//! (`*_per_thought_*`) and as a total over 64 thoughts (`*_total_k64`). Run: `cd crates/mork-pathmap-probe && cargo run --release --example cost` +//! (add `--features leapfrog` for MORK's leapfrog join). +//! +//! Timings are one run per cell on whatever machine runs this; read them as orders of magnitude. + +use std::time::Instant; + +use mork_pathmap_probe::{ + Backend, JoinInput, RelationView, Semantics, bind, boundary::mork_join, reference, +}; + +// No allocation counter: MORK turns on PathMap's `jemalloc` feature, which +// installs a `#[global_allocator]` inside the library, so a second one here +// does not link. That is itself a finding (see the plan's entropy ledger). + +/// Deterministic rows: R(x, y) with y in a small domain so the join fans out. +fn rows(n: usize, seed: u32, domain: u32) -> Vec<[u32; 2]> { + let mut v = Vec::with_capacity(n); + let mut s = seed; + for i in 0..n as u32 { + s = s.wrapping_mul(1_664_525).wrapping_add(1_013_904_223); + v.push([i, s % domain]); + } + v +} + +fn main() { + println!("mork join: {}", mork_join()); + println!( + "backend,n_rows,answers,build_ms,build_ns_per_row,query_ms,shared_per_thought_ms_k8,shared_per_thought_ms_k64,rebuilt_per_thought_ms,shared_total_ms_k64,rebuilt_total_ms_k64" + ); + for &n in &[1_000usize, 10_000, 50_000] { + let domain = (n as u32 / 4).max(1); + let r = rows(n, 7, domain); + // S(y, z): one z per y, so |answers| ~ |R|. + let s: Vec<[u32; 2]> = (0..domain).map(|y| [y, y + 1_000_000]).collect(); + let inp = JoinInput { + r: RelationView { + rows: &r, + generation: 1, + }, + s: RelationView { + rows: &s, + generation: 1, + }, + }; + let want = reference::answers(&inp, Semantics::Set); + for backend in [Backend::PathMapZipper, Backend::Mork] { + let t = Instant::now(); + let mut idx = bind(backend, &inp, Semantics::Set).unwrap(); + let build = t.elapsed(); + let t = Instant::now(); + let got = idx.answers(&inp).unwrap(); + let query = t.elapsed(); + assert_eq!(got, want, "{backend:?} n={n}"); + let ms = |d: std::time::Duration| d.as_secs_f64() * 1e3; + // One index built once and queried by k thoughts, per thought. + let shared_per_thought = |k: f64| ms(build) / k + ms(query); + let rebuilt_per_thought = ms(build) + ms(query); + println!( + "{backend:?},{n},{},{:.3},{:.0},{:.3},{:.3},{:.3},{:.3},{:.3},{:.3}", + got.rows.len(), + ms(build), + build.as_nanos() as f64 / (r.len() + s.len()) as f64, + ms(query), + shared_per_thought(8.0), + shared_per_thought(64.0), + rebuilt_per_thought, + ms(build) + 64.0 * ms(query), + 64.0 * rebuilt_per_thought, + ); + } + } +} diff --git a/crates/mork-pathmap-probe/src/boundary.rs b/crates/mork-pathmap-probe/src/boundary.rs new file mode 100644 index 000000000..f4802ac33 --- /dev/null +++ b/crates/mork-pathmap-probe/src/boundary.rs @@ -0,0 +1,281 @@ +//! The Schnellverschluss: the one module that names `mork` and `pathmap`. +//! +//! In: [`JoinInput`] (borrowed Lance rows + generations). Out: [`Answers`] +//! (owned `[u32; 3]` rows) or a typed [`Refusal`]. Nothing upstream-typed is +//! returned, so a backend can be replaced or deleted by editing this file and +//! `Cargo.toml` only. +//! +//! Both upstream indexes are COPIES. [`bind`] encodes the borrowed rows into a +//! PathMap byte trie (directly, or through MORK's S-expression parser) and the +//! index owns those bytes. That is the translation cost the probe measures; it +//! is not zero-copy, and nothing here claims it is. A stale copy is caught by +//! the generation witness ([`Refusal::Stale`]), not by a lifetime. + +use std::time::{Duration, Instant}; + +use mork::space::Space; +use pathmap::PathMap; +use pathmap::zipper::{ZipperIteration, ZipperPath}; + +use crate::{Answers, JoinInput, Semantics}; + +/// A physical implementation of the fixed query. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum Backend { + /// MORK's MM2 `exec` over its own `Space`. Which join MORK runs is fixed at + /// compile time by its `leapfrog` feature (this crate's `leapfrog` feature); + /// see [`mork_join`]. + Mork, + /// PathMap byte trie, read through read-only zippers. + PathMapZipper, + /// lance-graph's own RelAlg / Quack / mask-risc lowering. + Native, +} + +/// Why a backend did not answer. Every refusal is returned, never papered over. +#[derive(Clone, Debug, PartialEq, Eq)] +pub enum Refusal { + /// The backend stores a set (`PathMap<()>`, a MORK `Space`), so it cannot + /// report multiplicities. Asking it for [`Semantics::Bag`] is refused rather + /// than answered with every count collapsed to 1. + SetOnly(Backend), + /// No admitted native join exists. The join reads `S` at positions chosen by + /// `R`'s values, the shape the 2026-10-16 no-gather ruling forbids; the + /// admitted form is a co-addressed layout (D-HOLO H14c/H14d), which these + /// inputs are not in. + NoAdmittedNativeJoin, + /// The index was built from other generations than the caller's current + /// views. + Stale { + bound: (u64, u64), + current: (u64, u64), + }, + /// The upstream index rejected the encoded input. + Upstream(String), +} + +/// Which join the compiled MORK runs for a conjunctive body. +pub const fn mork_join() -> &'static str { + if cfg!(feature = "leapfrog") { + "leapfrog" + } else { + "product-zipper" + } +} + +// Proof that the leapfrog module is compiled when the feature is on: this +// fails to build if `mork::leapfrog` is absent. +#[cfg(feature = "leapfrog")] +#[allow(dead_code)] +type LeapfrogCursor<'a> = + mork::leapfrog::SubtermCursor>; + +/// Answers what a backend can do before anything is built. +pub fn capability(backend: Backend, semantics: Semantics) -> Result<(), Refusal> { + match (backend, semantics) { + (Backend::Native, _) => Err(Refusal::NoAdmittedNativeJoin), + (_, Semantics::Bag) => Err(Refusal::SetOnly(backend)), + (_, Semantics::Set) => Ok(()), + } +} + +enum Index { + Mork(Box), + Path(PathMap<()>), +} + +/// An upstream index built from one pair of generations. +pub struct BoundIndex { + backend: Backend, + generations: (u64, u64), + index: Index, + /// Rows handed to the upstream builder (duplicates included). + pub rows_in: usize, + /// Wall time of the encoding and index build. + pub build_time: Duration, +} + +/// Builds the backend's index from the borrowed rows. This is where the +/// translation cost is paid. +pub fn bind( + backend: Backend, + input: &JoinInput<'_>, + semantics: Semantics, +) -> Result { + capability(backend, semantics)?; + let start = Instant::now(); + let index = match backend { + Backend::Mork => Index::Mork(Box::new(mork_space(input)?)), + Backend::PathMapZipper => Index::Path(path_trie(input)), + Backend::Native => return Err(Refusal::NoAdmittedNativeJoin), + }; + Ok(BoundIndex { + backend, + generations: input.generations(), + index, + rows_in: input.r.rows.len() + input.s.rows.len(), + build_time: start.elapsed(), + }) +} + +impl BoundIndex { + pub fn backend(&self) -> Backend { + self.backend + } + + /// Answers the query against the views the caller holds now. Refuses if + /// they are not the generations this index was built from. + pub fn answers(&mut self, current: &JoinInput<'_>) -> Result { + let now = current.generations(); + if now != self.generations { + return Err(Refusal::Stale { + bound: self.generations, + current: now, + }); + } + match &mut self.index { + Index::Mork(space) => mork_answers(space), + Index::Path(trie) => Ok(path_answers(trie)), + } + } +} + +// ---- MORK ------------------------------------------------------------------ + +/// Symbols are `n`, so the text round trip is exact. +fn mork_space(input: &JoinInput<'_>) -> Result { + let mut text = Vec::with_capacity(16 * (input.r.rows.len() + input.s.rows.len())); + for (name, rows) in [("R", input.r.rows), ("S", input.s.rows)] { + for &[a, b] in rows { + text.extend_from_slice(format!("({name} n{a} n{b})\n").as_bytes()); + } + } + let mut space = Space::new(); + space.add_all_sexpr(&text).map_err(Refusal::Upstream)?; + Ok(space) +} + +const MORK_QUERY: &[u8] = b"(exec 0 (, (R $x $y) (S $y $z)) (, (Q $x $y $z)))\n"; + +/// MM2 `exec` writes its answers into the space as `Q` facts. They are read +/// back and removed again, so the index holds only `R` and `S` between calls. +fn mork_answers(space: &mut Space) -> Result { + space.add_all_sexpr(MORK_QUERY).map_err(Refusal::Upstream)?; + space.metta_calculus(1); + let mut dump = Vec::new(); + space.dump_all_sexpr(&mut dump).map_err(Refusal::Upstream)?; + let mut out = Answers::default(); + let mut produced = Vec::new(); + for line in dump.split(|&b| b == b'\n') { + if let Some(rest) = line.strip_prefix(b"(Q ") { + let row = parse_q(rest) + .ok_or_else(|| Refusal::Upstream(String::from_utf8_lossy(line).into_owned()))?; + out.rows.insert(row, 1); + produced.extend_from_slice(line); + produced.push(b'\n'); + } + } + if !produced.is_empty() { + space + .remove_all_sexpr(&produced) + .map_err(Refusal::Upstream)?; + } + Ok(out) +} + +fn parse_q(rest: &[u8]) -> Option<[u32; 3]> { + let body = rest.strip_suffix(b")")?; + let mut ids = body.split(|&b| b == b' ').map(|tok| { + std::str::from_utf8(tok.strip_prefix(b"n")?) + .ok()? + .parse::() + .ok() + }); + let row = [ids.next()??, ids.next()??, ids.next()??]; + ids.next().is_none().then_some(row) +} + +// ---- PathMap --------------------------------------------------------------- + +const R_TAG: u8 = b'R'; +const S_TAG: u8 = b'S'; + +/// `R(x, y)` is keyed `R ‖ y ‖ x` and `S(y, z)` is keyed `S ‖ y ‖ z` +/// (big-endian), so both are ordered by the shared variable. +fn path_trie(input: &JoinInput<'_>) -> PathMap<()> { + let mut trie = PathMap::new(); + let mut key = [0u8; 9]; + for (tag, rows, swap) in [(R_TAG, input.r.rows, true), (S_TAG, input.s.rows, false)] { + key[0] = tag; + for &[a, b] in rows { + let (first, second) = if swap { (b, a) } else { (a, b) }; + key[1..5].copy_from_slice(&first.to_be_bytes()); + key[5..9].copy_from_slice(&second.to_be_bytes()); + trie.insert(key, ()); + } + } + trie +} + +/// For every `R ‖ y ‖ x`, walks the `S ‖ y` subtrie. Oracle code: the seek into +/// `S` at a position chosen by `R`'s value is exactly what lance-graph's +/// admitted path may not do. +fn path_answers(trie: &PathMap<()>) -> Answers { + let mut out = Answers::default(); + let mut r = trie.read_zipper_at_path([R_TAG]); + while r.to_next_val() { + let p = r.path(); + let y = u32::from_be_bytes(p[0..4].try_into().unwrap()); + let x = u32::from_be_bytes(p[4..8].try_into().unwrap()); + let mut prefix = [S_TAG; 5]; + prefix[1..5].copy_from_slice(&y.to_be_bytes()); + let mut s = trie.read_zipper_at_path(prefix); + while s.to_next_val() { + let z = u32::from_be_bytes(s.path()[0..4].try_into().unwrap()); + out.rows.insert([x, y, z], 1); + } + } + out +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::RelationView; + + fn lines(space: &Space) -> usize { + let mut dump = Vec::new(); + space.dump_all_sexpr(&mut dump).unwrap(); + dump.split(|&b| b == b'\n') + .filter(|l| !l.is_empty()) + .count() + } + + /// The MORK index holds exactly the `R` and `S` facts before and after a + /// query: the `Q` answers `exec` writes are removed again. + #[test] + fn mork_query_leaves_only_the_input_facts() { + let r = [[1, 10], [2, 10]]; + let s = [[10, 5]]; + let inp = JoinInput { + r: RelationView { + rows: &r, + generation: 1, + }, + s: RelationView { + rows: &s, + generation: 1, + }, + }; + let mut idx = bind(Backend::Mork, &inp, Semantics::Set).unwrap(); + let Index::Mork(space) = &idx.index else { + panic!("mork index") + }; + assert_eq!(lines(space), 3); + assert_eq!(idx.answers(&inp).unwrap().rows.len(), 2); + let Index::Mork(space) = &idx.index else { + panic!("mork index") + }; + assert_eq!(lines(space), 3); + } +} diff --git a/crates/mork-pathmap-probe/src/lib.rs b/crates/mork-pathmap-probe/src/lib.rs new file mode 100644 index 000000000..341d2b0c9 --- /dev/null +++ b/crates/mork-pathmap-probe/src/lib.rs @@ -0,0 +1,78 @@ +//! MORK / PathMap differential probe behind one Lance-owned boundary. +//! +//! STATUS: PROBE (results only). Not a production executor, not wired into +//! Quack, R2IL or mask-risc, and not a fact store. Plan: +//! `.claude/plans/mork-pathmap-schnellverschluss-v1.md`. +//! +//! The query is fixed: `Q(x, y, z) :- R(x, y), S(y, z)` — two relations that +//! share one variable. Every arm answers the same question; [`reference`] is the +//! independent oracle that names no upstream type. +//! +//! Ownership, as the module layout states it: +//! +//! - [`RelationView`] is the Lance side: borrowed rows plus the generation of the +//! sealed view they came from. It is the probe's stand-in for a sealed +//! `RelationOrder` view, which does not exist in code yet (D-RMAP is PROPOSED). +//! - [`boundary`] is the only module that names `mork` or `pathmap`. Replacing +//! or removing a backend touches that module and `Cargo.toml`, nothing else. +//! - No upstream type crosses the boundary in either direction: callers pass +//! `[u32; 2]` rows in and get `[u32; 3]` rows out. A MORK `Expr` is a raw +//! pointer with no lifetime, so it never leaves the boundary module. + +use std::collections::BTreeMap; + +pub mod boundary; +pub mod reference; + +pub use boundary::{Backend, BoundIndex, Refusal, bind, capability}; + +/// Borrowed rows of one binary relation and the generation of the sealed view +/// they were read from. Row order and duplicates are the caller's: a bag. +#[derive(Clone, Copy, Debug)] +pub struct RelationView<'a> { + pub rows: &'a [[u32; 2]], + pub generation: u64, +} + +/// The two inputs of `Q(x, y, z) :- R(x, y), S(y, z)`. +#[derive(Clone, Copy, Debug)] +pub struct JoinInput<'a> { + pub r: RelationView<'a>, + pub s: RelationView<'a>, +} + +impl JoinInput<'_> { + pub fn generations(&self) -> (u64, u64) { + (self.r.generation, self.s.generation) + } +} + +/// One answer row `[x, y, z]`. +pub type Row = [u32; 3]; + +/// Whether duplicate derivations are counted. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum Semantics { + /// Each distinct answer once. + Set, + /// Each answer as many times as it is derived (`|R row| × |S row|` pairs). + Bag, +} + +/// Answers with their multiplicities. Under [`Semantics::Set`] every count is 1. +#[derive(Clone, Debug, Default, PartialEq, Eq)] +pub struct Answers { + pub rows: BTreeMap, +} + +impl Answers { + pub fn to_set(&self) -> Answers { + Answers { + rows: self.rows.keys().map(|r| (*r, 1)).collect(), + } + } + + pub fn total(&self) -> u64 { + self.rows.values().map(|&c| u64::from(c)).sum() + } +} diff --git a/crates/mork-pathmap-probe/src/reference.rs b/crates/mork-pathmap-probe/src/reference.rs new file mode 100644 index 000000000..497baf9fe --- /dev/null +++ b/crates/mork-pathmap-probe/src/reference.rs @@ -0,0 +1,20 @@ +//! Independent oracle: a nested loop over the borrowed rows. Names no upstream +//! type and shares no code with [`crate::boundary`]. + +use crate::{Answers, JoinInput, Semantics}; + +/// `Q(x, y, z) :- R(x, y), S(y, z)` by nested loop. +pub fn answers(input: &JoinInput<'_>, semantics: Semantics) -> Answers { + let mut out = Answers::default(); + for &[x, y] in input.r.rows { + for &[y2, z] in input.s.rows { + if y == y2 { + *out.rows.entry([x, y, z]).or_insert(0) += 1; + } + } + } + match semantics { + Semantics::Bag => out, + Semantics::Set => out.to_set(), + } +} diff --git a/crates/mork-pathmap-probe/tests/differential.rs b/crates/mork-pathmap-probe/tests/differential.rs new file mode 100644 index 000000000..c9421f0de --- /dev/null +++ b/crates/mork-pathmap-probe/tests/differential.rs @@ -0,0 +1,145 @@ +//! Every admitted arm against the independent nested-loop oracle. + +use mork_pathmap_probe::{ + Backend, JoinInput, Refusal, RelationView, Semantics, bind, boundary::mork_join, capability, + reference, +}; + +fn input<'a>(r: &'a [[u32; 2]], s: &'a [[u32; 2]], gen_r: u64, gen_s: u64) -> JoinInput<'a> { + JoinInput { + r: RelationView { + rows: r, + generation: gen_r, + }, + s: RelationView { + rows: s, + generation: gen_s, + }, + } +} + +const ARMS: [Backend; 2] = [Backend::Mork, Backend::PathMapZipper]; + +fn check_set(r: &[[u32; 2]], s: &[[u32; 2]]) -> usize { + let inp = input(r, s, 1, 1); + let want = reference::answers(&inp, Semantics::Set); + for arm in ARMS { + let mut idx = bind(arm, &inp, Semantics::Set).expect("set semantics is admitted"); + let got = idx.answers(&inp).expect("fresh generations"); + assert_eq!( + got, + want, + "{arm:?} ({}) disagrees with the oracle", + mork_join() + ); + // Asking twice gives the same answer: the MORK arm removes its `Q` facts. + assert_eq!(idx.answers(&inp).unwrap(), want, "{arm:?} second call"); + } + want.rows.len() +} + +#[test] +fn shared_variable_join_matches_the_oracle() { + let r = [[1, 10], [1, 11], [2, 10], [3, 12]]; + let s = [[10, 100], [11, 101], [10, 102], [99, 103]]; + // (1,10,100) (1,10,102) (1,11,101) (2,10,100) (2,10,102); y = 12 and 99 match nothing. + assert_eq!(check_set(&r, &s), 5); +} + +#[test] +fn empty_and_disjoint_inputs_give_no_answers() { + assert_eq!(check_set(&[], &[[1, 2]]), 0); + assert_eq!(check_set(&[[1, 2]], &[]), 0); + assert_eq!(check_set(&[[1, 2]], &[[3, 4]]), 0); +} + +/// `x` and `z` occur in one atom each; only `y` is shared. A cross product +/// over `x` and `z` per `y` is the whole answer, no more. +#[test] +fn variable_scope_is_per_atom() { + let r = [[1, 7], [2, 7], [3, 7]]; + let s = [[7, 4], [7, 5]]; + assert_eq!(check_set(&r, &s), 6); +} + +/// Values that differ only in byte order or width must not alias. +#[test] +fn large_ids_do_not_alias() { + let r = [[u32::MAX, 0x0100_0000], [0, 1]]; + let s = [[1, u32::MAX], [0x0100_0000, 2], [0x0000_0001, 3]]; + assert_eq!(check_set(&r, &s), 3); +} + +/// Duplicate rows: the oracle counts 2 × 2 = 4 derivations; the upstream +/// indexes are sets and must refuse a bag request instead of answering 1. +#[test] +fn duplicates_are_refused_under_bag_semantics() { + let r = [[1, 10], [1, 10]]; + let s = [[10, 5], [10, 5]]; + let inp = input(&r, &s, 1, 1); + assert_eq!( + reference::answers(&inp, Semantics::Bag).rows[&[1, 10, 5]], + 4 + ); + for arm in ARMS { + assert!(matches!(bind(arm, &inp, Semantics::Bag), Err(Refusal::SetOnly(a)) if a == arm)); + } + // Under set semantics they agree with the oracle's set. + assert_eq!(check_set(&r, &s), 1); +} + +#[test] +fn a_rebound_generation_is_refused() { + let r = [[1, 10]]; + let s = [[10, 5]]; + let bound = input(&r, &s, 4, 9); + for arm in ARMS { + let mut idx = bind(arm, &bound, Semantics::Set).unwrap(); + // Same rows, but the R view was resealed: the copy is stale. + let current = input(&r, &s, 5, 9); + assert_eq!( + idx.answers(¤t), + Err(Refusal::Stale { + bound: (4, 9), + current: (5, 9) + }) + ); + // Silence twin: the matching generations still answer. + assert!(idx.answers(&bound).is_ok()); + } +} + +/// A generation that comes back after a reseal (ABA) is accepted: the witness +/// compares numbers, so the caller must never reuse one. Pinned, not hidden. +#[test] +fn a_reused_generation_number_is_not_detected() { + let r1 = [[1, 10]]; + let r2 = [[2, 10]]; + let s = [[10, 5]]; + let mut idx = bind( + Backend::PathMapZipper, + &input(&r1, &s, 1, 1), + Semantics::Set, + ) + .unwrap(); + let got = idx.answers(&input(&r2, &s, 1, 1)).unwrap(); + assert!( + got.rows.contains_key(&[1, 10, 5]), + "answers come from the bound copy" + ); + assert!(!got.rows.contains_key(&[2, 10, 5])); +} + +#[test] +fn native_join_is_refused_not_emulated() { + assert_eq!( + capability(Backend::Native, Semantics::Set), + Err(Refusal::NoAdmittedNativeJoin) + ); + let r = [[1, 10]]; + let s = [[10, 5]]; + assert!(matches!( + bind(Backend::Native, &input(&r, &s, 1, 1), Semantics::Set), + Err(Refusal::NoAdmittedNativeJoin) + )); +}