D-SRAM-2 spike: masked Lance 13 reads and the addressed semijoin

.claude/board/STATUS_BOARD.md

@@ -355,8 +355,8 @@ Plan: `.claude/plans/ce64-hypothesis-layers-v1.md`. Plateau: `.claude/knowledge/
 | **D-TGA-4** | land through the revision gate into the next Lance version (D-HL-8) | Queued | landing a thought that is behind is refused |
 | **D-TGA-5** | squash_into (partial squash) and blame | Queued | blame returns original sources after slot reuse |
 | **D-TGA-6** | MergeQueue | Queued | independent thoughts == landed by hand in FIFO order; a thought reading another negated lands after it; a cycle through negation refused at enqueue |
-| **D-TGB-0** | thought git basics: fixtures per write-intention outcome, squash-stack and anomaly fixtures, absent-not-zero fixture, from-scratch oracle (`thought-git-basics-v1.md`) | In PR (lab probe `crates/thought-git-probe`, 19 tests; `PREREGISTRATION.md`) | 5 oracle disables verified red-then-green; every signal has a can-fire and a can-stay-silent case on real contract code; the pearl half on edges moves to D-TGB-3 |
-| **D-TGB-1** | slot pool, autonomous: taken by `branch`, freed by `squash`/`discard`/`land` (the write is the Rubicon decision, 2026-10-20) | In PR (lab probe `crates/thought-git-probe/src/slots.rs`) | at 63 taken `branch` picks a FIFO or least-used candidate, squashes it along its dependency chain if it has one, else evicts it (DECISIONS 2026-10-20; was: squash unremarkable first, else refuse); stale handles refused; 6 tests |
+| **D-TGB-0** | thought git basics: fixtures per write-intention outcome, squash-stack and anomaly fixtures, absent-not-zero fixture, from-scratch oracle (`thought-git-basics-v1.md`) | Shipped (lab, #1630; probe `crates/thought-git-probe`, 19 tests; `PREREGISTRATION.md`) | 5 oracle disables verified red-then-green; every signal has a can-fire and a can-stay-silent case on real contract code; the pearl half on edges moves to D-TGB-3 |
+| **D-TGB-1** | slot pool, autonomous: taken by `branch`, freed by `squash`/`discard`/`land` (the write is the Rubicon decision, 2026-10-20) | Shipped (lab, #1630/#1631; `crates/thought-git-probe/src/slots.rs`). Slot pool only: no production land, rebase or persistence (D-TGB-2..5 Queued) | at 63 taken `branch` picks a FIFO or least-used candidate, squashes it along its dependency chain if it has one, else evicts it (DECISIONS 2026-10-20; was: squash unremarkable first, else refuse); stale handles refused; 6 tests |
 | **D-TGB-2** | `w_slot` lane labels (W = thought layer, not a pointer) qualified by base version / slot generation | Queued (lab) | no live cell carries a landed slot; old-generation edge refused |
 | **D-TGB-3** | `Commit` lands one sparse batch through `LanceCycleWriter`; `DurableRef` = dataset + `Ref::Version(branch, version)` | Queued (lab) | empty cycle writes nothing; re-submit reconciles; two branches at version 7 told apart |
 | **D-TGB-4** | receipt (base ref, program id, seeds, slot generation) instead of history | Queued (lab) | replay from receipt reproduces writes cell for cell |

.claude/board/entries/2026-10-20-crepe-rule-optimizer-spike.md

@@ -18,4 +18,6 @@
 
 - Dynamic MeTTa rules need crepe's analysis as a library: crepe is a proc-macro crate (follow-up A, `crepe-core` split).
 - Semijoin reduction built (`Options::semijoin`): laid-out rows 36 at every size instead of growing with n; preparation 3.47 ms → 0.28 ms at 50,000. The scan stays O(n).
-- ndarray for the row filter, measured (`examples/row_filter.rs`): over the row-major store the strided compare is slower than the scalar test at every needle count, so it is not wired; over a contiguous column it is 5.8 µs per needle at 50,000 rows and beats the scalar test up to 64 needles. Next step: a column-major relation store.
+- ndarray for the row filter, measured (`examples/row_filter.rs`): over the row-major store the strided compare is slower than the scalar test at every needle count, so it is not wired; over a contiguous column it is 5–6 µs per needle at 50,000 rows and beats the scalar test up to 64 needles; at 128 needles two runs disagree because of benchmark noise (the scalar time moved 1.8× between runs), so that crossover is not pinned (re-measured after the #1632 Codex fix). Next step: a column-major relation store.
+- Stage 3 over Lance 13 / lancedb 0.40 built (`lance_store`): each side read under a relalg address mask with `with_row_addr_prefilter`; every masked read scans exactly its mask (P1–P4 green). The relalg membership mask holds 4 centers at most: `k` centers take `2k - 1` scratch slots (no slot reuse inside a fold), and `MAX_SLOTS = 8` is an r2il setting, not a hard limit (mask-risc addresses 65,536). Above that a scalar test builds it.
+- Arm B built (`addressed`): lanes in key order, attested with `SealedFacetLane::attest_sorted`, read only at the `seek_from` ranges of the demanded keys. At 50,000 rows it scans `3k` rows where Arm A scans at least 150,000. It is 2.5–12× faster than Arm A up to 1% of centers and pays back its 7 ms attest after 2–3 queries. From 10% of centers up a full scan wins, and with all centers Arm B never breaks even. Open: the selectivity crossover between 1% and 10%; multi-fragment lanes (an append currently forces a compaction); a second lane for demands on the second column.

.claude/board/entries/2026-10-20-d-tgb-thought-git-basics.md

@@ -14,3 +14,7 @@ OPEN: card granularity (mailbox card or per-move card); the PR entry under MemWA
 Correction (2026-10-20, review #1628): `LanceCycleWriter` commits one batch per supervisor cycle, so every `Commit` in a cycle coalesces into one batch and one version; a landed thought is (version, thought id within the batch). D-TGB-2 is blocked on the `w_slot` reading decision.
 
 Correction (2026-10-20, operator): D-TGB-2 is not blocked. W = thought layer 0..63 is the 2026-10-17 decision (#1597); D-TGB-2 carries the migration of the competing readers (`MailboxSoA::apply_edges`, `deepnsm-v2`'s `WitnessTable` index).
+
+## Status note (2026-10-20, later)
+
+The full thought git API (`thought-git-api-v1.md`, D-TGA) is still a PROPOSAL. `crates/thought-git-probe` implements the slot pool (D-TGB-1), the from-scratch oracle and the tests (D-TGB-0). It does not implement the production land, rebase or persistence pipeline (D-TGB-2..5 are Queued).

.claude/plans/crepe-rule-optimizer-spike-v1.md

@@ -28,6 +28,9 @@ Pipeline, one stage per module of the probe:
 - [x] Phase 5: differential tests A–M against `crepe!`, datafrog, MORK, reference; 9 disable runs
 - [x] Phase 6 (limited): Soufflé harvest table (§3)
 - [x] Semijoin reduction before preparation (`Options::semijoin`, `native::prepare_semijoin`; §2b, tests N)
+- [x] Stage 3 over Lance 13 / lancedb 0.40: masked read per side, P1–P4 green (§2d); P5 measured at 50,000 rows only (§2e)
+- [x] CI workflow for this probe (`.github/workflows/crepe-plan-probe.yml`)
+- [x] Arm B, the addressed semijoin over attested lanes (§2e, tests B1–B9)
 
 ## 1. Integration point
 
@@ -95,17 +98,76 @@ The semijoin filter (and the constant restriction) is a scalar test per row: bin
 
 | needles | scalar µs | ndarray over the rows µs | ndarray over a contiguous column µs |
 |---|---|---|---|
-| 1 | 17.4 | 82.4 | 5.8 |
-| 8 | 144.4 | 654.8 | 29.6 |
-| 32 | 284.9 | 2,869.7 | 120.7 |
-| 64 | 329.0 | 5,434.8 | 242.8 |
-| 128 | 375.4 | 10,603.4 | 485.2 |
+| 1 | 30.9 | 82.7 | 5.4 |
+| 8 | 146.4 | 665.9 | 30.5 |
+| 32 | 225.3 | 2,892.2 | 114.9 |
+| 64 | 278.2 | 4,988.1 | 238.0 |
+| 128 | 518.1 | 10,043.3 | 462.0 |
+
+Re-measured 2026-10-20 after the Codex P2 fix on #1632. The contiguous arm used to return the mask; it now collects the kept rows like the other two arms, and the example asserts that its rows equal the scalar test's. Collecting the rows costs at most 1.8 µs per run.
 
 - Over the store as it is (`[u32; 2]` rows, the D-MPS-8 `RelationView` shape) the strided kernel is slower than the scalar test at every needle count: its field loads are scalar by design, and it pays one pass per needle where the binary search pays one pass in total. It is therefore **not** used in `prepare`.
-- Over a contiguous column the same compares are 14–20× faster than the strided ones and beat the scalar test up to 64 needles; at 128 the scalar test wins again (the per-needle passes add up).
+- Over a contiguous column the same compares are 14–22× faster than the strided ones, and they beat the scalar test up to 64 needles in both runs. At 128 needles the two runs disagree: this run 462 vs 518 µs, the first run 485 vs 375 µs. The disagreement is benchmark noise, not a property of the kernels: the scalar time moved by up to 1.8× between runs (17.4 vs 30.9 µs at one needle). The crossover lies near 64–128 needles; pinning it needs a quieter benchmark (more repetitions, pinned CPU). At 1,000 rows the scalar test wins at 128 (10.0 vs 7.2 µs).
 - The one-pass alternative, a needle bitmap read through `mask_gather_u32`, is a gather and stays out (2026-10-16 ruling, test H).
 - Consequence for the next step: a column-major relation store (one contiguous `u32` lane per column) is what makes ndarray applicable here, and it is the same change the key-column index needs.
 
+## 2d. Stage 3 over Lance 13 (pre-registered 2026-10-20, before any run)
+
+DECISION (operator, 2026-10-20): "ja, bau Stufe 3 mit Lance 13 und CI-Workflow" / "Lance 13 Lancedb 0.40 is mandatory".
+
+The relations are stored column-major in Lance: one table per relation, columns `c0`, `c1` (`UInt32`), written through lancedb 0.40 and read through lance 13. Preparation becomes H14d's stages 2 and 3 per side:
+
+1. Read the one column the address mask needs (the restriction column, or the key column for the semijoin), with `_rowaddr`.
+2. Build the address mask `R` through relalg over that column as a `U32` lane: `eq_u32(col, k)` for a restriction, `Or` of `eq_u32(key, c)` for the semijoin centers (lowered by mask-risc onto `ndarray::simd::eq_u32_to_mask`). Above 4 centers the mask is built by the scalar membership test instead. relalg lowers `Or(a1..ak)` left-associatively, and every atom and every `OR` takes a scratch slot of its own (no reuse inside a fold), so `k` centers take `2k - 1` slots. One fold may use `lance_graph_r2il::MAX_SLOTS = 8`, a setting the crate calls "sized with headroom", so 5 centers (9 slots) fail with `ScratchBufferTooSmall`. mask-risc addresses up to 65,536 slots. Either a larger arena or slot reuse inside the fold (an `Or` chain needs only the accumulator and the current atom) would lift the limit; both are r2il/relalg changes. This is not the §2c speed crossover, which is ndarray's row filter.
+3. Read the side's columns with `with_row_addr_prefilter(R)`.
+4. Hand those rows to `JoinTable::prepare_star`, as the in-memory path does.
+
+A side's generation is its Lance table version.
+
+Predictions, each with its falsifier:
+
+- P1. Answers equal the in-memory path and the nested-loop reference on every existing fixture (A, B, C, F, J, N) for set, bag, count and exists. Falsifier: any difference.
+- P2. For every masked read, Lance's `rows_scanned` equals `|R|` exactly, and the rows returned equal `|R|`. Falsifier: any read that scans more rows than `R` admits.
+- P3. Rows handed to `prepare_star` equal the in-memory semijoin path's `rows_prepared` exactly (11 and 5 on the N fixtures). Falsifier: any difference.
+- P4. A write that adds a table version makes a table prepared under the old version stale (`JoinRefusal::Stale`). Falsifier: the old table is accepted.
+- P5 (measured, not predicted as a number): bytes read and wall time per stage at 1,000 / 10,000 / 50,000 rows, against reading every row. No direction is predicted for bytes: H14d found bytes follow page locality, not `|R|`.
+
+Disables: build `R` from the wrong column (P1, P2 red); drop the prefilter (P2 red); skip the version check (P4 red).
+
+## 2e. Arm B: the addressed semijoin (measured 2026-10-20)
+
+DECISION (operator, 2026-10-20): "They are ordered by design. The only issue is the relationship maps proposed earlier pulling in global to local which would make the nodeguid order unusual. Build arm b".
+
+A relation is written in key order: row `(a, b)` carries `FacetCascade::from_semantic_tiles([0, 0, a_hi, a_lo, b_hi, b_lo, 0, 0])`, which compares as `(a, b)`. The probe never orders a lane itself. `addressed::attest` reads the keys once and calls `SealedFacetLane::attest_sorted`, which refuses an unsorted lane. A side then reads only the lane ranges of its demanded keys, located with `seek_from` under the lane's `OrderedLaneWitness`, through `with_row_addr_prefilter`. The arm adds no mask column, index, opcode, gather or global-to-local mapping.
+
+The arm refuses, by name:
+- `NotLeadingColumn`: a restriction or demand on the second column (that would need a second lane, ordered by that column);
+- `Stale`: a table that moved on since its lane was attested;
+- `NotSorted`: an unsorted table, at attest;
+- `NotContiguous`: a table split over more than one fragment, at attest. An append adds a fragment, so after one the table must be compacted before it can be addressed again.
+
+Measured with `cargo run --release --example addressed` (lance 13, lancedb 0.40, rustc 1.99.0). The query is `Answer(17, z)` over `family`. There are N = 50,000 background edges with distinct `y`, all `Allowed`, and `k` children of 17 spread over them, so there are exactly `k` centers and `k` answers. Times are medians of 9 runs. "Rows" counts rows Lance scanned, mask columns included.
+
+| k | full ms | Arm A ms | Arm B warm ms | attest ms | full rows | Arm A rows | Arm B rows | break-even |
+|---|---|---|---|---|---|---|---|---|
+| 0 | 9.81 | 4.46 | 0.37 | 7.28 | 150,000 | 150,000 | 0 | 2 |
+| 1 | 9.95 | 5.02 | 2.19 | 6.83 | 150,003 | 150,005 | 3 | 3 |
+| 8 | 12.30 | 6.60 | 2.24 | 6.67 | 150,024 | 150,040 | 24 | 2 |
+| 500 (1%) | 10.42 | 8.86 | 3.47 | 6.71 | 151,500 | 152,500 | 1,500 | 2 |
+| 5,000 (10%) | 11.70 | 15.02 | 12.11 | 7.81 | 165,000 | 175,000 | 15,000 | 3 |
+| 25,000 (50%) | 18.55 | 41.39 | 34.54 | 7.74 | 225,000 | 275,000 | 75,000 | 2 |
+| 50,000 (all) | 30.71 | 36.15 | 38.98 | 9.84 | 300,000 | 400,000 | 150,000 | never |
+
+Break-even is `ceil(P / S)`, where `P` is the attest time and `S` is Arm A's per-query time minus Arm B warm's.
+
+What this says:
+- **Source-row reads do go down, by construction, and exactly.** Arm B scans `3k` rows: one row per demanded key on each of the three sides. Arm A scans at least `150,000`, because each side reads its key column in full to build the mask. The answer to "does demand-aware semijoin reduction decrease source-row reads" is yes for Arm B and no for Arm A. Arm A reduces the rows laid out (§2b), not the rows read.
+- **Time follows the rows only at low selectivity.** Up to 1% of centers, Arm B is 2.5–12× faster than Arm A and the attest pays for itself after 2–3 queries. Its floor is about 2 ms for three masked reads, even for 3 rows; with no centers it reads nothing and takes 0.37 ms.
+- **From 10% of centers up, every masked read loses to a full scan.** Arm A is already slower than the full read at 10%, and both arms are slower at 50%. With all centers, Arm B never breaks even. Many small ranges cost Lance more per row than one sequential scan. A planner must switch to the full read above a crossover between 1% and 10%; that crossover is not measured more finely here.
+- `P` (reading and attesting 100,002 keys over three tables) is 7–10 ms: a cold cost the size of one full query.
+
+P5 is measured at 50,000 rows only; 1,000 and 10,000 were not run.
+
 ## 3. Soufflé harvest (souffle-lang/souffle `f53dab8`, UPL-1.0; algorithms only, no source copied)
 
 | Soufflé | transformation | precondition | in crepe | in lance-graph | taken |
@@ -148,6 +210,9 @@ Known limits: rule dedup does not search body permutations; the star must have o
 ## 6. Follow-ups (not built)
 
 - ~~Semijoin reduction before preparation~~ — built, §2b. Layout O(k); the scan stays O(n).
+- ~~Addressed read~~ — built, §2e. Reads O(k) rows over attested lanes.
+- Selectivity crossover: measure between 1% and 10% of centers and switch Arm B to the full read above it.
+- Multi-fragment lanes: map lane positions to row addresses per fragment, so that an append does not force a compaction before Arm B can read the table again.
 - **Next (recommended):** a column-major relation store (contiguous `u32` lanes). It lets the restriction and semijoin filters run through `ndarray::simd::eq_u32_to_mask` (§2c: 4–6 µs per needle at 50,000 rows instead of a scalar or strided scan), and it is the layout a key-column index would sit on, which then removes the O(n) scan.
 - A: `crepe-core` library split in the fork for dynamic rules.
 - B: full magic sets for derived body relations and recursion (waits on D-MTP-3 / D-SRAM-4).

.github/workflows/crepe-plan-probe.yml

@@ -0,0 +1,51 @@
+name: Crepe plan probe
+# `crates/crepe-plan-probe` is excluded from the workspace (own [workspace],
+# git deps on the crepe and MORK forks, lance 13 / lancedb 0.40), so no
+# workspace job builds it. It runs from its own directory so the cargo config
+# above it and its own `.cargo/config.toml` apply. It path-deps relalg,
+# mask-risc, mork-pathmap-probe and lance-graph-contract (the ordered lane and
+# facet keys of arm B), so changes there run it too.
+on:
+  pull_request:
+    paths:
+      - 'crates/crepe-plan-probe/**'
+      - 'crates/lance-graph-relalg/**'
+      - 'crates/lance-graph-contract/**'
+      - 'crates/lance-graph-mask-risc/**'
+      - 'crates/mork-pathmap-probe/**'
+      - '.github/workflows/crepe-plan-probe.yml'
+      - 'rust-toolchain.toml'
+  push:
+    branches: [main]
+    paths:
+      - 'crates/crepe-plan-probe/**'
+      - 'crates/lance-graph-relalg/**'
+      - 'crates/lance-graph-contract/**'
+      - 'crates/lance-graph-mask-risc/**'
+      - 'crates/mork-pathmap-probe/**'
+      - '.github/workflows/crepe-plan-probe.yml'
+      - 'rust-toolchain.toml'
+  workflow_dispatch:
+permissions:
+  contents: read
+jobs:
+  probe:
+    runs-on: ubuntu-latest
+    timeout-minutes: 60
+    defaults:
+      run:
+        working-directory: crates/crepe-plan-probe
+    env:
+      CARGO_PROFILE_DEV_DEBUG: "0"
+      CARGO_INCREMENTAL: "0"
+    steps:
+      - uses: actions/checkout@v4
+      - name: Setup rust toolchain (pinned by rust-toolchain.toml)
+        run: rustup show
+      # lance's build compiles .proto files: `protoc` alone is not enough, the
+      # well-known includes (libprotobuf-dev) are needed too.
+      - name: Install protoc and the protobuf includes (for lance)
+        run: sudo apt-get update && sudo apt-get install -y protobuf-compiler libprotobuf-dev
+      - run: cargo fmt --check
+      - run: cargo clippy --all-targets -- -D warnings
+      - run: cargo test

crates/crepe-plan-probe/Cargo.toml

@@ -21,12 +21,25 @@ lance-graph-mask-risc = { path = "../lance-graph-mask-risc" }
 # The D-MPS-8 versioned relationship table (shared y domain, presence planes,
 # generations) and the MORK/PathMap oracle behind boundary.rs.
 mork-pathmap-probe = { path = "../mork-pathmap-probe" }
+# Arm B: the attested key-ordered lane (`SealedFacetLane`, `OrderedLaneWitness`)
+# and the FacetCascade keys.
+lance-graph-contract = { path = "../lance-graph-contract" }
+# Stage 3 (D-HOLO-H14d): the relations live in a Lance table and are read
+# under the address mask (`with_row_addr_prefilter`). lance 13 / lancedb 0.40
+# move in lockstep, patch floated, as in the workspace manifest.
+lance = { version = "13.*", default-features = false }
+lancedb = { version = "0.40.*", default-features = false, features = ["remote"] }
+arrow-array = "58"
+arrow-schema = "58"
+futures = "0.3"
+tokio = { version = "1.37", features = ["macros", "rt-multi-thread"] }
 
 [dev-dependencies]
 # The SIMD facade, for the row-filter comparison in examples/row_filter.rs.
 # Same coordinate as mask-risc and mork-pathmap-probe, so the graph keeps one
 # ndarray package.
 ndarray = { git = "https://github.com/AdaWorldAPI/ndarray", branch = "master", default-features = false, features = ["std"] }
+tempfile = "3"
 datafrog = { git = "https://github.com/AdaWorldAPI/datafrog", branch = "master" }
 
 [patch."https://github.com/AdaWorldAPI/lance-graph"]