D-CMR-2b probe: no win on whole extents; short runs pay per-run executor cost

.claude/board/LATEST_STATE.md

@@ -1,3 +1,7 @@
+## 2026-10-21 — D-CMR-2b measured: no win on whole extents; short runs pay per-run cost (in PR)
+
+- **lance-graph-benches (lab example only, no library change):** `examples/pred_scratch_probe.rs`. On whole extents and 40k-row runs a fused compare-and-count fold does not beat today's tiled `execute_extent`. On `set_lower`'s run shape with 200-row runs, per-run `execute_extent` is 2.5–4× slower than the fold; compiling once cuts it by only 5–18 %. Remedy (fewer edge-tile passes, a run-batched entry point, or a short-run fold) not decided, not built. Entry `entries/2026-10-21-d-cmr-2b-scratch-probe.md`. Contract inventory: unchanged.
+
 ## 2026-10-21 — main after #1654 (merged `d754128`)
 
 - **On main:** #1652, #1653, #1654 (D-RA-2 repoint of relalg `symbolic` onto `contract::set_rules`; `Rule::{EmptyMeasure, CountAll}`; #1651 closeout). Contract inventory delta: two `set_rules::Rule` variants (see the D-RA-2 section below).

.claude/board/STATUS_BOARD.md

@@ -256,6 +256,7 @@ Plan: `.claude/plans/d-cmr-concept-chain-normalization-v1.md`. Parent: D-CMR-RES
 | **D-CMR-0** | Plan, identity matrix, inventory, board rows | Shipped #1532 (docs) | docs only; every inventory row cites code |
 | **D-CMR-1** | Set-theory IR core in the contract: `Runs` (∩ ∪ ∖, complement), residual `Res`, measures; pure | Shipped (#1644, `c05a313`): `contract::set_algebra::{SetExpr, Res, Runs, normalize}`; measures not yet (`entries/2026-10-20-d-cmr-1-set-algebra.md`) | algebra laws vs materialized bitsets as oracle |
 | **D-CMR-2** | Lowering: runs → extents, residual → predicates per run; measurement levels geometry / optics / etch | Shipped (#1650, `f031089`): `lance_graph::fold::set_lower::{SetPlan, SortedBasis, Level, Measured}`; geometry reads nothing, optics/etch run the residual per run via `execute_extent`. The scratch-free `Pred` fold in mask-risc is D-CMR-2b, not built (`entries/2026-10-21-d-cmr-2-lowering.md`) | == row oracle; `Count` without residual reads zero rows |
+| **D-CMR-2b** | Scratch-free `Pred` fold in mask-risc (lab first) | Measured (`lance-graph-benches/examples/pred_scratch_probe.rs`; `entries/2026-10-21-d-cmr-2b-scratch-probe.md`): no win on whole extents or 40k-row runs; on 200-row runs per-run `execute_extent` is 2.5–4× slower than the fold, compile-once cuts A by 5–18 %. Remedy not decided, not built | answers == oracle before timing; interleaved paired timing with a null arm |
 | **D-CMR-3** | Quack frontend: `Filter` → set IR (extent hoisting falls out) | Queued | identical answers on the quack suite; other-basis range stays residual |
 | **D-CMR-4** | Two dimensions: `Rect` over composite coordinates, row-major or Morton runs | Queued | rectangle ∩ == bitset oracle |
 | **D-CMR-5** | **BLOCKED: lawful design proposed (operator 2026-10-21).** Preimage `f⁻¹(S)` as a co-addressed transposed tile fold over a version-witnessed projection of `f`, built in preparation by a sequential same-table scan with frozen-codebook endpoint translation (ADMISSION B; reusable across queries). `S` enters only as a membership plane translated into the same frozen coordinates. No seek or load of `f` keyed by `S`'s values or a previous fold (Q1 = C), no `Gather`, no scatter terminal. R1 (`rmap` `rebind` / `discover` escape-driven seeks) stays option C. ⊘ Was: "VIOLATION — GATHER. DO NOT BUILD" (entry `2026-10-21-d-cmr-5-rescope.md`) | BLOCKED — needs a contract-level attested witness for the frozen codebook (`rmap::Witness` is a probe-local stand-in); no preimage operator exists; not admitted, not ready to build | == row oracle; program checked Gather-free with a silence twin; attest / projection / query costs reported apart; a many-to-one `f` gives a valid distinct-source preimage (tested) and a bag-sensitive Count/Sum over it is refused (tested); refusal on a stale or missing witness |

.claude/board/entries/2026-10-21-d-cmr-2b-scratch-probe.md

@@ -0,0 +1,23 @@
+# 2026-10-21 — D-CMR-2b phase 1: a scratch-free `Pred` fold does not pay on whole extents; on `set_lower`'s short runs, per-run `execute_extent` is 2.5–4× slower
+
+**Status:** MEASURED (lab probe, no production change). **Basis:** D-CMR-2's open item (`entries/2026-10-21-d-cmr-2-lowering.md`); operator authorization 2026-10-21; conditions of `gpt-ada-01/007`; claim EXO-50 `session_013z2e1YXkm6o6HNrDn5BPWA/023`.
+
+- **Dependencies:** ndarray sibling at `origin/master` `37a3b59` (workspace `[patch]`); `ndarray::simd` has every compare→mask primitive (`*_i32_to_mask[_under]`, `*_u32_to_mask[_under]`), `mask_ternlog_popcount`, `popcount_batch_u64`; no fused compare-and-count primitive. mask-risc lib 47/47 against it. Host AVX-512 (Xeon @ 2.10 GHz, 4 cores), `target-cpu=native`; the probe prints the tier.
+- **What scratch is today:** caller-owned, `slots × TILE_WORDS` (256 words = 2 KiB per slot), allocated once per `Scratch::for_program`, never population-sized. So the question was per-op tile write-back / re-read and the separate `And` + count passes, not heap memory.
+- **Probe:** `crates/lance-graph-benches/examples/pred_scratch_probe.rs`, run with `cargo run --release -p lance-graph-benches --example pred_scratch_probe`. Shapes `S1` Count(a), `S2` Count(a∧b), `S2u` b gated `under` a, `S3` Count(a∧b∧plane) over `i32` lanes; ~1 / 50 / 99 % selectivity. Two workloads: **whole** (one extent `0..n`, n = 16k / 1M / 4M) and **runs** (`set_lower`'s shape: over n = 1M, K non-overlapping runs of 200, 3,000 or 40,000 rows at odd starts, about a quarter of the rows). Arms: **A** `execute_extent` per extent with reused scratch; **A+alloc** with `Scratch::for_program` once per call (what `SetPlan::count` does); **Ac** A with the program compiled once (`Program::compile` + `execute_compiled`); **B** compares into fresh chunks + one `mask_ternlog_popcount`, plane shifted into a window when a run starts mid-word; **B′** the same over reused chunks. Every configuration asserted against a bit-serial oracle before timing. Timing interleaved per rep with rotating order, median of 101 per-rep ratios, plus a null arm (A timed twice).
+- **Result (3 runs; ranges over the 3 selectivities × 3 runs):**
+
+  | workload | null A′/A | A+alloc/A | Ac/A | B′/A |
+  |---|---|---|---|---|
+  | whole 16k (1 tile) | 0.989–0.999 | 1.026–1.090 | 0.979–1.005 | S1 1.40–1.46, S2/S2u/S3 1.06–1.26 |
+  | whole 1M | 0.957–1.012 | 0.971–1.027 | 0.971–1.033 | S1 1.04–1.20, S2/S2u/S3 0.95–1.04 |
+  | whole 4M | 0.984–1.014 | 0.989–1.023 | 0.987–1.011 | 0.970–1.028 |
+  | 1,310 runs × 200 | 0.952–1.033 | 0.981–1.023 | 0.821–0.948 | **0.236–0.524** (S3 0.24–0.27, S2 0.30–0.33) |
+  | 87 runs × 3,000 | 0.971–0.997 | 0.989–1.023 | 0.905–1.027 | S2/S3 0.67–0.80, S2u 0.86–0.93, S1 0.74–1.06 |
+  | 6 runs × 40,000 | 0.971–0.999 | 0.985–1.029 | 0.993–1.029 | 1.00–1.45 |
+
+- **Reading, whole extents and long runs:** no win for the fold; it is equal or slower. The earlier version of this probe (same fold, whole extents only, before the move to `lance-graph-benches`) measured B′/A 0.95–1.05 at 16k; this version measures 1.06–1.46 with the same A. The cause is not found. 64-byte alignment of the chunks was tried and does not change it. Either way the fold does not beat A there.
+- **Reading, short runs:** at 200 rows per run, per-run `execute_extent` costs about 170–520 ns a run (S1 → S3) and the fold is 2.5–4× faster. Compiling once (`Ac`) cuts A by only 5–18 %, so re-deriving the lowering per call is a small part of the gap. From the code (not attributed by measurement): `extent_tiles` executes an unaligned extent as a head-edge word, a body and a tail-edge word, each running every op and the terminal, and `execute_compiled` validates on every call. Per-call allocation (`A+alloc`) costs nothing measurable.
+- **Decision input:** a `Pred` `Lowering` arm for whole extents (phase 2 as first framed) is not supported. The measured gap is per-run cost on short runs, which is what `set_lower` produces at optics/etch. Not decided which remedy (fewer passes per edge tile in `execute_extent`, a run-batched entry point, or a short-run fold); not built.
+
+OPEN: (1) where the per-run cost of `execute_extent` goes on a 200-row run — not attributed between edge-tile passes, validation and op dispatch; (2) why this probe's fold is slower at full tiles than the earlier version. Not measured: `u32` / strided lanes, more than two comparisons, `Keep` terminals, other tiers (AVX2, NEON), real `set_lower` run distributions.

.claude/board/entries/README.md

@@ -25,7 +25,7 @@ index row, (3) no duplicate entry id. Checks 1 and 2 are deliberately
 opposite directions; the stranding this convention prevents shows up in
 exactly one of them, never both.
 
-406 entries, 2026-08-06 .. 2026-10-21.
+407 entries, 2026-08-06 .. 2026-10-21.
 
 | date | entry id | finding | file |
 |---|---|---|---|
@@ -37,6 +37,7 @@ exactly one of them, never both.
 | 2026-10-21 | `D-RA-2` |  | [2026-10-21-d-ra-2-relalg-symbolic-repoint.md](2026-10-21-d-ra-2-relalg-symbolic-repoint.md) |
 | 2026-10-21 | `D-CMR-6` |  | [2026-10-21-d-cmr-6-rule-catalogue.md](2026-10-21-d-cmr-6-rule-catalogue.md) |
 | 2026-10-21 | `D-CMR-5` |  | [2026-10-21-d-cmr-5-rescope.md](2026-10-21-d-cmr-5-rescope.md) |
+| 2026-10-21 | `D-CMR-2b` |  | [2026-10-21-d-cmr-2b-scratch-probe.md](2026-10-21-d-cmr-2b-scratch-probe.md) |
 | 2026-10-21 | `D-CMR-2` |  | [2026-10-21-d-cmr-2-lowering.md](2026-10-21-d-cmr-2-lowering.md) |
 | 2026-10-20 | `D-RU-1` |  | [2026-10-20-u1-jc-batch.md](2026-10-20-u1-jc-batch.md) |
 | 2026-10-20 | `thought-git-replay-api-write-rubicon` |  | [2026-10-20-thought-git-replay-api-write-rubicon.md](2026-10-20-thought-git-replay-api-write-rubicon.md) |

.claude/plans/d-cmr-concept-chain-normalization-v1.md

@@ -211,6 +211,7 @@ when the first floating-point or quantized column arrives, not before.
 |---|---|---|
 | D-CMR-1 | **Set IR core** in the contract: `Runs` with `∩ ∪ ∖` and complement, `Res`, measures; pure, no lowering | algebra laws (closure, distributivity, De Morgan, `∅`) checked against materialized bitsets as oracle on random and edge-case runs |
 | D-CMR-2 | **Lowering** to mask-risc: runs → extents, residual → predicates within each run, the three measurement levels. Level 2 for residual comparisons needs predicate fusion in mask-risc (`fused_ternlog` declines `Pred` today) | == row oracle; a `Count` with no residual reads zero rows (instrumented); disabling level 1 or 2 keeps answers |
+| D-CMR-2b | **Scratch-free `Pred` fold** (lab first, phase 2 only on a measured win) | interleaved paired timing against today's tiled `execute_extent` with a null arm; measured 2026-10-21: no win on whole extents; on `set_lower`'s 200-row runs per-run `execute_extent` is 2.5–4× slower, remedy open (`entries/2026-10-21-d-cmr-2b-scratch-probe.md`) |
 | D-CMR-3 | **Quack normalization pass:** over `Filter`, computing set values (ranges on a proven basis → `Runs`, other leaves → `Res`, `sql_where` first). Extent hoisting falls out of this (Experiment C) | answers identical to today's `lower` path on the quack suite; a range on another basis stays residual (negative control) |
 | D-CMR-4 | **Two dimensions:** `Rect` over composite coordinates, linearized row-major or in Morton order | rectangle `∩` == bitset oracle; run count within the stated bound |
 | D-CMR-5 | **BLOCKED: lawful design proposed (operator 2026-10-21).** Preimage `f⁻¹(S)` as a co-addressed transposed tile fold over a version-witnessed projection of `f`, built in preparation by a sequential same-table scan with frozen-codebook endpoint translation (ADMISSION B; reusable across queries). `S` enters only as a membership plane translated into the same frozen coordinates. No seek or load of `f` keyed by `S`'s values or a previous fold (Q1 = C), no `Gather`, no scatter terminal. R1 (`rmap` `rebind` / `discover` escape-driven seeks) stays option C. ⊘ Was: "VIOLATION — GATHER. DO NOT BUILD" (entry `2026-10-21-d-cmr-5-rescope.md`) | == row oracle; program checked Gather-free with a silence twin; attest / projection / query costs reported apart; a many-to-one `f` gives a valid distinct-source preimage (tested) and a bag-sensitive Count/Sum over it is refused (tested); refusal on a stale or missing witness |

crates/lance-graph-benches/Cargo.toml

@@ -24,6 +24,10 @@ datafusion = { workspace = true }
 serde_json = "1"
 lance-graph-quack = { path = "../lance-graph-quack" }
 lance-graph-mask-risc = { path = "../lance-graph-mask-risc" }
+# `examples/pred_scratch_probe.rs` (D-CMR-2b) calls the `ndarray::simd`
+# compare and fold primitives directly for its scratch-free arm. Same package
+# as mask-risc's git coordinate through the workspace `[patch]`.
+ndarray = { path = "../../../ndarray", default-features = false, features = ["std"] }
 
 [[bench]]
 name = "graph_execution"