lab(D-DR256-OD): on-demand global mask, u16 scatter, bounded reuse, L1 interference

.claude/board/entries/2026-10-26-dr256-on-demand-l1.md

@@ -0,0 +1,159 @@
+# 2026-10-26 — D-DR256-OD: on-demand global mask, native u16 scatter, bounded reuse, L1 interference
+
+**Status:** PRE-REGISTERED (this section is committed before any measurement). Results are appended below.
+Lab only: `crates/drf-focus-probe/examples/dr256_on_demand.rs`. Follows #1790 (D-DR256), does not extend it.
+ndarray primitive: `mask_scatter_or_u16` (AdaWorldAPI/ndarray, separate PR).
+
+## Pre-registration
+
+**Fixtures.** The five #1790 fixtures (64 registers × 4,096 slots, u16 maps): chaotic 50 %, clustered 64×64,
+scattered 4,096, empty, full. Each thought t carries its own global plane `X_t` (8 KiB, ~50 % dense, seeded).
+
+**Two questions a thought asks.**
+- `any_t = (G ∧ X_t) ≠ ∅`. This needs no global dedup and can be answered locally.
+- `count_t = |G ∧ X_t|`. This needs dedup over global addresses.
+
+**Arms (each answers the same question; all preparation charged).**
+- A, local only (any): per register, walk the set bits of `L_r` and test `X_t[ord_r[s]]`, with early exit. No
+  global mask.
+- B, u16 on demand (any, count): per thought, zero G, run 64 × `mask_scatter_or_u16`, then fold `G ∧ X_t`.
+- C, u32 on demand (any, count): widen the 64 maps once per binding (charged once over N), then per thought the
+  same as B with `mask_scatter_or_u32`.
+- D, bounded reuse (any, count): per thought, check the witness; build G once with u16 when the witness is new;
+  fold `G ∧ X_t`. The witness covers source, scope, every map (id, seal) and every register's selection
+  generation.
+- E, direct without a shared G (count): per thought, per register, `M_r = L_r ∧ gather(X_t, ord_r)`, then
+  `mask_scatter_or_u16(M_r)` into scratch and popcount. No G is shared or kept.
+- N ∈ {1, 2, 4, 8, 64}. Report the total and the cost per thought. Compatible case: one witness for all N.
+  Incompatible cases: one register's selection changes between thoughts; one map's seal changes. D must
+  rebuild, and the answer must match the oracle.
+
+**Correctness.** Every arm's mask and answer must equal an independent set oracle (BTreeSet), compared as whole
+masks, not popcounts. A D variant that skips the witness check must give a wrong answer on the incompatible
+case (disable run).
+
+**L1 interference.** The resident DR64 fold `F` is `masked_sum_i32` over one register's 512 B mask and its 16 KiB
+gathered i32 column, a ~16.5 KiB working set (L1d here is 48 KiB per core). F is timed inside four loops of N =
+64 thoughts:
+1. F alone;
+2. F interleaved with D's per-thought fold over the live G and `X_t`;
+3. F interleaved with B's per-thought rebuild;
+4. F interleaved with A's local walk.
+
+No hardware counters are available (no `perf`). Interference is inferred only from F's time, never from
+buffer sizes.
+
+**Accounting (stated, not measured as residency).** Live bytes, peak scratch bytes, number and lifetime of 8 KiB
+masks, u16 map bytes reachable versus selected ordinals actually read, u32 bytes eliminated. Hot-loop
+allocations come from a counting allocator and must be 0 after setup.
+
+**Decision rules.**
+- u16 = u32 bit for bit.
+- Reuse crossover: N* = the smallest N with D_total(N) < B_total(N), per fixture and question, or "no
+  crossover".
+- Interference: D harms the resident fold if F's median in loop 2 exceeds loop 1 by more than 10 % in both of
+  two runs.
+- Contract: a scattered G read by later thoughts is the shape `ir.rs:329-337` forbids. Any benefit is recorded
+  as lab evidence for a contract proposal, never applied to mask-risc.
+
+## Results (two runs, avx512f, 4 cores, L1d 48 KiB; µs per workload, median of 15 batches, run 1 / run 2)
+
+**Correctness.**
+- Every arm's mask and answer equals the BTreeSet oracle on all five fixtures. Masks were compared whole, and
+  E's mask equals `G ∧ X`.
+- u16 and u32 masks are identical.
+- Witness controls:
+  - a compatible second thought reuses the mask;
+  - a selection change rebuilds, and the stale mask gives a wrong count (load-bearing);
+  - a rebinding at a new seal rebuilds;
+  - release-then-regenerate is exact.
+- Hot-path allocations after setup: 0 (counting allocator, all arms).
+
+**count (needs global dedup), total µs over N thoughts**
+
+| fixture | N | B u16 per thought | C u32 per thought | D reuse | E no shared G |
+|---|---|---|---|---|---|
+| clustered 64×64 | 1 | 9.1 / 8.7 | 46.3 / 56.2 | 9.7 / 11.3 | 12.9 / 17.5 |
+| clustered 64×64 | 2 | 18.1 / 20.6 | 56.9 / 71.3 | 9.6 / 9.2 | 50.5 / 48.8 |
+| clustered 64×64 | 64 | 764 / 605 | 784 / 703 | 30.0 / 26.6 | 2,839 / 2,240 |
+| scattered 4,096 | 1 | 10.1 / 6.2 | 48.2 / 47.8 | 10.4 / 6.8 | 18.3 / 11.6 |
+| scattered 4,096 | 64 | 626 / 416 | 820 / 670 | 32.0 / 20.8 | 2,237 / 2,184 |
+| chaotic 50 % | 1 | 135 / 203 | 201 / 210 | 138 / 137 | 244 / 252 |
+| chaotic 50 % | 64 | 9,181 / 8,798 | 11,899 / 10,909 | 148 / 146 | 19,278 / 16,567 |
+| full | 1 | 252 / 276 | 348 / 351 | 255 / 260 | 407 / 398 |
+| full | 64 | 27,453 / 18,272 | 23,358 / 20,666 | 290 / 286 | 27,376 / 30,373 |
+
+- Reuse crossover N* = 2 on every fixture with a selection. At N = 1, D and B do the same work, and D's
+  witness check is within noise.
+- With a valid shared window the cost per thought falls to 0.3–0.5 µs (clustered, scattered) and 2.3–4.5 µs
+  (chaotic, full) at N = 64.
+- Rebuilding per thought at chaotic/full density crosses into milliseconds at N = 64. That is a failed gate
+  under the workspace rule, and it is the measured case against Mistake A.
+- E is the slowest count arm on every fixture: gathering X through the maps and scattering survivors costs more
+  than one scatter of the whole selection.
+- u16 vs u32: at N = 1 the 37–56 µs widening dominates the clustered and scattered cases (9 vs 46 µs). The
+  1 MiB u32 lane is gone. Amortised over 64 thoughts the u32 scatter itself is no faster.
+
+**any**
+- A (local, no global mask) answers in 3–21 ns per thought, because X is 50 % dense and the walk exits on the
+  first hit.
+- On the empty selection A has nothing to exit on and costs 2.4–4.9 µs per thought, against D's 0.3 µs at
+  N = 64.
+- A's advantage therefore depends on how early a hit appears. Sparse X was not measured (not registered).
+
+**Incompatible thoughts** (one register's selection changes every thought): D rebuilds each time, as required,
+and costs B ± 15 % (e.g. chaotic N = 64: 8,675 µs vs 9,405 µs; scattered 486 vs 413 µs). Reuse gives nothing
+without a stable window, and the witness check costs about 1 µs.
+
+**L1 interference** (resident fold F = `masked_sum_i32`, 512 B mask + 16 KiB column; ns p50, both runs)
+
+| fixture | F alone | F + D fold over live G | F + B rebuild | F + A walk |
+|---|---|---|---|---|
+| chaotic | 879 / 786 | 892 / 786 | 1,022 / 811 | 883 / 786 |
+| clustered | 255 / 160 | 254 / 160 | 285 / 181 | 257 / 160 |
+
+- Keeping the one 8 KiB G live and folding over it does not slow F: +1.5 % / 0 % and −0.4 % / 0 %. The
+  pre-registered harm rule does not fire for D.
+- Rebuilding per thought slows F by 11.8 % / 13 % on clustered, which is over 10 % in both runs, and by 16 % /
+  3 % on chaotic. That arm reads up to 512 KiB of u16 maps.
+- Absolute F differs between runs (879 vs 786 ns). Only within-run comparisons are valid, and no hardware
+  counters exist here.
+
+**Working set (accounting from buffer sizes, not measured residency)**
+- Authoritative and unchanged: 64 u16 maps = 512 KiB; 64 local masks = 32 KiB.
+- Derived: B, C, D, E hold one 8 KiB global mask (caller scratch); A holds none. D's lives for the compatible
+  window only.
+- E adds 512 B of scratch. C adds the 1 MiB u32 lane, which u16 removes.
+- Map bytes read per build = 2 B × selected slots: 8 KiB (clustered, scattered), 256 KiB (chaotic), 512 KiB
+  (full). Which cache lines those touch was not measured.
+
+## Decisions
+
+- **ndarray `mask_scatter_or_u16`: implemented** (separate ndarray PR). Same walker as the u32 path, no
+  widening, reference + disable tests, parity crate check `0xD12`.
+- **mask-risc exposure: designed, not implemented.**
+  - Smallest form: add `LaneRef::U16`, and let `ScatterOrU32` / `ScatterCountU32` read either lane width.
+    Width is a bind fact (law 2), resolved once per tile and never per element. No new terminal (law 6: the
+    semantics are identical).
+  - Deferred for two reasons: mask-risc pins ndarray `master`, so it waits for the ndarray PR; and a new
+    `LaneRef` variant touches every exhaustive match in mask-risc and its consumers.
+- **Bounded thought sharing is NOT lawful today.** A scattered G read by later thoughts is the shape
+  `ir.rs:329-337` forbids. D's numbers are lab evidence only.
+
+## Contract proposal (needs authorisation; not implemented)
+
+A **witnessed derived population**:
+- Produced only as a demanded `ScatterOr*` result into a caller-owned 8 KiB `Out::Mask`.
+- Read only as a `Planes::masks` input to folds over the same 65,536 universe: `Count`, `Any`, and `Keep` of
+  aligned `And`/`Ternlog`. Never the input of another scatter or gather, never mutated in place, never used as
+  a Lance block list, never persisted, no Thought-Git record.
+- Valid only while its witness holds: source (dataset, branch, version, `index_seal`), scope, universe size,
+  every map (id, seal), every register's selection generation. Any mismatch rebuilds or refuses, never a reuse
+  by equal dimensions.
+- Lifetime: one named compatible thought window. The scratch returns to the caller after the last reader.
+
+Evidence for it: crossover N* = 2, no resident-fold interference, and a load-bearing witness. Evidence against
+widening it: E and the incompatible case show no benefit without a stable window.
+
+OPEN: sparse-X `any` (where A may lose its early exit); hardware-counter confirmation of the L1 result; whether
+any consumer has a compatible window of N ≥ 2 in practice.

.claude/board/entries/README.md

@@ -25,10 +25,11 @@ index row, (3) no duplicate entry id. Checks 1 and 2 are deliberately
 opposite directions; the stranding this convention prevents shows up in
 exactly one of them, never both.
 
-479 entries, 2026-08-06 .. 2026-10-26.
+480 entries, 2026-08-06 .. 2026-10-26.
 
 | date | entry id | finding | file |
 |---|---|---|---|
+| 2026-10-26 | `D-DR256-OD` |  | [2026-10-26-dr256-on-demand-l1.md](2026-10-26-dr256-on-demand-l1.md) |
 | 2026-10-26 | `D-DR256` |  | [2026-10-26-dr256-global-union-lab.md](2026-10-26-dr256-global-union-lab.md) |
 | 2026-10-26 | `D-DQM` | FAIL | [2026-10-26-dqm-p1-chunk-touched.md](2026-10-26-dqm-p1-chunk-touched.md) |
 | 2026-10-26 | `coresearch-lance13-masked-read-dr64` |  | [2026-10-26-coresearch-lance13-masked-read-dr64.md](2026-10-26-coresearch-lance13-masked-read-dr64.md) |