diff --git a/.claude/board/INTEGRATION_PLANS.md b/.claude/board/INTEGRATION_PLANS.md index 6a337cee3..4d669202e 100644 --- a/.claude/board/INTEGRATION_PLANS.md +++ b/.claude/board/INTEGRATION_PLANS.md @@ -1,3 +1,8 @@ +## 2026-11-03 — D-DRAM-0: DR address memoization + adaptive Abakus mask lens → `.claude/plans/d-dram-abacus-address-memoization-v1.md` + +**Status:** PROPOSED / DOCS ONLY. **Board:** [STATUS_BOARD](STATUS_BOARD.md) D-DRAM; [entry](entries/2026-11-03-d-dram-abacus-address-memoization.md); [EXO-50](https://linear.app/exo-red/issue/EXO-50/agent-message-bus-gpt-xgrok-ccsession-id) coordination. Separate focus-scoped, witnessed receipt for already resolved canonical SoA locations, *no copy of values and no mutation of `LocalMap::globals`*. Quack dynamic ClassView/WideFieldMask scope can test shipped ndarray half/full-adder words for k-of-n masks; operator's exact 5×24 symbols require source verification. Existing Lance masked reads stay in their owner, D4/S-INGRESS gate the DR↔Lance binding, and generic payload gather remains prohibited outside the narrow D-DRF-1 §5 exception. Gates D-DRAM-1..5 are proposals, not authorized production tasks. +**Confidence:** source-grounded owner/ABI map; no executable implementation, measured benefit or public Abakus type. + ## 2026-11-03 — D-XLOG-WIRE: XLOG wiring gaps to the hot path → `.claude/plans/xlog-wiring-to-hot-path-v1.md` **Status:** PLAN. Gating input for D-CODEGRAPH-0-BIN B3 (operator 2026-11-03). **Board:** [STATUS_BOARD](STATUS_BOARD.md) D-XLOG-WIRE; [entry](entries/2026-11-03-a10-writers-retract-xlog-pr.md). diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index 942f098d9..45ccfa884 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -1,3 +1,18 @@ +## D-DRAM — DR address memoization + adaptive Abakus masking lens (2026-11-03) + +**Status:** PROPOSED / DOCS ONLY; [plan](../plans/d-dram-abacus-address-memoization-v1.md), [board entry](entries/2026-11-03-d-dram-abacus-address-memoization.md), [EXO-50](https://linear.app/exo-red/issue/EXO-50/agent-message-bus-gpt-xgrok-ccsession-id). No code, tests, new runtime, new row-id map or production authorization. + +| D-id | Deliverable | Status | Owner / gate | +| --- | --- | --- | --- | +| D-DRAM-0 | design and falsifier map: borrowed SoA locator receipts, dynamically scoped Abakus mask algebra | Proposed, documentation only | review; EXO-50 message is not approval | +| D-DRAM-1 | read-only census of existing DR focus locator lease and exact ndarray 5×24 symbols | Queued, not authorized | EXO-46, ndarray owner | +| D-DRAM-2 | lab memoized address receipts; no values copied, exact version/scope and lifetime refusal | GATED | EXO-46, EXO-89, EXO-151; no second coordinate authority | +| D-DRAM-3 | Quack k-of-n/threshold Abakus comparison with shipped facade words | Proposed lab | Quack/ndarray owners; no automatic new opcode/API | +| D-DRAM-4 | combined ClassView selective borrowing + Lance read probe | GATED for end-to-end DR→Lance | D4 and S-INGRESS from #1788 | +| D-DRAM-5 | engineering-parity and cost verdict, handoff | Queued | independent falsifiers and owner decision | + +Verified: ndarray #402 half/full adder words and #390 byte sum shipped, `Abacus` lab-only; DR `LocalMap` is identity-only; H14c/H14d and #1633 selective Lance reads are lab. Unverified: exact operator-mentioned `ndarray::simd::5x24::*` signature. **Borrow/prefetch default; materialized foreign facet only under D-DRF-1 §5 admission.** + ## D-DT64-7 — behaviour addressing and registration: Frozen, Learned, Explore over one classid (2026-10-31) Plan: `.claude/plans/d-dt64-7-behaviour-addressing-registration-v1.md`. Docs only. Frozen = compile time, at `0xFFFF :: 0xF###`, registered by DT64 as delegate for `lance-graph-ogar` without `ogar-vocab` (concern by range: relalg `F0__`, XLOG `F1__`, ONNX `F2__`, TF family incl. TFLite `F3__`, SQL-specific `F4__`, Gremlin-specific `F5__`; DECIDED O7); Learned = registered in the `BindStore` at `Domain::AppId :: 0xF###`; Explore = not registered. Right side = ClassView / `WideFieldMask` selection. O7–O10 DECIDED (plan §1d): host composition, full-classid basis for behaviour lenses, consumer-supplied `ClassView` with a basis witness. diff --git a/.claude/board/entries/2026-11-03-d-dram-abacus-address-memoization.md b/.claude/board/entries/2026-11-03-d-dram-abacus-address-memoization.md new file mode 100644 index 000000000..6f95f9cc0 --- /dev/null +++ b/.claude/board/entries/2026-11-03-d-dram-abacus-address-memoization.md @@ -0,0 +1,34 @@ +# D-DRAM-0: address memoization in DR, adaptive Abakus mask lens + +**Date:** 2026-11-03. **Status:** PROPOSED / DOCS ONLY. No executable code, public API, performance result or production authorization. +**Plan:** [D-DRAM-0](../../plans/d-dram-abacus-address-memoization-v1.md). +**Coordination:** [EXO-50](https://linear.app/exo-red/issue/EXO-50/agent-message-bus-gpt-xgrok-ccsession-id), new append-only message; issue status is not implementation consent. + +## Operator intention + +1. After resolving a canonical SoA address under an existing witnessed DR tile/local coordinate, **remember that locator receipt** in the active focus/binding lease for repeated borrow and prefetch. Do **not** copy payload values into DR or append physical locators into `LocalMap::globals` (global identity list). +2. Let ClassView/WideFieldMask select the logical property footprint and let Quack choose existing mask/bit-sliced SIMD algebra through ndarray. Abakus may be useful for arithmetic thresholds (k-of-n), not default AND/OR. +3. Borrow canonical values and use selective Lance 13 source fetch when necessary. Payload gathering remains the D-DRF-1 §5 narrowly admitted foreign-cache exception, never a generic default optimizer choice. + +## Verified anchors + +- [DR MapTable](https://github.com/AdaWorldAPI/lance-graph/blob/main/crates/lance-graph-dt64/src/dr.rs): stable identity mapping, append-only coordinate history, not physical SoA pointers. +- [ClassView](https://github.com/AdaWorldAPI/lance-graph/blob/main/crates/lance-graph-contract/src/class_view.rs) and [DT64](https://github.com/AdaWorldAPI/lance-graph/blob/main/crates/lance-graph-dt64/src/lib.rs): WideFieldMask borrowed words and ValueDemand separation. +- [D-RMAP-1A](../../plans/d-rmap-separate-tables-thought-palettes-v1.md), [D-DRF-1](../../plans/d-dr64-focus-lease-foreign-cache-v1.md), [D-DRAV-1](../../plans/d-dr-address-first-value-demand-v1.md): owners, borrowed default and admitted exception. +- [ndarray #402](https://github.com/AdaWorldAPI/ndarray/pull/402): half/full adder masks SHIPPED; `Abacus` remained LAB/DEFERRED. [ndarray #390](https://github.com/AdaWorldAPI/ndarray/pull/390): masked byte-sum shipped. Operator's exact `ndarray::simd::5x24::*` symbol is **UNVERIFIED**, census first. +- [#1573](https://github.com/AdaWorldAPI/lance-graph/pull/1573), [#1575](https://github.com/AdaWorldAPI/lance-graph/pull/1575), [#1633](https://github.com/AdaWorldAPI/lance-graph/pull/1633): Lance 13 masked reads already measured. [#1788](https://github.com/AdaWorldAPI/lance-graph/pull/1788) / [#1789](https://github.com/AdaWorldAPI/lance-graph/pull/1789): D4, S-INGRESS and physical-byte limitations. +- [#1943](https://github.com/AdaWorldAPI/lance-graph/pull/1943), [#1945](https://github.com/AdaWorldAPI/lance-graph/pull/1945): in-memory DR/FOLD experiments, not Lance I/O. + +## Planned gates, not authorized implementations + +| Gate | Deliverable | Evidence/owner boundary | +| --- | --- | --- | +| D-DRAM-1 | enumerate actual source locator leases and 5×24 SIMD signatures | READ-ONLY | +| D-DRAM-2 | lab memoized source-location receipt under one immutable map, L1–L10 | EXO-46, EXO-89, EXO-151 | +| D-DRAM-3 | lab k-of-n Abakus vs native mask words, A1–A7 | Quack/ndarray owner; no public Abacus type by default | +| D-DRAM-4 | combined ClassView WHERE, borrowed SoA, Lance source and mask algebra, C1–C4 | D4/S-INGRESS remain GATED | +| D-DRAM-5 | cost/evidence handoff and owner decision, not a merge | EXO-50 append-only bus | + +**Refusals:** no second coordinate authority; no historical raw source pointer; no per-query copied payload by default; do not confuse WHERE criterion counts with property widths, SoA lane widths or DR tiles; no unproven physical 4×8 semantics; no new F### / opcode; no SQL UNKNOWN / Bag conflation. All measurements must separate cold setup, witness work, locator miss/hit, Lance bytes, borrow, prefetch and FOLD. + +**Board references:** [STATUS_BOARD D-DRAM](../STATUS_BOARD.md), [INTEGRATION_PLANS](../INTEGRATION_PLANS.md). **Related issues:** [EXO-46](https://linear.app/exo-red/issue/EXO-46/a1-decide-counted-globalfocus-binding-membrane-for-production), [EXO-89](https://linear.app/exo-red/issue/EXO-89/d-rmap-1-1a-implicit-relationship-register-and-separate-thought), [EXO-151](https://linear.app/exo-red/issue/EXO-151/d-drf-immutable-dr64-focus-leased-foreign-cache-sparse-writeback), [EXO-121](https://linear.app/exo-red/issue/EXO-121/d-qr-cap-classviewwidefieldmask-relational-isa-binding-capstone), [EXO-140](https://linear.app/exo-red/issue/EXO-140/admission-b-call-sites-unclassified-semijoin-equ32via-gather-and-no). diff --git a/.claude/board/entries/README.md b/.claude/board/entries/README.md index 32bea6719..af5b7e668 100644 --- a/.claude/board/entries/README.md +++ b/.claude/board/entries/README.md @@ -25,13 +25,14 @@ index row, (3) no duplicate entry id. Checks 1 and 2 are deliberately opposite directions; the stranding this convention prevents shows up in exactly one of them, never both. -566 entries, 2026-08-06 .. 2026-11-03. +567 entries, 2026-08-06 .. 2026-11-03. | date | entry id | finding | file | |---|---|---|---| | 2026-11-03 | `D-XLOG-META-WIRE-P0` | | [2026-11-03-d-xlog-meta-wire-p0.md](2026-11-03-d-xlog-meta-wire-p0.md) | | 2026-11-03 | `D-GRAMMAR-CONSTRAINTS-0` | | [2026-11-03-d-grammar-constraints-0.md](2026-11-03-d-grammar-constraints-0.md) | | 2026-11-03 | `D-GLOBAL-WHERE` | | [2026-11-03-d-global-where-p0.md](2026-11-03-d-global-where-p0.md) | +| 2026-11-03 | `D-DRAM-0` | | [2026-11-03-d-dram-abacus-address-memoization.md](2026-11-03-d-dram-abacus-address-memoization.md) | | 2026-11-03 | `D-DR-FOLD-MAP` | | [2026-11-03-d-dr-fold-map.md](2026-11-03-d-dr-fold-map.md) | | 2026-11-03 | `D-COGNITIVE-POLYFILL` | | [2026-11-03-d-cognitive-polyfill-p0.md](2026-11-03-d-cognitive-polyfill-p0.md) | | 2026-11-03 | `D-BAG-FOLD` | | [2026-11-03-d-bag-fold-witnessed-existential.md](2026-11-03-d-bag-fold-witnessed-existential.md) | diff --git a/.claude/plans/d-dram-abacus-address-memoization-v1.md b/.claude/plans/d-dram-abacus-address-memoization-v1.md new file mode 100644 index 000000000..643fb8712 --- /dev/null +++ b/.claude/plans/d-dram-abacus-address-memoization-v1.md @@ -0,0 +1,208 @@ +# D-DRAM-0: DR address memoization and adaptive Abakus masking lens + +> **Status: ARCHITECTURE / PRE-REGISTRATION ONLY, 2026-11-03.** This PR adds no production or lab executable code, dependency, register, storage format, public API, opcode, planner switch or measured speedup. **Not merge or implementation authorization.** It maps the operator's two ideas to existing owners, names required falsifiers and preserves all outstanding EXO decisions. +> +> **Operator thesis:** (a) remember **already resolved SoA addresses** as append-like *receipts within the current witnessed DR binding*, without materializing/copying their values; (b) use the existing binary Abakus / SIMD facade under dynamically scoped ClassView/Quack lenses **where the algebra benefits**, without building a second mask engine. +> +> Distinguish **address memoization** from payload gather, **bit-sliced arithmetic** from Boolean conjunction, and a **proposed dynamic lens** from an already shipped 5×24 kernel. Evidence labels are normative: VERIFIED-IN-CODE, MEASURED-LAB, PROPOSED, GATED, UNVERIFIED. + +## 1. Why this plan exists + +The D-GLOBAL-WHERE P0/P2 discussion confused several independent operations: mapping identities into DR tiles, selecting qualifying identities with WHERE, resolving demanded properties through ClassView, locating canonical SoA bytes, and executing an already-bound mask algebra. Expensive re-routing or cold source reads do **not** imply that values should be copied into a compact per-query cache. + +The guiding invariant is **DR remembers where; ClassView names what; Quack describes which algebra; ndarray executes; canonical SoA keeps the values.** A read-only source locator is not relationship truth. A previously loaded source address is not a new identity, field value, or durable map revision. + +### 1.1 One flow, different lifetimes + +```text +ClassId -> ClassView + schema/permission witness -> WideFieldMask (fields demanded) + | | + Quack WHERE/DT WHEN source facet/layout contract + | | + DR population/relation mask + sealed Frame/MapTable + | + DR tile/local ordinal, already present in existing mapping + | + [optional, leased] remembered physical/source locator receipt + | + canonical SoA borrowed read + advisory lookahead + | + Quack / mask-risc -> ndarray::simd existing mask words + | + optional bit-sliced Abakus threshold lens + | + COUNT / EXISTS / other typed terminal +``` + +Read stages are not always sequential: a MAP-only COUNT can terminate before source value resolution; first evaluation of a value predicate legitimately reads source values. Fields can be physically packed (including proposed 4×8-bit words), without implying that property ordinal, physical SoA lane, predicate count, DR tile width or source row address are interchangeable. + +### 1.2 Semantics that must remain separate + +| Name | Meaning / authority | Not | +| --- | --- | --- | +| Global ID | stable typed identity, outside a local tile | a Lance row address | +| DR Frame + MapTable | versioned global ID ↔ tile/local binding and replay; one route authority | a second source locator map or value copy | +| DR relation / population mask | bit-per-local-position support or WHERE survivor, under semiring and witness | field-presence mask; arbitrary SQL truth | +| ClassView / WideFieldMask | logical field basis, demanded fields, stable positions; >64 possible | row-address mask or physical column ordinal | +| Source locator receipt | validated route from a *bound* local position to canonical storage view, valid only under source/index/scope lease | new globally authoritative hash index; raw pointer persisted across remap | +| Lance RowAddrMask | physical source-row eligibility for Lance 13 scanner | DR mask, unless attested coordinate translation is proven | +| Abakus significance planes | independent bit-sliced integer counters/thresholds over *the same* selected positions | a way to mix row identity with carry position | + +The observed WHERE criterion-count working distribution is 1:34%, 2–4:46%, 5–16:14%, 17+:6%. It is operator-supplied and **not a measured selectivity distribution**. A 64-field ClassView, 32 physical SoA lanes, up to 256 WideFieldMask positions, 256×256 relation tile and 65,536-position DR context are different dimensions. More than 64K identities requires lawful composition across multiple DR relations/tiles, not automatic global dataset paging. + +## 2. Primary law: a memoized locator, never a gathered value + +**Proposed meaning of “append the address in DR”.** When a canonical SoA address has been resolved for a DR local position, a focus-scoped, version-attested *receipt* for that existing position becomes reusable by subsequent reads. The receipt can be sparsely populated (miss → resolve → memoize; hit → borrow), and prefetched positions may be prepared ahead of the consumer. + +**Do not append the source address to `globals` of `LocalMap`.** That vector is local → *global identity*, and `MapTable`'s append/prefix history is about changed coordinate membership. Mixing physical locators into it would change its semantics and corrupt old-frame replay. Before implementation, the DR/RelationshipMap owner must identify the existing focus/binding lease to host **disposable locator metadata**, with no second identity or coordinate authority. A receipt is derivative of a single already-attested canonical map. + +### 2.1 Conceptual receipt (not a proposed Rust type or ABI) + +```text +Key: DR map/frame identity + tile/local ordinal + + source dataset/branch/version + index/row-address seal + + ClassView schema/facet layout revision + scope/authorization +Value: validated source location / borrowed view descriptor + + its live source lease and optional prefetched-line hint +``` + +- **No bare pointer longevity:** a resident `&[T]` can be borrowed only while the canonical owner/Arrow-buffer lease lives. mmap, page-cache, decompression and storage reads are not presumed zero-copy. +- **No duplication:** subsequent methods sharing the *same* fully witnessed binding may reuse the resolved locator, not copy its payload. Changes in permission, projection or source location must not inherit a stale descriptor. +- **Versioning:** old DR frames remain replayable for coordinates. They can replay values only while the *corresponding* historical SoA snapshot and index seal are available; otherwise typed refusal. Epoch counters or frame-number equality alone cannot certify a foreign source row. +- **Mutation:** a value-only change can leave the DR relationship binding intact, but may invalidate a value-derived predicate and the affected value borrow. ADD/RETIRE of a relation changes support/delta, not global ID identity. Compaction, overwrite, remap, fragment split or stable-row-id policy changes require revalidation/rebind/refusal. +- **Eviction:** locator receipts may be forgotten independently of sealed relationship history. Losing a receipt causes a slower legal resolve, never lost relational truth or an empty answer. +- **Concurrency:** one admitted lease/witness for concurrent readers; no ABA from focus-slot reuse, speculative Thought-Git incarnation, scope changes or map-palette revisions. +- **Selective reads:** Lance 13 `Scanner::with_row_addr_prefilter`, `Scanner::project` and indexed/range reads are existing capabilities. Source ingress masks may be copied or reserialized; **do not label that seam zero-copy**. D4 (single DR ordinal→`_rowid` authority) and S-INGRESS remain gated. +- **Prefetch:** optional source-line or future-focus lookahead, never equivalent to materializing a new value array; test mispredictions and resident limits. No prefetch if the operation is MAP-only. + +### 2.2 Where actual gathered payload is lawful + +Default: MAP-only, or direct canonical borrow with optional attested source read and prefetch. A separately allocated facet cache is **not** a peer of borrowing in the general optimizer. D-DRF-1 `5 permits a disposable, witness-keyed *foreign-key/foreign-ontology* cache only for an explicitly admitted, repeated workload where measured whole-run gain exceeds build, allocation, eviction, invalidation and dirty-writeback costs. A numeric winner does not authorize bypassing the no-foreign-gather firewall (EXO-140 / EXO-46). The word `MaskOp::Gather` is an indexed mask operation, not automatically a copied SoA-value cache. + +## 3. Abakus as an adaptive physical realization of Quack's algebra + +**Verified library words:** ndarray PR #402 landed `ndarray::simd::half_add_mask` and `full_add_mask`, whose columns hold 64 independent bit-lanes and whose carries move to higher significance **within** a lane; `src/simd_masking_ops.rs` owns cross-ISA slice ergonomics. PR #390 landed `U64x8::sum8_u8_add_masked` for its specific masked byte-sum consumer. D-U9 PRs #389/#391/#393 provide research fixtures. + +**Not verified as a public API:** `Abacus` is a #402 lab fixture; its public type was deliberately deferred for lack of a consumer. The operator reports hand-rolled `ndarray::simd::5x24::*` kernels. **The exact 5×24 symbols/signatures, shipped status and intended bit-width are UNVERIFIED by this review.** Do not invent or import an API until the ndarray owner pins the actual file/symbol/commit. Do not infer that U9 numerical Abakus performance generalizes to Boolean masks. + +### 3.1 Dynamic lens, not a new VM + +- ClassView/DT64 resolves the *logical* field basis and read footprint using `WideFieldMask`, with no semantic bits hidden inside a presence mask. +- Quack/RelAlg owns typed predicate normalization, SQL TRUE/FALSE/UNKNOWN, Boolean / Bag choice and WHERE semantics. The execution consumer receives a **sealed, valid population mask** over one coordinate universe. +- The existing mask-risc/ndarray facade handles AND/OR/XOR/ANDNOT, mask compares, COUNT, and appropriate reductions. Do not reimplement those in a Quack loop or handwrite another vector backend. +- Only when the expression actually requires **arithmetic over many predicate masks**, such as “at least k of n conditions”, may bit-sliced `half_add_mask`/`full_add_mask`, vertical masked sums or a verified 5×24-family primitive be tested as a *physical realization*. Preserve identical lane order and tail bits. +- Scope width is chosen from the **actual demanded fields, admitted relation masks and selectivity**, not by materializing an entire 64-property or 256-predicate volume. A sparse predicate/threshold descriptor is an option, not a new data-storage contract. +- Possible 4×8-bit packed property/threshold routing is **hypothesis only** until codec, signedness, null/absence and overflow/refusal semantics are specified; do not silently reinterpret four bytes as four independent semantic fields. +- No new F### method ID, ClassId interpretation, DT64 algebra registry, mask-risc opcode or ndarray public type is authorized here. A lawful chosen realization would be admitted by current DT64/Quack ownership once supported by owner decisions. + +### 3.2 Strong negative controls + +- AND of 2–4 existing Boolean masks should usually lower directly to `mask_and`; proving Abakus correct does not justify extra carry operations here. +- Bag multiplicity, DISTINCT, SQL NULL/Kleene UNKNOWN, absent field versus present zero, negative U9/Palette512 sentinel and overflow must **not** be silently collapsed into Boolean thresholds. +- A “k-of-n” aggregate counts satisfied predicates **per identity**, not DR tiles, physical byte slots or join derivations. Bag counts must retain multiplicity if requested. +- A new kernel cannot bypass mask-risc/Quack ownership to get a better microbenchmark. First check existing facade and fixed-cost terminal options. +- Abakus is not a hash table, row-ID mapping, storage read mask, prefetch mechanism or reconstruction of source values. + +## 4. Existing shipped evidence / reuse map + +| Evidence | Grounded result | Implication | +| --- | --- | --- | +| `dt64::dr::{LocalMap,MapTable,Frame}`, lance-graph #1905/#1923 | versioned coordinate table and old-frame replay, identity-only `LocalMap` | keep authority; seek existing receipt lease, not a new global→local table | +| `contract::class_view::{FieldMask,WideFieldMask}`; `dt64::{ValueDemand,MethodBinding}` | demanded-field masks, borrowed mask words, map-only versus value demand | separate sparse logical projection from physical layout and WHERE | +| D-RMAP-1A; D-DRF-1; D-DRAV-1 | separate RelationshipMap owner, zero-copy default, admitted exceptional foreign cache, source leases | binding policy outranks microbenchmark winner | +| lance-graph #1573/#1575/#1633 | Lance 13 masked source reads already measured; selective range path with ordered-lane attestation | reuse, don't recreate selective reader | +| lance-graph #1788/#1789 | DR↔Lance wiring, S-INGRESS and D4 open; chunk-touched byte predictor failed | do not claim source zero-copy or infer physical bytes from logical rows | +| lance-graph #1943/#1946 | in-memory DR routing/fold, corrected DuckDB answer parity | no Lance I/O in those measurements; re-route per query is an avoidable cost hypothesis | +| lance-graph #1945 | top-down vector×tile beats reverse gather-shaped arm across 54 lab cells, with stated falsifier; no storage I/O | evaluation direction matters; C builds its own chunks over DR coordinates | +| ndarray #402 | half/full adder facade shipped; bit-sliced public counter deferred | choose a real Quack threshold consumer before new public type | +| ndarray #390 | vertical masked byte sums shipped, with independent per-ISA realizations | verify semantics and existing words first | + +Sources: +- https://github.com/AdaWorldAPI/lance-graph/blob/main/crates/lance-graph-dt64/src/dr.rs +- https://github.com/AdaWorldAPI/lance-graph/blob/main/crates/lance-graph-contract/src/class_view.rs +- https://github.com/AdaWorldAPI/lance-graph/blob/main/crates/lance-graph-dt64/src/lib.rs +- https://github.com/AdaWorldAPI/lance-graph/blob/main/.claude/plans/d-rmap-separate-tables-thought-palettes-v1.md +- https://github.com/AdaWorldAPI/lance-graph/blob/main/.claude/plans/d-dr64-focus-lease-foreign-cache-v1.md +- https://github.com/AdaWorldAPI/lance-graph/blob/main/.claude/plans/d-dr-address-first-value-demand-v1.md +- https://github.com/AdaWorldAPI/ndarray/blob/master/src/simd_masking_ops.rs +- https://github.com/AdaWorldAPI/ndarray/pull/402 +- https://github.com/AdaWorldAPI/ndarray/pull/390 + +## 5. Pre-registered, *separable* falsifiers + +### L: locator memoization, no payload copy + +| Gate | Fixture / independent oracle | Refusal or cost requirement | +| --- | --- | --- | +| L1 | Same frame/map, two distinct methods select overlapping properties at 64 / 256 local positions; oracle reads canonical SoA directly | second read reuses a validated locator, result equal, no copied values | +| L2 | Miss→hit, hit→evict→miss, scattered and clustered canonical 512-byte rows | correctness unaffected; count lookups, actual loads, allocations, cache lines | +| L3 | Changed values only, unchanged relationship, then value-dependent predicate change | DR identity remains; dependent derived mask refreshed/refused; other masks reusable | +| L4 | ADD, RETIRE, old frame replay, map pure extension versus replacement | no stale association, old frame requires correct historical source | +| L5 | Compaction/overwrite, stable-ID on/off, changed index seal and dataset version | old physical locator never silently reused | +| L6 | Foreign frame, scope change, palette reassignment, reused focus slot, revoked source lease | typed refusal; zero cross-scope bleed | +| L7 | Two concurrent readers with same witness and one invalidating writer | bounded locator construction; no ABA / dead borrow | +| L8 | No prefetch / lookahead 0/4/8/16/32/64, mispredicted lookahead | no reordering or payload copy; compare whole-run cost and memory | +| L9 | Full / Lance ALLOW / Lance BLOCK / attested range on 0.1%,1%,10%,50% scattered and clustered | account mask ingress and bytes; retain D4 refusal | +| L10 | 1M and 10M corpus, multiple DR256 relations across address domains | identity and Bag/Boolean oracles; no giant dense matrix, no new paging authority | + +### A: adaptive Abakus lens + +| Gate | Oracle and disable | Required verdict | +| --- | --- | --- | +| A1 | Native AND/OR/COUNT vs Abakus on k-of-n: n=1,2,4,5,16,24; k=0,1,half,n,n+1 | exact lane-by-lane threshold, no assumed win | +| A2 | Adversarial masks: zero, full, alternating, one lane, tail, sparse, dense, random | cross-backend parity and no carry into neighboring lane | +| A3 | FieldMask 64 vs WideFieldMask 256, 32-lane physical SoA mapping, noncontiguous fields | same logical projection with no field/row ordinal alias | +| A4 | Packed 4×8 threshold codec experiment only after encoding attestation | signedness, saturating vs exact, missing/present zero and overflow pinned | +| A5 | SQL NULL/UNKNOWN, absent property, Boolean set vs Bag multiplicity | never invent 0 from missing, nor equate witness count and support | +| A6 | Replace Abakus with naive per-row adds, then deliberately misplace one carry | independent oracle catches changed answers | +| A7 | Full Quack lowering vs isolated SIMD-only microkernel, cold/warm, all ISA tiers | calculate end-to-end benefit including dispatch; NO-EXISTING-KERNEL is allowed | + +### C: combined recall + dynamic lens + +C1: one ClassView lens, 3 of 64 demanded fields, 1–4 WHERE criteria, reused DR frame, first value read vs repeated hit vs changed source. Compare **MAP-only**, borrow, prefetch-borrow, and **authorized exception** only; no generic gather planner. + +C2: fixed DR frame with multiple relation compositions beyond 64K. Vary predicate count, result cardinality and number of DR domains independently; oracle must not share lookup/index code. + +C3: use DuckDB's published engineering mechanisms as **learning evidence** (selection lifetime, late materialization, join direction, index pruning) alongside semantic and performance parity. Do not invent feature parity because answers match. + +C4: if any stage needs DR ordinal→Lance `_rowid` without an attested single authority, report `BLOCKED_D4`. Likewise report `S_INGRESS_COPY` on masked Lance ingress. Never force a full scan under a hidden failure, call it zero-copy or silently materialize. + +## 6. Cost accounting (no free amortization) + +Report **separately**, in absolute units and by query count Q: + +```text +class/schema resolve | WHERE | DR coordinate map preparation | DR route reuse +locator miss/hit | mask construction | mask ingress copy +physical Lance rows/bytes/pages/ranges | SoA borrow loads | prefetch +optional exceptional gather build/eviction (separate admitted arm) +bit-sliced vs native Boolean fold | terminal | total wall | peak bytes +``` + +Do not hide a cold mapping in a warm number, count result rows as physical bytes, call a scan mask a copied payload, or assert CPU-cache residency from a source pointer. Stratify mask locality, field count, predicate count, selected cardinality and repeated-use Q. Statistical spread and runs required before any performance verdict. + +For an optional receipt, the break-even hypothesis is saved repeated locator resolution and avoided latency **minus** receipt insertion, witness validation, indirection, cache footprint, invalidation and prefetch errors. For Abakus, compare actual typed expressions, not invented workloads that force carries to win. Keep the hot-path 1 ms failure gate. + +## 7. Execution order, owners and stopping points + +| Step | Deliverable | Authority / status | +| --- | --- | --- | +| D-DRAM-0 | this map + board entry + EXO-50 message | docs only, operator can review | +| D-DRAM-1 | symbol & caller census, exact 5×24 ndarray signature, existing focus lease source, one DR map owner | READ-ONLY; no implementation | +| D-DRAM-2 | isolated memoized locator receipt lab with L1–L8; only existing owned source/leases | GATED on EXO-46 + EXO-89/EXO-151 admission; no new identity table | +| D-DRAM-3 | Abakus Quack threshold bench A1–A7 with existing public words | lab proposal; SIMD/Quack owners; no public type without consumer | +| D-DRAM-4 | combined C1–C4 and Lance source-read cost | D4/S-INGRESS remain explicitly blocked for end-to-end binding | +| D-DRAM-5 | owner review, measured decision per ability: REUSE / WIRE / DEFER / REFUSE | no production action implied | + +**Owner registers:** EXO-50 is an append-only coordination bus, **not** an approval. EXO-46 (binding admission), EXO-89 (RelationshipMap authority), EXO-151 (focus lease/foreign cache), EXO-121 (ClassView/WideFieldMask), EXO-140 (foreign gather classification), EXO-108 (Bag vs set), EXO-112 (contract/ndarray dependency shape), and D4/S-INGRESS (#1788). DT64/Quack own semantic admission; ndarray owns facade and architecture backend; Lance owns physical source reads. EXO-50 links are handoffs, not new runtime authority. + +**STOP immediately** on: new global→row-id store, fresh SIMD backend in Quack, changed field-presence meaning, duplicate physical payload by default, source locator surviving a dead lease, unverified 5×24 API claimed as shipped, unauthorized Gather, new ClassId/F###, or production patch while this remains a docs-only proposal. + +## 8. Board / literature links + +- Board: [D-DRAM-0 entry](../board/entries/2026-11-03-d-dram-abacus-address-memoization.md), [STATUS_BOARD](../board/STATUS_BOARD.md), [INTEGRATION_PLANS](../board/INTEGRATION_PLANS.md). +- EXO-50 bus: https://linear.app/exo-red/issue/EXO-50/agent-message-bus-gpt-xgrok-ccsession-id +- EXO-46: https://linear.app/exo-red/issue/EXO-46/a1-decide-counted-globalfocus-binding-membrane-for-production +- EXO-89: https://linear.app/exo-red/issue/EXO-89/d-rmap-1-1a-implicit-relationship-register-and-separate-thought +- EXO-151: https://linear.app/exo-red/issue/EXO-151/d-drf-immutable-dr64-focus-leased-foreign-cache-sparse-writeback +- DuckDB's public engineering references: https://duckdb.org/internals/vector , https://duckdb.org/2024/11/14/optimizers , https://duckdb.org/library/selective-late-materialization/ ; shipped features and research prototype must be distinguished.