diff --git a/.claude/agents/zero-copy-warden.md b/.claude/agents/zero-copy-warden.md index 5caa296fe..52f8b7eef 100644 --- a/.claude/agents/zero-copy-warden.md +++ b/.claude/agents/zero-copy-warden.md @@ -89,6 +89,18 @@ excluded in both directions (see the 16 MB scale anchor): never let "it's expensive to recompute" argue FOR a store, and never let "it's cheap" argue against a lane. +**SCOPED-EXCEPTION — the DR64³ foreign-key cache** *(DECISION 2026-10-24)*. +The law's § "The DR64³ foreign-key cache exception" admits one +cast-reproducible cache on a cost basis, in one scope: a DR64 focus whose +foreign-key reads span several ontologies, reused across many thoughts, +feeding relationship learning and live-generated ONNX (D-DRF-1 §5). Before +returning MATERIALIZES for a gathered foreign facet, check that scope and +every condition (disposable, recomputable from sealed references, full +witness key, no uncommitted truth, build cost reported). All met → not a +finding; say which condition you checked. Any missing → the ordinary +verdict. This is the only place a cost argument is admitted; do not extend +it by analogy. + **LENS-CLEAN** — every read is a cast at the point of use: `from_register_ref(&rows[pos].value[a..b])`, offsets derived from the tenant descriptor, filtering done by predicate rather than by gather. diff --git a/.claude/board/INTEGRATION_PLANS.md b/.claude/board/INTEGRATION_PLANS.md index 32b0b64d4..532a99ae5 100644 --- a/.claude/board/INTEGRATION_PLANS.md +++ b/.claude/board/INTEGRATION_PLANS.md @@ -1,3 +1,7 @@ +## 2026-10-24 — d-dr64-focus-lease-foreign-cache-v1 + d-dr64-focus-cache-falsifiers-v1 — immutable DR64 focus, leased foreign cache, sparse writeback → `.claude/plans/d-dr64-focus-lease-foreign-cache-v1.md`, `.claude/plans/d-dr64-focus-cache-falsifiers-v1.md` + +**Status:** D-DRF-1 REFERENCE DESIGN / PRE-REGISTRATION ONLY (#1756); DRF-P1..P4 PROPOSED, not authorized. The DR map stays an immutable versioned reference; focus load acquires an ephemeral residency lease; caches are disposable; dirty rows coalesce by canonical destination and go through the existing CycleWriter. DECISION 2026-10-24: the DR64³ foreign-key cache is a scoped exception to the zero-copy law (cost basis, several ontologies, learning into live-generated ONNX), lapsing if DRF-P2 does not measure a whole-run win. Entry `2026-10-24-d-drf-1-focus-lease-foreign-cache.md`. + ## 2026-10-23 — d-nfr-6-soa-borrow-v2 — native operations over the SoaEnvelope borrow, no guard → `.claude/plans/d-nfr-6-soa-borrow-v2.md` **Status:** MEASURED, lab only (`crates/soa-view-probe`). Supersedes `d-nfr-6-validate-once-v1`: views are `NodeRowPacket` borrows, operations are one ndarray kernel each, mutation is the owner's `&mut`; no `Table`, no frame guard. Packet equals direct ndarray within noise at 0 allocations. D-NFR-6a withdrawn. diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index 2f3f43e9d..7f4674949 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -7,6 +7,18 @@ Plan: `.claude/plans/d-onnx-native-v1.md`. Crate: `crates/lance-graph-onnx` (ALI | **D-ONNX-NATIVE-0** | crosswalk + minimal facade: elementwise float arithmetic, int32 comparisons, Boolean ops, Where, ReduceSum, ReduceSum(Where(Greater)) | IMPLEMENTED, draft PR (`entries/2026-10-23-d-onnx-native-0-facade.md`) | 11 conformance tests + doctest; 4 disable runs red on their own test; alias = native symbol; asm diff | | **D-ONNX-NATIVE-1** | the NATIVE rows: approximate transcendentals, float ReduceSum tolerance, empty-population min/max | PROPOSAL | each row names its open decision (plan §2) | +## D-DRF — immutable DR64 focus, leased foreign cache, sparse writeback (2026-10-24) + +Docs only (#1756). Plans `d-dr64-focus-lease-foreign-cache-v1.md` and `d-dr64-focus-cache-falsifiers-v1.md`. The foreign-key cache is a scoped exception to the zero-copy law (DECISION 2026-10-24, `zero-copy-lens-law.md`). + +| D-id | scope | status | gate / falsifier | +|---|---|---|---| +| **D-DRF-1** | reference semantics: geometry, focus lease, lookahead prefetch, foreign-key cache exception, dirty-row writeback, learning overlay | In PR | falsifier matrix G1–G7, L1–L4, P1–P3, M1–M2, W1–W8, R1–R7 pre-registered, NOT RUN | +| **DRF-P1** | two-focus lazy load / lease / prefetch / last-reader eviction | Proposed | L1–L3, P1–P2; stale-map and race fixtures first | +| **DRF-P2** | 64-thought foreign-table A/B/C (direct+prefetch vs gather+reuse vs gather+reuse+writeback) | Proposed | whole-run cost incl. cold build; the foreign-key cache exception lapses if B/C does not win | +| **DRF-P3** | relationship cube / slices and typed dirty-row reduction oracle | Proposed | R1–R3, W1–W3; no Bag/NULL/evidence loss | +| **DRF-P4** | optional plasticity overlay + ONNX candidate stream | Proposed | R4–R7; base DR revision bit-identical | + ## D-NFR / D-NFC — native representation contract + dependency correction (2026-10-23) Docs only. Plans `d-native-repr-contract-v1.md` (D-NFR) and `d-native-first-dependency-correction-v1.md` (D-NFC). Nothing implemented; no ABI or execution contract changed. diff --git a/.claude/board/entries/2026-10-24-d-drf-1-focus-lease-foreign-cache.md b/.claude/board/entries/2026-10-24-d-drf-1-focus-lease-foreign-cache.md new file mode 100644 index 000000000..fe781ac92 --- /dev/null +++ b/.claude/board/entries/2026-10-24-d-drf-1-focus-lease-foreign-cache.md @@ -0,0 +1,8 @@ +# D-DRF-1 — immutable DR64 focus, leased foreign cache, sparse writeback + +**Date:** 2026-10-24 · **Status:** REFERENCE DESIGN, pre-registered, NOT RUN · **Plans:** `.claude/plans/d-dr64-focus-lease-foreign-cache-v1.md`, `.claude/plans/d-dr64-focus-cache-falsifiers-v1.md` · **PR:** #1756 + +- **Design:** the DR map is an immutable, versioned reference; a changed map is a new sealed revision. Focus load acquires an ephemeral residency lease, released with the last reader. Caches are disposable and recomputable from sealed references. Dirty local positions translate through the original map, coalesce by canonical destination under a declared reduction, and go as owned rows through `collect_casts → CycleWriterHandle::try_submit → DetachedCycleBatch::freeze → WalSink::commit_cycle`. +- **DECISION (2026-10-24):** the DR64³ foreign-key cache is a scoped exception to the zero-copy law. SCOPE: foreign-key reads spanning several ontologies (about 8), reused across many thoughts, feeding relationship learning and live-generated ONNX. BASIS: only dirty rows are written back; most accesses (expected 70–90%) are reads served from the versioned cache; many learning steps need only the Boolean relationship maps, not values. All three fractions are hypotheses for DRF-P2. REVISIT WHEN: DRF-P2; the exception lapses where the gather-and-reuse arm does not win on whole-run cost including cold build. Recorded in `zero-copy-lens-law.md` and `zero-copy-warden.md` (SCOPED-EXCEPTION). +- **Checked against source:** geometry arithmetic; the writer chain exists as named; `LanceCycleWriter` copies into Arrow builders (its own module doc); `DurableRef` / `ThoughtCommitId` are specified, not shipped. +- **Open:** whether the cache wins at all (DRF-P2); the residency lease's reader-lifetime mechanism (DRF-P1). diff --git a/.claude/board/entries/README.md b/.claude/board/entries/README.md index 9a3a2b2de..49b63145b 100644 --- a/.claude/board/entries/README.md +++ b/.claude/board/entries/README.md @@ -25,10 +25,11 @@ index row, (3) no duplicate entry id. Checks 1 and 2 are deliberately opposite directions; the stranding this convention prevents shows up in exactly one of them, never both. -462 entries, 2026-08-06 .. 2026-10-23. +463 entries, 2026-08-06 .. 2026-10-24. | date | entry id | finding | file | |---|---|---|---| +| 2026-10-24 | `D-DRF-1` | | [2026-10-24-d-drf-1-focus-lease-foreign-cache.md](2026-10-24-d-drf-1-focus-lease-foreign-cache.md) | | 2026-10-23 | `ps2-compose-chain-preregistration` | | [2026-10-23-ps2-compose-chain-preregistration.md](2026-10-23-ps2-compose-chain-preregistration.md) | | 2026-10-23 | `ledger-revision-1635-1726` | | [2026-10-23-ledger-revision-1635-1726.md](2026-10-23-ledger-revision-1635-1726.md) | | 2026-10-23 | `EXP-GF2-ELIM-PREREG` | Gaussian elimination over GF(2) with a replayable row-operation witness — pre-registration | [2026-10-23-gf2-elimination-preregistration.md](2026-10-23-gf2-elimination-preregistration.md) | diff --git a/.claude/knowledge/zero-copy-lens-law.md b/.claude/knowledge/zero-copy-lens-law.md index 3c2d4631a..78196c90b 100644 --- a/.claude/knowledge/zero-copy-lens-law.md +++ b/.claude/knowledge/zero-copy-lens-law.md @@ -29,6 +29,7 @@ | "Projection is not chasing" — indirection cost is zero here | **FINDING** (mechanism), *unbenchmarked* | Follows from `target = cur + off` being computed arithmetic over a contiguous strided slab. **No micro-benchmark has been run**; the claim is structural, not measured. | | The rung test (elevation licenses a store; a projection never does) | **CONJECTURE** | Operator ruling + one shipped precedent (`Locus::Quorum` / `Contradiction`). **No probe has run.** Falsifier stated in § "The one apparent exception"; gated as `PROBE-RUNG-ELIGIBILITY`, NOT RUN. | | Grey/white per-lane assignment | **CONJECTURE** | Framing only. The doc says so at the table: *"the governing frame, not a census."* Census = plan §6 Q0, NOT RUN. | +| The DR64³ foreign-key cache exception pays for itself | **HYPOTHESIS** | DECISION 2026-10-24, scoped (§ "The DR64³ foreign-key cache exception"). Rests on three unmeasured fractions: dirty rows, read hits (expected 70–90%), value-free learning steps; DRF-P2 is the gate and the exception lapses if it fails. | | The 16 MB / 2×16 MB scale anchor | **FINDING** (arithmetic) | 32k rows × 512 B = 16 MB, checked. The grey/white *split* of that budget is operator-stated, not measured. | **Nothing in this doc is promoted past its row above.** Where a section reads as @@ -250,6 +251,17 @@ They qualify **not** because reconciling observations is expensive, but because contradiction is a strictly higher epistemic object than the observations it reconciles. That is the shape every future derivation tenant must match. +## The DR64³ foreign-key cache exception (2026-10-24) + +**DECISION:** a second exception, separate from the rung test above and not derived from it. A gathered foreign facet cache is a cast-reproducible projection, so by the rung test it is a violation; this decision admits it anyway, in one scope only, on a cost basis that this law otherwise rejects. Admitting it here, by name, is what keeps it from becoming a general hatch. + +- **SCOPE:** foreign-key operations over a DR64 focus whose referenced rows span several ontologies (the motivating case is about 8), reused across many thoughts, with relationship learning over the result feeding live-generated ONNX as the mathematical compiler substrate. Plan: `.claude/plans/d-dr64-focus-lease-foreign-cache-v1.md` §5 (D-DRF-1). +- **BASIS:** the cost is amortized in three places. (1) Writes: only dirty rows are written back, coalesced by canonical destination, so a focus touched by many thoughts commits a few full-row images, not one write per operation. (2) Reads: most accesses (expected 70–90%) are reads served from the versioned cache; without it each of them is a scattered 512-byte-stride read into a foreign table, repeated per ontology, per thought and per learning iteration, which leaves the nanosecond fold. (3) Learning: the relationship maps are Boolean support planes, and many learning steps need only those, not the row values, so they skip the value gather entirely. The three figures (dirty fraction, read-hit fraction, value-free learning fraction) are HYPOTHESES until DRF-P2 measures them. The 16 MB scale anchor's "cost is not an argument" holds for lens reads inside one table; it does not cover this cross-table, repeated, learning-driven case. +- **CONDITIONS:** disposable and never authoritative; clearing and recomputing from the sealed references gives the same answers; keyed by the full witness (map revision, index seal, source snapshot, focus, palette generation, facet layout, operation, scope); holds no uncommitted truth; build cost reported separately. +- **REVISIT WHEN:** DRF-P2 measures the gather-and-reuse arm against the direct arm on whole-run cost including cold build. If it does not win at the operator's workload, the exception lapses there and the cache is removed. + +The warden applies this as **SCOPED-EXCEPTION**: inside the scope and meeting every condition, not a finding; outside it, or missing a condition, the ordinary verdicts apply. + ## The canonical measured instance (2026-07-28) `witness_fabric::{resolve_chain, standing_wave_grounded}` took diff --git a/.claude/plans/d-dr64-focus-cache-falsifiers-v1.md b/.claude/plans/d-dr64-focus-cache-falsifiers-v1.md new file mode 100644 index 000000000..bacc3120f --- /dev/null +++ b/.claude/plans/d-dr64-focus-cache-falsifiers-v1.md @@ -0,0 +1,90 @@ +# D-DRF-1 falsification preregistration — lease, prefetch, foreign gather and writeback + +**Status:** PRE-REGISTERED, NOT RUN (2026-10-24). Companion to [D-DRF-1](d-dr64-focus-lease-foreign-cache-v1.md). This is a **proposed** lab matrix, not a declaration that a DR focus cache, third relationship axis or ONNX learning stream exists. + +## 1. All-in decision rule + +We compare three **correct**, source-pinned execution strategies for the same typed query and the same 64-thought workload: + +- **A (reference):** direct immutable DR64 u16 map -> borrowed SoA, with optional software lookahead prefetch. +- **B (materialized):** one legal/gated foreign facet gather -> compact resident cache -> repeated native ndarray folds. +- **C (materialized + writeback):** B plus dirty-local marking, duplicate-destination deterministic reduction, owned dirty payload handoff to the existing CycleWriter. + +The winner is not allowed to change Boolean/Distinct vs Bag vs SQL K3, the source scope, ONNX numeric semantics, contribution coverage or the version horizon to obtain a lower time. Compare a CPU scalar oracle and existing native ndarray/mask-risc behavior first, *then* costs. The legal outcome may be **REFUSE**, direct baseline, or fallback. + +Cost breakdown per strategy: initial source/index bind; version/scope admission; cache miss and build; focus lease acquisition; cold page faults; prefetch requests and faults; execution; dirty reduction; full-row materialization; writer submit and backpressure; durability latency separately; memory retention; eviction; cross-focus churn. Do not amortize A differently from B/C or quote post-warm execution alone. + +Sweep reuse N = 1, 8, 64, 256 and switching patterns AAA... / ABCABC... / adversarial unique tables. For the operator's expected case use 64 thoughts and up to three mixed foreign views (Disease / Phenomenology / LOINC). Report *actual* reuse counts for each MapId. A cache cannot save random reads if each foreign facet is used only once. + +## 2. Geometry and identity falsifiers + +| ID | Setup | Expected decision / observation | Disable that must fail | +| --- | --- | --- | --- | +| G1 | 32 x 128-bit, 64 x 64-bit, 64x64 u16 mapping, and 64^3 Boolean cube independently constructed | Equal **4096-bit** Boolean tile views, 512 B; pointer map **8192 B**; Boolean cube **32768 B**; combined 40960 B when both allocated; no 64^3 pointers or extra SoA rows | Conflate bitmask bytes with u16 bytes | +| G2 | DR64 map with repeated ordinals and 4,096 different ordinals; target row ABI 512 B | Max distinct targets 4,096 = **2 MiB footprint**, not mandated RAM allocation or bytes transferred. Full DR256 map 65,536 distinct = **32 MiB** | Multiply by relationship axis or by 64 thoughts | +| G3 | Same u16 ordinal in two different SoA datasets / branches at equal numeric version | Never infer equal identity; cache reuse must refuse without dataset/branch/index witnesses | Key only on local ordinal and bare version | +| G4 | Sealed map A, append/reseal/compact source; map B has same focus-slot numerals but different row-address seal | Old map remains replayable with old snapshot; B requires new binding; stale A->B reuse refused | Drop row-index seal / stable-ID requirement | +| G5 | MapPalette slot P -> Q -> P; thought slot reused after squash; identical shape and facets | Reject stale palette/ThoughtCommitId association despite reused slot and shape | Cache key only on palette slot / thought slot | +| G6 | Same physical pointer, different authorized tenant/scope | Reject cross-scope cache reuse before exposing bytes | Remove authorization from cache admission | +| G7 | Active focus omits a dependency contributor, or closure exits 64-local aperture | Expand only via an explicitly authorized and measured binding, otherwise refuse; do not silently turn missing into FALSE or zero | Pretend focused support is global support | + +## 3. Lifetime, safety and prefetch falsifiers + +| ID | Setup | Expected decision / observation | Disable that must fail | +| --- | --- | --- | --- | +| L1 | Two simultaneous threads demand identical witnessed Focus A | At most one completed source gather per cache-entry construction; both read the same immutable contents; lease begins when focus loads, not first fold | Each reader creates a new uncontrolled gather | +| L2 | Reader 1 holds Focus A; focus manager loads/activates B and retires A; reader 1 continues | A remains valid to reader 1, B available to reader 2, no use-after-free; A reclaimed only after final pin and dependent writer references are detached | Reclaim A immediately on focus switch | +| L3 | Load focus B speculatively but never use it; memory budget asks for eviction | Eviction can reclaim B without changing durable map or fact data; no leaked forever-static LazyLock | Retain B in a global static | +| L4 | Borrow NodeRowPacket from real canonical SoA, then try mutable owner write while borrowed | The **Rust compiler** refuses the aliased mutation; avoid inventing per-fold FrameStore checks | Introduce unsafe aliasing or interior writer | +| P1 | Random, clustered, repeated and adversarial 64K-target ordinals; lookahead distances 0,4,8,16,32,64 | Identical scalar/native results with and without prefetch; time and L1/L2/L3/DRAM misses reported by distribution; no presupposed speedup | Skip bounds on prefetch address or assume locality | +| P2 | Predict Focus B then use C; or make 64 fast focus switches | Prefetch cancellation/mis-prediction must not corrupt A/C or leak leases; track wrong-prefetch bytes, hit rate and eviction churn | Treat prefetch as a semantic commit | +| P3 | Prepare DR64 8 KiB map; prefetch 512-byte-strided SoA facet offsets from u16 ordinals | Prefetch targets declared facet offsets only, never arbitrary payload-derived foreign addresses; no cold page-fault costs hidden | Use unadmitted value as address | + +**Note:** the repository's separate dataset idle-flush plan rejected a hot per-read lease protocol for its >3-day, >300 MB cold-dataset eviction policy. L1–L3 describe **DR focus cache lifetime**, not a retrofit of that storage policy. + +## 4. Materialization and writeback falsifiers + +| ID | Setup | Expected decision / observation | Disable that must fail | +| --- | --- | --- | --- | +| M1 | Three foreign 16-byte facets across 4,096 local pixels, repeat 64 mixed-ontology thoughts | Charge 3 x 64 KiB = **192 KiB** cached facet data plus references, metadata, masks and build; compare against direct on whole run and with cold/warm/focus switches | Report only 64 KiB or omit gather cost | +| M2 | Same base relationship question represented by direct borrowed view and materialized facet (including out-of-focus, NULL/unknown and Bag inputs) | Correctness/typed refusals identical to scalar and native baseline; materialization never broadens admission policy | Convert presence to truth or erase multiplicity | +| W1 | 4,096 local pixels, 37 unique dirty canonical destinations, 64 thoughts, 512-byte row ABI | **37 distinct full-row images = 18,944 B** of final row payloads at persistence, not 4,096 x 512 or 64 x 37 x 512. Count metadata and intermediate costs separately | Write each touched position/thought | +| W2 | Two local pixels alias one canonical row; two thoughts modify **different** facets; order intentionally shuffled | Final row preserves both updates via typed reduction against sealed Vn; stable outcome for all permitted interleavings | Apply raw last-payload-wins to independent partial patches | +| W3 | Two noncommuting same-facet updates with declared stream positions | Applied result exactly matches deterministic oracle, otherwise typed conflict/refusal; never accidental associative rewrite | Swap update order, silently add operands | +| W4 | Producer submits owned casts to bounded CycleWriter, WAL is delayed; thought continues on same source | Source owner borrow is released after truly owned handoff; reader lifetime depends only on outstanding reads, not WAL completion; durability mark does **not** advance prematurely | Await durability inline or report submit as durable | +| W5 | Alternative descriptor-only path points into live SoA and is queued | Focus/source lease must survive until writer captures/pins bytes. Early release forbidden; test invalidation race | Drop lease at mere descriptor enqueue | +| W6 | Queue full / writer poisoned / WAL precommit failure / lost acknowledgement | Ownership returned on submit failure; precommit regeneration reads sealed Vn; ambiguity resubmits SAME frozen batch; no duplicate published version; no stale focus re-read | Resample live source bytes during ambiguous retry | +| W7 | Intent-only or no artifact change, and a mixed artifact-backed cycle | Intent-only does not write a new version; changed dirty rows coalesce and publish one sealed successor version; historical DR map version unchanged | Force an empty WAL append | +| W8 | Active view still pinned after cache eviction under heavy memory pressure | Bound resident usage may exceed soft budget while live readers exist, but result remains correct and evicted inactive views rebuild deterministically | Evict live underlying SoA snapshot | + +**The three amortization fractions are measured, not assumed.** The foreign-key cache exception rests on them, so DRF-P2 reports each per workload, with its denominator: + +| ID | Quantity | Definition | Pre-registered expectation | +| --- | --- | --- | --- | +| A1 | dirty fraction | distinct canonical rows written back ÷ distinct rows read, per focus version | small (W1's 37 / 4,096 shape); no target | +| A2 | read-hit fraction | facet reads served from the resident cache ÷ all facet reads, after the cold build | 70–90% | +| A3 | value-free learning fraction | learning steps answered from Boolean relationship planes alone ÷ all learning steps | "many"; no target | + +If A2 lands below the expected band, or A3 is near zero, report it as the reason before reporting any timing; the exception's basis is then weaker than stated and the break-even below decides it. + +Break-even test: C wins only when its **full** gather+64 folds+dirty+submit cost is lower than the legitimate A baseline for equivalent work. Report the lower and upper crossover N and the conditions (density, locality, cache budget, dirty fraction, focus churn) under which the win reverses. + +## 5. Relationship and learning separation falsifiers + +| ID | Setup | Expected decision / observation | Disable that must fail | +| --- | --- | --- | --- | +| R1 | Typed identity x property x relationship-selector 64^3 Boolean cube; select one 64^2 relation slice | Slice and independent scalar ternary oracle agree; no forced allocation of full 32 KiB cube for a single sparse relation | Assume every cube dimension means the same type | +| R2 | Try reinterpreting relationship selector as a SoA ordinal, CE64 P/W map ID, or thought occupancy lane | Reject; mapping index, relation selector and speculative thought axis remain separate | Collapse axes into one numeric namespace | +| R3 | Relational composition over local focus with missing contributing intermediate | Exact only with complete dependency closure; else authorized expansion or refusal | Mask missing contribution as absent | +| R4 | Proposed plastic overlay learns new strength but base relation/map revision V stays pinned | Old V remains bit-identical and historical reads unchanged; overlay carries independent commit/model/source-root identities; admitted proposal produces V+1 only on explicit commit | Mutate base DR in place | +| R5 | Duplicate learning source roots / correlated evidence + tanh or palette saturation | Refuse unwarranted independent-evidence accumulation; numerical boundedness does not certify truth or NARS/Bayes equivalence | Count repeated evidence as independent | +| R6 | ONNX model/weight shape collision but different graph or codebook generation | Refuse cached compiled result unless complete source/model/layer/weight/calibration identity matches; independently check typed approximate/exact contract | Use shape-only/d_head key | +| R7 | Persist candidate learning artifacts in hypothetical S3-like object storage and reload | Object storage is an **optional hydration/versioned artifact path**, not a random-read tensor store or proof engine; replay from pinned local snapshot, not live remote page faults | Treat ONNX format or S3 as automatically executable, durable or certified | + +## 6. Measurement and status reporting + +Pre-register results as **PASS / FAIL / BLOCKED / NOT RUN**. For each cell include seed and workload generator, machine/CPU/ISA, compiler flags and build profile, source snapshot and MapId identity, operations/semiring, exposed bytes, allocation counts, cold versus warm times, cache-miss and focus-switch distribution, p50/p95 and tail, writeback reduction cardinalities and durability completion separately. Report model quality if comparing learned approximate outcomes, with an independent held-out oracle and explicitly declared tolerance. + +**Non-goals in this PR:** no CI implementation, no claim of actual latency, no DR64^3 ABI, no CE64 extension, no extra mutable table, no runtime Tensor/ONNX interpreter, no new storage lease policy and no EXO-50 post. + +**Follow-up after review:** one small falsifier-led probe PR at a time, then create the coding prompt and EXO-50 handoff from the accepted reference rather than re-deriving semantics from chat. diff --git a/.claude/plans/d-dr64-focus-lease-foreign-cache-v1.md b/.claude/plans/d-dr64-focus-lease-foreign-cache-v1.md new file mode 100644 index 000000000..dca54ab2b --- /dev/null +++ b/.claude/plans/d-dr64-focus-lease-foreign-cache-v1.md @@ -0,0 +1,153 @@ +# D-DRF-1 — immutable DR64 focus, leased foreign caches and amortized dirty-row writeback + +**Status:** REFERENCE DESIGN / PRE-REGISTRATION ONLY (2026-10-24). This document creates no Rust types, modifies no wire format, proves no speedup, and does not authorize a new executor. Its purpose is to make the operator's clarified design reviewable **before** a coding prompt or EXO-50 coordination. + +**Canonical authority:** D-RMAP-1A (separate RelationshipMap tables and thought palettes), D-NFC (native-first dependency direction), D-NFR (representation status), D-MBX persistence, and the Rust-owned SoA. This note specializes their **focus residency / cache / writeback / optional learning** gap; it does not replace their semantic ownership. + +## 0. Operator decision and thesis + +1. **One DR64 focus is the common case.** It is not a mandate to allocate DR256 or a complete cubic tensor. +2. **The original DR mapping is immutable and versioned.** A map is an address/provenance reference, not a mutable cache index. Any changed map means a distinct sealed revision; no in-place map edit or reused ordinal silently inherits identity. +3. A 16-bit pixel stores the ordinal of **one SoA position** in a 64K-position address space. The value lives in the canonical 512-byte SoA row, not inside DR. +4. The active focus may mix up to 64 identities (Disease, Phenomenology, LOINC, etc.), 64 properties, and potentially 64 **distinct relationship selectors**. These axes are typed and are not automatically interchangeable. +5. Relationships are cheap enough to process before values. The third relationship axis may be an independent **Boolean** plane, not a third full-value copy or a third pointer cube. +6. **Load creates a lease** over the mapping and source snapshot. Lookahead prefetch prepares both subsequent referenced cache lines and the next focus. The lease is removed with the last reader after retirement; it does not change durable DR history. +7. Zero-copy is the default borrowed path. The one exception is the DR64³ foreign-key cache, scoped and conditioned in §5. Repeated 64-thought joins may profit from one gathered, compact cache of selected facets. +8. A cache records local dirtiness, reduces duplicate global destinations under declared update semantics, and hands off a small owned dirty-row batch. Repeated thought operations do not each trigger a WAL commit. +9. Optional ONNX-derived learning output may live in a **separate relationship/plasticity overlay**, as proposed evidence or candidate changes. ONNX remains an external vocabulary/serialization/admission leaf; no internal ONNX graph interpreter, model, proof or guaranteed performance follows from this note. + +## 1. Geometry: do not conflate a bit tile, a pointer map and a relationship cube + +| Logical view | DR64 focus | DR256 global | +| --- | ---: | ---: | +| Population pixels | 64 x 64 = 4,096 | 256 x 256 = 65,536 | +| Boolean presence tile at one bit/pixel | **512 B** | **8 KiB** | +| u16 pixel-to-SoA ordinal mapping at 2 B/pixel | **8 KiB** | **128 KiB** | +| Max distinct referenced 512-byte SoA rows | **2 MiB** | **32 MiB** | +| Dense Boolean 3-axis relation volume at 1 bit/cell | **32 KiB** (64^3) | **2 MiB** (256^3) | + +- The 32 x 128-bit register view and the 64 x 64-bit DR64 Boolean view contain the same 4,096 bits: one 512-byte tile. +- A **64 x 64 u16 pointer map is 8 KiB**, sixteen times the Boolean tile. Its 16-bit ordinals each select an entire 512-byte row from up to 65,536 existing rows; they do not address arbitrary bytes. +- A **64 x 64 x 64 Boolean relation cube is 32 KiB**. If stored *in addition to* the 8 KiB pointer map, their combined allocated payload is **40 KiB**, excluding headers, occupancy/validity, version witnesses and caches. The phrasing "8 KiB becomes 32 KiB" is meaningful for the change of representation, **not** a claim that both arrays fit in 32 KiB. +- DR64^3 has **262,144 logical Boolean cells**. It is not 64^3 pointer entries, not 64^3 SoA rows, not 64 copies of a 2 MiB backing dataset, and not the separate 64-thought occupancy plane. +- DR256^3 is optional global validation/reference geometry, not an automatic execution allocation. +- The **2 MiB / 32 MiB figures are addressable distinct-row footprints**, not necessarily simultaneously resident cache bytes or transferred traffic. A focus may reference fewer distinct rows and repeated ordinals. +- Axis semantics must be declared by the caller: e.g. (identity, property, relationship selector) or (source identity, target identity, predicate). Choosing one layout must not silently turn relationship selectors into property IDs. A Boolean cell means support/presence under a declared semiring; it cannot invent Bag counts, SQL NULL, epistemic evidence or numeric strength. + +## 2. Ownership and identities + +| Layer / owner | Authority and lifetime | +| --- | --- | +| Canonical SoA owner | Mutable values only through its real Rust ownership boundary; native borrowed NodeRow/SoaEnvelope views. No second FrameStore, Table or per-fold mutation guard. | +| Sealed DR / RelationshipMap | Immutable global-to-local reference and relationship topology at a particular **dataset/branch/version + index seal + mapping identity**. Changing relationships creates a successor revision; it never edits the old one. | +| ThoughtMapPaletteBinding | Separate selected map palette and generation per thought. CE64 S/P/O/W bytes do not become MapIds or physical cache selectors. | +| Focus binding | Typed, authorized local aperture and source snapshot witness. One ordinal shared numerically across datasets is not evidence of identical identity. | +| Focus lease | **Ephemeral resident ownership** of a map, compatible source snapshot and any cached gathered facets. Acquired at **focus load**, not at first fold; released only when dependent readers and writer inputs are detached. Not an extra durable DR generation. | +| Native executor | Existing ndarray / mask-risc / Quack / R2IL operations on borrowed or compact aligned buffers. No new tensor VM or per-op dynamic router. | +| CycleWriter + WalSink | Independent owned dirty payloads, cycle ordering and durable publication. The writer never needs a mutable SoA-owner borrow across I/O. | +| Temporal / Thought-Git | Version-aware historical projection, provenance, candidate-vs-committed distinction, deterministic replay. No per-focus-lease durable micro-event requirement. | + +**Identity key:** not merely a raw u16, focus number, local palette slot, d_head, shape or bare u64 version. A cache hit needs the relevant original map/RelationFacts durable reference, row-index seal or stable-row-ID witness, source SoA dataset revision, focus identity/aperture, palette assignment revision, projection/facet layout, operation/semiring, and access scope. Where model-specific compiled palettes are involved, include model/layer/weight/codebook identity and calibration generation. Reuse existing identity authorities: D-RMAP-1A's DurableRef and ThoughtCommitId remain *specified*, **not universally shipped Rust types**. + +Immutable mapping **does not** automatically imply immutable target values. Historical replay requires the exact SoA snapshot(s), not simply ordinal equality. Index compaction that changes physical row offsets requires a stable-row-ID mapping or a new witnessed index seal; an old physical-ordinal mapping is never silently rebound. + +## 3. Focus load is the lease boundary + +Proposed state transitions (not an implemented Rust state machine): + +- **UNLOADED -> LOADING:** resolve map version, palette, authorized focus, index seal and SoA source snapshot. Initiate lazy preparation. Coalesce concurrent requests for the *same fully witnessed key* to avoid duplicate gathers. +- **LOADING -> RESIDENT:** materialize the 8 KiB u16 map if not already present and pin its read-only source snapshot. **Acquire the focus lease here**, even before the first operation. Refuse any failed witness/scope/binding. +- **RESIDENT -> PREFETCHING_NEXT:** prepare Focus B and optionally prefetch its selected SoA facets while Focus A remains borrowed. Do not mutate the immutable mapping in place to become B. +- **RESIDENT -> ACTIVE:** execute via Rust borrows and original native kernels. A materialized foreign facet is a derived cache only, never authoritative fact storage. +- **ACTIVE -> RETIRED:** publish the next focus independently of outstanding writer I/O. Existing readers can finish on A; new readers bind B. +- **RETIRED -> EVICTED:** reclaim A's disposable map/cache allocation once the last reader is gone and any writer payload depending on A is **owned/detached**. Cache eviction does **not** delete the sealed DR or SoA history. + +Use a **focus-scoped lazy initialization entry** with a reference-counted or otherwise proven reader lifetime. A global static LazyLock is appropriate for persistent method/kernel registry but **not** for an evictable focus cache. Existing Rust &[NodeRow] / &mut NodeRow exclusion does the mutation safety work: do not introduce a repeated synthetic frame guard in the fold hot path. + +A cache may retain a map and source snapshot longer than strictly needed for hot reuse. The runtime residency lease is not a distributed storage lease, not the deployment writer claim, and not the 3-day idle-flush policy's *explicitly rejected* read-lease protocol. Those are separate lifecycles and cost models. + +## 4. Two kinds of lookahead prefetch + +1. **Focus anticipation:** predict likely next Disease / Phenomenology / LOINC map bindings; initialize Focus B's lease and begin gathering selected facets while A is active. Predictions may be wrong; they must never change query semantics or pin unlimited memory. +2. **Intra-focus memory lookahead:** the 8 KiB map gives bounded ordinals; request cache lines at SoA base + ordinal * 512 + **declared facet offset** ahead of demand. Bounds and snapshot validity are established by the focus binding, not rechecked per pixel. Software prefetch is advisory, portable fallback is a direct load. + +Prefetch distance is **not fixed by the architecture**. Compare no prefetch, hardware-only, and explicit lookahead (0/4/8/16/32/64 references) across uniform random, clustered, repeated and adversarial ordinals. A random 512-byte-stride SoA access can incur cache-line latency even though the 8 KiB index fits near CPU cache. Budget cached foreign composites independently of the canonical SoA and do not claim the full 2 MiB is actually in L2. + +## 5. Allowed materialization: amortized foreign-table cache for 64 thoughts + +**DECISION (2026-10-24):** the DR64³ foreign-key cache is a scoped exception to the zero-copy law (`.claude/knowledge/zero-copy-lens-law.md` § "The DR64³ foreign-key cache exception"). It is not an elevation under the law's rung test: a gathered facet is a cast-reproducible projection, and the exception says so. + +- **SCOPE:** foreign-key operations over a DR64 focus whose referenced rows come from several ontologies (the motivating case is about 8, e.g. Disease / Phenomenology / LOINC and others), reused across many thoughts, with relationship learning over the result feeding live-generated ONNX as the mathematical compiler substrate. Nothing outside that shape inherits the exception: a single-ontology fold, a one-shot query, or a value path that a borrowed lens already serves stays under the law. +- **BASIS:** the cost is amortized in three places. (1) Writes: only dirty rows are written back, coalesced by canonical destination, so a focus touched by many thoughts commits a few full-row images, not one write per operation. (2) Reads: most accesses (expected 70–90%) are reads served from the versioned cache; without it each of them is a scattered 512-byte-stride read into a foreign table, repeated per ontology, per thought and per learning iteration, which leaves the nanosecond fold. (3) Learning: the relationship maps are Boolean support planes, and many learning steps need only those, not the row values, so they skip the value gather entirely. The three figures (dirty fraction, read-hit fraction, value-free learning fraction) are HYPOTHESES until DRF-P2 measures them. This is a cost argument, made explicitly for this scope, which is why it is recorded as a decision rather than derived from the law. +- **CONDITIONS:** the cache is disposable and never authoritative; clearing it and recomputing from the exact sealed references must give the same answers; its key is the full witness of §2 (map revision, index seal, SoA source snapshot, focus, palette generation, facet layout, operation, scope); it never holds uncommitted truth; its build cost is reported separately, never folded into post-warm timings. The gather itself still uses the native kernels, and in-fold execution stays under the no-millisecond rule. +- **REVISIT WHEN:** DRF-P2 measures it. If the gather-and-reuse arm (B or C) does not beat the direct arm (A) on whole-run cost at the operator's workload, including cold build, the exception lapses for that workload and the cache is removed rather than tuned. + +A pre-fold gather may be worthwhile **when the admission rules allow it** and repeated execution amortizes random-access cost. Do not silently repeal D-RMAP-1A, H14c/H14d or the no-foreign-gather firewall: the first implementation must show a typed, explicitly admitted coordinate translation, complete contributors, required authority and a separate measured cold preparation cost. + +Examples for a DR64 focus of 4,096 pixels: + +- Mapping: 8 KiB, retained as immutable provenance. +- Gathered 16-byte facet: **64 KiB** per materialized foreign projection. +- Three different 16-byte projections for Disease / Phenomenology / LOINC: **192 KiB** excluding keys, mapping and masks. +- Entire 512-byte rows: 2 MiB if all 4,096 destinations are distinct. **Do not default to that copy** when only a few facets are needed. +- Optional 64^3 Boolean relation cube: **32 KiB**; can be materialized, compressed or evaluated as a set of existing 64x64 slices under typed relation identity. It is not a replacement for the 8 KiB pointer map. + +The **cache is expendable**. It must be legal to clear all resident foreign views and recompute from the exact sealed references. It must not store uncommitted semantic truth, silently over-approximate a focused reduction, or make model/palette weights valid only by matching shape. + +Cost per thought for N reuses is roughly: local_fold + (gather + cache_build + dirty_reduction + writeback) / N, **when** the operation's effects and numerical result contract permit this amortization. For 64 thoughts, compare actual per-table reuse, not just a blanket divide-by-64 claim. + +## 6. Dirty-row propagation without a second writer + +At each local operation, mark changed local positions (4,096-position Boolean dirty plane = **512 B**). On flush: + +1. Take dirty local positions; translate **through the exact original immutable DR mapping** into canonical SoA row ordinals and owner/dataset keys. +2. **Coalesce by canonical destination**. Multiple pixels can point to one row, and several thoughts can update different facets of that row. Coalescing cannot use an unqualified "last payload wins" policy for partial deltas; specify additive, overwrite, Boolean union, guarded replacement or explicit refusal **per declared operation**. Preserve deterministic thought/stream ordering and source roots where required. +3. Build **one complete 512-byte final row image per changed destination**, against the sealed predecessor version Vn, or a properly typed sparse patch only if its storage ABI is explicitly admitted. Unchanged rows are inherited. +4. Move detached owned payloads into **BatchWriter -> collect_casts -> CycleWriterHandle::try_submit -> DetachedCycleBatch::freeze -> WalSink::commit_cycle**. The thought loop resumes on successful queue handoff; durability is **not** asserted until the sealed outcome is known. A rejected/full queue returns ownership; an ambiguous commit reconciles using the **same frozen batch**, never newly sampled mutable SoA bytes. +5. Release a source-dependent focus lease as soon as writer inputs are genuinely **independent of that source**, not merely when a descriptor names a future live SoA read. The current CycleWriter path moves owned Vec payloads; the current LanceCycleWriter still copies frozen payloads into Arrow builders. Do not call that physical seam end-to-end zero-copy. +6. A successful artifact-backed seal creates Vn+1 and permits replay/transition application. Intent-only/no-change cycles are not forced into the WAL. Never publish in-flight Vn+1 values as if they were part of Vn. + +**Correctness before throughput:** sparse row updates from an unrelated ontology must not overwrite a concurrently valid different facet merely because both resolve to one 512-byte record. Version conflicts either reconcile under an admitted deterministic reduction or refuse/rebase using the existing owner/writer rules. + +## 7. Relationship-first learning as a separate future overlay + +An optional future learner may treat the 64^3 relationship coordinate system as a **cheap address/support substrate**: + +- Immutable DR version identifies WHERE the canonical relationships/values live; it is **not trained**. +- Boolean relation plane identifies WHICH bounded candidate triples are present or addressable. This plane is distinct from confidence/weight magnitude and from the 64-thought occupancy register. +- A separate sparse or quantized **plasticity/evidence overlay** holds learnable values and their provenance. It can be revised and versioned **without rewriting the original DR mapping**. +- ONNX/Optimum can serve as **offline or external model interchange, candidate-update stream and oracle adapter**. They are not native storage engines, not certified proof generators and not a new internal graph/interpreter. An S3-like cheap durable artifact stream is a *storage architecture hypothesis*: object storage should hydrate/version relatively large artifacts, not service per-pixel random reads or replace local mmap-backed execution. +- ndarray/BGZ/Helix/Palette256 supply existing numerical methods; approximate compiled tables remain on a separately calibrated quality frontier. ONNX candidate updates are **not certified** merely because they came from a model. +- Thought-Git may associate each candidate delta with source roots, parent commit, model/calibration identity and deterministic replay witness; approved relation changes land only as a new sealed revision, preserving the old map. No inferred automatic approval of probabilistic rules. + +A discrete u16 address lookup has no generic differentiable gradient. Back-propagation, if admitted, is through **declared value/weight paths and explicit aggregation of duplicate references**, not by treating the ordinal selection itself as a smooth ONNX operation. The third relationship axis can reduce unnecessary value gathers, but that win remains to be measured. + +## 8. Audit boundary + +Record **durable mapping/relation revisions and admitted updates** at the existing sealed persistence/Thought-Git boundary with enough identity and data (or deterministic reconstruction inputs) to replay. A hash proves identity/integrity under its chosen mechanism; **a digest alone cannot reconstruct a discarded map**. Query-time temporal.rs can project the resulting versioned history with a concrete Strict reference. It is not itself a WAL or an event publisher. + +Lease acquire, prefetch, miss, hit, retirement and eviction are **runtime metrics** rather than mandatory durable semantic commits. A watcher with always-latest notifications can drive cache invalidation but cannot replace an audit trail because slow readers can skip intermediate events. + +Keep historical frames independently readable. A new focus does not mutate old revisions. Authorization/ACL revocation and cross-tenant scope changes remain an explicit admission policy for new/reused leases; a historical version is not a bypass for current access restrictions. + +## 9. Explicit ownership of next actions (no new implementation authorized here) + +- **D-DRF-1 / this PR:** reference semantics, geometry, lifetime, storage and falsification contract only; add no new Rust production API. +- **DRF-P1 (separate future lab PR):** two-focus lazy-load / lease / prefetch / last-reader-eviction probe using existing owner/snapshot/borrow seams. Negative race and stale-map fixtures first. +- **DRF-P2 (separate future lab PR):** 64-thought foreign-table A/B/C comparison (direct+prefetch versus gather+reuse versus gather+reuse+dirty writeback). Include cold build and realistic memory. The reported winner is physical, never semantic. +- **DRF-P3 (separate future lab PR):** admitted relationship cube/slices and typed dirty-row reduction oracle. Demonstrate mixed ontology domain seals and no accidental Bag/NULL/evidence loss. +- **DRF-P4 (OPTIONAL, separate research):** plasticity overlay and ONNX serialized candidate stream with independent oracle, provenance and failure gates. No change to native-first architecture. + +**No EXO-50 message, coding-agent prompt, new DR field, map registrar, CE64 bit redefinition or production cache API is part of this PR.** Use the landed discussion/reference as the source when coordination is explicitly requested. + +## 10. Independent falsifiers and measurement matrix + +Read [D-DRF-1 falsification preregistration](d-dr64-focus-cache-falsifiers-v1.md). Required gates: geometry; map revision replay and compaction; focus eviction under concurrent readers; lookahead correctness; legal pre-fold gather; 64-thought total economics; duplicate-target dirty reduction; queued/asynchronous failure and reconciliation; ONNX candidate/proof separation. + +## Source crosswalk + +- [D-RMAP-1A ownership and versioning](d-rmap-separate-tables-thought-palettes-v1.md), especially §§2, 6a, 7 and 8; [D-RMAP-1](d-rmap-implicit-relationship-register-v1.md). +- [D-NFC dependency correction](d-native-first-dependency-correction-v1.md); [D-NFR native representation](d-native-repr-contract-v1.md). [#1755](https://github.com/AdaWorldAPI/lance-graph/pull/1755) is the separate SoA borrow probe (merged 2026-10-24); do not copy its withdrawn FrameStore design. +- [DR64^3 holographic geometry](d-dr3-holo-onnx-s3a-s3c-v1.md), [physical comparison/reuse cost](d-onnx-relalg-s3d-physical-shootout-v1.md). +- Existing source: [NodeRow/SoaEnvelope](../../crates/lance-graph-contract/src/soa_envelope.rs), [temporal.rs](../../crates/lance-graph-planner/src/temporal.rs), [BatchWriter](../../crates/lance-graph-planner/src/batch_writer.rs), [persistence sink](../../crates/lance-graph-planner/src/persist_sink.rs), [CycleWriter](../../crates/lance-graph-supervisor/src/cycle_writer.rs), [LanceCycleWriter](../../crates/lance-graph/src/graph/cycle_sink.rs). +- [Existing Palette256 plasticity probe](../board/entries/2026-10-22-pal256-hebbian-overlay-plasticity.md), [NARS/Bayes evidence boundary](../board/entries/2026-10-22-pal256-nars-bayes-evidence-contract.md), [S3 hydration doctrine](../board/entries/2026-08-06-e-object-store-hydrates-it-does-not-store-1.md).