diff --git a/.claude/board/INTEGRATION_PLANS.md b/.claude/board/INTEGRATION_PLANS.md index 72efec009..969c3d944 100644 --- a/.claude/board/INTEGRATION_PLANS.md +++ b/.claude/board/INTEGRATION_PLANS.md @@ -1,3 +1,8 @@ +## 2026-11-04 — D-DRISA-0: virtual DR registers × Abakus ALU × async data-local cognitive shader + +**Status: PROPOSED / DOCS ONLY.** [Architecture](../plans/d-drisa-remote-register-shader-v1.md) · [staged PR charter](../plans/d-drisa-staged-pr-charter-v1.md) · [57 independent falsifiers](../plans/d-drisa-falsifier-matrix-v1.md) · [entry](entries/2026-11-04-d-drisa-0-remote-register-contract.md) · [STATUS_BOARD](STATUS_BOARD.md) D-DRISA · [EXO-153](https://linear.app/exo-red/issue/EXO-153) · [EXO-50](https://linear.app/exo-red/issue/EXO-50) bus. **New hypothesis:** use existing DR frame as data-local virtual register namespace and preexisting Quack/R2IL/mask-risc/ndarray words as the typed ALU, including shipped Abakus half/full adders where arithmetic helps, with a bounded asynchronous remote descriptor and compact result. Do not send raw pointers, transport payloads by default, copy canonical values into DR, or create new global row mappings. Network zero-copy is *not* claimed; Arrow C same-process ABI and optional Flight RPC are distinguished. Nine separately gated PR stages, no production implementation. This does not replace D-DRAM-0/P3 or claim #1950's copied-lane prepared binding as address-only memoization. **D4/S-INGRESS, EXO-46/89/151, security and THOUGHT/WORLD admission remain open.** +**Confidence:** existing local borrowed-operand instruction surface source-verified; remote/asynchronous contract, economics, result continuation and source leases unimplemented and unmeasured. + ## 2026-11-03 — D-DRAM-P3: P2 open ends integrated with the D-DRAM-0 falsifier map → `.claude/plans/d-dram-p3-integrated-v1.md` **Status:** PLAN, owned by this session (operator). **Board:** [STATUS_BOARD](STATUS_BOARD.md) D-DRAM-P3-1..8. Credits P2 (#1950) against the pre-registered #1949 gates (L9, L10, C2, C4 measured in the lab; L1/L2/L6/A3/A4 partial). Orders eleven owned items by signal per effort: three need no outside decision (ReadPolicy range count, non-vacuous projection cases, the D-DRAM-1 census). Production and gated items keep their owners; nothing merges automatically. diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index e27abdb64..42cf8d85d 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -1,3 +1,21 @@ +## D-DRISA — DR remote register + Abakus cognitive shader (2026-11-04) + +**Status: D-DRISA-0 PROPOSED / DOCS ONLY.** [Master architecture](../plans/d-drisa-remote-register-shader-v1.md) · [staged PR charter](../plans/d-drisa-staged-pr-charter-v1.md) · [57 falsifiers](../plans/d-drisa-falsifier-matrix-v1.md) · [board entry](entries/2026-11-04-d-drisa-0-remote-register-contract.md) · [EXO-153 owner tracking](https://linear.app/exo-red/issue/EXO-153) · [EXO-50 append-only handoff](https://linear.app/exo-red/issue/EXO-50). **No remote driver, API, copied payload cache, new map or performance measurement is implemented by this plan.** + +| D-id | Staged PR / deliverable | Status | Owner / gates | +| --- | --- | --- | --- | +| D-DRISA-0 | documented virtual-register contract, ALU semantic map, async state model, staged PR charter and independent oracles | PROPOSED, THIS DOCS PR | EXO-153 and EXO-50 coordinate only, neither admits runtime | +| D-DRISA-1 | local borrowed-register equivalence, exact frame/scope refusals | FUTURE LAB | DT64 / mask-risc / R2IL, no new ISA | +| D-DRISA-2 | async loopback lifecycle, bounded queue, cancellation and lease invalidation | FUTURE LAB, security review | runtime/transport owner; no live TCP | +| D-DRISA-3 | Abakus k-of-n using shipped half/full adders, independent scalar/Boolean oracle | FUTURE LAB | ndarray/Quack owner; U24x5 examples-only | +| D-DRISA-4 | two-process read-only data-local command vs payload-fetch baseline | FUTURE LAB, transport/security gate | explicit authorization before networking | +| D-DRISA-5 | optional leased source-location receipt, zero copied payload, prefetch | **BLOCKED** | EXO-46 / EXO-89 / EXO-151 (no production lease host) | +| D-DRISA-6 | witnessed DR→Lance _rowid authority and mask ingress | **BLOCKED_D4 / S_INGRESS_COPY** | D4 / S-INGRESS / EXO-114, #1633 | +| D-DRISA-7 | grammar/XLOG/NAL read-only cognitive consumers, limited continuation | GATED | EXO-140 / EXO-73 / EXO-78 / semantic owners | +| D-DRISA-8 | whole-query/async-fault audit and explicit production ruling | NOT AUTHORIZED | operator and named code/security owners | + +**Invariants:** DR retains unique coordinate authority; borrowed canonical SoA is default; local borrow ≠ network zero-copy; receipts store locations, never values; remote method and result mask require scope/source/frame lease; no implicit cross-call scratch-slot survival, F###, second executor or automatic THOUGHT→WORLD promotion. #1950 Q≥2 timing included gathered copied lanes, so is not standalone address-only evidence. #1951 4,096-page rule is a fragmentation proxy, not physical page count. [Integration map](INTEGRATION_PLANS.md) has the full crosswalk. + ## D-DRAM — DR address memoization + adaptive Abakus masking lens (2026-11-03) **Status:** PROPOSED / DOCS ONLY; [plan](../plans/d-dram-abacus-address-memoization-v1.md), [board entry](entries/2026-11-03-d-dram-abacus-address-memoization.md), [EXO-50](https://linear.app/exo-red/issue/EXO-50/agent-message-bus-gpt-xgrok-ccsession-id). No code, tests, new runtime, new row-id map or production authorization. diff --git a/.claude/board/entries/2026-11-04-d-drisa-0-remote-register-contract.md b/.claude/board/entries/2026-11-04-d-drisa-0-remote-register-contract.md new file mode 100644 index 000000000..0912d7115 --- /dev/null +++ b/.claude/board/entries/2026-11-04-d-drisa-0-remote-register-contract.md @@ -0,0 +1,45 @@ +# D-DRISA-0: DR virtual register + Abakus ALU + remote async cognitive shader (pre-registered) + +**Date:** 2026-11-04. **Status:** PROPOSED / DOCS ONLY; no executable remote driver, ABI, F###, ISA extension, pointer-transport, source authority or speedup claim. **Operator idea:** eliminate unnecessary hash-table-like lookups and payload transport by executing a typed program *where canonical SoA is already owned*, using a witnessed DR frame as virtual register-address space and Abakus as an optional arithmetic ALU. + +**Plan (authoritative):** [D-DRISA-0](../../plans/d-drisa-remote-register-shader-v1.md) · [staged PR charter](../../plans/d-drisa-staged-pr-charter-v1.md) · [57 falsifiers](../../plans/d-drisa-falsifier-matrix-v1.md). +**Linear:** [EXO-153 program](https://linear.app/exo-red/issue/EXO-153) and [EXO-50 append-only bus](https://linear.app/exo-red/issue/EXO-50). + +## Reuse, not reimplementation + +- **DR:** existing `lance-graph-dt64::dr::{MapTable,Frame,LocalMap}` owns coordinates, not values. No second global→`_rowid` store. +- **Lens:** existing `ClassView`, `WideFieldMask` and immutable DT64 `MethodBinding` describe only demanded fields and authorized behavior. +- **ISA:** Quack/RelAlg/R2IL → mask-risc borrowed `Plane/Scratch/LaneRef` → ndarray SIMD. No new instruction loop, graphics backend or automatic F###. +- **Abakus:** shipped `half_add_mask` and `full_add_mask` are valid words; #1951's `U24x5`/`Abacus` are examples-only. Arithmetic thresholds are a **candidate**, native AND remains the control. +- **Source:** use canonical borrowed SoA and optional prefetch at its owner. Remote descriptor and compact result may be transported, but **remote zero-copy is not proven or promised**. Arrow C zero-copy is same-process, not an RPC guarantee. +- **Continuations:** R2IL scratch slots have epochs. Remote `MaskReceipt` cannot silently outlive its program or become a new global mask store; owner/lifetime decision required. +- **Epistemic:** grammar and XLOG/NAL polyfills are candidate consumers; THOUGHT remains speculative and WORLD writes require separate hydration/Rubicon admission. + +## Staged PR sequence, not existing PRs + +| Stage | Deliverable | Status / gate | +| --- | --- | --- | +| D-DRISA-0 | docs, board, EXO-153 issue, EXO-50 handoff | THIS PR; research-only | +| D-DRISA-1 | local borrowed register oracle and source/lifetime refusal | FUTURE LAB | +| D-DRISA-2 | bounded async loopback, cancellation and lease revocation | FUTURE LAB; security review | +| D-DRISA-3 | k-of-n Abakus with shipped words and negative controls | FUTURE LAB; ndarray/Quack owners | +| D-DRISA-4 | two-process data-local command vs fetch-payload baselines | FUTURE LAB; transport/security gate | +| D-DRISA-5 | source-location receipt and prefetch without copied values | **BLOCKED EXO-46/89/151** | +| D-DRISA-6 | real DR→Lance `_rowid` binding, copied mask ingress measured | **BLOCKED D4/S-INGRESS**, EXO-114 | +| D-DRISA-7 | read-only cognitive consumers, bounded result masks | **GATED EXO-140/73/78**, semantic owners | +| D-DRISA-8 | independent full-cost audit, production ruling | NOT AUTHORIZED | + +## Falsifiers and cost + +- **57 pre-registered rows:** R1–R16 addressing/leasing; A1–A8 ALU; N1–N14 async and transport; C1–C10 cognition/semantics; P1–P9 full-path economics. All PROPOSED/UNEXECUTED here; earlier #1948/#1950/#1951 evidence is identified as partial, never credited as remote results. +- **Whole-query benchmark:** local borrow B0, remote payload fetch B1, data-local instruction B2, binding reuse B3, memoized locator B4, prefetch B5, Abakus B6. Count cold costs, all copies, RPC bytes, source pages, authentication, p95/p99 and refusers. +- **#1950 caveat:** its route-once `Binding::prepare` retains *copied* `lane_a/lane_b`; Q≥2 improvement is not locator proof. +- **#1951 caveat:** `range_count * 4096` is a fragmentation heuristic/policy pin, not a count of physically touched pages. No planner law inferred. + +## Required decisions and STOP conditions + +[EXO-46 admission](https://linear.app/exo-red/issue/EXO-46), [EXO-89 relationship owner](https://linear.app/exo-red/issue/EXO-89), [EXO-151 focus lease](https://linear.app/exo-red/issue/EXO-151), [EXO-140 gather/hop classification](https://linear.app/exo-red/issue/EXO-140), [EXO-121 ClassView](https://linear.app/exo-red/issue/EXO-121), [EXO-114 ordering](https://linear.app/exo-red/issue/EXO-114), [EXO-73 recursion](https://linear.app/exo-red/issue/EXO-73), [EXO-78 thought layers](https://linear.app/exo-red/issue/EXO-78). + +**STOP:** untrusted remote pointer, unauthorized field, stale frame, second row-ID map, default value gather, silently serialized payload, cross-tenant leak, runtime write, new F###, unsupported arithmetic semantics, fictitious public kernel, automatic merge. + +**Reference PRs:** [#1948](https://github.com/AdaWorldAPI/lance-graph/pull/1948), [#1949](https://github.com/AdaWorldAPI/lance-graph/pull/1949), [#1950](https://github.com/AdaWorldAPI/lance-graph/pull/1950), [#1951](https://github.com/AdaWorldAPI/lance-graph/pull/1951). [STATUS_BOARD](../STATUS_BOARD.md) and [INTEGRATION_PLANS](../INTEGRATION_PLANS.md) both list this D-id. diff --git a/.claude/board/entries/README.md b/.claude/board/entries/README.md index f4806b3d6..5e757307a 100644 --- a/.claude/board/entries/README.md +++ b/.claude/board/entries/README.md @@ -25,10 +25,11 @@ index row, (3) no duplicate entry id. Checks 1 and 2 are deliberately opposite directions; the stranding this convention prevents shows up in exactly one of them, never both. -571 entries, 2026-08-06 .. 2026-11-03. +572 entries, 2026-08-06 .. 2026-11-04. | date | entry id | finding | file | |---|---|---|---| +| 2026-11-04 | `D-DRISA-0` | | [2026-11-04-d-drisa-0-remote-register-contract.md](2026-11-04-d-drisa-0-remote-register-contract.md) | | 2026-11-03 | `D-XLOG-META-WIRE-P0` | | [2026-11-03-d-xlog-meta-wire-p0.md](2026-11-03-d-xlog-meta-wire-p0.md) | | 2026-11-03 | `D-GRAMMAR-WHEN-1` | | [2026-11-03-d-grammar-when-1.md](2026-11-03-d-grammar-when-1.md) | | 2026-11-03 | `D-GRAMMAR-CONSTRAINTS-0` | | [2026-11-03-d-grammar-constraints-0.md](2026-11-03-d-grammar-constraints-0.md) | diff --git a/.claude/plans/d-drisa-falsifier-matrix-v1.md b/.claude/plans/d-drisa-falsifier-matrix-v1.md new file mode 100644 index 000000000..ebf9b75f4 --- /dev/null +++ b/.claude/plans/d-drisa-falsifier-matrix-v1.md @@ -0,0 +1,129 @@ +# D-DRISA falsifier matrix: local register, Abakus ALU, async driver, cognition and economics + +> **2026-11-04 · PRE-REGISTERED / UNEXECUTED.** Every row below is a **proposed** test, unless a precisely matching earlier PR is cited as PARTIAL evidence. Passing previous P2/P3 correctness is **not** completion of remote or locator falsifiers. No performance value is fabricated. +> +> Owner queue and exact PR boundaries: [staged charter](d-drisa-staged-pr-charter-v1.md). Architecture: [D-DRISA-0](d-drisa-remote-register-shader-v1.md). Tracking: [EXO-153](https://linear.app/exo-red/issue/EXO-153), coordination: [EXO-50](https://linear.app/exo-red/issue/EXO-50). + +## A. Method for judging a gate + +Each experiment must name (1) immutable source/version and original independent scalar oracle, (2) exact input, including scope and truth mode, (3) expected answer **and** explicit refusal, (4) disable/mutation that should break an assertion, (5) warm/cold costs, allocations, bytes and source receipts, (6) owner and commit SHA. Every successful refusal needs a paired successful authorized control. An oracle that calls the same map, source-locator helper or mask program as the candidate is **not independent**. + +**Truth:** Canonical corpus fixture with source global IDs and known typed values generated independently of the register/packing code; Boolean support and Bag multiplicity calculated separately; Kleene UNKNOWN explicit for missing value; value 0 distinct from absence. Source leases and permissions are fake-but-attested only for the first *local* stages; remote authorization is not proven by a fake token. A null path does not silently become FALSE unless exactly the admitted SQL WHERE rule says so. + +**Measurements:** absolute p50/p95/p99 and variance, Q=1/2/8/16/64/256; single DR64 / DR256 / multiple tiles; 0, 0.1, 1, 10, 50, 95, 100% selection; clustered/scattered/fragmented; large and small result; in-memory, page cache and disk distinct. Include request/result bytes, logical and physical rows, *actual* pages read if provider counters support it, copies, allocations, CPU cycles, scratch footprint, prefetch hits/misses. No arbitrary 1ms remote-SLA claim. + +## R. DR register, identity, lease, locator: R1–R16 + +| ID | Exact fixture / independent oracle | Deliberate disable / expected refusal | Owner / status | +| --- | --- | --- | --- | +| R1 | Two frames map same 64 identities to permuted local positions; oracle by global ID compares COUNT/EXISTS/mask | Reuse old ordinal without frame seal → wrong answer/refusal | DT64/map-risc, PR1 LAB | +| R2 | DR256 tile with nontrivial end tail and one-hot at final active bit | Disable tail clear → false extra bit | mask-risc, PR1 LAB | +| R3 | 1M identities over multiple DR256 tiles with two disjoint selections, a shared global id | Join on local index rather than global identity → oracle divergence | DR, PR1 LAB | +| R4 | MAP-only COUNT over pre-bound population plane; input source value array not mapped | Any payload-lane read or source I/O fails; count remains exact | DR/Quack, PR1 LAB | +| R5 | ClassView demands exactly 3 of 64 fields in physical order permuted; oracle maps field IRIs | Physical ordinal shortcut or field label shortcut → refusal/wrong sum | ClassView, PR1 LAB, #1950 partial | +| R6 | Authorization allows 3 fields; ask for fourth and ask for unknown IRI | No widen-to-all; Unauthorized / UnknownProperty | DT64/security, PR1 LAB | +| R7 | Same map/frame, 2 methods, Q=1,2,8,16,256; source unchanged | Force reroute per call; measure extra cost but answer parity | DR, PR1 LAB; #1950 bundled reference | +| R8 | One frame remains, canonical value mutates and a derived WHERE mask is reused | Source-value witness changed → mask must refresh/refuse; frame may remain valid | source/Quack, PR5 BLOCKED | +| R9 | Canonical source compacts, reorders, splits fragment or changes stable row ID under old DR frame | Reuse stale physical locator → typed StaleLocation, never wrong data | D4/source, PR5/6 BLOCKED | +| R10 | Focus slot recycled to another tenant while old remote job queued | ABA / scope check bypass must turn test red | EXO-46/151/security, PR5 BLOCKED | +| R11 | Source lease expires after binding, before borrowed read | No use-after-release; stale typed refusal, not empty result | source owner, PR1/5 GATED | +| R12 | Memoized locator hit → evict → miss on same still-valid source | Answer invariant, only lookup/cost changes; no persisted truth in receipt | EXO-151, PR5 BLOCKED | +| R13 | Two readers share leased same canonical bytes; a writer invalidates during one execution | Correct pinned snapshot or typed refusal, no torn/read-after-free | source/lease owner, PR5 BLOCKED | +| R14 | Old frame read with historical SoA snapshot available, then snapshot dropped | Correct old value with valid snapshot, then HistoricalSourceUnavailable | DR/source, PR5 BLOCKED | +| R15 | DR bool mask, ClassView field mask, Lance row-address mask all same bit width but different meanings | Deliberately pass one as another → typed failure; no raw transmute | DT64/Lance, PR1/6 | +| R16 | Source location receipt and auth capability are removed, without removing sealed DR relationship | Relation truth unchanged; only value read refusal/slower rebind | EXO-46/89/151, PR5 BLOCKED | + +**Important measured predecessor:** #1950 P2 `corrections` uses `Binding::prepare`, which already copies `gather_lane` payload values. Its Q≥2 win is not R7/R12 proof of source-location memoization. #1951 reports no production locator-lease host; PR5 cannot become production by inventing one. + +## A. Abakus / shader ALU: A1–A8 + +| ID | Exact fixture / oracle | Disable / negative control | Owner / status | +| --- | --- | --- | --- | +| A1 | k-of-n with n=1,2,4,5,16,24; k=0,1,ceil(n/2),n,n+1; oracle per identity summing Boolean input truth | Wrong threshold boundary and k>n must be caught | ndarray/Quack, PR3 LAB | +| A2 | 64 independent identities: one-hot, alternating, 0/full, upper-tail/random, high/low carry collisions | Swap significance carry plane or shift by one *identity* → fail | ndarray, PR3 LAB | +| A3 | Same expression over DR64, DR256 and multi-tile frames with 3/64 demanded fields | Use physical lane ordinal as field position → wrong answer | ClassView/Quack, PR3 LAB | +| A4 | Packed 4×u8 in u32 with zero/present/absent sentinel, threshold 0,64,170,255 and out-of-range | Ordered-u8 comparator must not be claimed shipped; unauthorized codec refuses | ndarray, separate PR6 proposal, #1950 gap | +| A5 | SQL Kleene UNKNOWN for NULL, empty population, logical NOT, present zero | Replacing UNKNOWN with false *before* a permitted WHERE terminal fails | Quack/semantics, PR3 LAB | +| A6 | Boolean support vs Bag multiplicities 0,1,2,255; same support different total count | Count support instead of multiplicity → fail | Quack/RelAlg, PR3 LAB | +| A7 | Native AND/OR/ternlog vs Abakus on 2-mask AND and a true ≥3-of-5 expression | Counter extra work must not be default; compare cold+warm, disable adder carry | ndarray/mask-risc, PR3 LAB | +| A8 | 4 ISA categories where supported (scalar, AVX2, AVX-512, NEON/wasm); seeded independent oracle | Backend wrong tail/overflow/sign must turn test red; skip unsupported hardware honestly | ndarray owner, PR3 LAB | + +The `U24x5` packing microtile and `Abacus` examples remain **examples-only** per #1951 census, not a shipped API. `half_add_mask`/`full_add_mask` are the existing legal words. The packed-u8 ordered comparison is **separate** from k-of-n carry arithmetic. + +## N. Async driver and transport: N1–N14 + +| ID | Exact fixture / oracle | Deliberate break and expected status | Owner / status | +| --- | --- | --- | --- | +| N1 | Same typed COUNT locally synchronously and via loopback queued worker on one immutable source | Mismatch result or effective witness fails | runtime, PR2 LAB | +| N2 | 64 admitted concurrent requests with max_inflight=8 and bounded scratch; same oracle | Overfill queue → ResourceExhausted, bounded RSS | runtime, PR2 LAB | +| N3 | Two jobs reuse request_id but differ in operation digest or scope | Replay/Conflict refusal; never return first job's answer | security, PR2 LAB | +| N4 | Cancel before admit, after admit, queued, already running and after completion | Cancellation cannot fabricate FALSE/0; CancelAck not rollback | runtime, PR2 LAB | +| N5 | Timeout and network disconnect after job ran but before reply | Indeterminate, retry only same immutable snapshot and digest | transport, PR2/4 LAB | +| N6 | Authorization revocation while request queued and while running | Out-of-scope data must not escape; lease/policy decision explicit | security, PR2/4 GATED | +| N7 | Untrusted client changes class/schema revision but keeps same frame and result shape | SchemaMismatch, not automatic default | DT64/security, PR2 LAB | +| N8 | Untrusted program requests forbidden opcode/F### or invalid scratch slot | UnsupportedProgram/UnboundSlot, not arbitrary native evaluation | R2IL/mask-risc, PR2 LAB | +| N9 | Execution returns MaskReceipt then client reuses on a different frame, source, scope or expired TTL | Reject wrong binding; do not read stale scratch slot | register/results owner, PR2/7 GATED | +| N10 | B1: client fetches source data; B2: remote data-local executes the same query; compare outputs and bytes | Force a payload column on B2 wire → violation, even if faster | transport, PR4 LAB | +| N11 | B2 server sends only scalar; mask output size > maximum allowed result | ResultTooLarge or bounded receipt, not silent row transfer | transport, PR4 LAB | +| N12 | Delayed request after source overwrite or compaction | WrongSourceVersion, never silently rebase operation | source/security, PR4/6 GATED | +| N13 | Two source owners with coincident local DR ordinals, no cross-owner join witness | Refuse unsupported cross-owner operation; no accidental join | D4/EXO-46, PR4 LAB | +| N14 | Network partitions with 2 hosts, duplicate completions, bursty load, misrouted locality hint | No cross-tenant data, bounded resources and measured timeouts | transport/security, PR4/8 GATED | + +**Transport baseline:** packet capture or explicit transport byte counters must show command and result bytes. Sending only a handle is not enough if the server secretly fetches the entire source into a new buffer. + +## C. Cognitive semantics and continuation: C1–C10 + +| ID | Fixture and oracle | Deliberate break / refusal | Owner / gate | +| --- | --- | --- | --- | +| C1 | #1948 German grammar `geben/gefallen`, identical dependent relation and Case, distinct head frame; independent annotation oracle | Confuse frame/candidate role → mismatched result | grammar/DT64, PR7 GATED | +| C2 | #1948 cold co-addressed head-frame producer, source lexicon language + revision changed while DR frame stays | Revision-only check insufficient if language changes; stale producer refused | EXO-46/140, PR7 GATED | +| C3 | Two read-only methods consume the same scope-bound WHERE/result plane in *one* admitted program | Scratch slot reused across epoch → StaleSlot, not a valid continuation | R2IL/Quack, PR1/7 | +| C4 | Bounded remote result-mask continuation across *two* requests, same frame/scope/source and live result sink | Without admitted result sink, refuse; no global mask cache | EXO-46/result owner, PR7 BLOCKED | +| C5 | One grammatical input gives competing Direction/Purpose functions | Preserve both candidate masks and UNKNOWN; no arbitrary winner | grammar owner, PR7 | +| C6 | XLOG simple acyclic 3-way join with independent enumerative Boolean oracle | Row/tuple reconstructed in hot path or foreign gather bypass → refusal | XLOG/Quack, PR7 | +| C7 | NAL/polyfill truth mode, Boolean support versus Bag and epistemic status | Mislabel support as weighted truth/causal fact → refusal | NAL semantic owner, PR7 | +| C8 | Any unadmitted recursive request (`EXO-73` open) | typed UnsupportedRecursion, no hidden fixpoint driver | EXO-73, PR7 | +| C9 | THOUGHT candidate with witness and provenance, request asks to write WORLD | no hydration/Rubicon gate → NotAdmitted, no side effect | EXO-78 / Thought-Git, PR7 | +| C10 | Two methods reference one another through WHEN, or require hop-shaped `EqU32Via` | detect cycle / EXO-140 unclassified foreign lookup, not extra interpreter | DT64/EXO-140, PR7 | + +## P. Cost and experimental validity: P1–P9 + +| ID | A/B comparison and counted cost | Falsifier / decision consequence | +| --- | --- | --- | +| P1 | B0 local borrowed vs B1 remote fetch vs B2 remote instruction+scalar on same source | If B2 does not beat B1 on whole-query time/bytes for a meaningful region, no throughput claim | +| P2 | Q=1/2/8/16/64/256 on unchanged source: routing per query vs one permitted binding | Savings not present without copied payload? separate address-only result, no bundled inference | +| P3 | Borrow direct vs memoized location vs prefetch, input locality clustered/scattered and false lookahead | If warm borrow loses 2.3–2.5× at 50% as in #1950, investigate, not mandate gather | +| P4 | 0.1/1/10/50/95% selectivity, one/4/16 projected columns, range counts varied independently | `range_count × 4096` is a bound/proxy, not actual pages touched; choose measured source costs | +| P5 | Cold/warm page cache and real disk IO, plus S-INGRESS tree build and attest costs | Report physical source bytes, not just logical selected rows or lane hits | +| P6 | Identical end-to-end semantics with and without repeated mask egress, bounded result receipt if admitted | A hidden remote mask store, wrong scope or unknown lifetime fails | +| P7 | End-to-end k-of-n Abakus vs native Boolean on small and large K, cold+warm | A single faster microkernel does not prove a driver speedup | +| P8 | Two-process 1-host vs two-host with latency, TLS/auth and congestion varied | If network cost dominates or bytes moved are unchanged, report NO BENEFIT | +| P9 | Reproduce on another ISA/source version with disable tests, raw logs + SHA | No cherry-picked fastest cell; uncertainty and hardware explicitly reported | + +## Existing evidence crosswalk (never over-credit) + +| Earlier gate | What can be credited | Still missing | +| --- | --- | --- | +| #1949 L9 / C4 | #1950 Lance A-full/B-allow/B-block/C-range semantic parity, ingress as copied, D4 explicit | No real DR-to-row-id link, no remote driver | +| #1949 L10 / C2 | #1950 1M/10M multi-tile composition | Bag multiplicity separate and cross-source identity not fully falsified | +| #1949 L1/L2 / C1 | #1950 prepared binding reused Q≥2 | Bundled gathered values; **not** memoized SoA locators | +| #1949 A3/A4 | #1950 packed layout C2 cost/correctness; #1951 lab-only U24x5 census | Native ordered-u8 kernel and Abakus k-of-n still not measured | +| #1951 P3-1 | range fragmentation heuristic with local 63-test suite | physical pages touched, 95% cluster crossover, transport selection | +| #1948 grammar | 16/16 lab + co-addressed candidate | production producer admission, cross-call lifetime, source scope | + +## Verdict format required for every future PR + +```text +D-DRISA-- +SOURCE: repository path:line + commit SHA +OPERATION: semantic program / witness / truth mode +FIXTURE: independent versioned data source, expected answer/refusal +ARM: B0/B1/B2/B3/B4/B5/B6 and disable variant +MEASURED: median, p10, p90, p95, allocations, CPU, wire, page/disc bytes +ORACLE: independently named, never the same shared function +STATUS: PASS | FAIL | NOT-RUN | GATED | NOT-COMPARABLE +ADMISSION: owner and EXO issues, explicit operator decision if needed +NO-GO: source violation / unauthorized path / stale witness / semantics drift +``` + +**Refusal boundary:** If an experiment cannot use an admitted owner, answer `BLOCKED_EXO` or `BLOCKED_D4` with a precise source citation. Do **not** fake a source identity, wire pointer, hash-table stand-in or ask a microbenchmark to ratify the architecture. diff --git a/.claude/plans/d-drisa-remote-register-shader-v1.md b/.claude/plans/d-drisa-remote-register-shader-v1.md new file mode 100644 index 000000000..00343794f --- /dev/null +++ b/.claude/plans/d-drisa-remote-register-shader-v1.md @@ -0,0 +1,194 @@ +# D-DRISA-0: DR remote async register / Abakus ALU / cognitive shader driver + +> **2026-11-04. Status: PRE-REGISTERED ARCHITECTURE, DOCS ONLY.** No new runtime, RPC endpoint, public ABI, wire protocol, GPU shader, source map, F### word, ISA implementation, or speedup is authorized or claimed. **Operator thesis:** replace repetitive hash-table-like lookups and transported/materialized payloads with a sealed DR register-address basis, a typed ClassView/DT64 method lens, existing Quack/R2IL/mask-risc instructions and ndarray/Abakus ALU words. Submit work asynchronously to the source-owning executor and return only the necessary result. +> +> This adds a **new hypothesis and a remote-execution boundary** to merged D-DRAM-0 (#1949), D-GLOBAL-WHERE P2 (#1950), and D-DRAM-P3 (#1951). It does **not** alter their evidence or lift the open owner decisions. + +**Tracking:** [EXO-153](https://linear.app/exo-red/issue/EXO-153); [EXO-50](https://linear.app/exo-red/issue/EXO-50) is append-only handoff, not approval. [Staged-PR charter](d-drisa-staged-pr-charter-v1.md); [falsifiers](d-drisa-falsifier-matrix-v1.md); [board](../board/entries/2026-11-04-d-drisa-0-remote-register-contract.md). + +## 1. The sharp thesis and its boundaries + +A normal remote database interaction often fetches records or columns and executes client-side work. **D-DRISA sends a typed program and witnessed coordinate context to the data-local executor.** The executor binds canonical borrowed value lanes if needed and runs the existing masking algebra there. DR coordinates remain the *virtual register-address space*. A compact scalar, witnessed mask or candidate relation returns to the caller. Abakus is a candidate **bit-sliced arithmetic** realization of certain ALU operations, **not** the key/identity map itself. + +The research question is whether this yields lower **end-to-end** latency, payload movement and repeated preparation without loss of semantics, isolation or zero-copy local borrow laws. It is **not** already a proven distributed cognitive processor. + +**Not:** a new hash table; a copied facet cache; a new mask store; a GPU backend; arbitrary executable shaders; a second Quack/R2IL interpreter; a DR→Lance global row-ID table; a remote Rust pointer; a magic claim of zero-copy network I/O. + +```text +REMOTE CALLER DATA-OWNING EXECUTOR + intent + method + result demand | + | | verify source/frame/schema/scope + +-- versioned descriptor ---------->| bind existing DR frame/tiles + | | (MAP-only may finish here) + | | resolve ClassView demanded fields + | | optional admitted locator receipt + | | local canonical SoA BORROW/read + | | Quack/RelAlg or R2IL → mask-risc + | | ndarray::simd + optional Abakus + |<------- compact result -----------| COUNT/EXISTS/mask/candidate/refusal + | async completion | release owned scratch/source lease +``` + +**Shader** means a small, bounded read-only data-local program over one witnessed register window, not graphics shader code or unbounded distributed execution. + +## 2. Owners: the already-existing ISA is unusually close + +| Concept | Existing actual owner / status | Remote boundary required | +| --- | --- | --- | +| Identity / coordinate register | `lance-graph-dt64/src/dr.rs`: `MapTable`, `LocalMap`, `Frame`, `Guards`, `Axis::for_survivors`. SHIPPED. Identity-only; no values | Opaque, versioned register *name*, never an exported pointer or second identity authority | +| Typed class and behavior | `lance-graph-dt64/src/lib.rs`: immutable `MethodBinding`, `MethodId`, `SigRev`, ON×WHEN×DO, `ValueDemand`. SHIPPED store | Source/field permission and read-only remote admission; DT64 store alone is not authorization | +| Property scope | `lance-graph-contract/src/class_view.rs`: `ClassView` / `FieldMask` / `WideFieldMask`. SHIPPED | Resolve property IRI to physical source column under schema/source seal, no silent widening | +| Algebra/compiler | Quack/RelAlg; R2IL and OGAR body ownership. SHIPPED partial execution pathways | Executor validates allowlisted described program; do not mint new F### or own OGAR body | +| Virtual mask register | `mask-risc/src/ir.rs` `Operand::{Plane,Scratch}`; `Planes`, `LaneRef::{I32,U32,U64,Strided}`, `Foreign`. SHIPPED LOCAL | Safe remote *descriptor*, not transported `&[T]` / `StridedRef` memory pointer | +| Execution/ISA | `mask-risc/src/exec.rs`, caller-owned reusable scratch, `execute_into` delegating to ndarray. SHIPPED | An admission-aware dispatch/finish boundary, not a second op loop | +| Arithmetic ALU | ndarray `half_add_mask` / `full_add_mask` (#402) and native Boolean/comparison words. SHIPPED | Capability/predicate-signature check; no public `Abacus` or `U24x5` claimed (#1951 census) | +| Source I/O | #1633 and #1950: `SealedFacetLane` / `OrderedLaneWitness`, `RowAddrMask`, `Scanner::project`. LAB | D4 single DR→Lance `_rowid` authority and S-INGRESS measured copy remain BLOCKED | +| Cached *location* receipt | D-DRAM-0 `2; #1951: **no production owner or lease found** | EXO-46/89/151 gate; never mutate `LocalMap::globals` | +| Grammar cognitive consumer | #1948 co-addressed `head_frame`, two read-only methods. LAB | Producer is derived semantic lane, not a locator; EXO-46/140 admission and source revision | +| Remote scheduler / transport | **NO IMPLEMENTATION ESTABLISHED BY THIS REVIEW** | Typed descriptor, async state, result lifecycle, security and locality; independent lab first | + +The 5×24 anecdote is now settled more narrowly: #1951 located examples-only `U24x5` microtile packing; no shipped `ndarray::simd::5x24::*` family. Never import lab example code as an alleged production ISA. + +## 3. Four meanings of zero-copy + +**(A) LOCAL-BORROW: established default.** Resident source-owning executor borrows canonical `&[T]` or `LaneRef::Strided` with a live source lease. mask-risc owns no payload bytes. Reusable scratch is *mask scratch*, not copied SoA. + +**(B) NO-INTERMEDIATE-PAYLOAD: the target.** Between remote caller and data-local executor pass only an admitted instruction descriptor and a compact result, not row/object payloads. This can yield a strong reduction in transferred bytes, **even if network request/result serialization exists**. + +**(C) PHYSICAL/NETWORK ZERO-COPY: unproven.** No assumed RDMA, pinned pages, NIC ownership, device access, mmap sharing, zero-copy compression, or guaranteed zero-copy transport. Arrow C Data Interface describes **same-process** sharing, and explicitly excludes distinct processes; Arrow Flight uses RPC/IPC. No transport chosen in this plan. + +**(D) STORAGE/INGRESS COPIES: explicitly accounted.** Lance page reads, Arrow batch creation, `RowAddrTreeMap` and `S_INGRESS_COPY` are separate. A source may not be resident; selective reads are allowed but costed. D4 remains `BLOCKED_D4` until one owner signs the attested mapping. + +**Non-negotiable:** A faster copied cache does not authorize changing BORROW into a generic GATHER physical plan. D-DRF-1 `5's exceptional foreign-cache admission remains exceptional. + +## 4. Logical register addressing and lifetime + +Register naming is a conceptual descriptor, **not an ABI definition**: + +```text +RegisterWindowRef: + coordinate_source: map_identity + Frame.issue + tile-range + route_witness + data_source: dataset identity + branch/version + index/rowaddr seal + class_basis: ClassId + schema revision + field-basis digest + method: MethodId + SigRev + guard digest + result kind + scope: authenticated principal + permission/scope + expiry + tenant + semantics: Boolean/Bag + TwoValued/Kleene + numeric/sentinel rules + program: validated existing-IR digest + bounded operand description +``` + +- Global ID is not a local DR ordinal; a DR ordinal is not a Lance `_rowid`; field position is not a physical column ordinal. Mixed dimensions receive typed refusals. +- A source-location memo is a **disposable, sparse receipt under an existing binding**. It can be forgotten with no loss of truth. It never becomes the new relation owner or is appended to the identity list. +- A binding may retain coordinates across value-only changes while **predicate planes derived from changed values** become stale. Separate versions: coordinate frame, source snapshot, field schema, permission scope, derived truth, program vocabulary. +- Reads from an old frame are lawful only with the corresponding historical source version and scope witness; otherwise refuse. +- Retired/overwritten source, compaction, stable-row-id change, index-rebuild, permission revocation, focus-slot reuse and tenant switch must invalidate or refuse stale physical descriptors. +- Live local references must never outlive their Arrow/source owner, pin/guard or release callback. Raw addresses never cross the RPC boundary. +- Result masks are tagged with their coordinate universe; a downstream method cannot reuse them on another frame solely because bit lengths agree. +- ClassView may select fewer fields. Dynamic **scope** does not license reinterpretation of presence/sign bits, NULL/unknown or codebook semantics. + +## 5. Execution request / result sketch (not public wire structs) + +```text +ExecuteDescriptorV0: + protocol_schema_version, request_id, operation_digest + target_dataset/branch/version/index_seal + frame/map/tile_range + source_class/field_schema revisions + method_id/sig_rev/guard_digest + fields_needed + program_ref (existing Quack/R2IL/mask-risc semantics; locally validated) + authentication/authorization claim (server-verifiable, never caller-declared) + Boolean|Bag, TwoValued|Kleene, null/absence, signed/overflow rules + result_kind (Scalar | MaskReceipt | RelationReceipt | Refusal) + read_only = true + deadline, quotas (max_tiles, max_slots, max_ops, max_memory, max_result_bytes) + optional locality/prefetch hint (ADVISORY, no authority) + +CompleteDescriptorV0: + request_id + operation_digest + effective source/frame/scope witnesses + status (Completed | Refused | Cancelled | Indeterminate) + scalar or bounded result + release token if returned handle + measured stage costs, bytes, allocations, source-read mode +``` + +The wire encoding, capability signatures, authentication, principal binding, revocation policy, version handshake, replay protection and opaque-handle issuance are **not designed or authorized** here. Explicitly require an owner decision and a threat-model review before an open TCP endpoint. The source-owning executor **validates the program and authorization**; a program digest and a caller-supplied scope string are not sufficient. + +### 5.1 Async state and resource rules + +```text +SUBMITTED → ADMISSION → BOUND → QUEUED → RUNNING → COMPLETED + | | | | + REFUSED STALE CANCEL EXPIRED / INDETERMINATE + \ + RELEASED after safe completion +``` + +- Source snapshot/frame/scope must remain valid while queued *and* running; either pin a lawful immutable source snapshot or re-check and refuse. +- Cancellation request is not rollback. An unknown network outcome remains **Indeterminate**, never logical FALSE or empty mask. +- Retry is allowed only for read-only deterministic operation against exactly the same effective source/frame/scope. No exactly-once guarantee. +- Bound queue depth, per-request operation count, scratch bytes, result bytes and in-flight leases. Denial of service and unbounded fan-out are refusals, not accepted workload. +- No effectful programs in stages 0–6, no automatic hydration, no undisclosed callback or script execution. +- Connection loss must not leak source lease or result receipt. TTL, explicit release and race tests required before promotion. +- Cross-owner joins are **not** automatically data-local; either require a separately admitted and costed data movement/join plan, or refuse. +- A true k-of-n must count satisfied predicates *per identity*, not multiply Bag derivations; SQL NULL/UNKNOWN cannot disappear. + +## 6. Abakus ALU / adaptive cognitive lens + +**Existing opcode words**: `ndarray::simd::half_add_mask`, `full_add_mask`; native `mask_and`, `mask_or`, `ternary_match_u32_to_mask`. Abakus is a bit-sliced **arithmetic** technique on identity-aligned significance planes. A carry moves to a higher significance *plane for the same identity*, not to a neighbor identity or a new row. + +**First positive consumer**: over one sealed population and predicates `P0..P4` compute `K3 = (P0 + P1 + P2 + P3 + P4 >= 3)` using existing half/full-add words, compare against an independent scalar-per-identity oracle and a native Boolean network. Verify all 64 tail/one-hot/alternating/random patterns and count/EXISTS. Benchmark n=1,2,4,5,16,24 with k=0,1,half,n,n+1. Report cold plane preparation and hot execution separately, with disable-carry tests. + +**Negative consumer:** two-mask AND must stay a native Boolean mask word; Abakus cannot justify adding operations. A packed unsigned-u8 comparison inside u32 is an **independent missing facade operation** identified by #1950, not proof that Abakus's carry adder should implement it. `U24x5` is lab packing, not the needed ordered-byte ISA. + +**Adaptive means:** lower only the admitted demanded fields from ClassView/WideFieldMask, select the existing word/facade compatible with the semantic contract, reuse frame-aligned masks while valid; do not reinterpret type/overflow/sentinel or bypass ndarray ownership. Numeric approximate U9/Palette512 and exact Boolean/Bag truth remain separate. + +**Cognitive shader** is a constrained, read-only method program over one leased DR population; it is not magical learned inference. #1948 grammar's `head_frame` is a **derived semantic lane**, not a memoized source locator. XLOG/NAL callers use the existing semantic polyfill boundary, no new XLOG runtime or recursion admitted by a transport adapter. THOUGHT hypotheses remain speculative; WORLD writes need the existing hydration/Rubicon authority. + +## 7. Exact baselines, not marketing extrapolation + +For each same-source, same-predicate, same-scope, same-result fixture: + +| Arm | Description | What it proves | +| --- | --- | --- | +| B0 | data-local borrowed planes, local call | local executor floor | +| B1 | remote fetch rows/columns, evaluate locally | actual transfer/materialization baseline | +| B2 | remote issue, data-local borrowed execution, scalar/mask result | remote no-intermediate-payload hypothesis | +| B3 | B2 + reused DR frame/selection binding | reuse without repeated routing | +| B4 | B3 + optional, validated **locator** receipt | address memoization, **not** copied value cache | +| B5 | B4 + prefetch lookahead (including wrong predictions) | prefetch economics | +| B6 | B2/B4 + justified Abakus k-of-n ALU | arithmetic physical-plan benefit | + +All arms account for: `C_total = C_request_serialization + C_transport + C_auth + C_admission + C_frame_bind + C_route + C_locator + C_source_io + C_mask_ingress + C_prefetch + C_fold + C_result_encoding + C_transport_return + C_release`. Count network bytes, disk/page-cache bytes, source allocations, result/receipt bytes, request count, queue delays, peak RSS, p50/p95/p99, CPU cycles. Compare Q=1,2,8,16,64,256, 64/256/multi-tile, scatter/cluster, 0.1%,1%,10%,50%,95%, k=1/2/4/5/16/24 and 1M/10M where feasible. Do not use a 1M-COUNT win to claim remote performance. + +**Critical #1950 correction:** `Binding::prepare` caches `gather_lane` copied `Vec>` as well as CSR and masks. Its Q≥2 timing is **bundled prepared-binding amortization**, not independent proof of locator memoization. The 50%-selectivity borrowed path being 2.3–2.5× slower remains a required falsifier. #1951's `range_count * 4096` is a **fragmentation proxy**, NOT the number of distinct pages touched, and the 4096 constant is a policy pin, not a calibrated general planner law. + +Keep <1000µs warm-in-fold gate **where already defined**; report network and IO cost separately rather than incorrectly claiming remote requests all meet it. Only measured whole-query benefit plus exact parity can justify an optimization decision. + +## 8. Gate map, staged PR sequence + +The precise PR charter and independent falsifier inputs are in the companion plans. This is the intended **order**, not automatic authorization: + +```text +D-DRISA-0 DOCS/BOARD/LINEAR (this PR) + ├── 1: in-process borrowed-register equivalence (LAB) + ├── 2: async loopback lifecycle / refusal (LAB, security-read first) + ├── 3: Abakus threshold consumer (LAB, ndarray owner) + ├── 4: two-process data-local transport comparison (LAB, transport/security gate) + ├── 5: admitted source-location receipt + prefetch (EXO-46/89/151 BLOCKED) + ├── 6: DR→Lance single authority / mask handoff (D4/S-INGRESS BLOCKED) + ├── 7: grammar/XLOG/NAL cognitive read-only consumers (EXO-140/73/78 GATED) + └── 8: fault/performance evidence register and promotion decision (NO AUTO MERGE) +``` + +Stages 1 and 3 are parallelizable **only as lab code** with pinned oracles; stage 2 does not grant permission for a network listener. The existing #1951 P3 plan continues unchanged; D-DRISA stages are a separate research direction. A failed stage reports `REFUSED / DEFERRED` and does not require adding a new engine. + +## 9. Scientific and security falsification + +See the [independent test matrix](d-drisa-falsifier-matrix-v1.md) for exact R1–R16 (register/lease), A1–A8 (ALU), N1–N14 (remote/fault), C1–C10 (cognition/semantics), P1–P9 (cost) fixtures, oracles, deliberate disable conditions, measurements and stopping points. Each result is graded VERIFIED-CODE, MEASURED-LAB, PROPOSED, GATED or UNVERIFIED. + +**Unconditional STOP**: new global→row-ID authority; wire-transmitted raw pointer; copy of canonical values silently cached as normal path; cross-scope/tenant leak; stale frame silently accepted; default value materialization; claiming U24x5 shipped; new F###/opcode ownership; setting UNKNOWN=FALSE except under an admitted SQL WHERE; THOUGHT→WORLD promotion; retry changing source version; auto merge; rewriting transport into Quack. + +## 10. Source and published-reference inventory + +- [DT64 DR map / frame](../../crates/lance-graph-dt64/src/dr.rs), [DT64 bindings](../../crates/lance-graph-dt64/src/lib.rs), [mask-risc IR](../../crates/lance-graph-mask-risc/src/ir.rs), [mask-risc execution](../../crates/lance-graph-mask-risc/src/exec.rs), [R2IL dialect](../../crates/lance-graph-r2il/src/lib.rs), [ClassView](../../crates/lance-graph-contract/src/class_view.rs), [ordered-lane witness](../../crates/lance-graph-contract/src/ordered_lane.rs). +- [#1948 grammar](https://github.com/AdaWorldAPI/lance-graph/pull/1948), [#1949 D-DRAM plan](https://github.com/AdaWorldAPI/lance-graph/pull/1949), [#1950 P2 measured labs](https://github.com/AdaWorldAPI/lance-graph/pull/1950), [#1951 P3 census](https://github.com/AdaWorldAPI/lance-graph/pull/1951), [ndarray #402](https://github.com/AdaWorldAPI/ndarray/pull/402). +- [D-DRAM-0](d-dram-abacus-address-memoization-v1.md), [D-DRAM-P3](d-dram-p3-integrated-v1.md), [D-DRF-1](d-dr64-focus-lease-foreign-cache-v1.md), [D-RMAP-1A](d-rmap-separate-tables-thought-palettes-v1.md), [D-DRAV-1](d-dr-address-first-value-demand-v1.md). +- [EXO-153](https://linear.app/exo-red/issue/EXO-153); [EXO-50 bus](https://linear.app/exo-red/issue/EXO-50); [EXO-46](https://linear.app/exo-red/issue/EXO-46); [EXO-89](https://linear.app/exo-red/issue/EXO-89); [EXO-151](https://linear.app/exo-red/issue/EXO-151); [EXO-121](https://linear.app/exo-red/issue/EXO-121); [EXO-140](https://linear.app/exo-red/issue/EXO-140); [EXO-114](https://linear.app/exo-red/issue/EXO-114); [EXO-73](https://linear.app/exo-red/issue/EXO-73); [EXO-78](https://linear.app/exo-red/issue/EXO-78). +- [Apache Arrow C Data Interface](https://arrow.apache.org/docs/format/CDataInterface.html): same-process zero-copy, release callbacks, **not** interprocess sharing. [Arrow Flight](https://arrow.apache.org/docs/format/Flight.html): IPC/gRPC, streaming and custom actions as **possible transport** only. [Arrow C Device Data Interface](https://arrow.apache.org/docs/format/CDeviceDataInterface.html): **experimental**, no device guarantee. diff --git a/.claude/plans/d-drisa-staged-pr-charter-v1.md b/.claude/plans/d-drisa-staged-pr-charter-v1.md new file mode 100644 index 000000000..7bb4ef6d0 --- /dev/null +++ b/.claude/plans/d-drisa-staged-pr-charter-v1.md @@ -0,0 +1,151 @@ +# D-DRISA staged PR charter: proof before driver, driver before promotion + +> **2026-11-04 · PRE-REGISTRATION.** This charter describes **future, separately reviewed** PRs, not PRs already opened. Stage 0 is the *only* PR authorized by this request. A passing lab is not permission to extend production APIs. The canonical tracking issue is [EXO-153](https://linear.app/exo-red/issue/EXO-153); [EXO-50](https://linear.app/exo-red/issue/EXO-50) remains append-only handoff. +> +> Invariants from #1949 and #1951 stay binding: existing DR coordinate authority, default canonical SoA borrowing, no generic gather, no invented public `U24x5`/`Abacus`, no new F### or mask backend, no hidden D4/S-INGRESS bridge, no production remote listener before authorization. + +## 0. Dependency map and route + +```mermaid +flowchart TD + P0["PR0: design, provenance, owner rulings"] --> P1["PR1: in-process register contract oracle"] + P0 --> P3["PR3: existing Abakus words / threshold lab"] + P1 --> P2["PR2: async loopback / bounded lifecycle"] + P2 --> P4["PR4: two-process read-only data-local execution"] + P1 --> P5["PR5: optional locator receipt"] + P4 --> P6["PR6: real Lance source reading"] + P5 --> P6 + P4 --> P7["PR7: cognitive shader consumers"] + P6 --> P7 + P3 --> P7 + P7 --> P8["PR8: stress / independent audit / production decision"] + G1{"EXO-46/89/151"} -.-> P5 + G2{"D4 + S-INGRESS + EXO-114"} -.-> P6 + G3{"EXO-140/73/78 + semantic owners"} -.-> P7 +``` + +Dashed arrows represent **unresolved gates**, never inferred approval. PR3 may proceed without PR2 in a lab; PR5 does **not** require PR4 for local proofs, but is owner-blocked. If the Mer­maid rendering is not supported, the stage table below is the canonical dependency record. + +## PR0: D-DRISA-0 docs-only map (this pull request) + +**Purpose:** Freeze the exact hypothesis and avoid reproducing a hash table or adding a new execution engine by accident. + +**New docs:** [architectural law](d-drisa-remote-register-shader-v1.md), this [stage charter](d-drisa-staged-pr-charter-v1.md), [independent falsifier matrix](d-drisa-falsifier-matrix-v1.md), [board entry](../board/entries/2026-11-04-d-drisa-0-remote-register-contract.md), `STATUS_BOARD.md` / `INTEGRATION_PLANS.md` / generated entries index, and append-only EXO-50 source-linked handoff. + +**Acceptance:** all source status claims labeled, no production code, distinct local borrow vs remote message copy, exact decisions named, falsifier matrix with independent oracles, no unbounded promise or invented codec/transport. CI board/index checks green. **No implementation authorization.** + +## PR1: D-DRISA-1, *local* ISA/register seam differential probe + +**Status:** FUTURE LAB PROPOSAL. **Owner review:** DT64 / mask-risc / R2IL; EXO-46 admission still needed for any indexed cross-population bind. + +**Purpose:** Demonstrate the same semantics in existing local execution with (a) direct borrowed planes, (b) a typed opaque local binding descriptor resolving to the same borrowed planes. No transport. Don't change `dr.rs` identity or row storage. + +**Suggested artifact:** workspace-excluded `crates/drisa-register-probe/`, fixture README, `oracles.rs`, test disable hooks, bench, RESULT with counters. Prefer wiring existing approved lab crate/fixture if it avoids an unnecessary crate. No new public mask-risc opcode or new registration mechanism. + +**Fixtures:** one DR64 tile, one DR256 tile, multiple tiles; distinct global IDs mapped to same local positions across frames; 0/1/all-set, sparse/clustered populations; projected 3 of 64 logical fields; P0 Boolean COUNT, EXISTS, Kleene WHERE; external foreign population must refuse unclassified navigation. Independent scalar oracle uses immutable global IDs, not probe's register map. + +**Positive:** repeated calls use the same frame and result, no payload copy in local borrow arm, warm fold uses existing ndarray facade, max scratch fixed. + +**Negative:** stale frame, unknown field, wrong scope, mismatched field basis, stale scratch epoch, deliberately swapped coordinate, expired source lease, unauthorized foreign lane. Every refusal has a success control; mutating the source map makes at least one test red. + +**Metrics:** hot allocations, `n_records`, borrowed bytes, ingress, route calls per Q, p50/p95, MB peak, same source version. Report ≥1ms warm fold as FAILED-GATE in this existing policy, no hidden full scan. + +**Exit:** source-backed evidence of semantics and local lifetime only, not network performance. + +## PR2: D-DRISA-2, bounded async loopback *control plane* + +**Status:** FUTURE LAB. **Predecessor:** PR1. **Owner:** runtime/transport + security review. No TCP listener by default. + +**Purpose:** Test descriptor → admission → queue → run → completion against the *same process* worker and borrowed canonical source. No raw pointer serialized; no untrusted bytecode; no new persistent result store. + +**Cases:** N1–N9: concurrent bounded requests, queue overflow typed refusal, duplicate request ID, cancellation before admission/queued/running, timeout, lease revocation, fairness with two tenants, stale frame while waiting, source change between admit and run. Completion includes operation and effective witness digests. + +**Oracle:** direct synchronous executor on same snapshot and rule; negative-disable controls bypass admission, drop cancellation, allow outdated permit, mix request IDs; each should turn a check red. Never equate CancelAck with rollback. + +**Exit:** deterministic read-only lifecycle and bounded resource accounting. **Not** an externally exposed endpoint. + +## PR3: D-DRISA-3, Abakus ALU / dynamic-lens lab + +**Status:** FUTURE LAB; can run in parallel with PR2 after PR0. **Owner:** ndarray + Quack + mask-risc. Does not change their public APIs by assumption. + +**Purpose:** Test semantic arithmetic on a bounded mask-register window using shipped `half_add_mask`, `full_add_mask`, native AND/OR/ternlog, and explicit scalar oracle. No `U24x5` example import. + +**Fixtures:** n=1/2/4/5/16/24 independent predicate masks, k=0,1,half,n,n+1, 64-lane one-hot/alternating/sparse/dense/random and tail bits, missing vs zero, Bag multiplicity, SQL NULL/Kleene unknown. Distinguish k-of-n count from Bag support and from u8-in-u32 ordered compare. + +**Cost test:** cold bit-plane construction + hot threshold, repeated Q, first and later values, scalar/AVX2/AVX-512/NEON/wasm where available. A simple 2-mask AND *must* remain preferable to extra adder work unless independently measured otherwise. + +**Stop:** if existing public words insufficient, report NO-EXISTING-WORD and ndarray-owner proposal; do not handroll ISA in Quack. A direct ordered-u8 compare remains a separate kernel/codec admission from #1950. + +## PR4: D-DRISA-4, two-process read-only data-local execution + +**Status:** FUTURE LAB, REQUIRES transport/security-owner approval. **Predecessors:** PR1, PR2. Not coupled to D4: use a single data owner's already-resident SoA with existing frame. + +**Purpose:** Differentially evaluate B1 (client receives rows/columns and executes) versus B2 (client sends a typed instruction; source executor borrows and returns scalar/mask). Same process-local semantic IR on the server. No external code execution; no claim of RDMA/zero-copy packets. + +**Transport:** perform a bounded comparative study of a minimal framed in-house lab protocol versus **optional** Arrow Flight action/exchange only if justified by features and security constraints. This charter does not choose gRPC, Flight, Cap'n Proto, UCX or HTTP. No new persistent daemon or broad network exposure. Prefer local Unix-domain sockets initially. + +**Cases:** 1M/10M in-memory data-owner single-host, 2 hosts only as a *separate* measured lane; payload sizes 0 / small / large, scalar vs mask output, repetitive Q and both resident vs cold-source setups; p50/p95/p99, wire bytes (request/response), TLS/auth cost if enabled, CPU, allocations, queue, compression, network contention. + +**Oracle:** same query fully evaluated on an isolated copy of the input under same source revision. Prove the B2 implementation transfers no payload columns on the wire; inspect packet bytes or endpoint counters. An identical-answer check is necessary but does not prove zero intermediate copies. + +**Stop:** if transport overhead dominates under relevant workloads, or remote ref accidentally becomes addressable outside the source owner, mark "NO BENEFIT / REFUSED" and do not introduce a production listener. + +## PR5: D-DRISA-5, optional location receipt and prefetch + +**Status:** **BLOCKED** EXO-46 + EXO-89 + EXO-151. **Predecessor:** PR1; proceed as architectural decision memo until admission. + +**Purpose:** Memoize validated canonical SoA *location* from already-attested DR frame/position, not value. The #1951 census found **no production lease host**: `dt64::Focus` is only a tile revision; `drf-focus-probe::FocusKey` is excluded lab code with a gathered-value cache, not a receipt. + +**Required owner decision:** designate exactly one lease owner without adding global→row-ID authority or editing `LocalMap::globals`; key by frame+source version+facet+scope+index seal; differentiate coordinate lease vs value snapshot. The receipt may be evicted without changing truth. + +**Fixtures:** L1–L8 from #1949, repeated random/clustered source locations, Q=1..256, source overwrite, compaction, tombstone, invalidation, focus-slot ABA, concurrent readers, cache evict, revoked scope, 0/4/8/16/32/64 lookahead. Compare: borrowed direct, borrowed with receipt, borrowed receipt+prefetch, and separately *admitted* exceptional foreign cache. No payload copy in main arms. + +**Stop:** if binding cannot be expressed under a lawful owner, block indefinitely rather than minting a map. A faster copied facet is not permission for an implicit gather planner. + +## PR6: D-DRISA-6, DR→Lance source-backed register execution + +**Status:** **BLOCKED D4 + S-INGRESS; EXO-114 ordering witness and #1633 ownership**. **Predecessors:** PR4 plus owner-approved source binding; optional PR5 receipt not required for first correctness arm. + +**Purpose:** Bridge one sealed DR ordinal and one Lance `_rowid` under exactly ONE source/version witness, then execute the same logical program with selective columns; no additional row mapping table. + +**Reuse:** #1950 `where_lance_arm`, `AddressedRelation`, `SealedFacetLane`, `ranges`/`range_mask`/`range_tree`, `Scanner::project` and `with_row_addr_prefilter`; mark `S_INGRESS_COPY`. Disable stable `_rowid` identity equality shortcut in a fixture and insist on refusal or lawful witness. Never treat `_rowaddr` as `_rowid` after remap. + +**Tests:** 0.1/1/10/50/95% clustered/scattered, 1 to 16 demanded columns, 1 vs 10M; page-cache and disk separately; physical pages touched rather than solely `range_count*4096`; exact source versions, fragment splits, abort/cancel, invalidation. Full-projection oracle and independent row-ID oracle. ReadPolicy heuristic recorded, not promoted to a calibrated optimizer. + +**Stop:** D4 missing, unguarded source patch, fake zero-copy or hidden source bytes → `BLOCKED_D4`, not a copied map. + +## PR7: D-DRISA-7, read-only cognitive/relational shader consumers + +**Status:** **GATED** EXO-46, EXO-140, EXO-73, EXO-78 and DT64/Quack/XLOG/NAL owners. **Predecessors:** PR1/3/4, and PR6 only for a storage-backed case. + +**Consumer 1:** #1948 German grammar dependent/head co-addressed candidate, two read-only methods. Producer has 35 cold head lookups; DO NOT call its `Vec head_frame` a source-locator receipt. Validate language/lexicon revision and frame/scope. Preserve ambiguity; no WORLD promotion. + +**Consumer 2:** XLOG/MeTTa/NAL syntax front end lowers through existing semantic Quack/RelAlg/R2IL only when already supported. A cross-frame join, recursion and unclassified `EqU32Via` are explicit stops, not an opportunity to reimplement execution. + +**In-program vs cross-call continuation:** Within one mask-risc `Program`, scratch slots are legal under their epoch. A returned `MaskReceipt` that survives *across* operations would require an explicit owner-approved, bounded result-sink lifetime and source/scope witness. Never reuse a prior `Scratch` slot across epochs (`StaleSlot`); never create a new unbounded global mask store. Until this decision, allow scalar result or materialized *egress mask* only, with counted bytes. + +**Refusal:** unknown method, unauthorized ClassView field, divergent frame, unknown truth, ambiguous candidate, missing source or Rubicon gate. No writing, hydration or separate grammar/NAL interpreter. + +## PR8: D-DRISA-8, adversarial and end-to-end evidence / promotion ruling + +**Status:** FUTURE, NOT AUTHORIZED. **Predecessors:** sufficient stage evidence, **all** owner decisions, explicit security review. Not automatically the merge of PR1–7. + +**Evidence packet:** independently reviewed oracle/falsifier matrix, negative-control disable logs, B0..B6 cost table with cold/warm separation, 2-host/1-host comparisons, source invalidation under concurrency, ACL bypass attempts, replay/dedup, cancellation/partitions, predictable resource ceilings, code coverage + CI, cross-ISA parity, actual storage/page bytes and transport bytes. 10M row performance is not inferred from in-memory microbench. + +**Owner decision:** one of `REUSE EXISTING / WIRE LAB / ADMIT LIMITED PRODUCTION / REVISE / REFUSE`. The production decision may authorize only a small read-only same-process path; full remote support and writeback are separate milestones. + +## Shared PR gates and publication policy + +1. **Every PR starts from current main** or explicitly states dependencies; no accidental stacking onto CC's active #1951 branch. Track base SHA and source revision. +2. **One purpose per PR** and an append-only cross-reference to EXO-153/EXO-50. Commit reports, raw evidence and independent oracles to an explicit research branch when large, with SHA and retrieval links; docs on main must not falsely imply raw logs are there (lesson #1950). +3. **One independent oracle and at least one deliberate disable** per semantic feature. A negative expected count of zero cannot certify a compare. Source, null and invalidation traps are required. +4. **No special low-level crate until needed**; reuse existing facade and algebra vocabulary. A proposed type name is not an implemented contract. +5. **Label evidence honestly**: source-verified, lab measured, proposed, owner-gated, refused. Answer parity ≠ performance parity ≠ engineering parity. +6. **No production merge without named owner approval**, and no EXO-50 message or passing CI substitutes for the approval. +7. **Board index & supersession index are generated** via their repository scripts; never overwrite curated rows, never claim CI green without a run on the head SHA. + +## Links + +- [D-DRISA master](d-drisa-remote-register-shader-v1.md) and [test matrix](d-drisa-falsifier-matrix-v1.md). +- [#1949](https://github.com/AdaWorldAPI/lance-graph/pull/1949) / [#1950](https://github.com/AdaWorldAPI/lance-graph/pull/1950) / [#1951](https://github.com/AdaWorldAPI/lance-graph/pull/1951) / [#1948](https://github.com/AdaWorldAPI/lance-graph/pull/1948). +- [EXO-153](https://linear.app/exo-red/issue/EXO-153), [EXO-50](https://linear.app/exo-red/issue/EXO-50), [EXO-46](https://linear.app/exo-red/issue/EXO-46), [EXO-89](https://linear.app/exo-red/issue/EXO-89), [EXO-151](https://linear.app/exo-red/issue/EXO-151), [EXO-140](https://linear.app/exo-red/issue/EXO-140), [EXO-114](https://linear.app/exo-red/issue/EXO-114), [EXO-73](https://linear.app/exo-red/issue/EXO-73), [EXO-78](https://linear.app/exo-red/issue/EXO-78).