From 6ebae555abd24d4842e838a28124b52641a4e2e9 Mon Sep 17 00:00:00 2001 From: AdaWorldAPI Date: Sun, 25 Oct 2026 18:19:43 +0100 Subject: [PATCH 1/4] docs(D-NAM-0): reapply own capstone on current main --- .../d-nam-native-algebra-dto-capstone-v1.md | 81 +++++++++++++++++++ 1 file changed, 81 insertions(+) create mode 100644 .claude/plans/d-nam-native-algebra-dto-capstone-v1.md diff --git a/.claude/plans/d-nam-native-algebra-dto-capstone-v1.md b/.claude/plans/d-nam-native-algebra-dto-capstone-v1.md new file mode 100644 index 000000000..09fa98874 --- /dev/null +++ b/.claude/plans/d-nam-native-algebra-dto-capstone-v1.md @@ -0,0 +1,81 @@ +# D-NAM — Native Algebra Methods × external DTO dialects (capstone v1) + +**Date:** 2026-10-25. **Status:** PROPOSED / docs only; this PR is NOT an implementation, executable proof or operator approval. **Linear:** [EXO-152](https://linear.app/exo-red/issue/EXO-152) (plan), [EXO-50](https://linear.app/exo-red/issue/EXO-50) (append-only session handoff). + +## Thesis +A native mathematical method and its admitted composed polyfill are primary. ONNX, TFLite, MLIR and SQL/RelAlg expressions are **versioned data-transfer-object (DTO) readings** of that mathematics, not separate execution engines. Normalization is *partial and typed*: an external expression is recognized only when its operation, carrier, address domain, shape, reduction, semantics and effects agree with the admitted native recipe. Export reverses this crosswalk only where a compatible external representation is witnessed. + +```text +ONNX / TFLite / MLIR / SQL native Quack / RelAlg / ndarray callers + \ / + semantic DTO mapping + | + normalized native method identity + + admitted polyfill + semantic guards + | + Quack binding / R2IL / mask-risc + | + SoA values + DR mask/focus + ndarray +``` + +The center is a *method identity/crosswalk*, not a universal tensor, graph or AST. ONNX ModelProto / TFLite FlatBuffer / MLIR remain external artifacts; any importer or exporter belongs outside the native execution hot path. + +## Binding decisions inherited, not reopened +- **D-RA-2=A** [EXO-86](https://linear.app/exo-red/issue/EXO-86), [#1644](https://github.com/AdaWorldAPI/lance-graph/pull/1644): `lance-graph-contract` owns single-table set algebra and rewrites. `lance-graph-relalg` owns planning/lowering. Do not grow `Rel` into `Rel::MatMul/Tanh/Conv` or create a second full graph IR. +- **D-NFC** [#1753](https://github.com/AdaWorldAPI/lance-graph/pull/1753), `d-native-first-dependency-correction-v1.md`: ONNX is external leaf facade. Native R2IL/ndarray work never depends on importing ONNX. `lance-graph-onnx` [#1754](https://github.com/AdaWorldAPI/lance-graph/pull/1754), [#1758](https://github.com/AdaWorldAPI/lance-graph/pull/1758) merged aliases and bounded Tanh/Exp. +- **Existing narrow reality:** `lance-graph-relalg/src/ir.rs` is one-table relational selection `Rel {All,Empty,Plane,Atom,And,Or,Not}` + `Measure {Count,Sum,Min,Max,Keep}`; `relalg/lower.rs` uses R2IL FoldDialect. `quack/bind.rs` resolves names, IDs and codebook values before execution; it does not currently consume relalg's whole IR. [EXO-115](https://linear.app/exo-red/issue/EXO-115) governs any normalization bridge. +- **SoA / DR:** [#1755](https://github.com/AdaWorldAPI/lance-graph/pull/1755) validates borrowed SoA kernel access in a lab. D-NFR contract describes DR64² lab, DR64³ proposed, codebook identity and DR→Palette link missing. Never claim DR64³ or Palette substitution shipped merely because a plan names them. +- **ADMISSION B** [EXO-102](https://linear.app/exo-red/issue/EXO-102): bounded, version-witnessed same-table preparation seeks; cross-table value-derived positional gathers forbidden. Preserve SQL K3 NULL and Bag/Distinct separate from Boolean sets ([EXO-126](https://linear.app/exo-red/issue/EXO-126)). +- **Proof:** Egglog/DialEgg may certify an admitted, typed rewrite offline; [EXO-127](https://linear.app/exo-red/issue/EXO-127) does not authorize FP reassociation, runtime egraph dispatch or a new theorem checker. Palette composition requires carrier closure evidence ([EXO-132](https://linear.app/exo-red/issue/EXO-132)). +- **Scope fence:** D-DTM-1 [#1777](https://github.com/AdaWorldAPI/lance-graph/pull/1777) can later transport methods/receipts, but transport, distributed state, Kafka and Flight are not dependencies of this arc. + +## The four dialect readings, one native example +`Σ{x_i | x_i > t}` with rank-1 i32 input: +- ONNX candidate: `Greater(X,t) -> Where(cond,X,0) -> ReduceSum`. +- TFLite candidate: `GREATER -> SELECT_V2 -> SUM`. +- MLIR candidate: `linalg.generic` predicate and reduction with explicit indexing/iterator maps. +- RelAlg candidate: `Rel::i32(lane,Cmp::Gt,t), Measure::Sum(lane)` on same table. +- Direct native precedent: `ndarray::simd::gt_i32_to_mask` + `masked_sum_i32`; ONNX facade already has `ReduceSum::sum_where_greater_i32`. + +**Important:** These are NOT universally equivalent schemas. They form one admitted *restricted fixture* only when scalar threshold, broadcasting, output shape, reduction axes, value carrier, zero filling, integer range and null policy coincide. The TFLite runtime bool byte format must not be mistaken for native packed mask layout. The `Rel` query selects rows; `Where` returns values. The zero-filled composition is equivalent to masked sum only under the agreed numerical domain and overflow behavior. Refuse mismatches. + +## Method contract, not another IR +A cold, versioned crosswalk row consists of: +`{native_method, composition?, dialect, domain/op/version, type, logical_shape, index_map, mask/validity, reduction_axes, carrier/semiring, overflow+FP contract, state/effects, proof/evidence, output_shape, export_policy, refusal}`. +This is an **engineering schema sketch**, not a proposed `struct` merged into contract. Owner of actual storage is decided in D-NAM-1 after locating existing ClassId/method registry. Native operation function pointers are bound once, not switched on dialect per call. + +## PR trajectory: independent branches from main +| Stage | Deliverable (separate PR) | Gate / owner | +|---|---|---| +| **D-NAM-0** | This alignment and read-only agent handoff; board index | docs-only, no ABI | +| **D-NAM-1** | Versioned crosswalk schema, method normalization, concrete four-dialect examples, typed refusals | contract/Quack owners; no duplicative method registry | +| **D-NAM-2** | ONNX + TFLite thin *external leaf* DTO mappings | ONNX facade/ndarray owners; official schema pins and differential oracle | +| **D-NAM-3** | MLIR indexing semantics and SQL/RelAlg normalizer bridge | EXO-86/115/126; no floats in set rewrite | +| **D-NAM-4** | Binding to SoA/DR/Quack/R2IL native prepared recipes | EXO-102/127/151; zero hot-path dispatch, focus and identity guards | +| **D-NAM-5** | Conditional native→DTO export, independent conformance and full-cost closure | no claim of full model conversion; reversibility only for admitted subset | + +**Dependency DAG:** S0→S1; S1→{S2,S3}; S2+S3→S4; S4→S5. All draft branches start at main; **dependency is approval/test dependency**, not a stacked commit dependency. After merges rebase each surviving branch and verify ancestry per [EXO-136](https://linear.app/exo-red/issue/EXO-136). + +## Acceptance and stop conditions +1. No native execution crate imports ONNX/TFLite/MLIR, no new graph AST, runtime interpreter, foreign evaluation fallback, or new SIMD implementation. +2. Type/shape/axis/quantization/NaN/±0/overflow/empty/K3/Bag/effects/version are retained or typed-refused; unknown domain/version never guessed. +3. Prove an admitted alias vs a direct native call; polyfill vs *independent* scalar oracle; compare external runtime as separate oracle where available. +4. An admitted normalization cannot broaden support: ONNX `MatMul` ≠ Boolean relation compose; SQL NULL ≠ false; `Gather` under a tensor backend ≠ permitted cross-table positional gather. +5. Count cold parse/map, preparation, bind, conversions, allocations, native calls and full output; zero hot-loop registry lookup. Kill a specialization if conversion erases the benefit. +6. Gated PR stages remain **PROPOSED** until code, tests, receipts and owner decisions exist. Draft green docs are not an implementation. + +## Canonical source breadcrumbs +- `.claude/plans/d-onnx-relalg-cap-v1.md`, `d-onnx-relalg-cap-capstone-1750-v1.md`, `d-onnx-relalg-s1-s2-semantic-intake-v1.md`, `d-onnx-relalg-s3d-physical-shootout-v1.md`, `d-onnx-relalg-s4-s5-palette-export-v1.md` +- `.claude/plans/d-native-repr-contract-v1.md`, `d-native-first-dependency-correction-v1.md`, `d-relalg-unified-v1.md`, `d-onnx-native-v1.md` +- [EXO-152](https://linear.app/exo-red/issue/EXO-152), [EXO-50](https://linear.app/exo-red/issue/EXO-50) (handoff only). + +## Opened draft PR receipts (2026-10-25) + +- D-NAM-0: [#1778](https://github.com/AdaWorldAPI/lance-graph/pull/1778) — capstone and board; this stage only. +- D-NAM-1: [#1779](https://github.com/AdaWorldAPI/lance-graph/pull/1779) — static crosswalk spec, candidate JSON manifest and fail-closed schema validator; runtime mapping still NOT implemented. +- D-NAM-2: [#1780](https://github.com/AdaWorldAPI/lance-graph/pull/1780) — ONNX/TFLite engineering spec. +- D-NAM-3: [#1781](https://github.com/AdaWorldAPI/lance-graph/pull/1781) — MLIR/SQL/RelAlg engineering spec. +- D-NAM-4: [#1782](https://github.com/AdaWorldAPI/lance-graph/pull/1782) — prepared Quack/SoA/DR recipes engineering spec. +- D-NAM-5: [#1783](https://github.com/AdaWorldAPI/lance-graph/pull/1783) — reverse DTO and conformance closure spec. + +All six are **drafts**, independent branches from main. Only the D-NAM-1 static manifest/validator are code tooling; no native Rust runtime/ABI integration exists in this arc yet. Every proposed implementation gate needs a separate code review/approval. From eb47173530c558bb4a7a2605bc6fcdb813043154 Mon Sep 17 00:00:00 2001 From: AdaWorldAPI Date: Sun, 25 Oct 2026 18:19:45 +0100 Subject: [PATCH 2/4] docs(D-NAM-0): reapply own capstone on current main --- .../d-nam-native-dto-claude-code-handoff.md | 28 +++++++++++++++++++ 1 file changed, 28 insertions(+) create mode 100644 .claude/prompts/d-nam-native-dto-claude-code-handoff.md diff --git a/.claude/prompts/d-nam-native-dto-claude-code-handoff.md b/.claude/prompts/d-nam-native-dto-claude-code-handoff.md new file mode 100644 index 000000000..d5b00e7b3 --- /dev/null +++ b/.claude/prompts/d-nam-native-dto-claude-code-handoff.md @@ -0,0 +1,28 @@ +# Claude Code handoff: D-NAM native algebra DTO normalization + +**Date 2026-10-25. Status: READ-ONLY SCOUT / IMPLEMENTATION GATED.** Source of truth: `.claude/plans/d-nam-native-algebra-dto-capstone-v1.md`; [EXO-152](https://linear.app/exo-red/issue/EXO-152) plan, [EXO-50](https://linear.app/exo-red/issue/EXO-50) append-only handoff. This is **not** approval to merge or refactor. + +## Task +Map ONNX, TFLite, MLIR and SQL/RelAlg operators as **versioned DTO semantics over existing native methods and their polyfills**. Never create a new tensor graph, interpreter, general AST, runtime dispatcher, library of duplicate kernels or independent method registry. + +## Before writing code +1. Fetch main and list exact source refs + merged status of #1644, #1682, #1752, #1753, #1754, #1755, #1758 and the D-ONNX S0 arc #1744/#1750; read current `.claude/board/STATUS_BOARD.md`, `INTEGRATION_PLANS.md`, `LATEST_STATE.md` and relevant owner decisions. +2. Read `contract::set_algebra` and `set_rules`, `relalg::ir/normalize/lower`, `quack::bind`, R2IL procedures / mask-risc, `lance-graph-onnx/src/lib.rs`, ndarray kernels; check actual ClassView/method signature API before proposing new types. +3. Read EXO-86, EXO-102, EXO-115, EXO-126, EXO-127, EXO-132 and newest EXO-50 comments. Do not overwrite others' sessions or claim their code. +4. Pin official ONNX and TFLite builtin operator schema versions; quote exact MLIR dialect documentation/version and SQL oracle semantics. Gemini ONNX IR overview is an **unverified external research overview**, not authority for this project's representation. + +## First deliverable (no mutation) +Output a table for four narrow cases: i32 Greater scalar; i32 Select/Where; guarded masked sum; f32 Tanh. Each row includes external name+version, concrete native symbol/composition, all shape/axis/dtype/mask/NULL/overflow/precision constraints, a source anchor, negative test, independent reference and refusal. Distinguish import semantics, normalization, execution, proof and export as separate columns. + +## Future implementation PRs (only when approved) +- D-NAM-1: cold crosswalk and typed admission, reuse method identity already available, no per-call interpretation. +- D-NAM-2: ONNX/TFLite **leaves**, TFLite bool/quantization boundary; no native dependency on the facades, no arbitrary model conversion claim. +- D-NAM-3: MLIR indexing and SQL/RelAlg legal normalization: contract owns set rules, relalg planning; no `Rel::MatMul`. +- D-NAM-4: same native method routes through Quack/SoA/DR and R2IL/ndarray; reuse bindings, zero redundant guards, no cross-table gathers. +- D-NAM-5: conditional reverse DTO and oracle/cost closure; no automatic inference of exportability. + +## Kill tests +Wrong opset, empty/invalid axes, shape mismatch, signed zero/NaN, i32 overflow, SQL K3 and Bag, TFLite BOOL packing and zero-point, synthetic-only oracle, wrong model weights/codebook, stale focus/frame, 0*Inf numeric contraction and effects. Counterexample → refuse, do not repair via a new runtime. + +## Response format +Propose the *smallest* code diff per owner crate, actual test command, expected failure of a deliberate mutation, end-to-end cost accounting and PR gate; clearly mark EXISTS/LAB/PROPOSED/MISSING/REFUSED. Link output to EXO-152 via its Linear issue and send **one new envelope comment** to EXO-50 with explicit sender/session ID; GitHub remains authoritative. Never infer operator approval from an agent's proposal. From b3469e3413351f932f1e7e9880a3dca13ff1d1d5 Mon Sep 17 00:00:00 2001 From: AdaWorldAPI Date: Sun, 25 Oct 2026 18:19:49 +0100 Subject: [PATCH 3/4] docs(D-NAM-0): additive board entries over fresh main --- .claude/board/STATUS_BOARD.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index 215d15243..e338887c8 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -1,3 +1,19 @@ +## D-NAM — Native Algebra Methods × ONNX/TFLite/MLIR/SQL DTO crosswalk (2026-10-25) + +**Status:** D-NAM-0 **PROPOSAL / DOCUMENTATION ONLY**, draft capstone; D-NAM-1..5 engineering specs are **not implemented, approved, tested or merged**. [Capstone](../plans/d-nam-native-algebra-dto-capstone-v1.md); [Claude Code source-first handoff](../prompts/d-nam-native-dto-claude-code-handoff.md). [EXO-152](https://linear.app/exo-red/issue/EXO-152) is the plan record; [EXO-50](https://linear.app/exo-red/issue/EXO-50) append-only messages only. + +| Stage | owner / target | current state and dependency gate | +| --- | --- | --- | +| **D-NAM-0** | docs + board alignment | DOCS ONLY: preserve D-RA-2=A and D-NFC #1753; no runtime change | +| **D-NAM-1** | existing method identity / cold semantic DTO normalization | SPEC / PROPOSED: real source owner, versioned admission, refused negative tests | +| **D-NAM-2** | ONNX #1754/#1758 leaf; proposed TFLite leaf | SPEC / PROPOSED: pinned external schemas, BOOL and affine quantization contract, independent oracle | +| **D-NAM-3** | contract set algebra + RelAlg/Quack SQL, MLIR indexing | SPEC / PROPOSED: K3/Bag/NaN and FP law refusals, no new Rel AST | +| **D-NAM-4** | Quack/SoA/DR bind; R2IL / ndarray kernels | SPEC / PROPOSED: prepared recipe, no per-call DTO dispatch, identity/ADMISSION B and measured costs | +| **D-NAM-5** | conditional export and full conformance closure | SPEC / PROPOSED: inverse mapping witnessed, end-to-end oracle and cost, all PR commits landed | + +**Owners:** [EXO-86](https://linear.app/exo-red/issue/EXO-86), [EXO-102](https://linear.app/exo-red/issue/EXO-102), [EXO-115](https://linear.app/exo-red/issue/EXO-115), [EXO-126](https://linear.app/exo-red/issue/EXO-126), [EXO-127](https://linear.app/exo-red/issue/EXO-127). No new universal tensor/graph AST; no ONNX/TFLite native runtime. D-DTM-1 transport [#1777](https://github.com/AdaWorldAPI/lance-graph/pull/1777) is adjacent future consumer, not dependency. + + ## D-DRAV — address-first DR map and demand-driven value residency (2026-10-25) **Status: PROPOSAL, DOCS ONLY (#1775 draft).** Plan: [d-dr-address-first-value-demand-v1.md](../plans/d-dr-address-first-value-demand-v1.md). Ledger: [2026-10-25-d-drav-1-address-first-value-demand.md](entries/2026-10-25-d-drav-1-address-first-value-demand.md). Specializes D-DRF-1/D-RMAP-1A without a new owner, executor, ABI, storage backend or admission rule. MAP/BORROW/GATHER/COMPILED/STORAGE are candidate physical paths; A1/A3 remain **NOT RUN**, the cache-exception lapse condition undecided. #1774 is a separate locality probe, not proof of cheap NVMe random access. From c1c81ff04bf01ba1f215a0bbf01057ab4f1e7da4 Mon Sep 17 00:00:00 2001 From: AdaWorldAPI Date: Sun, 25 Oct 2026 18:19:53 +0100 Subject: [PATCH 4/4] docs(D-NAM-0): additive board entries over fresh main --- .claude/board/INTEGRATION_PLANS.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/.claude/board/INTEGRATION_PLANS.md b/.claude/board/INTEGRATION_PLANS.md index 60cf53ee0..7340eeae2 100644 --- a/.claude/board/INTEGRATION_PLANS.md +++ b/.claude/board/INTEGRATION_PLANS.md @@ -1,3 +1,10 @@ +## 2026-10-25 — D-NAM native algebra method × DTO dialect capstone → `.claude/plans/d-nam-native-algebra-dto-capstone-v1.md` + +**Status:** D-NAM-0 PROPOSAL / DOCS ONLY; staged D-NAM-1..5 implementation **engineering specifications**, not yet approved/run. Source-first [handoff](../prompts/d-nam-native-dto-claude-code-handoff.md), [EXO-152](https://linear.app/exo-red/issue/EXO-152), append-only [EXO-50](https://linear.app/exo-red/issue/EXO-50). + +**Trajectory:** D-NAM-1 versioned method crosswalk and four concrete normalization fixtures → D-NAM-2 ONNX/TFLite leaf DTOs + quantization/BOOL format → D-NAM-3 MLIR indexing + SQL/RelAlg lawful normalization → D-NAM-4 Quack/SoA/DR/R2IL prepared binding → D-NAM-5 partial reverse DTO, oracle, cost and merge closeout. Separate draft PR plans: `d-nam-{1-native-method-crosswalk-spec,2-onnx-tflite-leaf-dto,3-mlir-sql-relalg-normalization,4-quack-soa-dr-native-recipes,5-reverse-dto-certification-gates}-v1.md` (on their own branches until merged). D-RA-2 owner #1644, native-first #1753, ONNX alias #1754/#1758 preserved; no new tensor AST, interpreter, runtime dispatch or production ABI. D-DTM-1 #1777 is a distinct transport arc. + + ## 2026-10-25 — D-DRAV-1 address-first DR: independently reusable map, demand-driven values → `.claude/plans/d-dr-address-first-value-demand-v1.md` **Status:** PROPOSAL / DOCUMENTATION ONLY ([#1775](https://github.com/AdaWorldAPI/lance-graph/pull/1775), draft), not a runtime, ABI, optimizer, storage-backend or ONNX/TFLite implementation. **Board entry:** [D-DRAV-1](entries/2026-10-25-d-drav-1-address-first-value-demand.md). **Falsifiers:** AV1–AV12 QUEUED / NOT RUN in [STATUS_BOARD.md](STATUS_BOARD.md); A1/A3 remain NOT RUN on actual workloads and D-DRF-1's scoped foreign-cache exception is undecided.