Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions .claude/agents/zero-copy-warden.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,18 @@ excluded in both directions (see the 16 MB scale anchor): never let
"it's expensive to recompute" argue FOR a store, and never let "it's
cheap" argue against a lane.

**SCOPED-EXCEPTION — the DR64³ foreign-key cache** *(DECISION 2026-10-24)*.
The law's § "The DR64³ foreign-key cache exception" admits one
cast-reproducible cache on a cost basis, in one scope: a DR64 focus whose
foreign-key reads span several ontologies, reused across many thoughts,
feeding relationship learning and live-generated ONNX (D-DRF-1 §5). Before
returning MATERIALIZES for a gathered foreign facet, check that scope and
every condition (disposable, recomputable from sealed references, full
witness key, no uncommitted truth, build cost reported). All met → not a
finding; say which condition you checked. Any missing → the ordinary
verdict. This is the only place a cost argument is admitted; do not extend
it by analogy.

**LENS-CLEAN** — every read is a cast at the point of use:
`from_register_ref(&rows[pos].value[a..b])`, offsets derived from the
tenant descriptor, filtering done by predicate rather than by gather.
Expand Down
4 changes: 4 additions & 0 deletions .claude/board/INTEGRATION_PLANS.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
## 2026-10-24 — d-dr64-focus-lease-foreign-cache-v1 + d-dr64-focus-cache-falsifiers-v1 — immutable DR64 focus, leased foreign cache, sparse writeback → `.claude/plans/d-dr64-focus-lease-foreign-cache-v1.md`, `.claude/plans/d-dr64-focus-cache-falsifiers-v1.md`

**Status:** D-DRF-1 REFERENCE DESIGN / PRE-REGISTRATION ONLY (#1756); DRF-P1..P4 PROPOSED, not authorized. The DR map stays an immutable versioned reference; focus load acquires an ephemeral residency lease; caches are disposable; dirty rows coalesce by canonical destination and go through the existing CycleWriter. DECISION 2026-10-24: the DR64³ foreign-key cache is a scoped exception to the zero-copy law (cost basis, several ontologies, learning into live-generated ONNX), lapsing if DRF-P2 does not measure a whole-run win. Entry `2026-10-24-d-drf-1-focus-lease-foreign-cache.md`.

## 2026-10-23 — d-nfr-6-soa-borrow-v2 — native operations over the SoaEnvelope borrow, no guard → `.claude/plans/d-nfr-6-soa-borrow-v2.md`

**Status:** MEASURED, lab only (`crates/soa-view-probe`). Supersedes `d-nfr-6-validate-once-v1`: views are `NodeRowPacket` borrows, operations are one ndarray kernel each, mutation is the owner's `&mut`; no `Table`, no frame guard. Packet equals direct ndarray within noise at 0 allocations. D-NFR-6a withdrawn.
Expand Down
12 changes: 12 additions & 0 deletions .claude/board/STATUS_BOARD.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,18 @@ Plan: `.claude/plans/d-onnx-native-v1.md`. Crate: `crates/lance-graph-onnx` (ALI
| **D-ONNX-NATIVE-0** | crosswalk + minimal facade: elementwise float arithmetic, int32 comparisons, Boolean ops, Where, ReduceSum, ReduceSum(Where(Greater)) | IMPLEMENTED, draft PR (`entries/2026-10-23-d-onnx-native-0-facade.md`) | 11 conformance tests + doctest; 4 disable runs red on their own test; alias = native symbol; asm diff |
| **D-ONNX-NATIVE-1** | the NATIVE rows: approximate transcendentals, float ReduceSum tolerance, empty-population min/max | PROPOSAL | each row names its open decision (plan §2) |

## D-DRF — immutable DR64 focus, leased foreign cache, sparse writeback (2026-10-24)

Docs only (#1756). Plans `d-dr64-focus-lease-foreign-cache-v1.md` and `d-dr64-focus-cache-falsifiers-v1.md`. The foreign-key cache is a scoped exception to the zero-copy law (DECISION 2026-10-24, `zero-copy-lens-law.md`).

| D-id | scope | status | gate / falsifier |
|---|---|---|---|
| **D-DRF-1** | reference semantics: geometry, focus lease, lookahead prefetch, foreign-key cache exception, dirty-row writeback, learning overlay | In PR | falsifier matrix G1–G7, L1–L4, P1–P3, M1–M2, W1–W8, R1–R7 pre-registered, NOT RUN |
| **DRF-P1** | two-focus lazy load / lease / prefetch / last-reader eviction | Proposed | L1–L3, P1–P2; stale-map and race fixtures first |
| **DRF-P2** | 64-thought foreign-table A/B/C (direct+prefetch vs gather+reuse vs gather+reuse+writeback) | Proposed | whole-run cost incl. cold build; the foreign-key cache exception lapses if B/C does not win |
| **DRF-P3** | relationship cube / slices and typed dirty-row reduction oracle | Proposed | R1–R3, W1–W3; no Bag/NULL/evidence loss |
| **DRF-P4** | optional plasticity overlay + ONNX candidate stream | Proposed | R4–R7; base DR revision bit-identical |

## D-NFR / D-NFC — native representation contract + dependency correction (2026-10-23)

Docs only. Plans `d-native-repr-contract-v1.md` (D-NFR) and `d-native-first-dependency-correction-v1.md` (D-NFC). Nothing implemented; no ABI or execution contract changed.
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
# D-DRF-1 — immutable DR64 focus, leased foreign cache, sparse writeback

**Date:** 2026-10-24 · **Status:** REFERENCE DESIGN, pre-registered, NOT RUN · **Plans:** `.claude/plans/d-dr64-focus-lease-foreign-cache-v1.md`, `.claude/plans/d-dr64-focus-cache-falsifiers-v1.md` · **PR:** #1756

- **Design:** the DR map is an immutable, versioned reference; a changed map is a new sealed revision. Focus load acquires an ephemeral residency lease, released with the last reader. Caches are disposable and recomputable from sealed references. Dirty local positions translate through the original map, coalesce by canonical destination under a declared reduction, and go as owned rows through `collect_casts → CycleWriterHandle::try_submit → DetachedCycleBatch::freeze → WalSink::commit_cycle`.
- **DECISION (2026-10-24):** the DR64³ foreign-key cache is a scoped exception to the zero-copy law. SCOPE: foreign-key reads spanning several ontologies (about 8), reused across many thoughts, feeding relationship learning and live-generated ONNX. BASIS: only dirty rows are written back; most accesses (expected 70–90%) are reads served from the versioned cache; many learning steps need only the Boolean relationship maps, not values. All three fractions are hypotheses for DRF-P2. REVISIT WHEN: DRF-P2; the exception lapses where the gather-and-reuse arm does not win on whole-run cost including cold build. Recorded in `zero-copy-lens-law.md` and `zero-copy-warden.md` (SCOPED-EXCEPTION).
- **Checked against source:** geometry arithmetic; the writer chain exists as named; `LanceCycleWriter` copies into Arrow builders (its own module doc); `DurableRef` / `ThoughtCommitId` are specified, not shipped.
- **Open:** whether the cache wins at all (DRF-P2); the residency lease's reader-lifetime mechanism (DRF-P1).
3 changes: 2 additions & 1 deletion .claude/board/entries/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,10 +25,11 @@ index row, (3) no duplicate entry id. Checks 1 and 2 are deliberately
opposite directions; the stranding this convention prevents shows up in
exactly one of them, never both.

462 entries, 2026-08-06 .. 2026-10-23.
463 entries, 2026-08-06 .. 2026-10-24.

| date | entry id | finding | file |
|---|---|---|---|
| 2026-10-24 | `D-DRF-1` | | [2026-10-24-d-drf-1-focus-lease-foreign-cache.md](2026-10-24-d-drf-1-focus-lease-foreign-cache.md) |
| 2026-10-23 | `ps2-compose-chain-preregistration` | | [2026-10-23-ps2-compose-chain-preregistration.md](2026-10-23-ps2-compose-chain-preregistration.md) |
| 2026-10-23 | `ledger-revision-1635-1726` | | [2026-10-23-ledger-revision-1635-1726.md](2026-10-23-ledger-revision-1635-1726.md) |
| 2026-10-23 | `EXP-GF2-ELIM-PREREG` | Gaussian elimination over GF(2) with a replayable row-operation witness — pre-registration | [2026-10-23-gf2-elimination-preregistration.md](2026-10-23-gf2-elimination-preregistration.md) |
Expand Down
12 changes: 12 additions & 0 deletions .claude/knowledge/zero-copy-lens-law.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@
| "Projection is not chasing" — indirection cost is zero here | **FINDING** (mechanism), *unbenchmarked* | Follows from `target = cur + off` being computed arithmetic over a contiguous strided slab. **No micro-benchmark has been run**; the claim is structural, not measured. |
| The rung test (elevation licenses a store; a projection never does) | **CONJECTURE** | Operator ruling + one shipped precedent (`Locus::Quorum` / `Contradiction`). **No probe has run.** Falsifier stated in § "The one apparent exception"; gated as `PROBE-RUNG-ELIGIBILITY`, NOT RUN. |
| Grey/white per-lane assignment | **CONJECTURE** | Framing only. The doc says so at the table: *"the governing frame, not a census."* Census = plan §6 Q0, NOT RUN. |
| The DR64³ foreign-key cache exception pays for itself | **HYPOTHESIS** | DECISION 2026-10-24, scoped (§ "The DR64³ foreign-key cache exception"). Rests on three unmeasured fractions: dirty rows, read hits (expected 70–90%), value-free learning steps; DRF-P2 is the gate and the exception lapses if it fails. |
| The 16 MB / 2×16 MB scale anchor | **FINDING** (arithmetic) | 32k rows × 512 B = 16 MB, checked. The grey/white *split* of that budget is operator-stated, not measured. |

**Nothing in this doc is promoted past its row above.** Where a section reads as
Expand Down Expand Up @@ -250,6 +251,17 @@ They qualify **not** because reconciling observations is expensive, but because
contradiction is a strictly higher epistemic object than the observations it
reconciles. That is the shape every future derivation tenant must match.

## The DR64³ foreign-key cache exception (2026-10-24)

**DECISION:** a second exception, separate from the rung test above and not derived from it. A gathered foreign facet cache is a cast-reproducible projection, so by the rung test it is a violation; this decision admits it anyway, in one scope only, on a cost basis that this law otherwise rejects. Admitting it here, by name, is what keeps it from becoming a general hatch.

- **SCOPE:** foreign-key operations over a DR64 focus whose referenced rows span several ontologies (the motivating case is about 8), reused across many thoughts, with relationship learning over the result feeding live-generated ONNX as the mathematical compiler substrate. Plan: `.claude/plans/d-dr64-focus-lease-foreign-cache-v1.md` §5 (D-DRF-1).
- **BASIS:** the cost is amortized in three places. (1) Writes: only dirty rows are written back, coalesced by canonical destination, so a focus touched by many thoughts commits a few full-row images, not one write per operation. (2) Reads: most accesses (expected 70–90%) are reads served from the versioned cache; without it each of them is a scattered 512-byte-stride read into a foreign table, repeated per ontology, per thought and per learning iteration, which leaves the nanosecond fold. (3) Learning: the relationship maps are Boolean support planes, and many learning steps need only those, not the row values, so they skip the value gather entirely. The three figures (dirty fraction, read-hit fraction, value-free learning fraction) are HYPOTHESES until DRF-P2 measures them. The 16 MB scale anchor's "cost is not an argument" holds for lens reads inside one table; it does not cover this cross-table, repeated, learning-driven case.
- **CONDITIONS:** disposable and never authoritative; clearing and recomputing from the sealed references gives the same answers; keyed by the full witness (map revision, index seal, source snapshot, focus, palette generation, facet layout, operation, scope); holds no uncommitted truth; build cost reported separately.
- **REVISIT WHEN:** DRF-P2 measures the gather-and-reuse arm against the direct arm on whole-run cost including cold build. If it does not win at the operator's workload, the exception lapses there and the cache is removed.

The warden applies this as **SCOPED-EXCEPTION**: inside the scope and meeting every condition, not a finding; outside it, or missing a condition, the ordinary verdicts apply.

## The canonical measured instance (2026-07-28)

`witness_fabric::{resolve_chain, standing_wave_grounded}` took
Expand Down
Loading
Loading