Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions .claude/board/INTEGRATION_PLANS.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,11 @@
## 2026-10-23 — d-nfr-6-soa-borrow-v2 — native operations over the SoaEnvelope borrow, no guard → `.claude/plans/d-nfr-6-soa-borrow-v2.md`

**Status:** MEASURED, lab only (`crates/soa-view-probe`). Supersedes `d-nfr-6-validate-once-v1`: views are `NodeRowPacket` borrows, operations are one ndarray kernel each, mutation is the owner's `&mut`; no `Table`, no frame guard. Packet equals direct ndarray within noise at 0 allocations. D-NFR-6a withdrawn.
**Confidence:** measured on one AVX-512 host; no independent review yet.

## 2026-10-23 — d-nfr-6-validate-once-v1 — validate a binding once, then execute without repeated guards → `.claude/plans/d-nfr-6-validate-once-v1.md`

**Status:** SUPERSEDED by d-nfr-6-soa-borrow-v2 (was MEASURED, lab only, `crates/dr-validated-view-probe`, removed). The per-call 27–28 ns guard is `FrameStore::check`'s refcount traffic; a view validated once by the existing `bind_dr64`, and kept valid by borrowing the table that owns data, frame and focus, runs native operations at direct cost. Externally owned revocation keeps a one-load check per operation. Proposals D-NFR-6a need the probe owner's review.
## 2026-10-23 — d-onnx-native-v1 — ONNX vocabulary over existing native functions → `.claude/plans/d-onnx-native-v1.md`

**Status:** D-ONNX-NATIVE-0 IMPLEMENTED as a draft PR (`crates/lance-graph-onnx`), not merged. The path is ONNX name → existing native function: ALIAS where the contract is identical, a small INLINE adapter where it is not, a typed `Refusal` otherwise. No graph, dispatcher or initialization. R2IL routing (D-MPF-4, #1752) is separate and is not imported.
Expand Down
13 changes: 11 additions & 2 deletions .claude/board/STATUS_BOARD.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,14 +13,23 @@ Docs only. Plans `d-native-repr-contract-v1.md` (D-NFR) and `d-native-first-depe

| D-id | scope | status | gate / falsifier |
|---|---|---|---|
| **D-NFC-0** | one canonical dependency diagram and seven root assumptions, applied as marked supersessions to #1746–#1751 and #1750 | In PR | each affected PR cites the diagram; S0 dispute left open |
| **D-NFR-0** | seam table Quack → DR64³ → Palette256³ → execution, cited to source | In PR | every seam SHIPPED / LAB / PROPOSED / MISSING with its source |
| **D-NFC-0** | one canonical dependency diagram and seven root assumptions, applied as marked supersessions to #1746–#1751 and #1750 | Shipped (#1753) | each affected PR cites the diagram; S0 dispute left open |
| **D-NFR-0** | seam table Quack → DR64³ → Palette256³ → execution, cited to source | Shipped (#1753) | every seam SHIPPED / LAB / PROPOSED / MISSING with its source |
| **D-NFR-1** | lab adapter: a Quack-bound query sealed under a frame | Proposed | after one revision moves, the stored binding is refused with the matching `Refusal`; before, it equals an oracle |
| **D-NFR-2** | codebook generation carried in the frame seal | Proposed | a palette value bound under generation g is refused under g+1 |
| **D-NFR-3** | one DR axis bound to one palette plane under a frame | Proposed | Cartesian and numerical oracles agree; a mismatched codebook is refused |
| **D-NFR-4** | end-to-end acceptance fixture (nine steps, each measured) | Proposed | all nine steps pass, each step's cost reported separately |
| **D-NFR-5** | one binding and one focus reused across several operations | Proposed | reuse saving measured against rebinding and a fused scalar / SIMD baseline, guards run on every use |

## D-NFR-6 — native operations over the SoaEnvelope borrow, no guard (2026-10-23)

Lab only: `crates/soa-view-probe`, plan `.claude/plans/d-nfr-6-soa-borrow-v2.md` (supersedes `d-nfr-6-validate-once-v1.md`). Parent D-NFR (#1753).

| D-id | scope | status | gate / falsifier |
|---|---|---|---|
| **D-NFR-6** | views are `NodeRowPacket` borrows (512-byte rows, 16-byte facets); operations are one ndarray kernel each; mutation through the owner's `&mut`; no `Table`, no frame guard | MEASURED (`entries/2026-10-23-d-nfr-6-soa-borrow.md`) | 3 compile_fail examples (E0502/E0597/E0599); 4 runtime falsifiers; 3 disable runs red; packet = direct within noise, 0 allocations, asm shows only kernel calls |
| ~~**D-NFR-6a**~~ | ~~`FrameStore::check` by pointer comparison~~ | WITHDRAWN (v2: a faster unneeded guard is not the fix) | — |

## D-MPF — mathematical polyfill API, Stage 1 (2026-10-23)

Lab only (`crates/math-polyfill-probe`). Plan `.claude/plans/d-mpf-math-polyfill-v1.md`; entry `entries/2026-10-23-d-mpf-1-polyfill-probe.md`.
Expand Down
9 changes: 9 additions & 0 deletions .claude/board/entries/2026-10-23-d-nfr-6-soa-borrow.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# D-NFR-6 v2 — native operations over the SoaEnvelope borrow

**Date:** 2026-10-23 · **Status:** MEASURED, lab only · **Plan:** `.claude/plans/d-nfr-6-soa-borrow-v2.md` · **Crate:** `crates/soa-view-probe` · supersedes `2026-10-23-d-nfr-6-validate-once.md`

- **Correction:** v1 put a `Table` with its own `FrameStore` beside the SoA and checked a frame per operation. That re-derived ownership `SoaEnvelope` already carries. Mutation safety is the borrow (`&self` views, the owner's `&mut`); frame provenance and authorization are separate questions and do not justify a per-operation check.
- **Design:** view = `NodeRowPacket` (512-byte rows, not `Clone`); address = `FacetSlot` (one of 32 × 16 bytes, in range by construction); operations = `FacetFold`, one ndarray kernel each (`eq_u32_strided_to_mask`, `ternary_match_strided16_to_mask`, `masked_strided_group_sum` over 12 × `u8`).
- **Pinned:** E0502 (no write during a live view), E0597 (no view outliving the owner), E0599 (no second holder); packet = direct ndarray = row loop at 8 sizes; 3 disable runs red.
- **Cost:** packet equals direct within noise at 64 / 4096 / 65536 rows, 0 allocations in both; asm: same kernel calls, B +4–7 instructions (`n_rows × 512` and register moves), no `lock`/`cpuid`.
- **Withdrawn:** D-NFR-6a (faster `FrameStore::check`).
11 changes: 11 additions & 0 deletions .claude/board/entries/2026-10-23-d-nfr-6-validate-once.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# D-NFR-6-v1 — validate once, execute without repeated guards (superseded)

> ⊘ Superseded by `2026-10-23-d-nfr-6-soa-borrow.md`: the `Table` + frame guard design is withdrawn.

**Date:** 2026-10-23 · **Status:** SUPERSEDED (was MEASURED, lab only) · **Plan:** `.claude/plans/d-nfr-6-validate-once-v1.md` · **Crate:** `crates/dr-validated-view-probe`

- **Q1, the guard:** the 27–28 ns per-call guard (#1752 route) is `FrameStore::check` (25.7–28.3 ns; 96 instructions; six `lock` RMWs) plus two `u64` compares (~1.3 ns). The cost is the temporary `Weak`s from `Arc::downgrade`; the pointer comparisons cost ~1 ns each. `bind_dr64` adds a `Frame` clone (two more `lock inc`): 59–67 ns total.
- **Q2/Q5, validate once:** a `Table` owns data, frame store and focus map; every change takes `&mut`. `validate` runs the existing `bind_dr64` once and returns a `View<'t>`. While a view lives, advance, refocus and revoke are compile errors (E0502), and a view cannot outlive its table or leak a `'static` borrow (E0597). After the borrow, stale guards are refused at re-acquisition.
- **Q6, the cost:** view operations equal direct `ndarray::simd` within noise, at 0 allocations. `View::sum` is a tail call with no `lock` op. One validation is ~65 ns per view. Per-operation alternatives: +25–40 ns (route guard), +50–75 ns (`bind_dr64`).
- **Q3, what stays at run time:** revocation owned by another component (one epoch load per operation, `SharedView`); data not owned with its frame (`bind_dr64` checks the frame, not the bytes); a focus whose revision is chosen by the caller (fixed by an owner-issued revision).
- **Proposed, not implemented:** `FrameStore::check` by `as_ptr` comparison (no refcount traffic); `BoundDr64` borrowing its frame. Estimated validation ~10 ns, not measured.
4 changes: 3 additions & 1 deletion .claude/board/entries/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ index row, (3) no duplicate entry id. Checks 1 and 2 are deliberately
opposite directions; the stranding this convention prevents shows up in
exactly one of them, never both.

460 entries, 2026-08-06 .. 2026-10-23.
462 entries, 2026-08-06 .. 2026-10-23.

| date | entry id | finding | file |
|---|---|---|---|
Expand All @@ -40,6 +40,8 @@ exactly one of them, never both.
| 2026-10-23 | `D-ONNX-RELALG-CAP` | S0 result: ONNX as import membrane, DR64³ plan not found | [2026-10-23-d-onnx-relalg-cap-s0.md](2026-10-23-d-onnx-relalg-cap-s0.md) |
| 2026-10-23 | `E-D-ONNX-RELALG-CAP-S0-1` | | [2026-10-23-d-onnx-relalg-cap-s0-capstone-1750.md](2026-10-23-d-onnx-relalg-cap-s0-capstone-1750.md) |
| 2026-10-23 | `D-ONNX-NATIVE-0` | | [2026-10-23-d-onnx-native-0-facade.md](2026-10-23-d-onnx-native-0-facade.md) |
| 2026-10-23 | `D-NFR-6-v1` | | [2026-10-23-d-nfr-6-validate-once.md](2026-10-23-d-nfr-6-validate-once.md) |
| 2026-10-23 | `D-NFR-6` | | [2026-10-23-d-nfr-6-soa-borrow.md](2026-10-23-d-nfr-6-soa-borrow.md) |
| 2026-10-23 | `D-NFR-0` | | [2026-10-23-d-nfr-0-native-repr-contract.md](2026-10-23-d-nfr-0-native-repr-contract.md) |
| 2026-10-23 | `D-MPF-4` | | [2026-10-23-d-mpf-4-classid-route.md](2026-10-23-d-mpf-4-classid-route.md) |
| 2026-10-23 | `D-MPF-2` | | [2026-10-23-d-mpf-2-fixed-cost.md](2026-10-23-d-mpf-2-fixed-cost.md) |
Expand Down
82 changes: 82 additions & 0 deletions .claude/plans/d-nfr-6-soa-borrow-v2.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
# D-NFR-6 v2 — native operations over the SoaEnvelope borrow, no guard

**Status:** MEASURED, lab only (`crates/soa-view-probe`). No production contract, ABI or ClassId/LazyLock change. Supersedes `d-nfr-6-validate-once-v1.md`.

**Rule:** mutation safety belongs to `&self` / `&mut self`; the mathematics belongs to ndarray. A view of SoA rows is the contract's existing borrow (`NodeRowPacket`), and an operation over it is the ndarray kernel it names. Nothing between them is checked at runtime.

## 1. What v1 got wrong

v1 built a `Table` that held its own `FrameStore` beside the SoA, issued frames, and checked a frame on each operation through `bind_dr64` (measured: `FrameStore::check` 25.7–28.3 ns, 96 instructions, six `lock` RMWs; `bind_dr64` 59–67 ns). That re-derived ownership the contract already has. `SoaEnvelope`'s own docs say views are `&self` borrows of the owner, a batch-writer `&mut` cannot begin while a view is live, and "nothing is checked at runtime." The frame check guarded against a mutation the borrow checker already makes impossible.

Frame provenance (who sealed these bytes) and external authorization (may this caller read them) are real questions. They are answered by whoever hands out the packet, once; they are not a reason for a per-operation check. v1's conflation of the two is withdrawn, and so is its D-NFR-6a proposal (a faster `FrameStore::check`): making an unneeded guard cheaper is not the fix.

## 2. Design (lab)

| part | what it is | where |
|---|---|---|
| owner | the `Vec<NodeRow>` the mailbox holds; writes through `&mut` (`set_value_facet`) | caller |
| view | `NodeRowPacket<'_>`: `&self` borrow, 512-byte row stride, not `Clone` | `lance-graph-contract` (unchanged) |
| facet address | `FacetSlot`, one of the 32 16-byte slots; built once, in range by construction | `soa-view-probe` |
| operations | `FacetFold` on `NodeRowPacket`: `classid_mask` → `eq_u32_strided_to_mask`; `facet_match_mask` → `ternary_match_strided16_to_mask`; `payload_byte_sum` → `masked_strided_group_sum` over 12 × `u8` (never widened); counts via `popcount_batch_u64` | ndarray |

No new ownership type, no frame, no epoch, no dispatch.

**Assumption:** all mutation goes through the owner by ordinary borrowing (`&mut`). The guarantee does not extend to interior mutability or an `unsafe` write path beside the owner; `NodeRow`'s fields are plain byte arrays, so neither exists here.

## 3. Falsifiers

| claim | how it is pinned | disable run |
|---|---|---|
| no owner write while a view is live | `compile_fail,E0502` | — (rustdoc checks the code) |
| no view outlives its owner | `compile_fail,E0597` | — |
| no second holder of a view | `compile_fail,E0599` (packet has no `clone`) | — |
| a view taken after a write sees the write | runtime test + the compiling doctest twin | — |
| packet result = direct ndarray = row loop, 8 sizes (1..4096), both classids, mask/match/sum | `packet_operations_equal_direct_ndarray_and_the_row_loop` | classid offset +1 → 3 tests red; payload width 11 → 1 red |
| fixture is not vacuous | `the_fixture_is_not_vacuous` | one class only → red |
| facet geometry (slot 0 key, value facet 0 at byte 32, 32 slots) | `facet_slots_are_the_canonical_16_byte_grid` | covered by the offset disable |

## 4. Cost (reported, not gated)

`examples/view_cost.rs`, release, AVX-512 host, load ≈ 0.45. Medians of 21 batches; three runs agree within noise. A = the ndarray kernel on the raw LE bytes; B = the same kernel through `FacetFold` on the packet. Allocations are counted around the operation loop only.

| rows | op | A µs/call | B µs/call | ns/row (A ≈ B) | allocs |
|---:|---|---:|---:|---:|---|
| 64 | classid mask + count | 0.13 | 0.12 | ~2 | 0 / 0 |
| 64 | mask + payload sum | 0.33 | 0.30 | ~5 | 0 / 0 |
| 64 | ternary match + count | 0.06 | 0.06 | ~0.9 | 0 / 0 |
| 4096 | classid mask + count | 11.1 | 10.5 | ~2.7 | 0 / 0 |
| 4096 | mask + payload sum | 27.0 | 26.8 | ~6.6 | 0 / 0 |
| 4096 | ternary match + count | 10.3 | 10.3 | ~2.5 | 0 / 0 |
| 65536 | classid mask + count | 356 | 357 | ~5.4 | 0 / 0 |
| 65536 | mask + payload sum | 903 | 861 | ~13.5 | 0 / 0 |
| 65536 | ternary match + count | 335 | 336 | ~5.1 | 0 / 0 |

The per-row cost is the strided read itself (one 64-byte line per 512-byte row; 65536 rows is 32 MB), and it is the same in both arms.

Assembly (`cargo rustc --release --example view_cost -- --emit=asm`, `probe_a_*` vs `probe_b_*`): each pair calls the same ndarray kernels and nothing else. B is 4–7 instructions longer: one `shlq $9` (the `n_rows × 512` length `as_le_bytes` computes) and register moves for the different argument shape. No `lock`, no `cpuid`, no lazy initialization in any probe.

## 5. Answers

**Q1 — what does the old 27–28 ns guard execute?** Unchanged from v1: `FrameStore::check`, dominated by creating and dropping two temporary `Weak`s. v2's answer is that it should not execute at all on this path.

**Q2 — what does borrowing guarantee?** That no write to the rows happens while any view is live, that no view outlives the rows, and that a view is not duplicated into a second holder (§3, compile errors).

**Q3 — what needs runtime validation?** Not mutation. Arguments that the type system does not bound are bounded once, by construction (`FacetSlot`), not per call. Frame provenance and authorization are decided where the packet is handed out; they are outside this probe.

**Q4 — at which boundary?** Where the owner gives out `&rows`. That is a single point, and it is already the borrow.

**Q5 — can several operations reuse one view?** Yes; any number, with no per-operation cost (§4).

**Q6 — what does it cost against direct ndarray?** Nothing measurable: equal within run-to-run noise, zero allocations in both arms, same kernel calls in the asm (§4).

## 6. Acceptance

- [x] no `Table`, no duplicate ownership, no frame guard
- [x] view = `NodeRowPacket`, 512-byte row stride, 16-byte facets
- [x] every operation is one ndarray kernel
- [x] compile-time refusals pinned with error codes
- [x] packet = direct = row loop; 3 disable runs red
- [x] cost: equal, 0 allocations, asm shows only the kernel calls
- [x] fmt, clippy `-D warnings`, tests (own CI workflow `soa-view-probe.yml`)
- [ ] independent review
- [ ] operator authorization before merge
Loading
Loading