diff --git a/.claude/board/INTEGRATION_PLANS.md b/.claude/board/INTEGRATION_PLANS.md index 918256687..32b0b64d4 100644 --- a/.claude/board/INTEGRATION_PLANS.md +++ b/.claude/board/INTEGRATION_PLANS.md @@ -1,3 +1,11 @@ +## 2026-10-23 — d-nfr-6-soa-borrow-v2 — native operations over the SoaEnvelope borrow, no guard → `.claude/plans/d-nfr-6-soa-borrow-v2.md` + +**Status:** MEASURED, lab only (`crates/soa-view-probe`). Supersedes `d-nfr-6-validate-once-v1`: views are `NodeRowPacket` borrows, operations are one ndarray kernel each, mutation is the owner's `&mut`; no `Table`, no frame guard. Packet equals direct ndarray within noise at 0 allocations. D-NFR-6a withdrawn. +**Confidence:** measured on one AVX-512 host; no independent review yet. + +## 2026-10-23 — d-nfr-6-validate-once-v1 — validate a binding once, then execute without repeated guards → `.claude/plans/d-nfr-6-validate-once-v1.md` + +**Status:** SUPERSEDED by d-nfr-6-soa-borrow-v2 (was MEASURED, lab only, `crates/dr-validated-view-probe`, removed). The per-call 27–28 ns guard is `FrameStore::check`'s refcount traffic; a view validated once by the existing `bind_dr64`, and kept valid by borrowing the table that owns data, frame and focus, runs native operations at direct cost. Externally owned revocation keeps a one-load check per operation. Proposals D-NFR-6a need the probe owner's review. ## 2026-10-23 — d-onnx-native-v1 — ONNX vocabulary over existing native functions → `.claude/plans/d-onnx-native-v1.md` **Status:** D-ONNX-NATIVE-0 IMPLEMENTED as a draft PR (`crates/lance-graph-onnx`), not merged. The path is ONNX name → existing native function: ALIAS where the contract is identical, a small INLINE adapter where it is not, a typed `Refusal` otherwise. No graph, dispatcher or initialization. R2IL routing (D-MPF-4, #1752) is separate and is not imported. diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index 4c34fb31a..2f3f43e9d 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -13,14 +13,23 @@ Docs only. Plans `d-native-repr-contract-v1.md` (D-NFR) and `d-native-first-depe | D-id | scope | status | gate / falsifier | |---|---|---|---| -| **D-NFC-0** | one canonical dependency diagram and seven root assumptions, applied as marked supersessions to #1746–#1751 and #1750 | In PR | each affected PR cites the diagram; S0 dispute left open | -| **D-NFR-0** | seam table Quack → DR64³ → Palette256³ → execution, cited to source | In PR | every seam SHIPPED / LAB / PROPOSED / MISSING with its source | +| **D-NFC-0** | one canonical dependency diagram and seven root assumptions, applied as marked supersessions to #1746–#1751 and #1750 | Shipped (#1753) | each affected PR cites the diagram; S0 dispute left open | +| **D-NFR-0** | seam table Quack → DR64³ → Palette256³ → execution, cited to source | Shipped (#1753) | every seam SHIPPED / LAB / PROPOSED / MISSING with its source | | **D-NFR-1** | lab adapter: a Quack-bound query sealed under a frame | Proposed | after one revision moves, the stored binding is refused with the matching `Refusal`; before, it equals an oracle | | **D-NFR-2** | codebook generation carried in the frame seal | Proposed | a palette value bound under generation g is refused under g+1 | | **D-NFR-3** | one DR axis bound to one palette plane under a frame | Proposed | Cartesian and numerical oracles agree; a mismatched codebook is refused | | **D-NFR-4** | end-to-end acceptance fixture (nine steps, each measured) | Proposed | all nine steps pass, each step's cost reported separately | | **D-NFR-5** | one binding and one focus reused across several operations | Proposed | reuse saving measured against rebinding and a fused scalar / SIMD baseline, guards run on every use | +## D-NFR-6 — native operations over the SoaEnvelope borrow, no guard (2026-10-23) + +Lab only: `crates/soa-view-probe`, plan `.claude/plans/d-nfr-6-soa-borrow-v2.md` (supersedes `d-nfr-6-validate-once-v1.md`). Parent D-NFR (#1753). + +| D-id | scope | status | gate / falsifier | +|---|---|---|---| +| **D-NFR-6** | views are `NodeRowPacket` borrows (512-byte rows, 16-byte facets); operations are one ndarray kernel each; mutation through the owner's `&mut`; no `Table`, no frame guard | MEASURED (`entries/2026-10-23-d-nfr-6-soa-borrow.md`) | 3 compile_fail examples (E0502/E0597/E0599); 4 runtime falsifiers; 3 disable runs red; packet = direct within noise, 0 allocations, asm shows only kernel calls | +| ~~**D-NFR-6a**~~ | ~~`FrameStore::check` by pointer comparison~~ | WITHDRAWN (v2: a faster unneeded guard is not the fix) | — | + ## D-MPF — mathematical polyfill API, Stage 1 (2026-10-23) Lab only (`crates/math-polyfill-probe`). Plan `.claude/plans/d-mpf-math-polyfill-v1.md`; entry `entries/2026-10-23-d-mpf-1-polyfill-probe.md`. diff --git a/.claude/board/entries/2026-10-23-d-nfr-6-soa-borrow.md b/.claude/board/entries/2026-10-23-d-nfr-6-soa-borrow.md new file mode 100644 index 000000000..d1c277717 --- /dev/null +++ b/.claude/board/entries/2026-10-23-d-nfr-6-soa-borrow.md @@ -0,0 +1,9 @@ +# D-NFR-6 v2 — native operations over the SoaEnvelope borrow + +**Date:** 2026-10-23 · **Status:** MEASURED, lab only · **Plan:** `.claude/plans/d-nfr-6-soa-borrow-v2.md` · **Crate:** `crates/soa-view-probe` · supersedes `2026-10-23-d-nfr-6-validate-once.md` + +- **Correction:** v1 put a `Table` with its own `FrameStore` beside the SoA and checked a frame per operation. That re-derived ownership `SoaEnvelope` already carries. Mutation safety is the borrow (`&self` views, the owner's `&mut`); frame provenance and authorization are separate questions and do not justify a per-operation check. +- **Design:** view = `NodeRowPacket` (512-byte rows, not `Clone`); address = `FacetSlot` (one of 32 × 16 bytes, in range by construction); operations = `FacetFold`, one ndarray kernel each (`eq_u32_strided_to_mask`, `ternary_match_strided16_to_mask`, `masked_strided_group_sum` over 12 × `u8`). +- **Pinned:** E0502 (no write during a live view), E0597 (no view outliving the owner), E0599 (no second holder); packet = direct ndarray = row loop at 8 sizes; 3 disable runs red. +- **Cost:** packet equals direct within noise at 64 / 4096 / 65536 rows, 0 allocations in both; asm: same kernel calls, B +4–7 instructions (`n_rows × 512` and register moves), no `lock`/`cpuid`. +- **Withdrawn:** D-NFR-6a (faster `FrameStore::check`). diff --git a/.claude/board/entries/2026-10-23-d-nfr-6-validate-once.md b/.claude/board/entries/2026-10-23-d-nfr-6-validate-once.md new file mode 100644 index 000000000..80d000977 --- /dev/null +++ b/.claude/board/entries/2026-10-23-d-nfr-6-validate-once.md @@ -0,0 +1,11 @@ +# D-NFR-6-v1 — validate once, execute without repeated guards (superseded) + +> ⊘ Superseded by `2026-10-23-d-nfr-6-soa-borrow.md`: the `Table` + frame guard design is withdrawn. + +**Date:** 2026-10-23 · **Status:** SUPERSEDED (was MEASURED, lab only) · **Plan:** `.claude/plans/d-nfr-6-validate-once-v1.md` · **Crate:** `crates/dr-validated-view-probe` + +- **Q1, the guard:** the 27–28 ns per-call guard (#1752 route) is `FrameStore::check` (25.7–28.3 ns; 96 instructions; six `lock` RMWs) plus two `u64` compares (~1.3 ns). The cost is the temporary `Weak`s from `Arc::downgrade`; the pointer comparisons cost ~1 ns each. `bind_dr64` adds a `Frame` clone (two more `lock inc`): 59–67 ns total. +- **Q2/Q5, validate once:** a `Table` owns data, frame store and focus map; every change takes `&mut`. `validate` runs the existing `bind_dr64` once and returns a `View<'t>`. While a view lives, advance, refocus and revoke are compile errors (E0502), and a view cannot outlive its table or leak a `'static` borrow (E0597). After the borrow, stale guards are refused at re-acquisition. +- **Q6, the cost:** view operations equal direct `ndarray::simd` within noise, at 0 allocations. `View::sum` is a tail call with no `lock` op. One validation is ~65 ns per view. Per-operation alternatives: +25–40 ns (route guard), +50–75 ns (`bind_dr64`). +- **Q3, what stays at run time:** revocation owned by another component (one epoch load per operation, `SharedView`); data not owned with its frame (`bind_dr64` checks the frame, not the bytes); a focus whose revision is chosen by the caller (fixed by an owner-issued revision). +- **Proposed, not implemented:** `FrameStore::check` by `as_ptr` comparison (no refcount traffic); `BoundDr64` borrowing its frame. Estimated validation ~10 ns, not measured. diff --git a/.claude/board/entries/README.md b/.claude/board/entries/README.md index d36a8cb8e..9a3a2b2de 100644 --- a/.claude/board/entries/README.md +++ b/.claude/board/entries/README.md @@ -25,7 +25,7 @@ index row, (3) no duplicate entry id. Checks 1 and 2 are deliberately opposite directions; the stranding this convention prevents shows up in exactly one of them, never both. -460 entries, 2026-08-06 .. 2026-10-23. +462 entries, 2026-08-06 .. 2026-10-23. | date | entry id | finding | file | |---|---|---|---| @@ -40,6 +40,8 @@ exactly one of them, never both. | 2026-10-23 | `D-ONNX-RELALG-CAP` | S0 result: ONNX as import membrane, DR64³ plan not found | [2026-10-23-d-onnx-relalg-cap-s0.md](2026-10-23-d-onnx-relalg-cap-s0.md) | | 2026-10-23 | `E-D-ONNX-RELALG-CAP-S0-1` | | [2026-10-23-d-onnx-relalg-cap-s0-capstone-1750.md](2026-10-23-d-onnx-relalg-cap-s0-capstone-1750.md) | | 2026-10-23 | `D-ONNX-NATIVE-0` | | [2026-10-23-d-onnx-native-0-facade.md](2026-10-23-d-onnx-native-0-facade.md) | +| 2026-10-23 | `D-NFR-6-v1` | | [2026-10-23-d-nfr-6-validate-once.md](2026-10-23-d-nfr-6-validate-once.md) | +| 2026-10-23 | `D-NFR-6` | | [2026-10-23-d-nfr-6-soa-borrow.md](2026-10-23-d-nfr-6-soa-borrow.md) | | 2026-10-23 | `D-NFR-0` | | [2026-10-23-d-nfr-0-native-repr-contract.md](2026-10-23-d-nfr-0-native-repr-contract.md) | | 2026-10-23 | `D-MPF-4` | | [2026-10-23-d-mpf-4-classid-route.md](2026-10-23-d-mpf-4-classid-route.md) | | 2026-10-23 | `D-MPF-2` | | [2026-10-23-d-mpf-2-fixed-cost.md](2026-10-23-d-mpf-2-fixed-cost.md) | diff --git a/.claude/plans/d-nfr-6-soa-borrow-v2.md b/.claude/plans/d-nfr-6-soa-borrow-v2.md new file mode 100644 index 000000000..27b6a35f2 --- /dev/null +++ b/.claude/plans/d-nfr-6-soa-borrow-v2.md @@ -0,0 +1,82 @@ +# D-NFR-6 v2 — native operations over the SoaEnvelope borrow, no guard + +**Status:** MEASURED, lab only (`crates/soa-view-probe`). No production contract, ABI or ClassId/LazyLock change. Supersedes `d-nfr-6-validate-once-v1.md`. + +**Rule:** mutation safety belongs to `&self` / `&mut self`; the mathematics belongs to ndarray. A view of SoA rows is the contract's existing borrow (`NodeRowPacket`), and an operation over it is the ndarray kernel it names. Nothing between them is checked at runtime. + +## 1. What v1 got wrong + +v1 built a `Table` that held its own `FrameStore` beside the SoA, issued frames, and checked a frame on each operation through `bind_dr64` (measured: `FrameStore::check` 25.7–28.3 ns, 96 instructions, six `lock` RMWs; `bind_dr64` 59–67 ns). That re-derived ownership the contract already has. `SoaEnvelope`'s own docs say views are `&self` borrows of the owner, a batch-writer `&mut` cannot begin while a view is live, and "nothing is checked at runtime." The frame check guarded against a mutation the borrow checker already makes impossible. + +Frame provenance (who sealed these bytes) and external authorization (may this caller read them) are real questions. They are answered by whoever hands out the packet, once; they are not a reason for a per-operation check. v1's conflation of the two is withdrawn, and so is its D-NFR-6a proposal (a faster `FrameStore::check`): making an unneeded guard cheaper is not the fix. + +## 2. Design (lab) + +| part | what it is | where | +|---|---|---| +| owner | the `Vec` the mailbox holds; writes through `&mut` (`set_value_facet`) | caller | +| view | `NodeRowPacket<'_>`: `&self` borrow, 512-byte row stride, not `Clone` | `lance-graph-contract` (unchanged) | +| facet address | `FacetSlot`, one of the 32 16-byte slots; built once, in range by construction | `soa-view-probe` | +| operations | `FacetFold` on `NodeRowPacket`: `classid_mask` → `eq_u32_strided_to_mask`; `facet_match_mask` → `ternary_match_strided16_to_mask`; `payload_byte_sum` → `masked_strided_group_sum` over 12 × `u8` (never widened); counts via `popcount_batch_u64` | ndarray | + +No new ownership type, no frame, no epoch, no dispatch. + +**Assumption:** all mutation goes through the owner by ordinary borrowing (`&mut`). The guarantee does not extend to interior mutability or an `unsafe` write path beside the owner; `NodeRow`'s fields are plain byte arrays, so neither exists here. + +## 3. Falsifiers + +| claim | how it is pinned | disable run | +|---|---|---| +| no owner write while a view is live | `compile_fail,E0502` | — (rustdoc checks the code) | +| no view outlives its owner | `compile_fail,E0597` | — | +| no second holder of a view | `compile_fail,E0599` (packet has no `clone`) | — | +| a view taken after a write sees the write | runtime test + the compiling doctest twin | — | +| packet result = direct ndarray = row loop, 8 sizes (1..4096), both classids, mask/match/sum | `packet_operations_equal_direct_ndarray_and_the_row_loop` | classid offset +1 → 3 tests red; payload width 11 → 1 red | +| fixture is not vacuous | `the_fixture_is_not_vacuous` | one class only → red | +| facet geometry (slot 0 key, value facet 0 at byte 32, 32 slots) | `facet_slots_are_the_canonical_16_byte_grid` | covered by the offset disable | + +## 4. Cost (reported, not gated) + +`examples/view_cost.rs`, release, AVX-512 host, load ≈ 0.45. Medians of 21 batches; three runs agree within noise. A = the ndarray kernel on the raw LE bytes; B = the same kernel through `FacetFold` on the packet. Allocations are counted around the operation loop only. + +| rows | op | A µs/call | B µs/call | ns/row (A ≈ B) | allocs | +|---:|---|---:|---:|---:|---| +| 64 | classid mask + count | 0.13 | 0.12 | ~2 | 0 / 0 | +| 64 | mask + payload sum | 0.33 | 0.30 | ~5 | 0 / 0 | +| 64 | ternary match + count | 0.06 | 0.06 | ~0.9 | 0 / 0 | +| 4096 | classid mask + count | 11.1 | 10.5 | ~2.7 | 0 / 0 | +| 4096 | mask + payload sum | 27.0 | 26.8 | ~6.6 | 0 / 0 | +| 4096 | ternary match + count | 10.3 | 10.3 | ~2.5 | 0 / 0 | +| 65536 | classid mask + count | 356 | 357 | ~5.4 | 0 / 0 | +| 65536 | mask + payload sum | 903 | 861 | ~13.5 | 0 / 0 | +| 65536 | ternary match + count | 335 | 336 | ~5.1 | 0 / 0 | + +The per-row cost is the strided read itself (one 64-byte line per 512-byte row; 65536 rows is 32 MB), and it is the same in both arms. + +Assembly (`cargo rustc --release --example view_cost -- --emit=asm`, `probe_a_*` vs `probe_b_*`): each pair calls the same ndarray kernels and nothing else. B is 4–7 instructions longer: one `shlq $9` (the `n_rows × 512` length `as_le_bytes` computes) and register moves for the different argument shape. No `lock`, no `cpuid`, no lazy initialization in any probe. + +## 5. Answers + +**Q1 — what does the old 27–28 ns guard execute?** Unchanged from v1: `FrameStore::check`, dominated by creating and dropping two temporary `Weak`s. v2's answer is that it should not execute at all on this path. + +**Q2 — what does borrowing guarantee?** That no write to the rows happens while any view is live, that no view outlives the rows, and that a view is not duplicated into a second holder (§3, compile errors). + +**Q3 — what needs runtime validation?** Not mutation. Arguments that the type system does not bound are bounded once, by construction (`FacetSlot`), not per call. Frame provenance and authorization are decided where the packet is handed out; they are outside this probe. + +**Q4 — at which boundary?** Where the owner gives out `&rows`. That is a single point, and it is already the borrow. + +**Q5 — can several operations reuse one view?** Yes; any number, with no per-operation cost (§4). + +**Q6 — what does it cost against direct ndarray?** Nothing measurable: equal within run-to-run noise, zero allocations in both arms, same kernel calls in the asm (§4). + +## 6. Acceptance + +- [x] no `Table`, no duplicate ownership, no frame guard +- [x] view = `NodeRowPacket`, 512-byte row stride, 16-byte facets +- [x] every operation is one ndarray kernel +- [x] compile-time refusals pinned with error codes +- [x] packet = direct = row loop; 3 disable runs red +- [x] cost: equal, 0 allocations, asm shows only the kernel calls +- [x] fmt, clippy `-D warnings`, tests (own CI workflow `soa-view-probe.yml`) +- [ ] independent review +- [ ] operator authorization before merge diff --git a/.claude/plans/d-nfr-6-validate-once-v1.md b/.claude/plans/d-nfr-6-validate-once-v1.md new file mode 100644 index 000000000..9b39db60e --- /dev/null +++ b/.claude/plans/d-nfr-6-validate-once-v1.md @@ -0,0 +1,210 @@ +# D-NFR-6 — validate a binding once, then execute without repeated guards + +> ⊘ **SUPERSEDED by `d-nfr-6-soa-borrow-v2.md` (2026-10-23).** The `Table` below kept its own `FrameStore` beside the SoA and checked a frame per access, re-deriving ownership that `SoaEnvelope`'s borrow already gives (views are `&self`, the owner's `&mut` cannot start while one is live). The crate was reworked into `crates/soa-view-probe`; D-NFR-6a is withdrawn. The measurements below remain the record of what the guard cost. + +**Status:** SUPERSEDED; was MEASURED, lab only (`crates/dr-validated-view-probe`, removed). No production contract, ABI or other crate is changed. Parent: D-NFR (`d-native-repr-contract-v1.md`, #1753); the per-call guard figure comes from D-MPF-4 (#1752 §10). + +**Question.** The #1752 ClassId route runs a frame/focus/scope guard on every call, and that guard measured 27–28 ns. Is that guard needed on every operation, or can ownership and borrowing guarantee the same thing at binding time? + +**Answer in one line.** For data owned together with its frame and focus, and for grants owned by the caller, one validation authorizes any number of operations. After it, each operation costs what the direct `ndarray::simd` call costs. Two cases still need a runtime check: + +- revocation owned by another component costs one plain load per operation; +- data the frame does not own keeps the per-use check. + +## 1. Three responsibilities + +| responsibility | can go stale? | where it is decided | mechanism | +|---|---|---|---| +| method resolution: which native function | no | once per process | `LazyLock`; a copied `fn` pointer captures nothing | +| data binding: frame, focus, scope | yes | when a view is acquired | the existing `bind_dr64`, unchanged, run once in `validate` | +| execution | no | never | `View` methods call `ndarray::simd` on borrowed data | + +## 2. Source audit (read in source at `origin/main`) + +`dr-math-binding-probe/src/lib.rs`: + +- `FrameStore { identity: Arc<()>, current: Arc }`. Changes go through `advance(&mut self)` only; there is no interior mutability. +- `seal()` returns `Frame { store: Weak<()>, seal: Weak, at }`. +- `check()` compares `Weak::ptr_eq(&frame.store, &Arc::downgrade(&self.identity))`, and the same for the seal. Each comparison creates a temporary `Weak` and drops it. +- `bind_dr64` runs, in order: method / axis / semantics compares, the validity–truth match, the Bag refusal, `store.check`, then focus and scope equality against caller-supplied `now`. It returns `BoundDr64 { frame: bound_at.frame.clone(), rows: &'a [u64;64], .. }`. + +`dr-math-binding-probe/src/relcomp.rs`: `Chain<'a>` holds `&'a FrameStore` and checks the frame once in `Chain::new`. Its `compile_fail` doctest shows `advance` does not compile while a chain lives. `Chain` is therefore already the validate-once pattern, but for the frame only. + +`lance-graph-quack/src/bind.rs`: a bound `Query` is an owned numeric value. It carries table, column and plane indices, no lifetime and no frame. That is the D-NFR-1 gap: a bound query does not know its frame. + +The seven audit questions, each answered with its evidence: + +1. **Can the data change through another path while borrowed?** For `bind_dr64`, yes. Its `rows` are any `&[u64;64]` the caller passes, unrelated to the frame (test `bind_dr64_checks_the_frame_not_the_data`). The bytes themselves cannot change during a shared borrow: `[u64; 64]` and `Vec` have no interior mutability. The gap is coupling, not mutation. In the probe, a `Table` owns the data and changes it only in `advance(&mut self)`, together with the revisions. +2. **Can focus change while borrowed?** `Focus` is a `Copy` revision number supplied by the caller. Nothing owns it, and a map could change while its number stays the same. In the probe, `Table::refocus(&mut self)` issues the next revision itself, so even an identical map gets a new revision (disable run D1). +3. **Can scope change independently?** `Scope { id }` equality says nothing about authorization. In the probe, the scope comes from the grant owner (`Grants`), never from the caller (disable run D3). When revocation is owned elsewhere (`SharedGrants`: `revoke(&self)` through an atomic), the borrow cannot stop it; see §4, F5. +4. **Are source and relation-map revisions coupled to the data?** Not by `bind_dr64`. Yes by `Table::advance`. +5. **Can a cloned frame outlive validity?** Yes, and it is harmless: a clone is a pair of `Weak`s. After `advance` the seal `Arc` is replaced, so `check` refuses (`SourceChanged`, or `FrameResealed` at equal numbers). A frame from another store is refused as `ForeignFrame`. +6. **Does a seal protect data identity or only revision equality?** It protects the identity of a revision issue: `Arc` pointer identity, so re-sealing equal numbers is a new frame. It does not protect the bytes; see item 1. +7. **Can a prepared callable keep references past the borrow?** No. `Kernels` holds `fn` pointers, which capture nothing, and data reaches them only as arguments. A view cannot outlive its table (E0597) or hand out a `'static` borrow (E0597). + +## 3. Design (lab) + +`Table` owns `FrameStore`, `Data { rows: [u64;64], values: Vec }` and the focus map. Every change needs `&mut Table`. + +`validate(&'t Table, &'t Grants, declared, requested, &Guards) -> View<'t>` runs `bind_dr64` once, taking the focus from the table and the scope from the grants. + +`View<'t>` offers `count`, `sum`, `min`, `max` and `count_gt` (Greater → AND → popcount, written into caller scratch). Each calls a `KERNELS` function pointer directly. + +`validate_shared(&'t Table, &'t SharedGrants, ..) -> SharedView<'t>` is the same, plus one `Acquire` load of the grant epoch per operation. + +## 4. Falsifiers + +**Compile time.** These are `compile_fail` doctests with pinned error codes. Each snippet was also compiled on its own and fails with exactly the pinned code: + +| id | attempt | result | +|---|---|---| +| F1 | `table.advance(..)` while a view lives | E0502 | +| F2 | `table.refocus(..)` while a view lives | E0502 | +| F3 (owned grants) | `grants.revoke()` while a view lives | E0502 | +| F4 (outlive) | keep a view after its table is dropped | E0597 | +| F7 | `let r: &'static [u64;64] = view.rows()` | E0597 | + +**Run time** (`tests/falsifiers.rs`, 8 tests): + +- **E arm:** after the borrow ends, the old guards are refused at re-acquisition: + - an `advance` → `SourceChanged`; + - an identical-map `refocus` → `FocusChanged`; + - a revocation → `ScopeRevoked`. + + Freshly taken guards validate again. +- Equal numbers re-sealed → `FrameResealed`; another table's frame → `ForeignFrame`. +- **F3, scope as grant:** guards for scope 3 under grants for scope 4 → `ScopeChanged`. +- **F5, shared revocation:** `revoke(&self)` compiles while a `SharedView` lives. The next operation returns `ScopeRevoked`, and operations before it answer. **The borrow cannot enforce this, so the check stays: one epoch load per operation.** +- **F6, semantics:** the typed refusals come through unchanged: + - another method → `MethodChanged`; + - other axes → `AxisChanged`; + - other truth → `SemanticsChanged`; + - three-valued without validity → `ThreeValuedNeedsValidity`; + - Bag → `BagNeedsMultiplicity`. +- **Results:** all five operations equal both the direct `ndarray::simd` calls and an independent row loop, at 1, 2, 63, 64, 65, 1000, 4095 and 4096 rows and five thresholds. An anti-vacuity test pins a partial selection. + +**Disable runs** (committed first; each red on exactly one test, then restored): + +| disable | red test | +|---|---| +| D1 `refocus` keeps the revision | `after_the_borrow_ends_every_change_is_refused_at_acquisition` | +| D2 `SharedView` never reads the epoch | `shared_revocation_during_a_view_is_refused_per_operation` | +| D3 owned grants ignored | `after_the_borrow_ends_every_change_is_refused_at_acquisition` | +| D4 focus taken from the caller's guards instead of the table | `after_the_borrow_ends_every_change_is_refused_at_acquisition` | + +**Removed as unreachable:** a post-bind "two-valued Distinct only" check. The view passes no validity plane, so `bind_dr64` already refuses every three-valued and every Bag method before it. A guard that cannot fire was deleted, not kept. + +## 5. Cost attribution (reported, not gated) + +Setup: `examples/guard_cost.rs`, release, AVX-512 at compile time, load 0.1–0.7. ns per call, median of 21; three runs; ranges are across the runs. Allocations per call: 0 in every arm. + +**The guard, taken apart** (64 rows; independent of data size): + +| piece | ns | +|---|---:| +| method + axis + semantics compares | 3.4–4.1 | +| validity–truth match | 0.9–1.0 | +| focus + scope equality | 1.2–1.3 | +| `FrameStore::check` (real) | 25.7–28.3 | +| `Frame::clone` + drop (real; `bind_dr64` clones the frame) | 23.0–26.2 | +| one `Arc::downgrade` + drop of the `Weak` (proxy) | 21.0–22.5 | +| one `Weak::ptr_eq` (proxy) | 1.0–1.2 | +| one `Weak::as_ptr == Arc::as_ptr` (proxy, no refcount) | 1.0–1.8 | +| `bind_dr64` total | 59–67 | +| `validate` total (`bind_dr64` + grant read) | 62.5–65.4 | +| `LazyLock` first initialisation (cold, single sample) | 651–1016 | +| first `validate` (cold, single sample) | 253–362 | + +**Assembly** (`--emit=asm`, each function read on its own): + +| function | instructions | `lock` RMWs | notes | +|---|---:|---:|---| +| `FrameStore::check` | 96 | 6 | 2 × `lock cmpxchg` (`Arc::downgrade`), 4 × `lock dec` (the temporaries' drops) | +| `bind_dr64` | 108 + `check` | 2 | 2 × `lock inc` (the `Frame` clone) | +| `validate` | 83 + `bind_dr64` | 2 | `lock dec` dropping the `BoundDr64`'s frame; `Once` check for `LazyLock` | +| `View::sum` | 20 | 0 | word count, one bounds check, tail `jmp` through the copied `fn` pointer | +| `SharedView::sum` | 38 | 0 | one plain load and compare of the epoch (`Acquire` is a plain `mov` on x86), then the same body | + +So the guard's cost is the reference-count traffic. Its comparisons cost 1–4 ns, and its six atomic RMWs (eight in `bind_dr64`) are the rest. + +**Per operation, 64 rows:** + +| op | A direct | B prebound | C `bind_dr64` per op | C′ guard per op (#1752 route) | D view | D′ shared view | +|---|---:|---:|---:|---:|---:|---:| +| count | 2.9–3.5 | 3.8–4.5 | 61–68 | 28–31 | 3.2–3.9 | 4.4–5.5 | +| sum | 28.6–30.9 | 28.6–32.8 | 78–83 | 42–49 | 27.8–39.3 | 28.8–43.8 | +| min | 30.3–49.0 | 31.2–57.5 | 80–114 | 48–89 | 29.9–61.6 | 30.7–60.2 | +| max | 30.7–53.3 | 31.5–36.2 | 79–87 | 48–54 | 30.2–34.2 | 30.5–34.5 | +| count_gt | 33.4–36.1 | 36.6–39.4 | 94–108 | 61–74 | 35.5–48.1 | 40.7–51.3 | + +The wide ranges (min, and one sum and count_gt sample) come from single noisy runs and appear in A and B as well as in D. Where D was above A in a run, the same run usually showed the same spike elsewhere. + +**Five operations, 64 rows:** + +| path | ns | +|---|---:| +| direct, inlined together | 107–119 | +| one `validate` + five view operations | 200–332 (≈ `validate` 65 + the five per-op view costs) | +| `bind_dr64` before each operation | 387–488 | + +**4096 rows:** the kernels cost 1.7–2.0 µs (masked sum/min/max), so all arms are within run-to-run noise except: +- `count`: direct 6.6, view 5.4, `bind_dr64` per op 64; +- `count_gt`: direct 292, view 307, `bind_dr64` per op 523. + +**E (refused re-validation):** 28–32 ns. + +## 6. Answers + +**Q1 — what does the 27–28 ns guard execute?** `FrameStore::check` (25.7–28.3 ns, 96 instructions, six `lock` RMWs) plus two `u64` equality tests for focus and scope (~1.3 ns). Inside `check`, the two `Weak::ptr_eq` comparisons cost about 1 ns each. Almost all the time goes to creating and dropping two temporary `Weak`s, which is atomic weak-count traffic. Comparing pointers does not cost it. + +**Q2 — what does borrowing already guarantee?** +- **In the probe's `Table` design**, while a view lives: + - the data, the frame store and the focus map cannot change (E0502); + - the view cannot outlive the table (E0597); + - no borrow escapes as `'static`; + - the prepared `fn` pointers hold no references. +- **In the existing code**, `Chain<'a>` already holds `&'a FrameStore`, so the frame cannot advance while a chain lives. `BoundDr64`, however, holds a cloned `Frame` rather than a borrow, so the store **can** advance while a `BoundDr64` lives. That is why the #1752 route has to re-check on every call. + +**Q3 — what genuinely needs runtime validation?** +1. **Acquisition, all of `bind_dr64`:** the guards were taken earlier and may be stale. +2. **Revocation owned by another component:** an atomic that a borrow cannot freeze. +3. **Data not owned with its frame:** `bind_dr64` checks the frame, not the bytes. +4. **A focus identified by a caller-chosen number:** no check, at run time or compile time, can see a map change behind an unchanged number. The fix is ownership (an owner-issued revision), not more checking. + +**Q4 — at which boundary?** + +| validation | boundary | +|---|---| +| method resolution | once per process (`LazyLock`) | +| frame, focus, scope, semantics | at view acquisition, and again only after a `&mut` change, which cannot happen while a view lives | +| shared revocation | per operation, one epoch load; per batch if the policy accepts revocation at batch granularity | +| data from outside the frame owner | per use, as today | + +**Q5 — can several operations reuse one validated binding without weakening the protections?** Yes, for table-owned data and caller-owned grants. Stale frame, focus and scope while a view lives are compile errors, not runtime events. After the borrow they are refused at re-acquisition. Semantic mismatches are refused at acquisition with the existing typed refusals. With externally revocable grants the same holds, plus one epoch load per operation. + +**Q6 — what does it cost against direct ndarray?** +- **Per operation:** D view equals A direct within run-to-run noise, at 0 allocations. In the assembly, `View::sum` is a tail call through the resolved `fn` pointer and executes no `lock` instruction. +- **Validation:** ~65 ns once per view; the first, cold one is 253–362 ns. +- **Shared view:** adds the epoch load, about 1–2 ns per operation. +- **The per-call paths it replaces cost:** + - +25–40 ns per operation (C′, the #1752 route guard); + - +50–75 ns per operation (C, `bind_dr64` per op). +- **Comparison points:** the 36.1 ns direct baseline and the ~0.5 ns `LazyLock` effect from #1752 are kept as comparison points, not as guarantees. Here the `fn`-pointer call measured 2.1–2.6 ns against 2.7–2.9 ns direct for a one-word count, i.e. no measurable cost. + +## 7. Acceptance + +- [x] A validated binding executes several operations with no routing and no repeated metadata validation. +- [x] Data, focus and scope remain valid for the whole view lifetime (compile-time for owned state; per-operation epoch for shared revocation). +- [x] Invalid reuse fails to compile, or is refused at acquisition (E arm, F4–F6). +- [x] No guard removed for speed. The one removed check was unreachable. The shared-revocation check is retained. +- [x] No correctness condition rests on an unsupported lifetime assumption. The two that a lifetime cannot carry (foreign data, external revocation) keep their runtime checks. +- [x] Zero allocation on the hot path (counted, all arms). +- [x] Direct and view results identical; also equal to an independent row loop. +- [x] Validate-once benchmarked against per-operation validation (§5). +- [x] Assembly supports the claim (§5). + +## 8. Proposals (not implemented; owner coordination needed) + +1. **`FrameStore::check` without refcount traffic.** Compare `Weak::as_ptr(&frame.store)` with `Arc::as_ptr(&self.identity)`, and the same for the seal, instead of creating temporaries with `Arc::downgrade`. This removes the six `lock` RMWs. The proxy measured ~1–2 ns per comparison against ~21 ns for the downgrade. The semantics are the same: `Weak::ptr_eq` compares `as_ptr`. A dangling `Weak` from a dropped store cannot alias a live `Arc` while the issuing store is alive, because the store holds its `Arc`. This needs its own review. +2. **`BoundDr64` borrows its frame** (`&'a Frame`) instead of cloning it. That removes two `lock inc` and two `lock dec`. With proposal 1, a full validation would fall from ~65 ns to roughly 10 ns (estimate, not measured). +3. **D-NFR-1:** the smallest seam (a Quack-bound query does not know its frame) takes the `View` shape here. The query is executed only through a value that borrows a frame owner, so frame validity rides on the borrow. diff --git a/.github/workflows/soa-view-probe.yml b/.github/workflows/soa-view-probe.yml new file mode 100644 index 000000000..2b8cc0173 --- /dev/null +++ b/.github/workflows/soa-view-probe.yml @@ -0,0 +1,64 @@ +name: SoA View Probe (D-NFR-6) +# Own [workspace] excluded from root tests; a lab gate, not production +# admission. Mirrors egglog-r2il-probe.yml's toolchain setup: the pinned +# toolchain from rust-toolchain.toml, never `stable`. +on: + pull_request: + paths: + - "crates/soa-view-probe/**" + - "crates/lance-graph-contract/**" + - ".github/workflows/soa-view-probe.yml" + - "rust-toolchain.toml" + push: + branches: [main] + paths: + - "crates/soa-view-probe/**" + - "crates/lance-graph-contract/**" + - ".github/workflows/soa-view-probe.yml" + - "rust-toolchain.toml" + workflow_dispatch: +permissions: + contents: read +concurrency: + group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }} + cancel-in-progress: true +env: + CARGO_TERM_COLOR: always + CARGO_INCREMENTAL: "0" + CARGO_PROFILE_DEV_DEBUG: "0" + CARGO_PROFILE_TEST_DEBUG: "0" + RUST_BACKTRACE: "1" +jobs: + isolated-probe: + name: excluded-crate fmt, clippy and tests + runs-on: ubuntu-24.04 + timeout-minutes: 60 + defaults: + run: + working-directory: lance-graph + steps: + - uses: actions/checkout@v4 + with: + path: lance-graph + persist-credentials: false + - name: Rust toolchain + run: rustup show + - name: Resolve target-cpu=native for this runner + id: cpu + run: | + flags=$(.github/scripts/resolve-native-rustflags.sh) + echo "RUSTFLAGS=$flags" >> "$GITHUB_ENV" + hash=$(printf %s "$flags" | sha256sum | cut -c1-16) + echo "flags=$hash" >> "$GITHUB_OUTPUT" + - uses: Swatinem/rust-cache@v2 + with: + key: cpu-${{ steps.cpu.outputs.flags }} + workspaces: lance-graph/crates/soa-view-probe + - name: fmt + run: cargo fmt --manifest-path crates/soa-view-probe/Cargo.toml --check + - name: clippy + run: cargo clippy --manifest-path crates/soa-view-probe/Cargo.toml --all-targets -- -D warnings + - name: tests + run: cargo test --release --manifest-path crates/soa-view-probe/Cargo.toml + - name: packet vs direct cost (reported, not gated) + run: cargo run --release --example view_cost --manifest-path crates/soa-view-probe/Cargo.toml diff --git a/crates/soa-view-probe/Cargo.toml b/crates/soa-view-probe/Cargo.toml new file mode 100644 index 000000000..db105fe4f --- /dev/null +++ b/crates/soa-view-probe/Cargo.toml @@ -0,0 +1,17 @@ +[package] +name = "soa-view-probe" +version = "0.0.0" +edition = "2024" +publish = false +description = "D-NFR-6 lab probe: native ndarray operations over a SoaEnvelope borrow (NodeRowPacket, 512-byte rows, 16-byte facets). Mutation safety is the borrow; no runtime guard. Lab only: no production contract or ABI." + +# Own workspace root: a lab crate excluded from the root workspace. +[workspace] + +[dependencies] +lance-graph-contract = { path = "../lance-graph-contract" } +# The native kernels. Same source string as mask-risc's copy. +ndarray = { git = "https://github.com/AdaWorldAPI/ndarray", branch = "master" } + +[patch."https://github.com/AdaWorldAPI/lance-graph"] +lance-graph-contract = { path = "../lance-graph-contract" } diff --git a/crates/soa-view-probe/examples/view_cost.rs b/crates/soa-view-probe/examples/view_cost.rs new file mode 100644 index 000000000..5e1740055 --- /dev/null +++ b/crates/soa-view-probe/examples/view_cost.rs @@ -0,0 +1,175 @@ +//! D-NFR-6: does an operation through a borrowed `NodeRowPacket` cost +//! anything beyond the ndarray kernel it names? Reported, not gated. +//! +//! Arm A: the ndarray kernel on the raw LE bytes. Arm B: the same kernel +//! through `FacetFold` on the packet. Both write into caller scratch, so +//! neither should allocate; a counting allocator checks that. +//! +//! `cargo run --release --example view_cost` +//! Assembly: `cargo rustc --release --example view_cost -- --emit=asm`, then +//! compare the `probe_a_*` / `probe_b_*` bodies. + +use std::alloc::{GlobalAlloc, Layout, System}; +use std::hint::black_box; +use std::sync::atomic::{AtomicUsize, Ordering}; +use std::time::Instant; + +use lance_graph_contract::NodeRowPacket; +use lance_graph_contract::canonical_node::NODE_ROW_STRIDE; +use lance_graph_contract::soa_envelope::SoaEnvelope; +use soa_view_probe::fixture::{CLASS_A, rows}; +use soa_view_probe::{FacetFold, FacetSlot}; + +struct Counting; +static ALLOCS: AtomicUsize = AtomicUsize::new(0); +unsafe impl GlobalAlloc for Counting { + unsafe fn alloc(&self, l: Layout) -> *mut u8 { + ALLOCS.fetch_add(1, Ordering::Relaxed); + unsafe { System.alloc(l) } + } + unsafe fn dealloc(&self, p: *mut u8, l: Layout) { + unsafe { System.dealloc(p, l) } + } +} +#[global_allocator] +static GLOBAL: Counting = Counting; + +const AT: usize = 32; // value facet 0 + +#[inline(never)] +fn probe_a_count(bytes: &[u8], n: usize, sel: &mut [u64]) -> u64 { + ndarray::simd::eq_u32_strided_to_mask(bytes, AT, NODE_ROW_STRIDE, n, CLASS_A, sel); + ndarray::simd::popcount_batch_u64(sel) +} + +#[inline(never)] +fn probe_b_count(view: &NodeRowPacket<'_>, sel: &mut [u64]) -> u64 { + view.classid_mask(FacetSlot::value(0).unwrap(), CLASS_A, sel); + ndarray::simd::popcount_batch_u64(sel) +} + +#[inline(never)] +fn probe_a_sum(bytes: &[u8], n: usize, sel: &mut [u64]) -> i64 { + ndarray::simd::eq_u32_strided_to_mask(bytes, AT, NODE_ROW_STRIDE, n, CLASS_A, sel); + ndarray::simd::masked_strided_group_sum(bytes, AT + 4, NODE_ROW_STRIDE, n, 12, 1, sel).unwrap() +} + +#[inline(never)] +fn probe_b_sum(view: &NodeRowPacket<'_>, sel: &mut [u64]) -> i64 { + let slot = FacetSlot::value(0).unwrap(); + view.classid_mask(slot, CLASS_A, sel); + view.payload_byte_sum(slot, sel) +} + +#[inline(never)] +fn probe_a_match(bytes: &[u8], n: usize, p: &[u8; 16], c: &[u8; 16], sel: &mut [u64]) -> u64 { + ndarray::simd::ternary_match_strided16_to_mask(bytes, AT, NODE_ROW_STRIDE, n, p, c, sel); + ndarray::simd::popcount_batch_u64(sel) +} + +#[inline(never)] +fn probe_b_match(view: &NodeRowPacket<'_>, p: &[u8; 16], c: &[u8; 16], sel: &mut [u64]) -> u64 { + view.facet_match_mask(FacetSlot::value(0).unwrap(), p, c, sel); + ndarray::simd::popcount_batch_u64(sel) +} + +/// Median ns per call over `reps` timed batches, plus the allocations made +/// by the operation itself (counted around the call loop only). +fn median_ns(reps: usize, iters: usize, mut f: impl FnMut()) -> (f64, usize) { + let mut t = [0f64; 21]; + assert!(reps <= t.len()); + let mut allocs = 0; + for slot in t.iter_mut().take(reps) { + let a0 = ALLOCS.load(Ordering::Relaxed); + let s = Instant::now(); + for _ in 0..iters { + f(); + } + *slot = s.elapsed().as_nanos() as f64 / iters as f64; + allocs += ALLOCS.load(Ordering::Relaxed) - a0; + } + let t = &mut t[..reps]; + t.sort_by(f64::total_cmp); + (t[reps / 2], allocs) +} + +fn main() { + let mut pattern = [0u8; 16]; + pattern[..4].copy_from_slice(&CLASS_A.to_le_bytes()); + let mut care = [0u8; 16]; + care[..4].fill(0xFF); + care[4] = 0x80; + + println!(" rows op A µs/call B µs/call A ns/row B ns/row allocs A/B"); + for n in [64usize, 4096, 65536] { + let owner = rows(n); + let view = NodeRowPacket::new(&owner, 1); + let bytes = view.as_le_bytes(); + let mut sel = vec![0u64; n.div_ceil(64)]; + let iters = (1 << 22) / n.max(64); + + assert_eq!( + probe_a_count(bytes, n, &mut sel), + probe_b_count(&view, &mut sel) + ); + assert_eq!( + probe_a_sum(bytes, n, &mut sel), + probe_b_sum(&view, &mut sel) + ); + assert_eq!( + probe_a_match(bytes, n, &pattern, &care, &mut sel), + probe_b_match(&view, &pattern, &care, &mut sel) + ); + + let row = |name: &str, a: &mut dyn FnMut(), b: &mut dyn FnMut()| { + let (ta, aa) = median_ns(21, iters, &mut *a); + let (tb, ab) = median_ns(21, iters, &mut *b); + println!( + "{n:>6} {name:<7} {:>10.2} {:>10.2} {:>9.2} {:>9.2} {aa}/{ab}", + ta / 1000.0, + tb / 1000.0, + ta / n as f64, + tb / n as f64 + ); + }; + let s = std::cell::RefCell::new(&mut sel); + row( + "count", + &mut || { + black_box(probe_a_count(black_box(bytes), n, &mut s.borrow_mut())); + }, + &mut || { + black_box(probe_b_count(black_box(&view), &mut s.borrow_mut())); + }, + ); + row( + "sum", + &mut || { + black_box(probe_a_sum(black_box(bytes), n, &mut s.borrow_mut())); + }, + &mut || { + black_box(probe_b_sum(black_box(&view), &mut s.borrow_mut())); + }, + ); + row( + "match", + &mut || { + black_box(probe_a_match( + black_box(bytes), + n, + &pattern, + &care, + &mut s.borrow_mut(), + )); + }, + &mut || { + black_box(probe_b_match( + black_box(&view), + &pattern, + &care, + &mut s.borrow_mut(), + )); + }, + ); + } +} diff --git a/crates/soa-view-probe/src/lib.rs b/crates/soa-view-probe/src/lib.rs new file mode 100644 index 000000000..b6aa87397 --- /dev/null +++ b/crates/soa-view-probe/src/lib.rs @@ -0,0 +1,250 @@ +//! D-NFR-6 lab probe: native operations over a [`SoaEnvelope`] borrow. +//! +//! **Mutation safety is the borrow; the mathematics is ndarray.** There is no +//! table type, no second ownership mechanism and no runtime guard here. +//! +//! - **Owner:** whoever holds the `Vec` (the mailbox). It mutates +//! through `&mut` and nothing else. +//! - **View:** [`NodeRowPacket`], the contract's existing `&self` borrow of +//! those rows. As [`SoaEnvelope`]'s own docs put it, a view cannot outlive +//! or alias the owner, and the owner's `&mut` cannot begin while any view is +//! live: "Rust's borrow rules ARE the inheritance; nothing is checked at +//! runtime." The packet is deliberately not `Clone`, so it cannot become a +//! second holder either. The `compile_fail` examples below pin all three. +//! - **Operations:** [`FacetFold`], one ndarray kernel per method, over the +//! packet's LE bytes at the 512-byte row stride and the 16-byte facet +//! stride every other SoA in lance-graph uses. Each method body is that one +//! call; there is no per-operation check to pay for. +//! +//! **Assumption:** every mutation goes through that same owner by ordinary +//! borrowing. The guarantee does not cover rows reachable through interior +//! mutability (`Cell`, atomics, a lock) or an `unsafe` write path beside the +//! owner; neither exists for `NodeRow`, whose fields are plain byte arrays. +//! +//! What this crate does not do, on purpose: it does not check frame +//! provenance or external authorization. Those are real questions (who sealed +//! these bytes; may this caller read them), but they belong to whoever hands +//! out the packet, once, and they are not a reason to re-check, per +//! operation, something the borrow checker already guarantees. The earlier +//! version of this crate did exactly that (a `Table` beside the SoA plus a +//! frame guard on each access) and is superseded; see the plan. +//! +//! # The borrow does the work +//! +//! The owner cannot write while a view is live (E0502): +//! +//! ```compile_fail,E0502 +//! use lance_graph_contract::NodeRowPacket; +//! let mut rows = soa_view_probe::fixture::rows(64); +//! let view = NodeRowPacket::new(&rows, 1); +//! rows[0].value[0] = 7; +//! let _ = view.rows().len(); +//! ``` +//! +//! A view cannot outlive its owner (E0597): +//! +//! ```compile_fail,E0597 +//! use lance_graph_contract::NodeRowPacket; +//! let view; +//! { +//! let rows = soa_view_probe::fixture::rows(64); +//! view = NodeRowPacket::new(&rows, 1); +//! } +//! let _ = view.rows().len(); +//! ``` +//! +//! A view cannot be duplicated into a second holder (E0599: no `clone`): +//! +//! ```compile_fail,E0599 +//! use lance_graph_contract::NodeRowPacket; +//! let rows = soa_view_probe::fixture::rows(64); +//! let view = NodeRowPacket::new(&rows, 1); +//! let second = view.clone(); +//! ``` +//! +//! The twin that must compile: once the view is gone the owner writes, and a +//! new view sees the new bytes. +//! +//! ``` +//! use lance_graph_contract::NodeRowPacket; +//! use soa_view_probe::{FacetFold, FacetSlot, fixture}; +//! let mut rows = fixture::rows(64); +//! let slot = FacetSlot::value(0).unwrap(); +//! let mut sel = [0u64; 1]; +//! for class in [fixture::CLASS_A, fixture::CLASS_B] { +//! rows[1].value[..4].copy_from_slice(&class.to_le_bytes()); +//! let view = NodeRowPacket::new(&rows, 2); +//! view.classid_mask(slot, class, &mut sel); +//! assert_eq!(sel[0] >> 1 & 1, 1); +//! } +//! ``` + +use lance_graph_contract::canonical_node::{ + GUIDS_PER_NODE, NODE_ROW_STRIDE, VALUE_SLAB_ROW_OFFSET, +}; +use lance_graph_contract::soa_envelope::SoaEnvelope; +use lance_graph_contract::{NodeRow, NodeRowPacket}; + +/// Bytes per facet: `classid(4) + 12-byte payload`. +pub const FACET_BYTES: usize = 16; + +/// One of the 32 facet slots of a 512-byte row (slot 0 is the key, slot 1 the +/// edge facet, slots 2.. the value slab). Built once; operations then trust +/// it, because a value of this type is in range by construction. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct FacetSlot(u8); + +impl FacetSlot { + /// The key facet. + pub const KEY: Self = Self(0); + + /// Slot `k` of the row, if it exists. + pub const fn new(k: usize) -> Option { + if k < GUIDS_PER_NODE { + Some(Self(k as u8)) + } else { + None + } + } + + /// Facet `k` of the value slab (bytes `32 + 16k ..`), if it exists. + pub const fn value(k: usize) -> Option { + Self::new(VALUE_SLAB_ROW_OFFSET / FACET_BYTES + k) + } + + /// Byte offset of this facet inside a row. + pub const fn offset(self) -> usize { + self.0 as usize * FACET_BYTES + } +} + +/// Native operations over a borrowed packet. Every method is one ndarray call +/// on [`SoaEnvelope::as_le_bytes`] at the row stride; masks have one bit per +/// row (`n_rows().div_ceil(64)` words). +pub trait FacetFold { + /// Rows whose facet `slot` carries `classid` in its first four bytes. + fn classid_mask(&self, slot: FacetSlot, classid: u32, out: &mut [u64]); + + /// Rows whose 16-byte facet `slot` agrees with `pattern` on every set + /// bit of `care`. + fn facet_match_mask( + &self, + slot: FacetSlot, + pattern: &[u8; 16], + care: &[u8; 16], + out: &mut [u64], + ); + + /// Sum of the twelve payload bytes (each `u8`, never widened) of facet + /// `slot` over the rows selected by `mask`. + fn payload_byte_sum(&self, slot: FacetSlot, mask: &[u64]) -> i64; +} + +impl FacetFold for NodeRowPacket<'_> { + #[inline] + fn classid_mask(&self, slot: FacetSlot, classid: u32, out: &mut [u64]) { + ndarray::simd::eq_u32_strided_to_mask( + self.as_le_bytes(), + slot.offset(), + NODE_ROW_STRIDE, + self.n_rows(), + classid, + out, + ); + } + + #[inline] + fn facet_match_mask( + &self, + slot: FacetSlot, + pattern: &[u8; 16], + care: &[u8; 16], + out: &mut [u64], + ) { + ndarray::simd::ternary_match_strided16_to_mask( + self.as_le_bytes(), + slot.offset(), + NODE_ROW_STRIDE, + self.n_rows(), + pattern, + care, + out, + ); + } + + #[inline] + fn payload_byte_sum(&self, slot: FacetSlot, mask: &[u64]) -> i64 { + // 12 bytes per row, at most 255 each: the i128 accumulator inside the + // kernel cannot leave i64 below ~3e15 rows, so `None` is unreachable + // for any slice that fits in memory. + ndarray::simd::masked_strided_group_sum( + self.as_le_bytes(), + slot.offset() + 4, + NODE_ROW_STRIDE, + self.n_rows(), + 12, + 1, + mask, + ) + .expect("12 u8 per row cannot overflow i64 for an in-memory slice") + } +} + +/// Owner-side write: facet `k` of the value slab of one row. The owner holds +/// `&mut`, so no view of these rows can be live while this runs. +pub fn set_value_facet(row: &mut NodeRow, k: usize, facet: [u8; FACET_BYTES]) { + let at = k * FACET_BYTES; + row.value[at..at + FACET_BYTES].copy_from_slice(&facet); +} + +/// Deterministic rows for tests and the cost example. +pub mod fixture { + use super::{FACET_BYTES, set_value_facet}; + use lance_graph_contract::canonical_node::TailVariant; + use lance_graph_contract::{EdgeBlock, NodeGuid, NodeRow}; + + /// Two classids (canon high: concept, low: app) for the value facet. + pub const CLASS_A: u32 = 0x0B05_0001; + /// The other classid. + pub const CLASS_B: u32 = 0x080B_0001; + + /// `n` rows: key minted V3, value facet 0 alternating between the two + /// classids by a fixed pseudo-random pattern, payload bytes varying. + pub fn rows(n: usize) -> Vec { + let mut s: u64 = 0x9E37_79B9_7F4A_7C15; + (0..n) + .map(|i| { + s ^= s << 13; + s ^= s >> 7; + s ^= s << 17; + let key = NodeGuid::mint_for( + TailVariant::V3, + CLASS_A, + 1, + 2, + 3, + 4, + 5, + (i % 0xFFFF) as u32, + ); + let mut row = NodeRow { + key, + edges: EdgeBlock::from_bytes(&[0; 16]), + value: [0; 480], + }; + let class = if s.is_multiple_of(3) { + CLASS_B + } else { + CLASS_A + }; + let mut facet = [0u8; FACET_BYTES]; + facet[..4].copy_from_slice(&class.to_le_bytes()); + for (j, b) in facet[4..].iter_mut().enumerate() { + *b = (s >> (j * 5)) as u8; + } + set_value_facet(&mut row, 0, facet); + row + }) + .collect() + } +} diff --git a/crates/soa-view-probe/tests/falsifiers.rs b/crates/soa-view-probe/tests/falsifiers.rs new file mode 100644 index 000000000..73869a7fd --- /dev/null +++ b/crates/soa-view-probe/tests/falsifiers.rs @@ -0,0 +1,141 @@ +//! D-NFR-6 runtime falsifiers. The compile-time ones (no write during a live +//! view, no view outliving its owner, no second holder) are the +//! `compile_fail` examples in `src/lib.rs`. + +use lance_graph_contract::NodeRowPacket; +use lance_graph_contract::canonical_node::NODE_ROW_STRIDE; +use lance_graph_contract::soa_envelope::SoaEnvelope; +use soa_view_probe::fixture::{CLASS_A, CLASS_B, rows}; +use soa_view_probe::{FacetFold, FacetSlot, set_value_facet}; + +const SIZES: [usize; 8] = [1, 2, 63, 64, 65, 1000, 4095, 4096]; + +fn slot0() -> FacetSlot { + FacetSlot::value(0).unwrap() +} + +fn facet(row: &lance_graph_contract::NodeRow) -> (u32, &[u8]) { + ( + u32::from_le_bytes(row.value[..4].try_into().unwrap()), + &row.value[4..16], + ) +} + +/// Row-loop oracle over the typed rows, not the bytes: count of `class`, +/// count matching (class, first payload byte < 128), payload byte sum. +fn oracle(r: &[lance_graph_contract::NodeRow], class: u32) -> (u64, u64, i64) { + let hit: Vec<_> = r.iter().map(facet).filter(|(c, _)| *c == class).collect(); + ( + hit.len() as u64, + hit.iter().filter(|(_, p)| p[0] < 128).count() as u64, + hit.iter() + .flat_map(|(_, p)| p.iter()) + .map(|&b| i64::from(b)) + .sum(), + ) +} + +/// FAILS IF: an operation through the packet disagrees with the same ndarray +/// call on the raw bytes, or with the row loop, at any size. +#[test] +fn packet_operations_equal_direct_ndarray_and_the_row_loop() { + for n in SIZES { + let owner = rows(n); + let view = NodeRowPacket::new(&owner, 1); + let w = n.div_ceil(64); + for class in [CLASS_A, CLASS_B] { + let mut pattern = [0u8; 16]; + pattern[..4].copy_from_slice(&class.to_le_bytes()); + let mut care = [0u8; 16]; + care[..4].fill(0xFF); + care[4] = 0x80; // first payload byte < 128 + + let (mut sel, mut hit) = (vec![0u64; w], vec![0u64; w]); + view.classid_mask(slot0(), class, &mut sel); + view.facet_match_mask(slot0(), &pattern, &care, &mut hit); + let got = ( + ndarray::simd::popcount_batch_u64(&sel), + ndarray::simd::popcount_batch_u64(&hit), + view.payload_byte_sum(slot0(), &sel), + ); + assert_eq!(got, oracle(&owner, class), "n={n} class={class:#x}"); + + let bytes = view.as_le_bytes(); + let at = slot0().offset(); + let (mut dsel, mut dhit) = (vec![0u64; w], vec![0u64; w]); + ndarray::simd::eq_u32_strided_to_mask(bytes, at, NODE_ROW_STRIDE, n, class, &mut dsel); + ndarray::simd::ternary_match_strided16_to_mask( + bytes, + at, + NODE_ROW_STRIDE, + n, + &pattern, + &care, + &mut dhit, + ); + let dsum = ndarray::simd::masked_strided_group_sum( + bytes, + at + 4, + NODE_ROW_STRIDE, + n, + 12, + 1, + &dsel, + ); + assert_eq!((sel, hit, Some(got.2)), (dsel, dhit, dsum), "n={n}"); + } + } +} + +/// FAILS IF: the fixture is vacuous (one class only, or the payload filter +/// keeps all or none). +#[test] +fn the_fixture_is_not_vacuous() { + let owner = rows(4096); + let (a, a_lo, _) = oracle(&owner, CLASS_A); + let (b, _, _) = oracle(&owner, CLASS_B); + assert_eq!(a + b, 4096); + assert!(b > 1000 && a > 1000, "a={a} b={b}"); + assert!(a_lo > 0 && a_lo < a); +} + +/// FAILS IF: a view taken after the owner wrote reports the old bytes. There +/// is no stale view to refuse: the old one cannot exist across the write. +#[test] +fn a_view_taken_after_a_write_sees_the_write() { + let mut owner = rows(130); + let (a, _, _) = oracle(&owner, CLASS_A); + let mut facet = [0u8; 16]; + facet[..4].copy_from_slice(&CLASS_B.to_le_bytes()); + for row in owner.iter_mut() { + set_value_facet(row, 0, facet); + } + let view = NodeRowPacket::new(&owner, 2); + let mut sel = vec![0u64; 3]; + view.classid_mask(slot0(), CLASS_A, &mut sel); + assert!(a > 0); + assert_eq!(ndarray::simd::popcount_batch_u64(&sel), 0); + view.classid_mask(slot0(), CLASS_B, &mut sel); + assert_eq!(ndarray::simd::popcount_batch_u64(&sel), 130); + assert_eq!(view.payload_byte_sum(slot0(), &sel), 0); +} + +/// FAILS IF: the facet geometry drifts: slot 0 is the key, value facet 0 sits +/// at byte 32, and there are exactly 32 slots of 16 bytes. +#[test] +fn facet_slots_are_the_canonical_16_byte_grid() { + assert_eq!(FacetSlot::KEY.offset(), 0); + assert_eq!(slot0().offset(), 32); + assert!(FacetSlot::new(31).is_some()); + assert!(FacetSlot::new(32).is_none()); + assert!(FacetSlot::value(29).is_some()); + assert!(FacetSlot::value(30).is_none()); + let owner = rows(3); + let view = NodeRowPacket::new(&owner, 7); + assert!(view.verify_layout().is_ok()); + assert_eq!(view.row_stride(), 512); + // Key facet: the minted classid is in key bytes 0..4. + let mut sel = [0u64; 1]; + view.classid_mask(FacetSlot::KEY, CLASS_A, &mut sel); + assert_eq!(sel[0], 0b111); +}