diff --git a/.claude/board/INTEGRATION_PLANS.md b/.claude/board/INTEGRATION_PLANS.md index 7340eeae2..9123efbc9 100644 --- a/.claude/board/INTEGRATION_PLANS.md +++ b/.claude/board/INTEGRATION_PLANS.md @@ -1,3 +1,9 @@ +## 2026-10-26 — DuckDB ↔ DR/Quack machine-room reuse → `.claude/plans/duckdb-dr-quack-machine-room-reuse-v1.md` + +**Status: M0 PROPOSAL / DOCS ONLY** (this draft PR). [Plan](../plans/duckdb-dr-quack-machine-room-reuse-v1.md). Extends the *existing* September `duckdb-to-v3-translation-matrix-v1.md`, rather than re-harvesting or porting DuckDB. Existing authority: Quack lowers; mask-risc executes; ndarray supplies SIMD peers; DR/D-DRAV own sealed global→local `u16` map, resident masks and optional independently accounted D-DRF facet cache. DuckDB's vector/dictionary/executor source is a read-only precedent, not a new ABI or compute owner. + +**Stages proposed, NOT RUN / NOT APPROVED:** M1 timed arm in existing DuckDB semantic differential; M2 DRF MAP/BORROW/GATHER amortization crossover with source/scope/seal and distinct-line/page controls; M3 one narrow, separately gated implementation *only if* a real difference survives full accounting; M4 explicit KEEP/ADAPT/REJECT decision. Continue the lane-fold capstone's open DuckDB timing item. The 2026-10-24 P2 figures remain host-bound and the 2026-10-25 TLB analysis does not establish huge-page causality. No Quack/mask-risc/ndarray/DataFusion runtime change in M0. + ## 2026-10-25 — D-NAM native algebra method × DTO dialect capstone → `.claude/plans/d-nam-native-algebra-dto-capstone-v1.md` **Status:** D-NAM-0 PROPOSAL / DOCS ONLY; staged D-NAM-1..5 implementation **engineering specifications**, not yet approved/run. Source-first [handoff](../prompts/d-nam-native-dto-claude-code-handoff.md), [EXO-152](https://linear.app/exo-red/issue/EXO-152), append-only [EXO-50](https://linear.app/exo-red/issue/EXO-50). diff --git a/.claude/plans/duckdb-dr-quack-machine-room-reuse-v1.md b/.claude/plans/duckdb-dr-quack-machine-room-reuse-v1.md new file mode 100644 index 000000000..e88dd490b --- /dev/null +++ b/.claude/plans/duckdb-dr-quack-machine-room-reuse-v1.md @@ -0,0 +1,138 @@ +# DuckDB ↔ DR/Quack: machine-room reuse without a second executor + +> **Status: PROPOSAL / DOCUMENTATION ONLY (2026-10-26).** +> No runtime code, new ABI, benchmark run, source copy, optimiser decision, or board admission in this PR. +> This is a focused continuation of the existing DuckDB harvest, **not** a new DuckDB translation matrix. +> **Owners:** Quack lowering → `lance-graph-quack`; mask execution → `lance-graph-mask-risc`; SIMD polyfills → AdaWorldAPI/ndarray; address-first focus and its scoped derived cache → existing D-DRF/D-DRAV owners. + +## Question + +Given that DR already supplies a canonical, sealed global-to-local ordinal mapping and resident membership/validity planes, **which of DuckDB's mature vector execution techniques can improve Quack without reintroducing a per-query SelectionVector, tuple materialization, duplicate storage ownership, or an interpreter?** + +This is **not** "build DuckDB in Rust." The test is whether any DuckDB-inspired specialization improves the **existing DR mapping + mask-risc terminal + ndarray** path when compared on the same semantic result and the same accounted work. + +## Source-first inventory: do not re-derive + +### Existing local authority + +| Source | What it already establishes | Consequence | +| --- | --- | --- | +| [DuckDB → V3 matrix](../plans/duckdb-to-v3-translation-matrix-v1.md) (§2 R1–R8, E*, A*) | September source-level KEEP/ADAPT/ELIMINATE decisions with falsifiers; deliberately no new executor | Extend only rows affected by DRF map/focus residency and *new* upstream evidence. Do **not** duplicate its 80-KiB harvest. | +| [Quack](../../crates/lance-graph-quack/src/lib.rs), [binder](../../crates/lance-graph-quack/src/bind.rs) | Bound numeric IR only; `lower` vs `lower_fused`; resident SQL K3 planes; no second evaluator | All implementation candidates must lower to existing mask-risc operations or explicitly refuse. | +| [mask-risc](../../crates/lance-graph-mask-risc/src/exec.rs), [IR](../../crates/lance-graph-mask-risc/src/ir.rs) | One execution owner, terminal folds, gated word skips | Don't add runtime compute to Quack. Don't claim fused ternlog skips under a gate. | +| [DuckDB differential](../../crates/lance-graph-quack/tests/duckdb_differential.rs) and [fixture/oracle](../../crates/lance-graph-quack/tests/duckdb/README.txt) | SQL semantic oracle + allocation and intermediate-byte accounting already exist | **Add timing, not a second correctness oracle.** Never hand-edit `cases.tsv` expected values. | +| [Lane-fold capstone](../../.grok/board/CLAUDE_LANE_FOLD_CAPSTONE.md), [integration](../../.grok/board/LANE_FOLD_INTEGRATION.md) | Timed DuckDB column is an **explicit open item**; 64k ordered lane, `u16` index, borrowed aperture, terminal collapse and skip gates | The DuckDB comparison must reuse this harness/cost model. Do not inflate a word-fold into a join/plane timing. | +| [DRF-P2 measured entry](../board/entries/2026-10-24-drf-p2-breakeven.md), [TLB correction](../board/entries/2026-10-25-drf-tlb-probe.md) | Gather-build and repeated cached fold are measured; the original 1,024-versus-65,536 comparison changed cache-line count | Reuse the actual P2 fixtures; report distinct lines/pages and randomized visit order before attributing a speedup. | +| [D-DRAV address-first plan](../plans/d-dr-address-first-value-demand-v1.md) and [DRF code](../../crates/drf-focus-probe/src/lib.rs) | Sealed map can be useful **without values**; optional gathered facet column is a derived disposable cache, not authority | Keep MAP / BORROW / GATHER distinct. Do not make materialization mandatory or call copied facet bytes zero-copy. | + +### DuckDB source pointers (read-only, pinned snapshot) + +Snapshot: [duckdb/duckdb@`1641475ef1db08ec063631ff6a8f08d9d6736ae7`](https://github.com/duckdb/duckdb/tree/1641475ef1db08ec063631ff6a8f08d9d6736ae7), captured 2026-10-26. These are **source entrypoints**, not evidence that any candidate already wins DR benchmarks. + +- [SelectionVector](https://github.com/duckdb/duckdb/blob/1641475ef1db08ec063631ff6a8f08d9d6736ae7/src/include/duckdb/common/types/selection_vector.hpp): index-carrier shape, owned buffer, `Slice`. +- [Vector::Slice / SelCache / Dictionary](https://github.com/duckdb/duckdb/blob/1641475ef1db08ec063631ff6a8f08d9d6736ae7/src/common/types/vector.cpp#L209-L270): indirection composition and reuse. +- [Vector::ToUnifiedFormat](https://github.com/duckdb/duckdb/blob/1641475ef1db08ec063631ff6a8f08d9d6736ae7/src/common/types/vector.cpp#L461-L473) and [vector.hpp](https://github.com/duckdb/duckdb/blob/1641475ef1db08ec063631ff6a8f08d9d6736ae7/src/include/duckdb/common/types/vector.hpp#L117-L127): normalized access to data, selection and validity. +- [ScalarExecutor](https://github.com/duckdb/duckdb/blob/1641475ef1db08ec063631ff6a8f08d9d6736ae7/src/include/duckdb/common/vector_operations/scalar_executor.hpp#L129-L453): flat/constant/generic specialization and nullable/selection loops. +- [Conjunction execution](https://github.com/duckdb/duckdb/blob/1641475ef1db08ec063631ff6a8f08d9d6736ae7/src/execution/expression_executor/execute_conjunction.cpp#L61-L144): selection permutations, intermediate buffers and runtime-adaptive ordering. + +**License boundary:** DuckDB is MIT licensed. A source-inspired Rust reimplementation still requires independent semantic tests and attribution where copying expression or code; this plan does not vendor or copy C++. + +## Representation: map ≠ mask ≠ cached values + +```text +source identity + version + scope + index seal (checked at bind/lease) + │ + ▼ + sealed DR pointer map ord[slot] : u16 + │ + ┌─────────────┴────────────┐ + ▼ ▼ + map-only membership value demanded + Boolean / relation mask ├ BORROW: SoA[ord[slot]] + no value read └ GATHER: derived focus column + │ │ + └─────────────┬────────────┘ + ▼ + existing Quack lowering + ▼ + mask-risc → ndarray terminal +``` + +- **Selection role:** packed DR resident mask (`u64` words) already says which slots participate. No reified `Vec` on a fold path. +- **Dictionary role:** the sealed `u16` global-to-local permutation is the **one admitted pointer index**. It is not an independently minted relational identity. Duplicate mapped slots remain duplicates when Bag semantics require them. +- **Cached value role:** DRF's optional `Focus::facet_col` stores derived values and is **not** the map. Account for its `FOCUS_BYTES`; require matching `SourceId`, map, scope and facet lease. Eviction must preserve map-only operations. +- **SQL role:** valid/present/unknown are **not interchangeable**. SQL K3, Bag/Distinct, `NULL`, order and multiplicities must match the independent oracle before any timing. +- **No unearned guarantees:** consistent ordinals follow only from attested source/bind invariants; stale seals, wrong scope and cross-version data must refuse before the hot path. + +## Candidate harvest: inspect, map, falsify + +| Upstream technique | Existing DR owner | Only plausible transfer | Kill/refusal criterion | +| --- | --- | --- | --- | +| SelectionVector + dictionary slicing | DR mask + sealed `u16` map | Compare cost of composing/using resident maps against directly borrowed SoA | Any proposal adding a second per-query row-id vector or losing duplicates/order is rejected | +| `SelCache` dictionary-on-dictionary reuse | D-DRF focus lease / D-DRAV address map | Separate **map reuse** from optional **facet cache** and measure each | Wrong generation/scope/facet cache hit, cache bytes hidden, or cache required for map-only work | +| Specialized flat/constant/generic loops | mask-risc and ndarray | Identify exactly one missing specialization; add *only* with native + AVX2/AVX-512/NEON/WASM/scalar polyfill discipline | Runtime ISA dispatch, Quack evaluator or a regression on existing paths | +| Adaptive conjunction ordering | Quack `Filter::and_by_skip`, gate input | Compare sparse-gated predicate order under **real dead-word/dead-tile fractions** | `lower_fused` claims under-skip, resident plane reordered, or gain disappears on shuffled data | +| Chunk and validity handling | borrowed SoA + resident planes | Borrow DuckDB's experiment design, not its arbitrary 2,048-element width | Extra allocation, copied tuples, K3/NULL discordance | +| Aggregation / terminal | `Agg` + mask-risc / ndarray fold | Verify terminal collapse opportunities already registered in lane-fold | Duplicated evaluator, widened `u16` address, hidden hash/row output | + +A DuckDB component being similar is **not** evidence that a matching DR implementation is faster. Record `already exists / missing / different / refused` against named code paths, not narrative analogies. + +## Follow-up PR ladder (none approved or executed by this document) + +**M0: This PR — baseline and decision register (docs only).** +- [x] Locate the existing matrix, Quack differential, lane-fold open item, D-DRF-P2 and D-DRAV. +- [x] Identify pinned DuckDB headers/implementation seams. +- [x] Define the one performance question: `MAP` vs `BORROW` vs disposable `GATHER`, and amortized fold, under equivalent semantics. +- [ ] Verify existing matrix verdicts against *new* DRF behaviors before changing code. Every candidate gets a named existing owner. + +**M1: Differential timing arm (small, separate PR).** +- Extend the **existing** `lance-graph-quack/tests/duckdb_differential.rs`/oracle workflow; do not fork cases or retype expected answers. +- Untimed checksum/parity first. Time native Quack after binding/scratch creation and DuckDB separately on the same logical fixture, projection, output cardinality and materialization boundary. +- Report DuckDB planning/binding and any Python/FFI conversion **outside** the fold separately; do not compare whole-API latency against an in-cache word operation. Do not put timing assertions in correctness tests. +- Preserve `alloc_bytes_exec`, `pair_relation_bytes`, `population_state_bytes`, `fixture_view_bytes`, `scratch_bytes` and add measured `t_compile_or_bind_ns`, `t_execute_ns` and oracle checksum. +- **Exit gate:** parity on all registered cases; no edits to committed expected values except re-generation by documented oracle; timings have identical measured boundaries and units. Otherwise STOP without a performance claim. + +**M2: DRF landing-zone threshold crossover (small, separate PR).** +- Reuse `crates/drf-focus-probe/examples/p2_breakeven.rs`, the AV1 map-only path and `tlb_probe.rs`, not a new production API. +- Three arms: (A) MAP-only membership/filter; (B) direct BORROW through sealed map; (C) derived GATHER once + N cached folds. Include `N ∈ {1,2,4,8,64}`, mask density `{1, 1/4, 1/64}`, clustered/balanced/shuffled maps and repeated ordinals. +- Equal source/version/seal/scope, same values and mask, identical K3/Bag target; if map-only has no numerical value answer, compare only a common complete Boolean query, **not** a numerical fold. +- Measure first build and subsequent reuse separately, warm/cold as reported, count distinct cache lines and 4-KiB pages, visit-order histogram, memory bytes and eviction/churn. Do not relabel correlation as TLB causality or quietly include an extra cache copy. +- For each workload, infer crossover from **observed** costs: `N* = min {N >= 1 | C_build + N·C_cached < C_first_direct + (N-1)·C_warm_direct}`; report "no crossover" if observed costs do not meet it. Do not hardcode the prior 1–3 threshold as universal. +- Add invalid-map, stale generation, wrong scope, changed facet and duplicate-ordinal tests. A derived cache must remain disposable. +- **Exit gate:** exact parity and bytes accounted, threshold seen in measured end-to-end path with can-fire negative controls, not hand-picked `get()` microbenchmarks. + +**M3: One targeted optimization only if M1/M2 expose a real gap (separate PR).** +- Candidate is one existing backend specialization or a plan-level reuse decision. Prioritize **no additional work** (reuse borrowed map, fuse mask terminal, avoid materialization). +- Keep Quack strictly as a lowerer; execute in mask-risc/ndarray; no new ubiquitous vector API, dynamic dispatch or hash/tuple fallback. +- Check every supported backend's peer implementation, scalar oracle, SQL K3, Bag/Distinct, 0 hot-path allocations, no pair relation, no silent `u32→u16` cast. +- **Kill:** improvement disappears when preparation, cache eviction and comparable source/target work are included; or any correctness/refusal gate fails. Revert candidate rather than widen architecture. + +**M4: Decide / document (not a default implementation).** +- KEEP / ADAPT / ELIMINATE / REJECT for each audited DuckDB technique. +- If there is no independent benefit beyond DRF-P2 and lane-fold, **close with evidence; no new crate, planner layer, or migration.** + +## Pre-registered reporting discipline + +Rows to report: `case, semantics, source_id, map_id, scope, n_slots, n_distinct_ordinals, n_distinct_lines, n_distinct_pages, mask_density, visit_order, N, map_bytes, facet_cache_bytes, scratch_bytes, fixture_view_bytes, pair_relation_bytes, alloc_bytes_exec, checksum, t_bind_ns, t_build_ns, t_first_fold_ns, t_reuse_fold_ns, total_ns, crossover`. + +- Compare the **same** physical work per arm; no "cached local fold" vs "uncached remote SQL" headline. +- Zero is a measured count for the defined boundary, not a claim that upstream preparation was free. +- Per-operation costs in ns; cold/preparation costs in µs where necessary and clearly marked **not fold gates**, following `CLAUDE.md`'s existing exception. Aggregate latencies cannot be passed off as a resident fold. +- Keep the already measured P2 figures with host/cache qualifications: at 65,536 rows/random/full mask, warm build 23,256 ns, cached fold 670 ns, direct warm 20,982 ns; warm crossover 2 folds. This is an **existing lab fixture**, not an expected universal ratio. +- The DRF-TLB page-locality probe isolated a page-count effect under controlled line counts, **not** the full cause of the original build gap. Huge-page causation remains unproven. + +## Scope firewall / no drift + +- Do **not** modify the DuckDB matrix's old verdicts by implication; amend them with source/date and an explicit reason only after a counterexample. +- Do **not** touch DataFusion grace-period planner, OGAR semantics, ONNX/TFLite, cognitive-shader-driver, P64, or a distributed KV backend in this arc. +- Do **not** treat `DuckDB POSITIONAL JOIN` as proof of a shared, sealed physical ordinal space; our admission contract is independently established in DR. +- Do **not** call row reconstruction, tuple lists, pointer-table copies, or a copied facet cache zero-copy. DR's legal global→local inversion materializes the `u16` pointer map; cache materialization is a distinct, audited concession of D-DRF. +- Do **not** substitute a timed `COUNT` for a requested Bag/Set relation with different semantics. +- No M1–M4 authorization, production implementation or benchmark result is implied by merging this docs PR. + +## References / breadcrumbs + +- [Lane-fold reference](../../.grok/board/LANE_FOLD.md) and [bound computation](../../.grok/board/BOUND_COMPUTATION.md). +- [Quack DuckDB source harvest](../plans/duckdb-to-v3-translation-matrix-v1.md), [DuckDB oracle files](../../crates/lance-graph-quack/tests/duckdb/). +- [DRF focus](../../crates/drf-focus-probe/src/lib.rs), [map-only membership](../../crates/drf-focus-probe/src/membership.rs), [DRAV plan](../plans/d-dr-address-first-value-demand-v1.md). +- [P2 measurement](../board/entries/2026-10-24-drf-p2-breakeven.md), [page-control correction](../board/entries/2026-10-25-drf-tlb-probe.md), PRs [#1774](https://github.com/AdaWorldAPI/lance-graph/pull/1774), [#1775](https://github.com/AdaWorldAPI/lance-graph/pull/1775).