Skip to content

docs(quack): audit DuckDB vector techniques against DR landing zones - #1787

Merged
AdaWorldAPI merged 2 commits into
mainfrom
docs/d-dqm-duckdb-quack-landing-zone-20261026
Oct 26, 2026
Merged

AdaWorldAPI merged 2 commits into
mainfrom
docs/d-dqm-duckdb-quack-landing-zone-20261026

Conversation

@AdaWorldAPI

Copy link
Copy Markdown
Owner

Why

We were about to re-invent vector-execution mechanisms that DuckDB already documents and that Quack already audited in September. The missing job is narrower: harvest only useful implementation techniques against the now-existing DR global→local sealed index and D-DRF landing-zone economics, without creating a second executor or violating the zero-copy fold rule.

The existing DuckDB→V3 matrix and lane-fold integration are authoritative. Lane-fold item 8 explicitly has no timed DuckDB column yet. The existing D-DRF-P2 benchmark establishes warm build vs direct vs amortized cached fold, while #1774 corrects the original cache-line/page attribution.

What this PR changes

  • NEW DuckDB ↔ DR/Quack machine-room reuse plan.
  • INDEX a concise status entry in .claude/board/INTEGRATION_PLANS.md.
  • Source-pinned upstream DuckDB entrypoints: SelectionVector, Vector::Slice/SelCache, ToUnifiedFormat, scalar specializations and conjunction selection loops (read-only; snapshot 1641475ef1db08ec063631ff6a8f08d9d6736ae7).
  • Separate resident DR mask, sealed u16 pointer map, borrowed SoA and optional derived facet cache; don't call the latter zero-copy.
  • Four separately gated next steps: M1 timed existing DuckDB differential, M2 MAP/BORROW/GATHER break-even with cache-line/page controls, M3 one evidence-selected optimization, M4 KEEP/ADAPT/REJECT decision.
  • Pre-registered semantics, accounting and falsifiers: SQL K3, Bag/Distinct, generation/scope/index-seal, same work, cache/churn, no pair tuple objects, zero hot-path allocation.

Existing measurement, NOT reproduced here

At 4,096 slots over 65,536 rows, random map, full mask, warm: P2 reported direct 20,982 ns, focus build 23,256 ns, cached fold 670 ns, crossover 2 folds. These are host/fixture-bound; TLB attribution is not established, and no huge-page control is claimed.

Invariants

  • Quack lowers, mask-risc executes, ndarray provides compile-time peer SIMD backends.
  • Do not recreate DuckDB's SelectionVector as a second per-query index; DR's versioned pointer map is the one admitted global→local index.
  • Source and scope witnesses checked at binding/lease; derived value-cache materialization is explicit and disposable; MAP-only paths must not fetch values.
  • No DataFusion migration, new executor, P64/cognitive shader wiring, tuple decoding, duplicate relational identity, or runtime ISA dispatch.

Checks

  • Current repository sources and DuckDB entrypoints inspected.
  • Compared branch against main: only the plan and integration index change.
  • No Rust build or benchmark run (documentation-only PR; M1–M4 NOT RUN/NOT APPROVED).
  • No generated entry index change (no new .claude/board/entries/ item).

Decision requested: review this audit and select the first tiny follow-up PR (M1 timed differential) rather than opening a broad refactor.

@coderabbitai

coderabbitai Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 512054e9-548c-4300-b5ec-f341b74432e3
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Warning

Billing warning: we have not been able to collect payment for this subscription for more than 72 hours. Please update the payment method or pay any pending invoices in Billing to avoid service interruption.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@AdaWorldAPI
AdaWorldAPI marked this pull request as ready for review October 26, 2026 02:45
@AdaWorldAPI
AdaWorldAPI merged commit 87b2899 into main Oct 26, 2026
5 checks passed
@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review 🔄 Running since 2026-10-26T02:45:21.025882Z ebe057f Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

AdaWorldAPI pushed a commit that referenced this pull request Oct 26, 2026
Amend six open fruit rows. They are re-read at main 87b2899 and ndarray 195844bd:
- ISS-DURABLEREF: D-DR64/D-DRF focus cache (G5) and D-DTM-1 (EXO-148) now also wait on it.
- ISS-BF16: ndarray #374's FMA did not touch the per-element vec! in bf16_tile_gemm_16x16.
- ISS-FISHERZ: codec8 is out of scope for Fisher-Z, so the fix stays in bgz-tensor.
- ISS-CAUSAL-EDGE clippy: give the section a Status line.
- TD-BGZ: there are now 11 test files and CI still runs --lib.
- TD-STATUS-BOARD: Status line, plus a Haiku census of 34 ids. Two substring hits are collisions. The #1787 DuckDB plan defines no D-ids, only M0-M4.

Add two fruits:
- Low-hanging: D-MPF-2's fold numbers and its "scalar is 3x faster" verdict predate ndarray #376. That PR implemented D-MPF-2's own proposal.
- High-signal: the D-NAM-2 TFLite int8 leaf and codec8's phase-(f) lance-graph facade are the same PR. The PZL-1 "no new codec" tension has to be settled first.

Regenerate FRUITS.md (27 open, 1 closed) and record the Haiku runs in AGENT_LOG, with their hits and misses.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ArVbbq3DsToBM7r79zGeEf
AdaWorldAPI added a commit that referenced this pull request Oct 26, 2026
docs(D-DQM): DR64 ↔ DuckDB ↔ Lance 13 capability crosswalk (follow-up to #1787)
AdaWorldAPI pushed a commit that referenced this pull request Oct 26, 2026
Amend six open fruit rows. They are re-read at main 87b2899 and ndarray 195844bd:
- ISS-DURABLEREF: D-DR64/D-DRF focus cache (G5) and D-DTM-1 (EXO-148) now also wait on it.
- ISS-BF16: ndarray #374's FMA did not touch the per-element vec! in bf16_tile_gemm_16x16.
- ISS-FISHERZ: codec8 is out of scope for Fisher-Z, so the fix stays in bgz-tensor.
- ISS-CAUSAL-EDGE clippy: give the section a Status line.
- TD-BGZ: there are now 11 test files and CI still runs --lib.
- TD-STATUS-BOARD: Status line, plus a Haiku census of 34 ids. Two substring hits are collisions. The #1787 DuckDB plan defines no D-ids, only M0-M4.

Add two fruits:
- Low-hanging: D-MPF-2's fold numbers and its "scalar is 3x faster" verdict predate ndarray #376. That PR implemented D-MPF-2's own proposal.
- High-signal: the D-NAM-2 TFLite int8 leaf and codec8's phase-(f) lance-graph facade are the same PR. The PZL-1 "no new codec" tension has to be settled first.

Regenerate FRUITS.md (27 open, 1 closed) and record the Haiku runs in AGENT_LOG, with their hits and misses.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ArVbbq3DsToBM7r79zGeEf
AdaWorldAPI pushed a commit that referenced this pull request Oct 26, 2026
…#1802

- PR_ARC_INVENTORY: compact entries for #1772, #1774–#1787, #1794–#1803,
  recorded from merge commits, changed files and each PR's own Status
  line. #1773 and #1793 are board-hygiene-only and get none.
- INTEGRATION_PLANS: one index row for thirteen plans merged without one
  (D-NAM-1..5 + PZL, DT64-1..5 + QB pointer, D-ONNX-INTAKE-0).
- LATEST_STATE: "main after #1803" section.
- d-onnx-relalg-cap-v1: OP-ONNX-2 status cell names #1802's Burn
  onnx-ir vs Tract proposal; still OPEN.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VncKoyG6JkpsoqhqTGTv3f
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant