Skip to content

docs(D-DQM): DR64 ↔ DuckDB ↔ Lance 13 capability crosswalk (follow-up to #1787) - #1788

Merged
AdaWorldAPI merged 6 commits into
mainfrom
claude/dqm-lance13-crosswalk
Oct 26, 2026
Merged

AdaWorldAPI merged 6 commits into
mainfrom
claude/dqm-lance13-crosswalk

Conversation

@AdaWorldAPI

Copy link
Copy Markdown
Owner

Follow-up to merged #1787. Documentation only: no runtime code, no new crate, no executor, no DataFusion, ONNX or cognitive-shader change.

What changed

  • .claude/plans/duckdb-dr-quack-machine-room-reuse-v1.md: the M2 "Lance 13 masks the READ too" input note, plus a new Appendix A, appended. Nothing existing was rewritten (238 lines added, 0 removed).
    • A.0 Explicit corrections:
      • the input note's "the handoff must stay a fold" is wrong. Building Lance's mask is an ingress copy outside the fold, and Lance re-serialises it (scanner.rs:3852-3866).
      • September matrix R6 is superseded by Quack's SQL three-valued validity planes (Filter::sql_where).
      • R5's evidence (the hex tenant) is superseded by the Moore tenant.
      • Two different gaps are both called "G8" in different files.
    • A.1 Coordinate domains: a DR ordinal, a Lance _rowid, a _rowaddr and an output batch position are four different spaces. Two lab coordinate maps exist today with different staleness witnesses.
    • A.2 Seventeen DuckDB capabilities, each with a source location at 1641475, our owner by file and symbol, a status, the falsifier, the duplication risk and a decision. Also a re-audit of September gaps G1–G8:
      • G1 (u8 only), G2 (unsigned only), G4 and G6 are CLOSED;
      • G3 and G8 are OPEN;
      • G5 stays do-not-build;
      • G7 is deliberately absent.
    • A.3 Lance 13 masked read as an existing layer (H14c D-HOLO H14c: thought-first positional reads, Hamming presence and Lance masked I/O #1573, H14d D-HOLO H14d: logic → address mask → masked read → terminal (C only), measured on v4 #1575), with the byte results explained from Lance's own source:
      • narrow columns are read in mini-block chunks of at most 4,096 values;
      • a whole-fragment selection becomes one range;
      • a scattered 1 % mask touches every chunk, so it reads 1.00× the bytes;
      • clustered and narrow-aperture masks touch few chunks.
    • A.4 End-to-end map and cost ledger across five paths: MAP-only, resident BORROW, GATHER plus cached folds, Lance masked fetch, and Lance fetch followed by a DRF cache.
    • A.5 Missing-feature register, split three ways:
      • missing code: u16 and signed i64 compares, G3, G8;
      • missing wiring: one coordinate authority, DR ordinal → _rowid composition, bulk mask construction, u8/u64 compares in the IR;
      • intentional exclusions.
    • A.6 Probes P1–P5 with kill conditions. One recommended next PR: P1, offline, comparing a chunk-touched estimator with the bytes already recorded in the committed H14c/H14d CSVs. No implementation is recommended.
  • .claude/board/entries/2026-10-26-coresearch-lance13-masked-read-dr64.md: the co-research exploration map (premise gate, five scouts, three reviewers, a verdict per idea, and what was not searched). The entries index is regenerated.
  • .claude/board/INTEGRATION_PLANS.md: one entry prepended.

Open, for the operator

  • S-INGRESS. Converting resident mask words into Lance's RowAddrMask is a copy into a Lance-owned container, and Lance copies it twice more. The earlier "Roaring under KEEP" ruling covers our own containers; whether it extends to a foreign-owned one is not ruled. Until it is, nothing on that path is called zero-copy or a fold, and the DR → Lance wiring rows stay gated on this decision.
  • One coordinate → _rowid authority. Choose the single authority: the dataset-version witness, the index-seal witness, or one that records both. A third map would add a third authority.
  • Storage-read timing. The millisecond exception covers cold compiles and oracle arms only, so storage results are reported in bytes, ranges and rows scanned, not wall time.

Checks

  • Every claim names a file and symbol. Claims not read first-hand are marked UNVERIFIED, including lance-io's 64 KiB merge window and Lance's 8,192-row zone-map granularity.
  • Spot-checked against source:
    • mask_any has no early exit;
    • mask-risc Pred has no u8/u16 or ordered-u64 variant;
    • Quack's gate choice (lib.rs:2280) and fused slot assignment (:2369);
    • Pred::Range has no gated variant;
    • Lance's whole-fragment range (filtered_read.rs:1198-1209) and mini-block limits (miniblock.rs:19,21);
    • DuckDB TryLateMaterialization and RowGroup::CheckZonemap at the pinned commit.
  • Local gates: entries_index --check OK; citation_decay --since origin/main reports 0 new decays (a line citation was replaced with a section anchor); append_only_gate OK; fruits_index --check OK; the supersession index is regenerated and unchanged.

Generated by Claude Code

Records the read-side mask, the measured H14c byte behaviour (uniform vs
clustered), the word-level handoff via RoaringBitmap::from_lsb0_bytes, and
the _rowid/stable-id caveat.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X6y3drwKSE2zSgoexheLFX
…lk (#1787 follow-up)

Source-backed crosswalk of 17 DuckDB capabilities (pinned @1641475) against
DR64 / Quack / mask-risc / ndarray / DRF / Lance 13; re-audit of September
gaps G1-G8; Lance 13 masked read (H14c #1573, H14d #1575) recorded as an
existing layer with its byte behaviour explained from Lance source; coordinate
domain table; cost ledger over five paths; missing-feature register; probes
P1-P5 with one recommended (offline P1). Explicit corrections: the M2 input's
"handoff must stay a fold", September R6 and R5. Coresearch exploration map in
entries/. Documentation only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X6y3drwKSE2zSgoexheLFX
@coderabbitai

coderabbitai Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: f451f0f7-44df-4fdb-8152-ab5a4709e896
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Warning

Billing warning: we have not been able to collect payment for this subscription for more than 72 hours. Please update the payment method or pay any pending invoices in Billing to avoid service interruption.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Append-only amendment to Appendix A. Records today's live PostgreSQL 16.15
parity runs (K3 WHERE, NULL operands, aggregates, contract normalize_kleene),
the remaining SQL gaps for the scan use case, the ALLOW(K) == BLOCK(U\K)
equivalence and its K3 counterexample (BLOCK(K(NOT P)) admits UNKNOWN rows),
Lance 13's block universe and fail-open asymmetry, six BLOCK admission gates,
and the chain verdict (semantics legal, ALLOW blocked by wiring, BLOCK by
gates). Next step unchanged: P1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X6y3drwKSE2zSgoexheLFX
…n, P1 unchanged)

Short delta over A.0-A.8, no rerun: SQL verified/missing status, ALLOW
wiring blockers, why BLOCK stays gated, PgLance as a tuple-conversion
reference arm only (Lance 1.0, no pushdown, no mask), and two DuckDB
rules for the native ONNX/TFLite polyfill (table amortisation guard;
complement only on the no-NULL path). Op-shape -> native-route table
cites existing D-TFLP / D-NAM-PZL / D-NAM-3 lines. P1 stays next.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X6y3drwKSE2zSgoexheLFX
…tence corrected

Documentation only. Records that Lance row masks (ALLOW/BLOCK via
with_row_addr_prefilter) apply only to reading sealed Lance versions;
the lancedb 0.40 MemWAL has an internal PK block list, no external mask
input and no stable _rowid. Writer ownership (MailboxSoA, BatchWriter,
CycleWriter try_submit/seal_cycle) verified in source and left unchanged.
A.9's DuckDB sentence now says amortisation only; the LUT/SIMD crossover
stays a separate D-TFLP-2 measurement. P1 remains next.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X6y3drwKSE2zSgoexheLFX
P1 reads only committed H14c/H14d Lance measurements and seeded masks;
no M1 output feeds it. M1 status unchanged: parity passed, timing
comparison inconclusive (boundaries differ).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X6y3drwKSE2zSgoexheLFX
@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review 🔄 Running since 2026-10-26T17:47:47.298755Z c33eb05 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@AdaWorldAPI
AdaWorldAPI merged commit 10291a8 into main Oct 26, 2026
5 checks passed
@coderabbitai

coderabbitai Bot commented Oct 26, 2026

Copy link
Copy Markdown

Add Carrot credits or activate Agent usage billing to use Autofix

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants