Skip to content

D-GLOBAL-WHERE P0: direct-routed virtual SoA views across selections + DuckDB parity gate (lab) - #1943

Merged
AdaWorldAPI merged 4 commits into
mainfrom
ccr-85acaff0-oszq40-global-where
Nov 3, 2026
Merged

AdaWorldAPI merged 4 commits into
mainfrom
ccr-85acaff0-oszq40-global-where

Conversation

@AdaWorldAPI

Copy link
Copy Markdown
Owner

Lab only. No production change. dt64, Quack, mask-risc and ndarray are used unchanged. No Gather, no new identity table, no witness store, no per-object allocation. The full write-up is in .claude/probes/2026-11-03-d-global-where/REPORT.md.

Falsifier: 21/21 (probe/ → falsify)

Two guards were disabled to prove they are load-bearing; both checks then went red:

  • F4: join by local index instead of the DR route.
  • F6: leave the presence bit set.

What the checks establish:

  • Routing (F1–F5, F18): production DR routing (LocalMap, MapTable::{map, map_at, locate}, Frame) is exact and bidirectional. It reconciles two selections that hold the same global id at different local positions.
    • Joining the two selections by local index gives 32; the oracle and the DR-aligned path both give 14.
    • A cross-selection relation evaluates correctly across two DR256 tiles.
  • Presence change (F6, F7): deleting an edge leaves the identity resolvable and turns the answer false. Deleting an unrelated edge changes nothing.
  • XOR consistency (F8, F9): the forward plane XOR the transposed reverse plane is 0. A single flipped bit is found at exactly its edge.
  • Frames (F10, F11):
    • An old frame replays its old layout.
    • A foreign frame, a future version and an ambiguous id are all refused.
  • Two methods, one selection (F12, F13): where_dr3::LocalSelection projects bit-identically for two methods. Its SchemaChanged on a second method is therefore a coupling, not a correctness need. Scope refuses on its own.
  • No allocation (F14, F15): the folds allocate 0 bytes in 0 calls.
  • Boolean vs bag (F17): Boolean and bag answers differ (156 vs 1095), and both equal their oracles.

DuckDB 2.0.0 parity gate (benchmark gate preserved here)

Setup:

  • 8 workloads × {DR64, DR256, multi-tile 16×DR256} × {sparse, medium, dense, skewed}.
  • DuckDB uses its normal optimizer at threads 1 and 4, with PREPARE/EXECUTE and 201 warm runs. The Python call floor is subtracted.
  • Our side is single-threaded.

Answers: 336/336 agree (duckdb/compare.py exits 1 on any mismatch).

Warm, best arm: 12/0/0 W/P/L for every workload. This does not establish general superiority.

  • DuckDB's fixed cost per query is about 600 µs: 703 µs for a filtered count over 65,536 rows, 12,106 µs over 10M rows. That cost dominates most warm rows at these sizes (domain ≤ 4,096 nodes).
  • Rows where the gap is work avoided rather than overhead:
workload ours DuckDB (t=1) why
W4 3-hop, multi_dense 131 µs 1,500,724 µs —
W4 3-hop, multi_skewed 89 µs 57,066 µs DuckDB builds 3,726,779 join rows for a 1,164 answer (EXPLAIN ANALYZE in results/explain/)
W3, multi_dense 7.7 µs 29,097 µs —
W6 bag count, multi_dense 119 µs 22,726 µs —

Losses (all recorded): every loss is caused by addressing; none is caused by the fold.

  • Cold (prep + first query) on every multi_* dataset: 8.7–520 ms ours against 1.9–35 ms DuckDB. DuckDB's load (40–117 ms) is excluded from its cold figure.
  • W7 per-call routing: 650–970 µs, against 5 µs when the routing is done once and reused.
  • In both cases the cost is MapTable::locate, which reads every listed tile per id.
  • Updates on multi_dense take 4.8 ms. That beats DuckDB's DELETE but is over the workspace's 1 ms gate.

Plan choice is by measured cost:

  • W3: csr wins everywhere.
  • W4: csr wins on small or sparse data; selection-first frontier masks win on medium, dense and skewed data.
  • Tile-first never won.

Remaining parity gap: scale ≥ 10M rows is unmeasured on our side.

Verdict

  • ALREADY WORKS: DR routing, frames, presence change, XOR consistency, Quack WHERE, allocation-free folds.
  • WIRE NOW (lab-proven): route a selection into the relation frame once per frame and reuse it.
  • OWNER DECISION:
    • May one selection serve several MethodRefs under one Scope (EXO-46)?
    • Is a global → tile index wanted beside locate's ambiguity scan?
    • Production has no relation-presence type outside Cube.
  • PROBE: ≥ 10M-row scale; building the plane via Cube::new_tiled.
  • PARK: EXO-73 recursion, EXO-108 Bag, physical prefetch.
  • SKIP: duplicate witnesses, object allocation, a second routing store, mask transfers between programs, new Gather operators.

Files (all under .claude/probes/2026-11-03-d-global-where/)

  • SPEC.md, REPORT.md
  • probe/: its own workspace, binaries falsify and bench
  • duckdb/: bench_duckdb.py, compare.py
  • results/: logs, JSON, compare tables, EXPLAIN ANALYZE output
  • Board entry .claude/board/entries/2026-11-03-d-global-where-p0.md, with the index regenerated.

Not included: benchmark commit a84464205 (D-DR-FOLD), #1941 and #1942. No merge requested.


Generated by Claude Code

@coderabbitai

coderabbitai Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

Important

  • Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 1c79fb02-6387-46db-90d9-0351f76da891
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

CodeRabbit Verify

  • Run CodeRabbit Verify to test the changes in this pull request.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-03T18:27:05.549204Z 65c89b3 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 65c89b3264

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@AdaWorldAPI
AdaWorldAPI force-pushed the ccr-85acaff0-oszq40-global-where branch from d18651f to 771ec92 Compare November 3, 2026 18:31
Spec for the cross-selection routing falsifier and the DuckDB parity gate.
No production change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAqKMNZqKXSaeb9g2ypywF
Cross-selection routing through existing dt64 MapTable/LocalMap, Quack
WHERE, ndarray mask folds. Lab only; no production change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAqKMNZqKXSaeb9g2ypywF
Falsifier 21/21 (F4, F6 disable-verified). DuckDB 2.0.0 gate: 336/336
answers agree; warm best arm 12/0/0 per workload but dominated by DuckDB's
fixed per-query cost except the multi-hop rows; cold multi-tile losses from
per-id MapTable::locate. W8 label fixed (after-delete answer is W8_after).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAqKMNZqKXSaeb9g2ypywF
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants