Skip to content

Zero-copy joins: relations are lanes, a side is an index map $i (mork-pathmap-probe, crepe-plan-probe) - #1659

Merged
AdaWorldAPI merged 14 commits into
mainfrom
ccr-0455e606-wmtsor
Oct 21, 2026
Merged

AdaWorldAPI merged 14 commits into
mainfrom
ccr-0455e606-wmtsor

Conversation

@AdaWorldAPI

Copy link
Copy Markdown
Owner

Lab crates only. Implements the no-boundaries ruling for joins. Values stay in the relation's lanes, and the only thing preparation builds is the global-to-local index map $i.

What changed

mork-pathmap-probe

  • RelationView holds two lanes (arrow_buffer::ScalarBuffer<u32>, refcounted, shared with Lance column buffers). The row fixture is from_rows.
  • Side is an index map. It holds a presence plane, per-slot offsets, and row positions into the lanes. Run reads lane[pos[k]] in place.
  • JoinExec::answers is gone. Answers are folds:
    • bag_count reads offsets only.
    • matching(pattern) with a pinned center inverts the center to one slot.
    • for_each_answer yields a JoinRow that is read in place.
  • JoinDomain handles dense id sets as a presence mask plus a running count per word: slot = rank + popcount, in value order, with no sort or search. Sparse sets keep the sorted list.
  • A side's keep is a mask over row positions. prepare_where and prepare_star_where walk only the set bits.
  • The reference is a counting oracle. MORK/PathMap row sets stay as outside comparators, checked in both directions.

crepe-plan-probe

  • Store holds RelationViews. Lance reads hand over the column buffers with no to_vec, and restricted reads are no longer trimmed with retain.
  • A restriction is one ndarray::simd::eq_u32_to_mask over the lane. Semijoin centers are a domain probed per row.
  • Derivations and slot views read Run with a one-slot prefetch lookahead (prefetch_read_t0).

Verification

  • cargo +1.99.0 fmt, clippy --release --all-targets -D warnings, and test --release --all-targets are green in both crates.
  • Disable runs, each red and then restored:
    • Run::get and iter skip positions.
    • for_each_set_ahead drops the last slot.
    • prepare_star_where ignores keep.
    • The dense rank drops the popcount.
    • The restriction compare uses the wrong constant.
    • Semijoin centers admit everything.
    • The kept-row iterator drops the last row or yields bits past n.
  • examples/measure.rs: with pushdown and semijoin, prepare is µs at every n up to 50 000, the mask and terminal steps stay ns to low µs, and every oracle agrees.

Open: one failed gate

The ablation arms without pruning (no-semijoin, none) at n = 50 000 index 56k and 119k rows and go over the µs budget. Building $i from unordered lanes is a counting sort, and its count and scatter passes cost about 10 ns per row on this host. A rank-free dense counting sort was tried and did not help, because the passes are bound by the random writes.

The candidate fix is lanes sealed in key order at write time, using the OrderedLaneWitness path that addressed.rs already attests. Then $i is just run offsets. That is a write-path decision and is not taken here. Recorded in .claude/board/entries/2026-10-21-zero-copy-join-index-map.md.

Known remaining tuple sites, not touched: crepe reference::eval (the oracle) and the external-engine timings in mork-pathmap-probe/examples/cost.rs.


Generated by Claude Code

@coderabbitai

coderabbitai Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 60a3410a-098e-4b64-b48d-9402c8def6f1
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Warning

Billing warning: we have not been able to collect payment for this subscription for more than 72 hours. Please update the payment method or pay any pending invoices in Billing to avoid service interruption.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-21T13:16:40.957057Z cd499c4 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

claude added 13 commits October 21, 2026 15:17
…answers are folds

- RelationView holds two lanes (arrow ScalarBuffer<u32>, shared by reference
  count) instead of [u32; 2] pairs. from_rows builds the lanes once from
  authored fixture rows.
- Side keeps present + off and holds pos, the row positions per slot (the
  global-to-local index map), instead of a copy of the value column. at(slot)
  returns a Run read in place (lane[pos[k]]); counts come from the offsets.
- JoinExec: answers() (BTreeMap rows) is gone. bag_count reads only offsets;
  matching(pattern) is a fold (a pinned y reads one slot); for_each_answer
  hands out a JoinRow read in place. The next slot's runs are prefetched
  through ndarray::simd while the current one is read.
- reference is a counting oracle over the lanes; it builds no row.
- MORK and PathMap keep their own representation; their row sets are checked
  against the folds in both directions. The native BoundIndex answers through
  native(), and answers() refuses with FoldOnly.
- Examples report microseconds, not milliseconds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G22yT6htkcdyXsihxxXdrg
…a Run::get misread)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G22yT6htkcdyXsihxxXdrg
…e), never copies rows

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G22yT6htkcdyXsihxxXdrg
Store holds RelationView lanes (arrow ScalarBuffer, shared with Lance's
column buffers, no to_vec). Restriction and semijoin centers select row
positions (prepare_views); restricted reads are no longer trimmed by
retain. Derivations and slot views read Run (lane[pos[k]]) in place, with
a one-slot prefetch lookahead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G22yT6htkcdyXsihxxXdrg
…ne compare into a mask

JoinDomain lays a dense value set (span <= 8x count) out as a presence mask
plus a running count per word: slot = rank + popcount, in value order, with
no sort and no search; sparse sets keep the sorted list. A side's keep is a
mask over row positions, and the index map walks its set bits. In
crepe-plan-probe a restriction is one eq_u32_to_mask over the lane, and the
semijoin centers are a domain probed per row, not a binary search.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G22yT6htkcdyXsihxxXdrg
… the end

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G22yT6htkcdyXsihxxXdrg
… at n=50k)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G22yT6htkcdyXsihxxXdrg
A compound hypothesis is split into atoms (one subject, one predicate, one
condition), each anchored to the proposer's words with its qualifiers, and
given its own verdict. A bundle verdict is blocked. Recorded incident: the
2026-10-21 ClassView/rails/QR verdict (EXO-50 /026, corrected in /027).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G22yTnyw
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G22yTnyw
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G22yTnyw
…he class's compute_dag

The user class declares through the contract's existing ClassView surface
that effective_roles is computed from member_of. A direct-routing table maps
that target position to the existing compiled closure fold
(Compiled::roles_of); execute_compute_dag runs it. No new relationship,
association or planner type. Checked on every user against dir-sim and the
crepe oracle, before and after removing one of two support paths; a class
that does not declare the behavior computes nothing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G22yTnyw
…irect paths

Per-user ns for the existing dir-sim walk, the prepared CSR walk, the bare
compiled fold, the fold reached through execute_compute_dag, and the
declaration resolved once per class. Preparation is reported as a failed
gate (millisecond scale), never as a measurement. Board entry
2026-10-21-classview-dr-iam.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G22yTnyw
…main

#1658's co-addressed join used Side::values(), a slot-order copy of the
non-key column. #1659 removed that copy. The restriction now runs over the
relation's lane in row order, and each side row reads its bit through
Side::positions(); values_at reads lane()[positions()[j]]. Store is lane
based (Store::from_rows) in the module, its tests and its example.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G22yT6htkcdyXsihxxXdrg
…ebra polyfill

Five scouts, three co-architects, 33-row crosswalk. Adopt-now: X9
(normalize vs Kleene oracle) and X10 (realization conformance helper).
Eight pre-registered probes; one contract bridge left open for the
operator (defaulted ClassView::semantics). Names a soundness gap:
production normalize is two-valued on NULL-capable lanes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G22yT6htkcdyXsihxxXdrg
@AdaWorldAPI
AdaWorldAPI merged commit 1c16153 into main Oct 21, 2026
15 checks passed
AdaWorldAPI pushed a commit that referenced this pull request Oct 21, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G22yT6htkcdyXsihxxXdrg
AdaWorldAPI added a commit that referenced this pull request Oct 21, 2026
board: #1659 closeout — PR_ARC entry, LATEST_STATE
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants