Repository navigation
Zero-copy joins: relations are lanes, a side is an index map $i (mork-pathmap-probe, crepe-plan-probe) - #1659
Conversation
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configuration
Warning Billing warning: we have not been able to collect payment for this subscription for more than 72 hours. Please update the payment method or pay any pending invoices in Billing to avoid service interruption. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
…answers are folds - RelationView holds two lanes (arrow ScalarBuffer<u32>, shared by reference count) instead of [u32; 2] pairs. from_rows builds the lanes once from authored fixture rows. - Side keeps present + off and holds pos, the row positions per slot (the global-to-local index map), instead of a copy of the value column. at(slot) returns a Run read in place (lane[pos[k]]); counts come from the offsets. - JoinExec: answers() (BTreeMap rows) is gone. bag_count reads only offsets; matching(pattern) is a fold (a pinned y reads one slot); for_each_answer hands out a JoinRow read in place. The next slot's runs are prefetched through ndarray::simd while the current one is read. - reference is a counting oracle over the lanes; it builds no row. - MORK and PathMap keep their own representation; their row sets are checked against the folds in both directions. The native BoundIndex answers through native(), and answers() refuses with FoldOnly. - Examples report microseconds, not milliseconds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G22yT6htkcdyXsihxxXdrg
…a Run::get misread) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G22yT6htkcdyXsihxxXdrg
…e), never copies rows Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G22yT6htkcdyXsihxxXdrg
Store holds RelationView lanes (arrow ScalarBuffer, shared with Lance's column buffers, no to_vec). Restriction and semijoin centers select row positions (prepare_views); restricted reads are no longer trimmed by retain. Derivations and slot views read Run (lane[pos[k]]) in place, with a one-slot prefetch lookahead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G22yT6htkcdyXsihxxXdrg
…ne compare into a mask JoinDomain lays a dense value set (span <= 8x count) out as a presence mask plus a running count per word: slot = rank + popcount, in value order, with no sort and no search; sparse sets keep the sorted list. A side's keep is a mask over row positions, and the index map walks its set bits. In crepe-plan-probe a restriction is one eq_u32_to_mask over the lane, and the semijoin centers are a domain probed per row, not a binary search. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G22yT6htkcdyXsihxxXdrg
… the end Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G22yT6htkcdyXsihxxXdrg
… at n=50k) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G22yT6htkcdyXsihxxXdrg
A compound hypothesis is split into atoms (one subject, one predicate, one condition), each anchored to the proposer's words with its qualifiers, and given its own verdict. A bundle verdict is blocked. Recorded incident: the 2026-10-21 ClassView/rails/QR verdict (EXO-50 /026, corrected in /027). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G22yTnyw
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G22yTnyw
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G22yTnyw
…he class's compute_dag The user class declares through the contract's existing ClassView surface that effective_roles is computed from member_of. A direct-routing table maps that target position to the existing compiled closure fold (Compiled::roles_of); execute_compute_dag runs it. No new relationship, association or planner type. Checked on every user against dir-sim and the crepe oracle, before and after removing one of two support paths; a class that does not declare the behavior computes nothing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G22yTnyw
…irect paths Per-user ns for the existing dir-sim walk, the prepared CSR walk, the bare compiled fold, the fold reached through execute_compute_dag, and the declaration resolved once per class. Preparation is reported as a failed gate (millisecond scale), never as a measurement. Board entry 2026-10-21-classview-dr-iam. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G22yTnyw
…main #1658's co-addressed join used Side::values(), a slot-order copy of the non-key column. #1659 removed that copy. The restriction now runs over the relation's lane in row order, and each side row reads its bit through Side::positions(); values_at reads lane()[positions()[j]]. Store is lane based (Store::from_rows) in the module, its tests and its example. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G22yT6htkcdyXsihxxXdrg
1be156e to
a9f05b2
Compare
…ebra polyfill Five scouts, three co-architects, 33-row crosswalk. Adopt-now: X9 (normalize vs Kleene oracle) and X10 (realization conformance helper). Eight pre-registered probes; one contract bridge left open for the operator (defaulted ClassView::semantics). Names a soundness gap: production normalize is two-valued on NULL-capable lanes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G22yT6htkcdyXsihxxXdrg
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G22yT6htkcdyXsihxxXdrg
board: #1659 closeout — PR_ARC entry, LATEST_STATE
Lab crates only. Implements the no-boundaries ruling for joins. Values stay in the relation's lanes, and the only thing preparation builds is the global-to-local index map
$i.What changed
mork-pathmap-probe
RelationViewholds two lanes (arrow_buffer::ScalarBuffer<u32>, refcounted, shared with Lance column buffers). The row fixture isfrom_rows.Sideis an index map. It holds a presence plane, per-slot offsets, and row positions into the lanes.Runreadslane[pos[k]]in place.JoinExec::answersis gone. Answers are folds:bag_countreads offsets only.matching(pattern)with a pinned center inverts the center to one slot.for_each_answeryields aJoinRowthat is read in place.JoinDomainhandles dense id sets as a presence mask plus a running count per word: slot = rank + popcount, in value order, with no sort or search. Sparse sets keep the sorted list.prepare_whereandprepare_star_wherewalk only the set bits.crepe-plan-probe
StoreholdsRelationViews. Lance reads hand over the column buffers with noto_vec, and restricted reads are no longer trimmed withretain.ndarray::simd::eq_u32_to_maskover the lane. Semijoin centers are a domain probed per row.Runwith a one-slot prefetch lookahead (prefetch_read_t0).Verification
cargo +1.99.0 fmt,clippy --release --all-targets -D warnings, andtest --release --all-targetsare green in both crates.Run::getanditerskip positions.for_each_set_aheaddrops the last slot.prepare_star_whereignores keep.n.examples/measure.rs: with pushdown and semijoin, prepare is µs at every n up to 50 000, the mask and terminal steps stay ns to low µs, and every oracle agrees.Open: one failed gate
The ablation arms without pruning (
no-semijoin,none) at n = 50 000 index 56k and 119k rows and go over the µs budget. Building$ifrom unordered lanes is a counting sort, and its count and scatter passes cost about 10 ns per row on this host. A rank-free dense counting sort was tried and did not help, because the passes are bound by the random writes.The candidate fix is lanes sealed in key order at write time, using the
OrderedLaneWitnesspath thataddressed.rsalready attests. Then$iis just run offsets. That is a write-path decision and is not taken here. Recorded in.claude/board/entries/2026-10-21-zero-copy-join-index-map.md.Known remaining tuple sites, not touched: crepe
reference::eval(the oracle) and the external-engine timings inmork-pathmap-probe/examples/cost.rs.Generated by Claude Code