D-SRAM-2 spike: masked Lance 13 reads and the addressed semijoin by AdaWorldAPI · Pull Request #1633 · AdaWorldAPI/lance-graph

AdaWorldAPI · 2026-10-20T19:30:30Z

Follow-up to #1632. This is still the D-SRAM-2 lab research spike, not authorization.

What changes

Stage 3 (lance_store): each star side is read from a Lance 13 table (lancedb 0.40). The read uses with_row_addr_prefilter under an address mask that relalg builds. The memory path is unchanged.
Arm B (addressed, pipeline::evaluate_addressed), the addressed semijoin:
- Each relation is written in key order. The key of row (a, b) is FacetCascade::from_semantic_tiles([0, 0, a_hi, a_lo, b_hi, b_lo, 0, 0]).
- The keys are attested once with SealedFacetLane::attest_sorted. The probe never sorts a lane itself; attest refuses an unsorted table.
- A side reads only the lane ranges of its demanded keys, found with seek_from.
- It adds no mask column, index, opcode or gather, and no global-to-local mapping.
- It refuses, by name:
  - a demand on the second column (NotLeadingColumn);
  - a lane whose table changed since attest (Stale);
  - an unsorted table (NotSorted);
  - a table split over several fragments (NotContiguous).
CI: .github/workflows/crepe-plan-probe.yml runs fmt, clippy -D warnings and the tests. The probe sits outside the workspace, so no other job builds it. The workflow sets CARGO_PROFILE_DEV_DEBUG=0.
Board: the D-TGB-0 and D-TGB-1 rows now read Shipped (lab). The full thought git API stays a proposal: the probe has the slot pool, the oracle and the tests, but no production land, rebase or persistence.

Fixes found on the way

The relalg membership mask takes at most 4 centers, not 64. Each eq atom of the Or holds one scratch slot, and a fold may use MAX_SLOTS = 8. With 5 centers it fails with ScratchBufferTooSmall.
With two restricted sides, the Lance path did not trim those sides to the intersected centers. It prepared 7 rows where the memory path prepares 5.
examples/row_filter.rs: the contiguous arm compared a mask with rows (Codex P2 on D-SRAM-2 spike: semijoin reduction before preparation #1632). The arm is fixed and §2c is re-measured.

Evidence

cargo run --release --example addressed: N = 50,000 edges, k centers, medians of 9 runs. "Rows" counts rows Lance scanned, mask columns included.

k	full ms	Arm A ms	Arm B warm ms	attest ms	Arm A rows	Arm B rows	break-even
0	9.81	4.46	0.37	7.28	150,000	0	2
1	9.95	5.02	2.19	6.83	150,005	3	3
8	12.30	6.60	2.24	6.67	150,040	24	2
500 (1%)	10.42	8.86	3.47	6.71	152,500	1,500	2
5,000 (10%)	11.70	15.02	12.11	7.81	175,000	15,000	3
25,000 (50%)	18.55	41.39	34.54	7.74	275,000	75,000	2
50,000 (all)	30.71	36.15	38.98	9.84	400,000	150,000	never

Arm B reduces source-row reads to exactly 3k. Arm A reduces only the rows laid out.
Up to 1% of centers Arm B is 2.5–12× faster, and its attest pays for itself after 2–3 queries.
From 10% of centers up a full scan wins. With all centers Arm B never breaks even.

Tests:

Stage 3, P1–P4, and Arm B, B1–B9; 35 in all, green. fmt and clippy are clean.
Disable runs, each red:
- no prefilter;
- no stale check;
- seal instead of attest_sorted;
- wrong center column;
- no trimming of the restricted sides;
- 64 atoms;
- no contiguity check.

Open

The selectivity crossover between 1% and 10%. Above it, a planner should switch Arm B to the full read.
Multi-fragment lanes: today an append forces a compaction before Arm B can read the table again.
A second lane ordered by the second column, for demands on that column.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VncKoyG6JkpsoqhqTGTv3f

Stage 3 (D-HOLO-H14d): each star side is read from a Lance 13 table (lancedb 0.40) under a relalg-built address mask with with_row_addr_prefilter. A side scans exactly the rows of its mask. Arm B: each relation is stored in key order and its keys are attested once with SealedFacetLane::attest_sorted. A side then reads only the lane ranges of its demanded keys, located with seek_from. It reads no mask column and adds no index, opcode or gather. It refuses a demand on the second column, a stale lane, an unsorted table and a table split over several fragments. Fixes found on the way: - The relalg Or holds one scratch slot per eq atom and a fold may use eight, so the relalg membership mask takes at most 4 centers (5 fail with ScratchBufferTooSmall). It was set to 64. - With two restricted sides the Lance path did not trim the restricted sides to the intersected centers, so it prepared 7 rows where the memory path prepared 5. - row_filter's contiguous arm compared a mask with rows (Codex P2). CI: a workflow builds, lints and tests the probe, which sits outside the workspace. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VncKoyG6JkpsoqhqTGTv3f

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VncKoyG6JkpsoqhqTGTv3f

coderabbitai · 2026-10-20T19:30:39Z

Important

🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: Organization UI
Review profile: CHILL
Plan: Advanced
Run ID: 9f141e6b-41b7-416c-bce6-1276c4ecd127

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Warning

Billing warning: we have not been able to collect payment for this subscription for more than 72 hours. Please update the payment method or pay any pending invoices in Billing to avoid service interruption.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

chatgpt-codex-connector · 2026-10-20T19:31:36Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-20T19:34:29.624154Z	`a31c61b`	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

coderabbitai · 2026-10-20T19:31:48Z

Add Carrot credits or activate Agent usage billing to use Autofix

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a31c61bb84

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Plan §2e records the addressed semijoin at 50,000 rows: - Arm B scans 3k rows where Arm A scans at least 150,000. - Up to 1% of centers it is 2.5-12x faster than Arm A, and its 7 ms attest pays for itself after 2-3 queries. - From 10% of centers up a full scan wins; with all centers Arm B never breaks even. §2c is re-measured with the row_filter fix. The contiguous column beats the scalar test up to 64 needles in both runs. At 128 needles the order differs between runs, so that crossover is not pinned. The example also passes clippy (manual_checked_ops). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VncKoyG6JkpsoqhqTGTv3f

The full thought git API (thought-git-api-v1.md) is still a proposal. crates/thought-git-probe implements the slot pool, the oracle and the tests. It does not implement the production land, rebase or persistence pipeline; D-TGB-2..5 stay Queued. D-TGB-0 and D-TGB-1 merged with #1630/#1631, so their board rows now read Shipped (lab) instead of In PR. The module doc no longer says that no thought-git code exists. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VncKoyG6JkpsoqhqTGTv3f

…ntity, CI paths - A demand constant above u32::MAX now matches nothing on both Lance arms, as on the memory path. Before, Arm A refused it with Lance and Arm B refused it with NotLeadingColumn. - Arm B no longer refuses a side keyed by its second column when it has no centers to look up. Such a side is read in full, so the lane order does not matter; only a lookup needs the leading column. - An attested lane records its dataset's URI. A lane used against another table with the same relation id and version is refused (OtherDataset). Before, its row addresses were applied to the wrong table. - The probe workflow also runs on lance-graph-contract changes. - Plan §2c: the 128-needle disagreement is called benchmark noise, which is what it is. Tests B10-B12. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VncKoyG6JkpsoqhqTGTv3f

An Or of k atoms takes 2k - 1 scratch slots, not one per atom: each atom and each OR gets a slot of its own, and no slot is reused inside a fold. MAX_SLOTS = 8 is an r2il setting ("sized with headroom"); mask-risc addresses 65,536 slots. A larger arena, or slot reuse inside the fold, would lift the 4-center limit. The comment, the plan and the board entry now say so. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VncKoyG6JkpsoqhqTGTv3f

claude added 2 commits October 20, 2026 19:29

crepe-plan-probe: Arm A vs Arm B measurement by selectivity

cc578f2

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VncKoyG6JkpsoqhqTGTv3f

AdaWorldAPI marked this pull request as ready for review October 20, 2026 19:31

chatgpt-codex-connector Bot reviewed Oct 20, 2026

View reviewed changes

Comment thread crates/crepe-plan-probe/src/lance_store.rs Outdated

Comment thread crates/crepe-plan-probe/src/addressed.rs Outdated

Comment thread crates/crepe-plan-probe/src/addressed.rs

Comment thread .github/workflows/crepe-plan-probe.yml

claude added 4 commits October 20, 2026 19:46

AdaWorldAPI merged commit cd9a340 into main Oct 20, 2026
14 checks passed

AdaWorldAPI mentioned this pull request Oct 20, 2026

relalg/r2il: width costs no scratch slots #1634

Merged

Navigation Menu

Repository navigation

D-SRAM-2 spike: masked Lance 13 reads and the addressed semijoin - #1633

D-SRAM-2 spike: masked Lance 13 reads and the addressed semijoin#1633
AdaWorldAPI merged 6 commits into
mainfrom
ccr-d3ce1263-ied9yh

AdaWorldAPI commented Oct 20, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Oct 20, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot commented Oct 20, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Oct 20, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

AdaWorldAPI commented Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes

Fixes found on the way

Evidence

Open

Uh oh!

coderabbitai Bot commented Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chatgpt-codex-connector Bot commented Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

coderabbitai Bot commented Oct 20, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

AdaWorldAPI commented Oct 20, 2026 •

edited

Loading

coderabbitai Bot commented Oct 20, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Oct 20, 2026 •

edited

Loading