Skip to content

D-RU-1 (U1): JC statistics batch over shared power-sum passes (lab probe) - #1643

Merged
AdaWorldAPI merged 3 commits into
mainfrom
ccr-d3ce1263-ied9yh
Oct 21, 2026
Merged

AdaWorldAPI merged 3 commits into
mainfrom
ccr-d3ce1263-ied9yh

Conversation

@AdaWorldAPI

Copy link
Copy Markdown
Owner

Lab probe crates/jc-batch-probe; no production path. Plan: .claude/plans/d-relalg-unified-v1.md §3 (U1). Entry: .claude/board/entries/2026-10-20-u1-jc-batch.md.

What it does

A batch of JC statistical requests is answered in two ways:

  • Separately: each request evaluates its relalg selection into a mask, then runs one power or cross pass.
  • Batched: the requests are first reduced:
    • Selections are deduplicated after relalg::normalize.
    • For each selection, each distinct lane pair gets one GroupCrossPowerSumsI32 pass keyed by the group lane.
    • A lane that no pair covers gets one GroupPowerSumsI32 pass.
    • Ungrouped answers merge the groups with checked_merge. Univariate answers read one side of the cross sums.

Statistics covered: mean, variance, one-way ANOVA, covariance, Pearson, R², and the simple regression line in both directions. A request built for another version is refused by both paths.

Measured

Battery: 252 requests over 2 selections and 7 I32 lanes. It reduces to 2 selection passes and 42 cross passes. The table gives medians of 5 warm runs.

rows separate batched time rows traversed values read mask bytes
100k 40.2 ms 5.8 ms 6.9× 11.5× 8.5× 126×
1M 424.5 ms 66.2 ms 6.4× 11.5× 8.5× 126×
4M 1,819 ms 279 ms 6.5× 11.5× 8.5× 126×

Cost is counted from the programs that actually run, not from a model.

Correctness

  • Batched answers equal the separate answers bit for bit.
  • They also agree with an independent f64 oracle over directly selected rows: JC's pearson and anova_one_way, plus two-pass mean and variance.
  • At i32 extremes, Σx² passes u64 and Σxy passes i64 (the test asserts both); mean and variance still equal the exact i128 values.
  • At an offset of 10⁹, the naive one-pass f64 Pearson is off by more than 10⁻³, while the exact path is within 10⁻¹².

8 tests pass. Six disable runs each turned red, with every anchor asserted:

  • swap of a reversed pair;
  • reading the y side as x;
  • pair dedup;
  • normalize;
  • group merge;
  • version check.

Not covered

  • R reference: none, because R is not installed. JC's slice functions plus exact integer sums stand in for it.
  • MergeOverflow: untested; it cannot occur below the 2³²-row cap.
  • Batch shapes: only one is timed. The break-even request count is not measured.
  • Overlapping selections: not shared yet; that is U2.

Board and CI

  • STATUS_BOARD: D-RU-1 is MEASURED.
  • The plan records the U1 result.
  • New entry; the entries index is regenerated; the supersession index is unchanged.
  • Root Cargo.toml excludes the crate.
  • New workflow jc-batch-probe.yml: clones the sibling ndarray, then runs fmt, clippy -D warnings, the tests and a benchmark smoke run.

Generated by Claude Code

…P, pre-disable)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VncKoyG6JkpsoqhqTGTv3f
…sured)

252 requests over 2 selections reduce to 2 selection and 42 cross passes.
At 1M rows the batch is 6.4x faster than answering each request alone, with
11.5x fewer rows traversed, 8.5x fewer values read and 126x fewer mask bytes;
answers are bit-identical. Six disable runs turn the tests red.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VncKoyG6JkpsoqhqTGTv3f
@coderabbitai

coderabbitai Bot commented Oct 20, 2026 •

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: c9b2ecb0-fb33-4a72-a4ee-928f91394d62
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Warning

Billing warning: we have not been able to collect payment for this subscription for more than 72 hours. Please update the payment method or pay any pending invoices in Billing to avoid service interruption.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@AdaWorldAPI
AdaWorldAPI marked this pull request as ready for review October 21, 2026 00:15
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-21T00:17:48.851852Z 7c9ead9 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7c9ead98c8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread crates/jc-batch-probe/src/lib.rs
@AdaWorldAPI
AdaWorldAPI merged commit 10c403e into main Oct 21, 2026
14 checks passed
AdaWorldAPI pushed a commit that referenced this pull request Oct 21, 2026
A Rel::Plane(i) selection was accepted and then failed inside the fold,
because the probe passes no resident masks. Both paths now refuse it up front
with BatchError::ResidentPlane; resident planes are U2's let-planes, and Cost
does not count plane reads. Codex review on #1643.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VncKoyG6JkpsoqhqTGTv3f
AdaWorldAPI pushed a commit that referenced this pull request Oct 21, 2026
A Rel::Plane(i) selection was accepted and then failed inside the fold,
because the probe passes no resident masks. Both paths now refuse it up front
with BatchError::ResidentPlane; resident planes are U2's let-planes, and Cost
does not count plane reads. Codex review on #1643.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VncKoyG6JkpsoqhqTGTv3f
AdaWorldAPI pushed a commit that referenced this pull request Oct 21, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013z2e1YXkm6o6HNrDn5BPWA
AdaWorldAPI pushed a commit that referenced this pull request Oct 21, 2026
…entries index after #1643

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013z2e1YXkm6o6HNrDn5BPWA
AdaWorldAPI added a commit that referenced this pull request Oct 21, 2026
jc-batch-probe: refuse selections that name a resident plane (Codex follow-up to #1643)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants