Skip to content

feat(core): dedup-cluster builder for curation (3A) - #136

Merged
PhysShell merged 6 commits into
mainfrom
claude/curation-cluster-builder
Jul 18, 2026
Merged

PhysShell merged 6 commits into
mainfrom
claude/curation-cluster-builder

Conversation

@PhysShell

@PhysShell PhysShell commented Jul 18, 2026 •

Copy link
Copy Markdown
Owner

3A — dedup-cluster builder

Curation step 3A: turn the corpus's duplicate links into navigation units so a curator reviews one representative per near-duplicate group instead of ~25% almost-identical phrases. Pure ChunkMeta in, clusters out — no Score, no import, no I/O, no UI. Deliberately not a graph framework: one link relation, resolved and validated.

Base b034fc6 (main)
Commits 4 (2 RED → GREEN pairs)

API

pub struct CurationCluster { key: ChunkId, representative: ChunkId, variants: Vec<ChunkId>, size: usize }
pub enum ClusterDiagnostic { UnresolvableProvenance, DanglingLink, SelfLink, ForwardLink, CrossRunLink, Cycle }
pub fn build_clusters(chunks: &[ChunkMeta]) -> (Vec<CurationCluster>, Vec<ClusterDiagnostic>)
  • representative is the linkless root; variants are all chunks that quote it, resolved transitively (p0 <- p1 <- p2 is one cluster of 3), sorted for determinism;
  • duplicate.of (a phrase index) resolves to the _p<N>-suffixed sibling — the pairing the schema itself documents — and the target's ensemble must match, or it is a CrossRunLink;
  • quote_share rides along as link metadata and is never used to reclassify;
  • no automatic reject, not even for an exact (quote_share == 1.0) repeat — a cluster is a unit of comparison, not of fate; the decision stays per chunk (3B/3C);
  • a malformed link is excluded and reported, never a silent singleton — corrupt provenance surfaces instead of masquerading as a clean corpus.

Acceptance contract — all seven hold

Unit tests (10) pin the shape and every diagnostic. The full-corpus invariants were checked over corpus/ingest/ (9,909 chunks, not committed — copyrighted):

chunks:      9,909
clusters:    7,405
variants:    2,504
size sum:    9,909      every chunk in exactly one cluster
diagnostics: 0
order-independent: true
singletons:  5,595      biggest cluster: 14
# contract result
1 each chunk in exactly one cluster ✓ clustered == 9,909
2 sizes sum == 9,909 ✓
3 variants == the duplicate-link count ✓ 2,504
4 clusters == chunks − variants ✓ 9,909 − 2,504 = 7,405
5 deterministic, order-independent ✓ clusters and diagnostics sorted (review-fix)
6 zero Score imports ✓ structural — pure ChunkMeta
7 typed diagnostics for bad links ✓ all six variants tested (Cycle via the resolution seam)

One reconciliation, stated plainly: the scope predicted 2,476 variants / 7,433 clusters. The real current corpus has 2,504 chunks carrying a duplicate link (verified: grep -l '"duplicate"' → 2,504), so 2,504 variants and 9,909 − 2,504 = 7,405 clusters. The 2,476 figure was mine, quoted from the earlier pre-GP7 9,687-chunk run; the builder is exactly right — variants equals the duplicate-link count to the record. The invariant that mattered (clusters + variants == chunks, zero diagnostics) holds.

Review-fix (7bb7c7e → 91412d3)

Two holes the first cut left, both caught in review:

  • diagnostics weren't deterministic — clusters were sorted but the error list came out in input order, invisible on a 0-diagnostic corpus. Now diagnostics sort too (derived Ord); a permuted broken corpus proves it.
  • the "cycle" test never built a cycle — p0↔p1 is caught as a ForwardLink, so no loop ever reached the graph, and the test only asserted !diags.is_empty(). The resolution is now a private resolve_roots a test feeds a real p0→p1→p2→p0 loop, asserting the exact Cycle for every member. The builder was right; the claim "all six tested" was the lie.

Next

3B — a separate, versioned curation-store keyed on stable chunk identity with reviewer provenance and restart/reingest tests; then 3C — a thin vertical cockpit flow (open cluster → play representative/variants → decide per chunk → restart → decision survived). Tags, filters, and counters come after the persistence contract is proven, not before.

Not self-accepted, not merged.

🤖 Generated with Claude Code

https://claude.ai/code/session_01NkqJUU6d1sW1RAfvyHrqVM

PhysShell and others added 4 commits July 18, 2026 21:54
A full-corpus load benchmark over the 9,909-chunk ingest measured 111.7s and
9,909 source imports for only 400 unique sources — each tab parsed ~25×. A
per-session sha256→Score cache cut it to 4.5s / 400 imports with an identical
result (9,909 loaded both ways).

`load_corpus_material_with` injects the importer so the count is observable.
The test writes two chunks that share one source and asserts it is imported
once; the loader currently imports per chunk, so it fails 2 != 1. The cache is
the GREEN.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NkqJUU6d1sW1RAfvyHrqVM
`load_chunk` keys the parsed `Score` by the source's sha256 (filename for pre-v9
records) in a per-load `HashMap`, so a tab shared by many chunks is parsed once.
The count test passes (1 import for 2 chunks), and the existing correctness test
is unchanged — `prepare_chunk` slices an immutable `&Score`, so reuse is
bit-identical to per-chunk parsing. The hash check lives on the cache-miss path,
so a mismatched source is still a strict load failure, never bypassed by reuse.

Full-corpus effect (measured): 111.7s → 4.5s, 9,909 source imports → 400, same
9,909 chunks loaded. No persisted-schema change; the cockpit's OPFS loader is a
separate path and unaffected.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NkqJUU6d1sW1RAfvyHrqVM
…k (3A)

Curation step 3A. The ingest corpus is ~26% near-duplicate (2,476 of 9,909
chunks carry a `duplicate` link), and sending a curator through all 9,909 would
mean re-reviewing almost-identical phrases — the most expensive resource in the
system. `build_clusters` groups a corpus's chunks into near-duplicate clusters
(a representative + its variants) purely from `ChunkMeta`: no Score, no import,
no I/O.

The contract, pinned by ten tests: a transitive `p0 <- p1 <- p2` chain is one
cluster of three; independent roots are separate singletons; the result is
independent of input order; every chunk lands in exactly one cluster with sizes
summing to the count. A malformed link never silently becomes a singleton — it
is a typed diagnostic: dangling, self, forward (not-earlier), cross-run
(different ensemble), cycle, unresolvable provenance. No automatic reject, not
even for an exact quote.

The function is a stub returning empty, so the cluster and diagnostic tests
fail; `split_phrase` (the id `_p<N>` reader the schema documents) is real.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NkqJUU6d1sW1RAfvyHrqVM
`build_clusters` resolves each chunk's `duplicate.of` (a phrase index) to the
`_p<N>`-suffixed sibling in the same run, follows the chain transitively to the
linkless root, and groups every chunk under its root — the representative plus
sorted variants. A malformed link is excluded and reported as a typed
diagnostic (dangling / self / forward / cross-run / cycle / unresolvable), never
folded into a silent singleton. `quote_share` rides along as link metadata and
is never used to reclassify; nothing is auto-rejected. Deterministic: the output
is sorted by id and independent of input order. Ten RED tests pass.

Pure ChunkMeta in, clusters out — no Score, no import, no I/O, so a curation
queue can be built without touching the source tabs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NkqJUU6d1sW1RAfvyHrqVM
@coderabbitai

coderabbitai Bot commented Jul 18, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

@PhysShell, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 17 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 5004f834-b612-43d8-aa28-9ee7503dad9f

📥 Commits

Reviewing files that changed from the base of the PR and between b034fc6 and 91412d3.

📒 Files selected for processing (4)
  • cli/src/generation_input.rs
  • core/src/corpus.rs
  • core/src/curation.rs
  • core/src/lib.rs
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch claude/curation-cluster-builder

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

The determinism claim covered the clusters (sorted) but not the diagnostics,
which came out in input order — a hole a clean 0-diagnostic corpus can never
expose (zero errors sort themselves). A permuted broken corpus (two danglings
and a self-link) makes `build_clusters(&broken) != build_clusters(&shuffled)`,
so the test fails.

Replaces the earlier "cycle" test, which asserted only `!diags.is_empty()` and
never built a real cycle (the `of < own` rule makes one unreachable through the
public builder). A genuine cycle-defense test comes with the GREEN, against the
extracted resolution.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NkqJUU6d1sW1RAfvyHrqVM

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6f402f7928

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread core/src/curation.rs
path.push(cur.clone());
cur = next.clone();
}
None => break None,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Report chains through invalid duplicate links

When a later valid-looking link reaches a chunk that was already excluded in the first pass, e.g. p1 is a CrossRunLink/DanglingLink and p2 has duplicate.of = 1, parent.get(&cur) returns None here and the code just marks p2 bad without emitting any diagnostic. That violates the function contract that chunks on broken links are reported instead of silently disappearing, so curation can lose descendants of a malformed link with no explanation.

Useful? React with 👍 / 👎.

Two corrective fixes to 3A, both from review:

- **Deterministic diagnostics.** `build_clusters` now sorts the diagnostics
  (via derived `Ord` on `ClusterDiagnostic`, and `Ord` on `ChunkId`), so the
  error list no longer depends on input order — the determinism guarantee held
  only for the clusters before, a hole a clean 0-diagnostic corpus can't expose.
  The RED permutation-of-a-broken-corpus test passes.

- **A real cycle test.** The root resolution is extracted into `resolve_roots`,
  and a test feeds it a hand-built loop `p0 -> p1 -> p2 -> p0` and asserts the
  exact `ClusterDiagnostic::Cycle` for every member. The public builder cannot
  make a cycle (the earlier-only rule strictly decreases the phrase index), so
  the old test asserted only `!diags.is_empty()` and never exercised the loop —
  the "all six variants tested" claim was the lie, not the builder.

Full-corpus numbers are unchanged (0 diagnostics, so the sort is a no-op there):
7,405 clusters, 2,504 variants, 9,909 chunks.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NkqJUU6d1sW1RAfvyHrqVM
@PhysShell
PhysShell merged commit ad71cd3 into main Jul 18, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant