Repository navigation
feat(core): dedup-cluster builder for curation (3A) - #136
Conversation
A full-corpus load benchmark over the 9,909-chunk ingest measured 111.7s and 9,909 source imports for only 400 unique sources — each tab parsed ~25×. A per-session sha256→Score cache cut it to 4.5s / 400 imports with an identical result (9,909 loaded both ways). `load_corpus_material_with` injects the importer so the count is observable. The test writes two chunks that share one source and asserts it is imported once; the loader currently imports per chunk, so it fails 2 != 1. The cache is the GREEN. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NkqJUU6d1sW1RAfvyHrqVM
`load_chunk` keys the parsed `Score` by the source's sha256 (filename for pre-v9 records) in a per-load `HashMap`, so a tab shared by many chunks is parsed once. The count test passes (1 import for 2 chunks), and the existing correctness test is unchanged — `prepare_chunk` slices an immutable `&Score`, so reuse is bit-identical to per-chunk parsing. The hash check lives on the cache-miss path, so a mismatched source is still a strict load failure, never bypassed by reuse. Full-corpus effect (measured): 111.7s → 4.5s, 9,909 source imports → 400, same 9,909 chunks loaded. No persisted-schema change; the cockpit's OPFS loader is a separate path and unaffected. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NkqJUU6d1sW1RAfvyHrqVM
…k (3A) Curation step 3A. The ingest corpus is ~26% near-duplicate (2,476 of 9,909 chunks carry a `duplicate` link), and sending a curator through all 9,909 would mean re-reviewing almost-identical phrases — the most expensive resource in the system. `build_clusters` groups a corpus's chunks into near-duplicate clusters (a representative + its variants) purely from `ChunkMeta`: no Score, no import, no I/O. The contract, pinned by ten tests: a transitive `p0 <- p1 <- p2` chain is one cluster of three; independent roots are separate singletons; the result is independent of input order; every chunk lands in exactly one cluster with sizes summing to the count. A malformed link never silently becomes a singleton — it is a typed diagnostic: dangling, self, forward (not-earlier), cross-run (different ensemble), cycle, unresolvable provenance. No automatic reject, not even for an exact quote. The function is a stub returning empty, so the cluster and diagnostic tests fail; `split_phrase` (the id `_p<N>` reader the schema documents) is real. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NkqJUU6d1sW1RAfvyHrqVM
`build_clusters` resolves each chunk's `duplicate.of` (a phrase index) to the `_p<N>`-suffixed sibling in the same run, follows the chain transitively to the linkless root, and groups every chunk under its root — the representative plus sorted variants. A malformed link is excluded and reported as a typed diagnostic (dangling / self / forward / cross-run / cycle / unresolvable), never folded into a silent singleton. `quote_share` rides along as link metadata and is never used to reclassify; nothing is auto-rejected. Deterministic: the output is sorted by id and independent of input order. Ten RED tests pass. Pure ChunkMeta in, clusters out — no Score, no import, no I/O, so a curation queue can be built without touching the source tabs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NkqJUU6d1sW1RAfvyHrqVM
|
Warning Review limit reached
Next review available in: 17 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (4)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
The determinism claim covered the clusters (sorted) but not the diagnostics, which came out in input order — a hole a clean 0-diagnostic corpus can never expose (zero errors sort themselves). A permuted broken corpus (two danglings and a self-link) makes `build_clusters(&broken) != build_clusters(&shuffled)`, so the test fails. Replaces the earlier "cycle" test, which asserted only `!diags.is_empty()` and never built a real cycle (the `of < own` rule makes one unreachable through the public builder). A genuine cycle-defense test comes with the GREEN, against the extracted resolution. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NkqJUU6d1sW1RAfvyHrqVM
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6f402f7928
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| path.push(cur.clone()); | ||
| cur = next.clone(); | ||
| } | ||
| None => break None, |
There was a problem hiding this comment.
Report chains through invalid duplicate links
When a later valid-looking link reaches a chunk that was already excluded in the first pass, e.g. p1 is a CrossRunLink/DanglingLink and p2 has duplicate.of = 1, parent.get(&cur) returns None here and the code just marks p2 bad without emitting any diagnostic. That violates the function contract that chunks on broken links are reported instead of silently disappearing, so curation can lose descendants of a malformed link with no explanation.
Useful? React with 👍 / 👎.
Two corrective fixes to 3A, both from review: - **Deterministic diagnostics.** `build_clusters` now sorts the diagnostics (via derived `Ord` on `ClusterDiagnostic`, and `Ord` on `ChunkId`), so the error list no longer depends on input order — the determinism guarantee held only for the clusters before, a hole a clean 0-diagnostic corpus can't expose. The RED permutation-of-a-broken-corpus test passes. - **A real cycle test.** The root resolution is extracted into `resolve_roots`, and a test feeds it a hand-built loop `p0 -> p1 -> p2 -> p0` and asserts the exact `ClusterDiagnostic::Cycle` for every member. The public builder cannot make a cycle (the earlier-only rule strictly decreases the phrase index), so the old test asserted only `!diags.is_empty()` and never exercised the loop — the "all six variants tested" claim was the lie, not the builder. Full-corpus numbers are unchanged (0 diagnostics, so the sort is a no-op there): 7,405 clusters, 2,504 variants, 9,909 chunks. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NkqJUU6d1sW1RAfvyHrqVM
3A — dedup-cluster builder
Curation step 3A: turn the corpus's
duplicatelinks into navigation units so a curator reviews one representative per near-duplicate group instead of ~25% almost-identical phrases. PureChunkMetain, clusters out — noScore, no import, no I/O, no UI. Deliberately not a graph framework: one link relation, resolved and validated.b034fc6(main)API
p0 <- p1 <- p2is one cluster of 3), sorted for determinism;duplicate.of(a phrase index) resolves to the_p<N>-suffixed sibling — the pairing the schema itself documents — and the target's ensemble must match, or it is aCrossRunLink;quote_sharerides along as link metadata and is never used to reclassify;quote_share == 1.0) repeat — a cluster is a unit of comparison, not of fate; the decision stays per chunk (3B/3C);Acceptance contract — all seven hold
Unit tests (10) pin the shape and every diagnostic. The full-corpus invariants were checked over
corpus/ingest/(9,909 chunks, not committed — copyrighted):ScoreimportsChunkMetaOne reconciliation, stated plainly: the scope predicted 2,476 variants / 7,433 clusters. The real current corpus has 2,504 chunks carrying a
duplicatelink (verified:grep -l '"duplicate"'→ 2,504), so 2,504 variants and 9,909 − 2,504 = 7,405 clusters. The 2,476 figure was mine, quoted from the earlier pre-GP7 9,687-chunk run; the builder is exactly right —variantsequals the duplicate-link count to the record. The invariant that mattered (clusters + variants == chunks, zero diagnostics) holds.Review-fix (
7bb7c7e→91412d3)Two holes the first cut left, both caught in review:
Ord); a permuted broken corpus proves it.p0↔p1is caught as aForwardLink, so no loop ever reached the graph, and the test only asserted!diags.is_empty(). The resolution is now a privateresolve_rootsa test feeds a realp0→p1→p2→p0loop, asserting the exactCyclefor every member. The builder was right; the claim "all six tested" was the lie.Next
3B — a separate, versioned curation-store keyed on stable chunk identity with reviewer provenance and restart/reingest tests; then 3C — a thin vertical cockpit flow (open cluster → play representative/variants → decide per chunk → restart → decision survived). Tags, filters, and counters come after the persistence contract is proven, not before.
Not self-accepted, not merged.
🤖 Generated with Claude Code
https://claude.ai/code/session_01NkqJUU6d1sW1RAfvyHrqVM