Review project PRs per rubric, with the verdict computed, not asked

[Vilin97]

Adopts Tau Ceti's single-angle review structure for project PRs, as discussed — the same rubric idea, sized to the pool.

Why split the call

The 148-review audit showed the monolithic prompt's failure signature clearly: one narrative call lets the overall impression swamp the specific checks. The two worst misses (#275, #278) were long approving summaries with zero findings; fit was ~96% collinear with the verdict it sat next to; code_quality produced one finding in 148 reviews. Tau Ceti runs each angle as its own agent precisely so a reviewer with one job can't be lulled by strength elsewhere, and orders them by observed block-rate per dollar. ArkLib's pipeline computes the verdict deterministically from findings instead of asking the model for one. This PR takes both ideas.

The rubrics (.github/review-rubrics/)

Each is an independent model call on the shared core (REVIEW_RULES.md, trimmed to identity + out-of-scope + style) plus one angle file:

Rubric	Owns	Verdict power
faithfulness	claim-vs-Lean, assumed-not-proved, vacuity, boundary probes	blocking
novelty	duplication vs Mathlib + pool; gets the #306 prior-art evidence	blocking
significance	what the pool wants, calibration, generated bulk	blocking
sources	does the citation say what the PR says it says; uncredited formalizations	blocking
quality	AI-slop, maintenance debt	advisory — its block records as discuss

Quality is advisory on the audit's own evidence: in 148 reviews that dimension never justified a veto, so it flags for a human instead of blocking. Not copied from Tau Ceti: their roadmap-fit scope rubric (the pool has no roadmap — significance is judged at PR time) and their placement/naming/generality angles (pool projects are independent by design).

The verdict is computed, never asked

aggregate_verdict: any blocking block → request_changes; any discuss → needs_discussion; all pass → approve. Two invariants a prompt could only request are now enforced in code: an elided-diff review cannot approve (nobody read those files — the #278 failure), and an unreadable rubric verdict aggregates as discuss, never as a silent pass.

One sticky comment

Rubric table with per-rubric verdicts and bottom lines, the combined facts table (claim / assumptions / prior art / source / fit / quality — same icons as before, including the #303 hedge rendering), findings grouped by rubric with their evidence field shown, and a usage footer summing tokens across calls with each call priced at its own tier.

Cost and scope

Five calls instead of one: ~$0.50–0.70 per project review at flex/xhigh vs ~$0.13 — a fraction of Tau Ceti's own per-round spend ($2.73–$6.33 on their scoreboards). Refactor, challenge, and solution modes unchanged. Sequential calls; parallelizing is a follow-up if latency ever matters for a comment bot.

326 tests: aggregation invariants, advisory clamping, orchestration (each call gets core + its own rubric; only novelty gets the search evidence), comment rendering, cost summing.

After merge I'll run /review on an open project PR to see the five rubrics live.

🤖 Generated with Claude Code

[greptile-apps]

Greptile Summary

This PR replaces the single monolithic project-PR review call with five independent per-rubric model calls (faithfulness, novelty, significance, sources, quality), each on top of a trimmed shared core, with the overall verdict computed deterministically from their outputs rather than asked of the model. Two invariants that the old prompt could only request are now enforced in code: an advisory rubric's block is clamped to discuss, and an elided-diff review cannot produce approve.

• run_project_rubrics runs the five calls sequentially, isolating prior-art evidence to the novelty rubric only; aggregate_verdict applies block-beats-discuss-beats-pass logic plus the truncation guard; render_rubric_comment assembles the per-rubric verdict table, the combined facts table, findings grouped by rubric, and a summed cost footer.
• REVIEW_RULES.md is trimmed to the identity + out-of-scope + style shared core; the five new files under .github/review-rubrics/ carry each dimension's full criteria and output schema.
• 326 new tests cover aggregation invariants, advisory clamping, rubric isolation (core + its own file; only novelty gets search evidence), comment rendering, and cost summing.

Confidence Score: 5/5

Safe to merge; the new rubric orchestration, verdict aggregation, and rendering logic are well-tested and cleanly isolated from the unchanged refactor/challenge/solution paths.

The refactor is thorough and self-contained: the two architectural invariants (advisory clamping, elided-diff cannot approve) are now enforced in code and covered by tests, the non-project review paths are untouched, and the one rendering nit is cosmetic and low-probability in practice.

Files Needing Attention: No files require special attention; python/lean_pool/review.py carries the most new logic but it is well-tested.

Important Files Changed

Filename	Overview
python/lean_pool/review.py	Core orchestration rewrite: replaces the monolithic project review with five sequential per-rubric calls, deterministic verdict aggregation, and a new rendering path; advisory quality clamping and elided-diff invariant are both enforced in code.
python/tests/test_review.py	Old monolithic-assessment tests replaced with targeted rubric tests covering advisory clamping, verdict aggregation invariants, per-rubric isolation, cost summing, and comment rendering — good coverage of the new code paths.
.github/REVIEW_RULES.md	Trimmed to a shared core (identity, out-of-scope, style); dimension-specific guidance moved to the five rubric files, reducing prompt length per call.
.github/review-rubrics/quality.md	New advisory rubric for code quality; schema correctly omits block, but the Verdict prose section still mentions block parenthetically, creating a minor internal inconsistency (already flagged in a prior review comment).
.github/review-rubrics/faithfulness.md	New rubric for the claim-vs-Lean dimension; schema includes all three verdict values and the full set of faithfulness checks from the original monolithic rules.
.github/review-rubrics/novelty.md	New rubric for prior-art checking; correctly scoped to receive the pre-computed search evidence.
.github/review-rubrics/significance.md	New rubric for fit/level/mode judgment; pulls the calibration examples and what-we-want prose directly from the original rules.
.github/review-rubrics/sources.md	New rubric for citation integrity; correctly scoped to only the source-holds-up question, with unverifiable as a normal non-blocking outcome.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Project PR arrives] --> B[run_project_rubrics]
    B --> C1[faithfulness call]
    B --> C2[novelty call + prior art]
    B --> C3[significance call]
    B --> C4[sources call]
    B --> C5[quality call]
    C1 --> D[normalize_rubric_verdict blocking=True]
    C2 --> D
    C3 --> D
    C4 --> D
    C5 --> E[normalize_rubric_verdict blocking=False]
    D --> F[aggregate_verdict]
    E --> F
    F -->|any block| G[request_changes]
    F -->|any discuss OR truncated| H[needs_discussion]
    F -->|all pass, not truncated| I[approve]
    G --> J[render_rubric_comment]
    H --> J
    I --> J
    J --> K[post_comment]

Loading

_{Reviews (2): Last reviewed commit: "Show the tier, price honestly, and keep ..." | Re-trigger Greptile}

[Vilin97]

All four addressed — the tier cell (deduped across calls), the honest unpriced footer, the empty-comment line, and the quality.md Verdict/schema mismatch (the prose no longer offers a block the schema doesn't; the rubric now says outright it has no block).