Skip to content

feat: rate findings at system scale against a project profile (12.2.0) - #111

Open
lklimek wants to merge 10 commits into
mainfrom
feat/global-severity
Open

lklimek wants to merge 10 commits into
mainfrom
feat/global-severity

Conversation

@lklimek

@lklimek lklimek commented Nov 9, 2026 •

Copy link
Copy Markdown
Owner

TL;DR: Review findings are rated by how much they matter to the whole project in production, not by how bad they look inside the changed component.

User story

As a maintainer triaging review findings, I want a rare edge case that only delays one node out of hundreds to be rated low, while recurring friction in a frequently used app flow or a footgun in an SDK is rated high, to spend fix effort where it matters.

Scenario

Base flow

A reviewer finds a defect in a small PR and rates it.

Actual behavior

The rating looks only at the component: a Tenderdash edge case that needs at least 1/3 Byzantine validators and all traffic routed through a Byzantine node, and that only slows one node, comes out as serious. Ratings drift upward, and nothing learns from the maintainer's corrections.

Expected behavior

Each finding opens with what production users or operators would actually notice, and is rated against the project's own scale, threat model, and priorities. Ratings are compared with the project's existing priority-labelled issues. Corrections the maintainer makes during triage are saved and reused by later ratings.

Detailed discussion

What was done

  • severity: new § System scale with an incident headline in impact_description, a per-project-type table (network/node, GUI app, SDK/library, server, CLI), and rules for local vs system harm, dominance, conjunction, and attacker economics. Dominance and economics need evidence. impact covers what fails together in one incident; how many users independently reach the path is likelihood. The tie-break takes the lower band outside Sovereign/Boundary zones when the headline is known.
  • severity/references/system-rating.md (new): project profile (MemCan type: project-profile, or .claudius/project-profile.md read from the base ref in PR reviews), a rating pass (compare with anchors, then an evidence-backed counter-argument for HIGH+), anchors from priority-labelled issues, and calibration memory (MemCan type: severity-calibration).
  • review-changes: consolidation step 3 runs the rating pass, inline in basic mode and through a fresh rater in full mode, which rates only the findings left after duplicate merging and INTENTIONAL downgrades. Ratings go through finding_updates (which now accepts impact_description), or through a merged cluster's own updates so they are not overwritten. gh issue list added to allowed-tools.
  • triage-findings: new per-decision severity_feedback (overrated/underrated) in the schema and triage UI, with a confirm before feedback without a decision is dropped. Each value is saved as a calibration example. check-pr-comments and triage capture user re-ratings the same way.
  • review-pr: the Context Digest carries the project profile.

Testing

python3 -m pytest -q: 1773 passed, 26 skipped. New tests cover schema acceptance and rejection, triage UI controls and the orphan-feedback confirm, the _save_triage round-trip, and finding_updates applying impact_description. claude plugin validate . passes; its warnings were there before this PR. Three independent basic reviews covered the whole branch; all their findings are fixed and the fixes verified by the reviewer.

Breaking changes

None. severity_feedback is optional and finding_updates only gains a field.

Prior work

Attribution

🤖 Co-authored by Claudius the Magnificent AI Agent

🤖 Generated with Claude Code

- severity: System scale rules (incident headline, local vs system harm,
  dominance, conjunction, attacker economics); tie-break takes the lower
  band outside Sovereign/Boundary zones when the headline is known.
- severity/references/system-rating.md: project profile, independent
  rating pass with backlog anchors and counter-argument, calibration memory.
- review-changes: rating pass replaces ad-hoc severity re-evaluation;
  full mode dispatches it to a fresh rater.
- triage-findings: per-finding severity_feedback (overrated/underrated)
  in schema and UI, saved as calibration examples; check-pr-comments and
  triage capture user re-ratings the same way.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Rater reads full finding text from intermediate.json; an unevidenced
  counter-argument no longer lowers a band.
- Dominance and attacker economics require evidence; no assumed threat model.
- Project profile is read from the base ref in PR reviews.
- finding_updates accepts impact_description so the headline and rating
  reason reach the report.
- Rating pass skips INTENTIONAL downgrades and targets cluster bases;
  callers and anchor filtering clarified; gh issue list allowlisted.
- Triage UI confirms before dropping severity feedback without a decision;
  save-path and behavior tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
GUI apps rate recurring friction on frequent flows as real impact, SDKs
rate by consuming apps and developer friction; profile gains Project type.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
GUI friction maps to the ~0.4 rung only when it makes users abandon or
redo a task, with frequency in likelihood; SDK row maps to impact; zones
override share of users for irreversible loss; mixed repos map paths to
types; consistent trap wording.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@lklimek
lklimek marked this pull request as ready for review November 16, 2026 09:45
@lklimek
lklimek requested a balanced review from Copilot November 16, 2026 09:47

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Duplicate-resolution updates can overwrite corrected ratings, and the full-mode rater receives stale findings that can produce invalid updates and block finalization.

2 open issues
What changed in this PR

This PR extends Claudius reviews to rate findings by production impact across the whole project, building on the earlier real-world-significance rules.

Changes:

  • Adds project profiles, system-scale rating guidance, and comparisons with existing priority-labelled issues.
  • Integrates re-rating into review consolidation and carries project profiles through review context.
  • Adds optional triage severity feedback for future calibration, with schema and regression coverage.
File Description
tests/​test_triage_server.py Tests severity-feedback persistence and report validity.
tests/​test_render_v4.py Tests feedback controls, discard warnings, and schema validation.
tests/​test_merge_findings_helper.py Tests incident-headline updates and type validation.
skills/​triage/​SKILL.md Applies system-scale severity and captures corrections.
skills/​triage-findings/​SKILL.md Saves severity feedback as calibration examples.
skills/​severity/​SKILL.md Defines system-scale rating and revised tie-break rules.
skills/​severity/​references/​system-rating.md Documents profiles, rating passes, anchors, and calibration.
skills/​severity/​references/​merge-classification.md Updates the reference to severity tie-break rules.
skills/​review-pr/​SKILL.md Adds project profiles to the Context Digest.
skills/​review-changes/​SKILL.md Loads profiles and permits issue-anchor lookups.
skills/​review-changes/​references/​consolidation.md Integrates system-scale re-rating before merge classification.
skills/​check-pr-comments/​SKILL.md Captures user re-ratings for calibration.
scripts/​merge_findings_helper.py Allows validated impact_description overrides.
scripts/​generate_review_report.py Adds severity-feedback controls and discard confirmation.
schemas/​review-report.schema.json Defines optional triage severity feedback.
.claude-plugin/​plugin.json Bumps the plugin version to 12.1.0.

Review effort: Balanced

A full-mode rater given only intermediate.json could rate merged-away
keys, failing finalization; cluster updates applied after
finding_updates could overwrite its floats. Pass the keys to rate with
merged text, and route a merged cluster's rated fields into its updates.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tep 3

The impact section still folded blast radius into impact, double-counting
the user share the per-type table moves into likelihood; the rating pass
Record step still sent all output to finding_updates.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… reach

What fails together in one incident stays in impact (a chain halt on
every node); how many users independently reach the path is likelihood.
Align the schema description, report tooltip and v3-migration comments.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@lklimek lklimek changed the title feat: rate findings at system scale against a project profile (12.1.0) feat: rate findings at system scale against a project profile (12.2.0) Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants