Repository navigation
Conversation
- severity: System scale rules (incident headline, local vs system harm, dominance, conjunction, attacker economics); tie-break takes the lower band outside Sovereign/Boundary zones when the headline is known. - severity/references/system-rating.md: project profile, independent rating pass with backlog anchors and counter-argument, calibration memory. - review-changes: rating pass replaces ad-hoc severity re-evaluation; full mode dispatches it to a fresh rater. - triage-findings: per-finding severity_feedback (overrated/underrated) in schema and UI, saved as calibration examples; check-pr-comments and triage capture user re-ratings the same way. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Rater reads full finding text from intermediate.json; an unevidenced counter-argument no longer lowers a band. - Dominance and attacker economics require evidence; no assumed threat model. - Project profile is read from the base ref in PR reviews. - finding_updates accepts impact_description so the headline and rating reason reach the report. - Rating pass skips INTENTIONAL downgrades and targets cluster bases; callers and anchor filtering clarified; gh issue list allowlisted. - Triage UI confirms before dropping severity feedback without a decision; save-path and behavior tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
GUI apps rate recurring friction on frequent flows as real impact, SDKs rate by consuming apps and developer friction; profile gains Project type. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
GUI friction maps to the ~0.4 rung only when it makes users abandon or redo a task, with frequency in likelihood; SDK row maps to impact; zones override share of users for irreversible loss; mixed repos map paths to types; consistent trap wording. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Duplicate-resolution updates can overwrite corrected ratings, and the full-mode rater receives stale findings that can produce invalid updates and block finalization.
2 open issues
What changed in this PR
This PR extends Claudius reviews to rate findings by production impact across the whole project, building on the earlier real-world-significance rules.
Changes:
- Adds project profiles, system-scale rating guidance, and comparisons with existing priority-labelled issues.
- Integrates re-rating into review consolidation and carries project profiles through review context.
- Adds optional triage severity feedback for future calibration, with schema and regression coverage.
| File | Description |
|---|---|
tests/test_triage_server.py |
Tests severity-feedback persistence and report validity. |
tests/test_render_v4.py |
Tests feedback controls, discard warnings, and schema validation. |
tests/test_merge_findings_helper.py |
Tests incident-headline updates and type validation. |
skills/triage/SKILL.md |
Applies system-scale severity and captures corrections. |
skills/triage-findings/SKILL.md |
Saves severity feedback as calibration examples. |
skills/severity/SKILL.md |
Defines system-scale rating and revised tie-break rules. |
skills/severity/references/system-rating.md |
Documents profiles, rating passes, anchors, and calibration. |
skills/severity/references/merge-classification.md |
Updates the reference to severity tie-break rules. |
skills/review-pr/SKILL.md |
Adds project profiles to the Context Digest. |
skills/review-changes/SKILL.md |
Loads profiles and permits issue-anchor lookups. |
skills/review-changes/references/consolidation.md |
Integrates system-scale re-rating before merge classification. |
skills/check-pr-comments/SKILL.md |
Captures user re-ratings for calibration. |
scripts/merge_findings_helper.py |
Allows validated impact_description overrides. |
scripts/generate_review_report.py |
Adds severity-feedback controls and discard confirmation. |
schemas/review-report.schema.json |
Defines optional triage severity feedback. |
.claude-plugin/plugin.json |
Bumps the plugin version to 12.1.0. |
Review effort: Balanced
A full-mode rater given only intermediate.json could rate merged-away keys, failing finalization; cluster updates applied after finding_updates could overwrite its floats. Pass the keys to rate with merged text, and route a merged cluster's rated fields into its updates. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tep 3 The impact section still folded blast radius into impact, double-counting the user share the per-type table moves into likelihood; the rating pass Record step still sent all output to finding_updates. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… reach What fails together in one incident stays in impact (a chain halt on every node); how many users independently reach the path is likelihood. Align the schema description, report tooltip and v3-migration comments. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

TL;DR: Review findings are rated by how much they matter to the whole project in production, not by how bad they look inside the changed component.
User story
As a maintainer triaging review findings, I want a rare edge case that only delays one node out of hundreds to be rated low, while recurring friction in a frequently used app flow or a footgun in an SDK is rated high, to spend fix effort where it matters.
Scenario
Base flow
A reviewer finds a defect in a small PR and rates it.
Actual behavior
The rating looks only at the component: a Tenderdash edge case that needs at least 1/3 Byzantine validators and all traffic routed through a Byzantine node, and that only slows one node, comes out as serious. Ratings drift upward, and nothing learns from the maintainer's corrections.
Expected behavior
Each finding opens with what production users or operators would actually notice, and is rated against the project's own scale, threat model, and priorities. Ratings are compared with the project's existing priority-labelled issues. Corrections the maintainer makes during triage are saved and reused by later ratings.
Detailed discussion
What was done
severity: new § System scale with an incident headline inimpact_description, a per-project-type table (network/node, GUI app, SDK/library, server, CLI), and rules for local vs system harm, dominance, conjunction, and attacker economics. Dominance and economics need evidence.impactcovers what fails together in one incident; how many users independently reach the path islikelihood. The tie-break takes the lower band outside Sovereign/Boundary zones when the headline is known.severity/references/system-rating.md(new): project profile (MemCantype: project-profile, or.claudius/project-profile.mdread from the base ref in PR reviews), a rating pass (compare with anchors, then an evidence-backed counter-argument for HIGH+), anchors from priority-labelled issues, and calibration memory (MemCantype: severity-calibration).review-changes: consolidation step 3 runs the rating pass, inline in basic mode and through a fresh rater in full mode, which rates only the findings left after duplicate merging and INTENTIONAL downgrades. Ratings go throughfinding_updates(which now acceptsimpact_description), or through a merged cluster's ownupdatesso they are not overwritten.gh issue listadded to allowed-tools.triage-findings: new per-decisionseverity_feedback(overrated/underrated) in the schema and triage UI, with a confirm before feedback without a decision is dropped. Each value is saved as a calibration example.check-pr-commentsandtriagecapture user re-ratings the same way.review-pr: the Context Digest carries the project profile.Testing
python3 -m pytest -q: 1773 passed, 26 skipped. New tests cover schema acceptance and rejection, triage UI controls and the orphan-feedback confirm, the_save_triageround-trip, andfinding_updatesapplyingimpact_description.claude plugin validate .passes; its warnings were there before this PR. Three independent basic reviews covered the whole branch; all their findings are fixed and the fixes verified by the reviewer.Breaking changes
None.
severity_feedbackis optional andfinding_updatesonly gains a field.Prior work
Attribution
🤖 Co-authored by Claudius the Magnificent AI Agent
🤖 Generated with Claude Code