Skip to content

feat: add design and nit severities - #86

Merged
brentvatne merged 2 commits into
mainfrom
tsapeta/design-and-nit-severities
Oct 23, 2026
Merged

brentvatne merged 2 commits into
mainfrom
tsapeta/design-and-nit-severities

Conversation

@tsapeta

@tsapeta tsapeta commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Why

expo/expo's reviewer config wants the classes its deep-code-review skill uses (critical, design, suggestion, nit), but the engine only has critical, warning and suggestion. So expo/expo's general reviewer reports design problems as warning with category quality, and nits as suggestion with a Nit: title prefix. The skill has no level for a non-critical bug, so warning stays for those, and the scale is now critical, warning, design, suggestion, nit.

How

SEVERITIES and SEVERITY_RANK in src/core/schema.ts gain design (rank 2) and nit (rank 4), and the model-facing JSON schema follows from the zod enum, so providers accept the new values. applyReviewPolicy drops nit together with suggestion when includeSuggestions is false, while design is never dropped by severity, like warning. Decision logic, verification, the inline cap and the dismissal floors key only on critical or on rank, so they need no change.

Comment headings now use the skill's icons (🚨 Critical, ⚠️ Warning, 🏗️ Design, 💡 Suggestion, 🐧 Nit), so the Critical, Warning and Suggestion icons change from 🔴 🟡 🔵.

The templates ask agents for critical, warning and design findings, and both shared.md and coordinator.md let a design finding name a concrete cost in place of a failure path, since a design problem rarely has one. Suggestions and nits stay off. Only ecr init reads the templates, so an existing adopter that doesn't change its prompts only sees the new heading icons.

Finding ids don't include severity (fingerprintFinding hashes file, category and evidence), so moving a finding from warning to design keeps its id and existing dismissals still apply. Downgrading is the one catch: /dismiss re-renders the comment from the stored state, and an engine without this change has no design or nit group in groupBySeverity, so after a review that posted those levels it would break /dismiss on that PR. I got this from reading the code, not from running it.

Test Plan

bun test (798 pass), bun run typecheck, bun run lint, bun run fmt:check and bun run llp:check all pass. The new tests fail before the change and pass after: the reviewer parser accepts design and nit, applyReviewPolicy drops nits with suggestions and keeps design, and the comment renders the groups in rank order with the new headings. One existing assertion changed for the new Warning icon. I haven't run it against a live PR yet.

@github-actions

github-actions Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

🤖 AI code review

Decision: Ready for human review

Overall PR risk: Medium. This change modifies existing behavior by adding design and nit severities to the SEVERITIES enum and SEVERITY_RANK, which affects all review output paths: filtering, sorting, rendering, markdown headings, and template prompts. The core decision logic remains unchanged—blocking still keys only on critical. Material concerns exist: older engines without this change would crash when re-rendering stored state containing design or nit findings, heading emoji changes will alter existing comment text on re-render, and adopters using copied templates will not receive design findings. Rollback is straightforward, though any state already written with these severities would require older engines to tolerate them.

🔗 Review setup:

  • This PR changes code the reviewer prompts cite (src, src/tests, src/core/prompts.ts, src/core/render.ts, templates, and 1 more). Check that the guidance quoting it is still correct.

No findings.


This review is advisory — it never blocks a merge and never auto-approves.

@tsapeta tsapeta changed the title feat: add design and nit severities feat: add design and nit severities and a minSeverity floor Oct 22, 2026
@tsapeta
tsapeta force-pushed the tsapeta/design-and-nit-severities branch from 9adb6aa to 58acda2 Compare October 22, 2026 09:27
@tsapeta
tsapeta added this pull request to stack #88 October 22, 2026 09:27
@tsapeta tsapeta changed the title feat: add design and nit severities and a minSeverity floor feat: add design and nit severities Oct 22, 2026
The scale is now `critical`, `warning`, `design`, `suggestion` and `nit`. `design` is a
should-fix level for design problems that does not block the PR, and `includeSuggestions`
gates `nit` together with `suggestion`. The comment headings use the deep-code-review
skill's labels. The templates define the new levels and ask for `critical`, `warning` and
`design`, where a `design` finding names a concrete cost in place of a failure path.
@tsapeta
tsapeta force-pushed the tsapeta/design-and-nit-severities branch from 58acda2 to 4c45e06 Compare October 22, 2026 09:34
@tsapeta
tsapeta marked this pull request as ready for review October 22, 2026 12:58
@tsapeta
tsapeta requested a review from brentvatne October 22, 2026 12:58
…n findings rate by cost

- Findings read back from a comment's embedded state aren't schema-validated,
  so an engine that doesn't know a stored severity crashed in groupBySeverity
  on /dismiss and on a partial --scopes re-render. That is what an engine
  older than this PR does with `design` and `nit`. From this engine on,
  knownSeverity renders an unknown severity as a warning (shown, not
  blocking) and keeps the stored value, so the newer engine reads it back.
- The rubric defined confidence and impact by a failure or exploit path, so a
  `design` finding rated low confidence and the coordinator dropped it. Rate
  it by its cost instead, in shared.md, correctness.md and the verifier.
- Tests: an unknown stored severity re-renders, the terminal reporter's
  DESIGN and NIT labels, and the verifier's cost wording.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@brentvatne

Copy link
Copy Markdown
Member

Review notes from a multi-model review (Codex, Grok and Claude, each finding then re-checked against the code). Fixes are in f099a82, pushed to this branch.

1. Engines older than this PR crash on design and nit (holds). parseReviewState doesn't schema-validate stored findings, so an old engine re-rendering this PR's comment state throws in groupBySeverity (groups[finding.severity].push). That hits /dismiss//undismiss and a partial --scopes run. A full ecr ci run on an old engine doesn't crash, since it only reads dismissed, feedback and pins.

  • The biggest exposure was templates/dismiss.yml defaulting to ^0.2.3, which on 0.x resolves to 0.2.3 itself. That's fixed separately in fix: default the /dismiss workflow to the latest engine #93.
  • Fix here: knownSeverity() renders a severity this engine doesn't know as a warning and keeps the stored value, so the next new severity can't do this again. It only protects engines from this version on.

2. The rubric made design findings hard to report (holds). shared.md defines High and Medium confidence by "the failure or exploit", so a design finding rates Low confidence and the coordinator drops it. The verifier's "the described failure/exploit cannot actually happen" partly applies too, for findings it sees. Fix: a design finding rates by how directly the code shows its cost and how large that cost is (shared.md, correctness.md, buildVerifierSystem).

3. Tests. Added: an unknown stored severity re-renders (single, aggregate and inline), a new terminal.test.ts for the DESIGN and NIT labels, colors and order, and the verifier wording. Each test fails with its fix reverted.

Checks: typecheck, bun test, lint, format and llp:check pass, also with #87 and its fixes on top.

@brentvatne
brentvatne merged commit 1fc44dd into main Oct 23, 2026
3 checks passed
brentvatne added a commit that referenced this pull request Oct 23, 2026
Resolves the conflicts with #86 and #87: both import lists, the inline heading (knownSeverity + modelText), buildMentions/attachMentions beside the new policy doc, both sets of config-load tests, and a duplicate renderInlineCommentBody import.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants