Skip to content

feat(cursor-approve): make the card say WHY, and split "nothing to judge" out of yellow - #395

Merged
mattmillerai merged 2 commits into
mainfrom
matt/axis-na-and-headline
Oct 9, 2026
Merged

mattmillerai merged 2 commits into
mainfrom
matt/axis-na-and-headline

Conversation

@mattmillerai

Copy link
Copy Markdown
Member

The problem, from the card itself

This is what a reader got on cloud#12712:

Axis Verdict Confidence Summary
correctness 🟢 green 0.90 Checked the only changed file.
conformance 🟡 yellow 0.60 I read root AGENTS.md and CLAUDE.md.

Neither cell says anything. The yellow is the worse half: the PR is blocked and
the card gives no reason for it.

Two separate causes.

1. The column was fed the wrong field. It rendered the first sentence of
summary, and summary is where a model writes its process log. The prompt
even asked for that ("Say what you checked and what decided the verdict" — the
models answer the first half). So the card reported what the axis did.

2. yellow meant three different things. prompt-common.md defined it as
"you are uncertain, or you found a material concern", and added "When the
evidence is thin, answer yellow with low confidence". So "I found a problem",
"I couldn't tell", and "nothing on my axis applies to this change" all came out
yellow and all blocked identically. The screenshot's conformance yellow is the
third kind — that is why no concern is named: there isn't one.

What this changes

n/a becomes its own verdict. For an axis with no purchase on the change.
It counts as REPORTING (so it is not a missing axis, which withholds) but not
as a yellow (so it does not block), and the prompt requires it to name what the
axis looked for and where — an n/a that doesn't is untrusted.

It is deliberately not a green. green means "I checked and it's clean";
letting "no applicable standard" claim that would quietly approve a conformance
gap the day guidance does appear, and nobody would learn the axis was never
exercised. And a round where every axis reports n/a is still withheld —
nothing was judged, so there is nothing to approve on. That is the one way this
could have become a path to a free approval, and it is closed.

A required headline. One line, ≤100 chars, stating what decided the
verdict
. The prompt gives per-verdict examples and bans the openers models
reach for:

  • red / yellow → the concern itself, and where. Not "reviewed the migration"
    but "Backfill rewrites rows the API still reads."
  • green → what you verified holds. Not "checked the changed file" but "Both
    new branches handle an empty payload."
  • n/a → what you looked for and did not find. Not "no issues" but "No
    AGENTS.md governs .github/."

Never begin with "Checked", "Read", "Reviewed", "Looked at", "I read" or
"Verified that I" — those spend the line on process. Begin with the finding.

A missing or over-long headline makes that axis untrusted, not truncated —
truncating would reproduce the reason-free row one step later.

The card becomes three columns, and the detail moves behind the link:

Axis Verdict Why
correctness 🟢 green Both new branches handle an empty payload.
conformance ⚪ n/a No AGENTS.md governs .github/.

Confidence goes with it. 0.60 on its own does not help anyone understand a
verdict, and it was taking a column from the one thing that does.

The run link now leads somewhere. No axis wrote GITHUB_STEP_SUMMARY
before, so "workflow run" led to a job list. The decide step now writes the
full summaries and confidences there. Card = the glance; job summary = the
record.

Scope

No input changes, no caller changes, no permission changes. aggregate.py extract already passed the whole object through, so headline flows end to
end without touching cursor-axis-base.yml beyond its header comment.

The one behavioural tightening: an axis that returns no headline is untrusted,
which withholds the approval. That is the intended failure mode (it is how
every other malformed field already behaves) and it only fires for an axis
running a prompt older than its aggregate.py — impossible here, since both
load from the same pinned ref.

Tests

196 passed in .github/cursor-approve/tests (+22), cursor-review's 1280
unaffected, check_workflow_pins.py and check_agents_md.py clean.

New: the headline contract (required / blank / non-string / the cap / the
whitespace fold / the folded length being what's measured) and the n/a rules
(not counted as yellow, named in the reason, all-n/a withholds, does not rescue
a red, does not excuse a missing axis, exact spelling) in test_aggregate.py;
the three-column card, the ⚪ icon, the rule line, the summary fallback, and the
job summary — including that card.py decide actually writes it to
GITHUB_STEP_SUMMARY — in test_card.py.

Card sanitization moved to the headline, since that is now the only model text
the card renders; the job summary's own sanitization is covered too (a #
heading, a mention and a forged card marker in a summary).

Not in this PR

The axes receive no prior-round context at all — build-ledger.py feeds
the review panel, and nothing feeds the axes — so a yellow the author has
already addressed comes back every round, and the card is a plain issue comment
that the ledger's reviews + reviewThreads query never sees either. Worth
fixing, separately.

🤖 Generated with Claude Code

…dge" out of yellow

The approve card showed each axis's verdict beside the first sentence of its
`summary`. That sentence is where a model writes its process log — "Checked the
only changed file...", "I read root AGENTS.md and CLAUDE.md..." — so the card
said what the axis DID and never why it ruled as it did. On a yellow that is
unreadable: the reader sees a blocked PR and no reason for it.

Two causes, both fixed here.

1. `yellow` conflated "I found a material concern" with "I could not tell" and
   with "nothing on my axis applies here", and all three blocked identically.
   `n/a` is now its own verdict for the third: it counts as REPORTING (so it is
   not a missing axis) but not as a yellow (so it does not block), and it must
   name what the axis looked for and where. It is deliberately not a `green` —
   an axis with no applicable guidance has not given the PR a clean bill, and
   recording that as one would quietly approve the gap the day guidance
   appears. A round where EVERY axis says `n/a` is still withheld: nothing was
   actually judged, which is the one way this could have become a free
   approval.

2. The card had no field whose job was the reason. Axes now return a required
   `headline` — one line, 100 characters, stating what decided the verdict, with
   per-verdict examples and the process-log openers banned outright. A missing
   or over-long one makes the axis untrusted rather than being truncated, since
   a verdict with no reason beside it is the exact failure the field exists to
   fix.

The card is now `Axis | Verdict | Why` — three columns, one line each. The full
`summary` and the confidence move to the run's job summary, which is what the
card's "workflow run" link now actually leads to: the card is the glance, the
job summary is the record.

Tests: the headline contract (required, blank, non-string, the cap, the fold)
and the n/a rules (not a yellow, all-n/a withholds, does not rescue a red or
excuse a missing axis, exact spelling) in test_aggregate.py; the three-column
card, the n/a icon, the rule line, the headline fallback and the job summary —
including that `decide` writes it to `GITHUB_STEP_SUMMARY` — in test_card.py.
Card sanitization now bites on the headline, which is the only model text the
card renders. 196 pass; cursor-review's 1280 unaffected.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

  • Run on-demand review

This review includes 9 billable files and costs up to $2.25.

  • Ask an admin to make reviews automatic

Open in CodeRabbit

Reviews can continue after your included limit without a manual trigger. An admin must approve usage-based billing.

Or wait 12 minutes for your next included review.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available. Your 131 included PR review attempts over the past 7 days set your current allowance at 1 review per hour.

Learn how review limits work.

Review configuration:

⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: ASSERTIVE
  • Plan: Team
  • Run ID: 49a743f3-d5ad-43fb-82ad-1f16b1bf6a73

📥 Commits

Reviewing files that changed from the base of the PR and between 0bfc0cb and 0b45dfe.


📒 Files selected for processing (9)
  • .github/cursor-approve/README.md
  • .github/cursor-approve/aggregate.py
  • .github/cursor-approve/card.py
  • .github/cursor-approve/prompt-common.md
  • .github/cursor-approve/tests/test_aggregate.py
  • .github/cursor-approve/tests/test_card.py
  • .github/workflows/cursor-approve.yml
  • .github/workflows/cursor-axis-base.yml
  • docs/callers/cursor-approve.md


  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@mattmillerai mattmillerai added the cursor-review Multi-model cursor review label Oct 9, 2026
…fact

`extract` validates the headline but dumps the model's original object, and
`extract_object` has no key allowlist — so the field reaching `decide` is
whatever the model wrote, not what `validate_output` returned. That wiring is
what makes this work end to end without a change to cursor-axis-base.yml, and
nothing covered it: every other test exercises `validate_output` or `decide`
directly.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 Cursor Review — Consolidated panel

Triggered by @mattmillerai.

Found 7 finding(s).

Severity Count
🟠 High 2
🟡 Medium 2
🟢 Low 3

Panel: 4/6 reviewers contributed findings.

Reviewers that did not contribute: kimi-k3-max:adversarial (error), kimi-k3-max:edge-case (error)

event, reasons = NONE, [
f"every axis ({', '.join(axes)}) reported n/a, so nothing was actually judged"]
else:
event = APPROVE

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High — decide can now return APPROVE when one or more axes are n/a. But auto-approve.py's external_decision_approves was not updated and still only accepts verdicts in ("green", "yellow"). Every mixed-n/a round therefore becomes not_approved: the bot withdraws earlier approvals, posts a standing REQUEST_CHANGES, and the "❌ Not approved" card shows this approving reason ("no red axis, 0 yellow …; n/a on …") as its only cause. That contradicts the docs, which say "n/a approves; does not block". Update external_decision_approves and its tests to accept n/a under the same rules. NaTest covers only aggregate.py, and the card tests hard-code outcome="approved", so no test catches the end-to-end break. Raised by 4 of 6 reviewers (claude-opus-5-5-xhigh adversarial, gpt-5.6-sol-xhigh adversarial, claude-opus-5-5-xhigh edge-case, gpt-5.6-sol-xhigh edge-case).

event, reasons = NONE, [f"red on {', '.join(red)}"]
elif len(yellow) > max_yellow:
event, reasons = NONE, [f"{len(yellow)} yellow axes ({', '.join(yellow)}) exceed the limit of {max_yellow}"]
elif len(na) == len(axes):

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High — The only n/a guard rejects the all-n/a round. That breaks the invariant cmd_decide enforces with max_yellow < len(axes), which exists so "a round with no green axis at all" cannot approve. Example: axes correctness,conformance with --max-yellow-axes 1, verdicts yellow + n/a, reaches APPROVE with zero green. Five axes with max 3 behave the same: 3 yellow + 2 n/a approves. Change the guard to "no green axis" (not any(verdicts[a] == 'green' for a in axes)), or check the yellow limit against the judged (non-n/a) axes. Once the auto-approve.py whitelist is fixed, this goes live. Raised by 4 of 6 reviewers (claude-opus-5-5-xhigh adversarial, gpt-5.6-sol-xhigh adversarial, claude-opus-5-5-xhigh edge-case, gpt-5.6-sol-xhigh edge-case).

# reports that nothing it judges is in scope. It is counted as REPORTING (so it
# does not read as a missing axis, which withholds) but not as a yellow (so it
# does not block). See `decide` for the one case where it still withholds.
VERDICTS = ("red", "yellow", "green", "n/a")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium — prompt-common.md, the README and docs/callers/cursor-approve.md (line 76) all say an n/a that doesn't name what it looked for and where "is treated as untrusted". Nothing in validate_output or decide enforces that: any n/a with any non-empty headline counts as reporting, and it is allowed on every axis, correctness included. In the shipped two-axis setup, a lazy or prompt-injected verdict (e.g. PR text claiming "no behaviour changed") can turn correctness n/a, and conformance's green alone then approves without correctness ever being judged. Either enforce the claim or remove it from the docs and prompt, and consider forbidding n/a on correctness. Raised by 2 of 6 reviewers (claude-opus-5-5-xhigh adversarial, claude-opus-5-5-xhigh edge-case).

try:
with open(path, "a", encoding="utf-8") as f:
f.write(body)
except OSError as e:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium — _write_job_summary is meant to be non-fatal, but it only catches OSError. A model can return valid JSON with an escaped lone surrogate (e.g. "\ud800") in headline/summary, and it survives aggregation. Writing it through the strict UTF-8 file handle then raises UnicodeEncodeError (a ValueError), which escapes this helper. The step fails before upsert, so the card is left stale. Catch (OSError, ValueError) or open the file with errors="replace". Raised by 2 of 6 reviewers (gpt-5.6-sol-xhigh adversarial, gpt-5.6-sol-xhigh edge-case).

# field was added to fix. Truncating silently would do the same thing one
# step later, so an over-long one is rejected too.
headline = data.get("headline")
if not isinstance(headline, str) or not headline.strip():

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Low — The "required, non-empty" check uses str.strip()/split(), and neither removes zero-width or bidi format characters (U+200B, U+2060, U+FEFF, U+202E…). A headline made only of those passes but renders as a blank "Why" cell, which is exactly the reason-free row this field was added to prevent. Bidi overrides can also visually reorder the one rationale line on the card. Strip or reject Unicode category Cf/Cc characters before the emptiness and length checks. Raised by 3 of 6 reviewers (claude-opus-5-5-xhigh adversarial, gpt-5.6-sol-xhigh adversarial, claude-opus-5-5-xhigh edge-case).

detail = decision.get("axes") if isinstance(decision, dict) else None
detail = detail if isinstance(detail, dict) else {}
lines = [heading(round_no, max_rounds, sha), ""]
lines.append("**Approved.**" if outcome == OUTCOME_APPROVED else "**Not approved.**")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Low — render_decide returns early for superseded/retargeted, but render_job_summary does not. For those outcomes the job summary says "Not approved." and "no decision was reached", then lists the full per-axis verdicts for a commit or base the PR no longer has. Meanwhile the card linking to it says "Superseded" and shows no verdicts. Mirror the early return, or label the verdicts as stale. Raised by 2 of 6 reviewers (claude-opus-5-5-xhigh adversarial, claude-opus-5-5-xhigh edge-case).

# stamped from this job's input, not taken from the model). A failing run uploads nothing,
# object `{"verdict", "confidence", "headline", "summary", "commit_sha"}`
# (commit_sha is stamped from this job's input, not taken from the model).
# `verdict` is red/yellow/green/n-a; `headline` is the one line the PR card

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Low — This artifact-contract comment, and README.md (the "red/yellow/green/n-a" lines), spell the verdict n-a. VERDICTS accepts only the exact string n/a, and test_the_spelling_is_exact asserts that n-a is rejected. Anyone writing a custom axis or prompt from this contract would produce outputs that silently fail closed. Spell it n/a everywhere. Raised by 1 of 6 reviewers (claude-opus-5-5-xhigh edge-case).

@mattmillerai
mattmillerai merged commit d38bd7e into main Oct 9, 2026
8 checks passed
@mattmillerai
mattmillerai deleted the matt/axis-na-and-headline branch October 9, 2026 19:12
mattmillerai added a commit that referenced this pull request Oct 9, 2026
…ing the axis (#398)

* fix(cursor-approve): clamp an over-long axis headline instead of failing the axis

#395 made `headline` required and capped at 100 characters, and rejected an
over-long one. The cap is enforced in `extract`, which the axis job's "Extract
the verdict" step runs, so a model that wrote a few extra words (106 and 113
characters on cloud PR #12748) exited 1 there and turned the whole axis job
red. Model output length is not deterministic, so neither the prompt nor a
retry can guarantee the cap.

Enforce it deterministically instead: `validate_output` clamps the folded
headline to HEADLINE_MAX at a word boundary with "…" (hard cut for a single
long token), and `extract` writes the clamped value into the artifact. A
missing or empty headline is still untrusted.

* fix(cursor-approve): fail closed on a separator-only clamp, keep the reason on an early space

- A headline that clamps to nothing but separators ("-" * 101) now raises
  instead of returning a bare "…", so the axis is untrusted as before.
- The word-boundary cut falls back to the hard cut when the last fitting space
  is in the first half ("Fix: <long path>" no longer clamps to "Fix…").
- The boundary search now sees a space at index HEADLINE_MAX - 1, so a word
  that fits exactly with "…" is kept.
- docs/callers/cursor-approve.md describes the clamp instead of the old reject.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
mattmillerai added a commit that referenced this pull request Oct 12, 2026
…ate.py does (#421)

* fix(cursor-approve): approve-external accepts an n/a axis like aggregate.py does

aggregate.py approves a round where one axis is n/a and the rest are
green/yellow (#395), but approve-external's re-check still demanded
green/yellow on every axis, so it withheld every such round with
"the axes did not approve". Share aggregate's rule: n/a reports and
does not block, all-n/a withholds. Add a test that pins the two halves
to the same decision on every verdict combination.

* test(cursor-approve): parity test covers every yellow cap; document the cap is not re-checked

external_decision_approves never receives max_yellow_axes, so say so
instead of claiming it mirrors aggregate.py's whole rule, and run the
aggregate.py parity test under caps 0 and 1 (the only ones the CLI
accepts for two axes) so the yellow-overflow branch is exercised.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cursor-review Multi-model cursor review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants