Repository navigation
feat(cursor-approve): make the card say WHY, and split "nothing to judge" out of yellow - #395
Conversation
…dge" out of yellow The approve card showed each axis's verdict beside the first sentence of its `summary`. That sentence is where a model writes its process log — "Checked the only changed file...", "I read root AGENTS.md and CLAUDE.md..." — so the card said what the axis DID and never why it ruled as it did. On a yellow that is unreadable: the reader sees a blocked PR and no reason for it. Two causes, both fixed here. 1. `yellow` conflated "I found a material concern" with "I could not tell" and with "nothing on my axis applies here", and all three blocked identically. `n/a` is now its own verdict for the third: it counts as REPORTING (so it is not a missing axis) but not as a yellow (so it does not block), and it must name what the axis looked for and where. It is deliberately not a `green` — an axis with no applicable guidance has not given the PR a clean bill, and recording that as one would quietly approve the gap the day guidance appears. A round where EVERY axis says `n/a` is still withheld: nothing was actually judged, which is the one way this could have become a free approval. 2. The card had no field whose job was the reason. Axes now return a required `headline` — one line, 100 characters, stating what decided the verdict, with per-verdict examples and the process-log openers banned outright. A missing or over-long one makes the axis untrusted rather than being truncated, since a verdict with no reason beside it is the exact failure the field exists to fix. The card is now `Axis | Verdict | Why` — three columns, one line each. The full `summary` and the confidence move to the run's job summary, which is what the card's "workflow run" link now actually leads to: the card is the glance, the job summary is the record. Tests: the headline contract (required, blank, non-string, the cap, the fold) and the n/a rules (not a yellow, all-n/a withholds, does not rescue a red or excuse a missing axis, exact spelling) in test_aggregate.py; the three-column card, the n/a icon, the rule line, the headline fallback and the job summary — including that `decide` writes it to `GITHUB_STEP_SUMMARY` — in test_card.py. Card sanitization now bites on the headline, which is the only model text the card renders. 196 pass; cursor-review's 1280 unaffected. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Warning Review limit reached
This review includes 9 billable files and costs up to $2.25.
Reviews can continue after your included limit without a manual trigger. An admin must approve usage-based billing. Or wait 12 minutes for your next included review. View limit detailsLimit details: You’ve used the included review currently available. Your 131 included PR review attempts over the past 7 days set your current allowance at 1 review per hour. Review configuration: ⚙️ Run configuration
📒 Files selected for processing (9)
Comment |
…fact `extract` validates the headline but dumps the model's original object, and `extract_object` has no key allowlist — so the field reaching `decide` is whatever the model wrote, not what `validate_output` returned. That wiring is what makes this work end to end without a change to cursor-axis-base.yml, and nothing covered it: every other test exercises `validate_output` or `decide` directly. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
There was a problem hiding this comment.
🔍 Cursor Review — Consolidated panel
Triggered by @mattmillerai.
Found 7 finding(s).
| Severity | Count |
|---|---|
| 🟠 High | 2 |
| 🟡 Medium | 2 |
| 🟢 Low | 3 |
Panel: 4/6 reviewers contributed findings.
Reviewers that did not contribute: kimi-k3-max:adversarial (error), kimi-k3-max:edge-case (error)
| event, reasons = NONE, [ | ||
| f"every axis ({', '.join(axes)}) reported n/a, so nothing was actually judged"] | ||
| else: | ||
| event = APPROVE |
There was a problem hiding this comment.
🟠 High — decide can now return APPROVE when one or more axes are n/a. But auto-approve.py's external_decision_approves was not updated and still only accepts verdicts in ("green", "yellow"). Every mixed-n/a round therefore becomes not_approved: the bot withdraws earlier approvals, posts a standing REQUEST_CHANGES, and the "❌ Not approved" card shows this approving reason ("no red axis, 0 yellow …; n/a on …") as its only cause. That contradicts the docs, which say "n/a approves; does not block". Update external_decision_approves and its tests to accept n/a under the same rules. NaTest covers only aggregate.py, and the card tests hard-code outcome="approved", so no test catches the end-to-end break. Raised by 4 of 6 reviewers (claude-opus-5-5-xhigh adversarial, gpt-5.6-sol-xhigh adversarial, claude-opus-5-5-xhigh edge-case, gpt-5.6-sol-xhigh edge-case).
| event, reasons = NONE, [f"red on {', '.join(red)}"] | ||
| elif len(yellow) > max_yellow: | ||
| event, reasons = NONE, [f"{len(yellow)} yellow axes ({', '.join(yellow)}) exceed the limit of {max_yellow}"] | ||
| elif len(na) == len(axes): |
There was a problem hiding this comment.
🟠 High — The only n/a guard rejects the all-n/a round. That breaks the invariant cmd_decide enforces with max_yellow < len(axes), which exists so "a round with no green axis at all" cannot approve. Example: axes correctness,conformance with --max-yellow-axes 1, verdicts yellow + n/a, reaches APPROVE with zero green. Five axes with max 3 behave the same: 3 yellow + 2 n/a approves. Change the guard to "no green axis" (not any(verdicts[a] == 'green' for a in axes)), or check the yellow limit against the judged (non-n/a) axes. Once the auto-approve.py whitelist is fixed, this goes live. Raised by 4 of 6 reviewers (claude-opus-5-5-xhigh adversarial, gpt-5.6-sol-xhigh adversarial, claude-opus-5-5-xhigh edge-case, gpt-5.6-sol-xhigh edge-case).
| # reports that nothing it judges is in scope. It is counted as REPORTING (so it | ||
| # does not read as a missing axis, which withholds) but not as a yellow (so it | ||
| # does not block). See `decide` for the one case where it still withholds. | ||
| VERDICTS = ("red", "yellow", "green", "n/a") |
There was a problem hiding this comment.
🟡 Medium — prompt-common.md, the README and docs/callers/cursor-approve.md (line 76) all say an n/a that doesn't name what it looked for and where "is treated as untrusted". Nothing in validate_output or decide enforces that: any n/a with any non-empty headline counts as reporting, and it is allowed on every axis, correctness included. In the shipped two-axis setup, a lazy or prompt-injected verdict (e.g. PR text claiming "no behaviour changed") can turn correctness n/a, and conformance's green alone then approves without correctness ever being judged. Either enforce the claim or remove it from the docs and prompt, and consider forbidding n/a on correctness. Raised by 2 of 6 reviewers (claude-opus-5-5-xhigh adversarial, claude-opus-5-5-xhigh edge-case).
| try: | ||
| with open(path, "a", encoding="utf-8") as f: | ||
| f.write(body) | ||
| except OSError as e: |
There was a problem hiding this comment.
🟡 Medium — _write_job_summary is meant to be non-fatal, but it only catches OSError. A model can return valid JSON with an escaped lone surrogate (e.g. "\ud800") in headline/summary, and it survives aggregation. Writing it through the strict UTF-8 file handle then raises UnicodeEncodeError (a ValueError), which escapes this helper. The step fails before upsert, so the card is left stale. Catch (OSError, ValueError) or open the file with errors="replace". Raised by 2 of 6 reviewers (gpt-5.6-sol-xhigh adversarial, gpt-5.6-sol-xhigh edge-case).
| # field was added to fix. Truncating silently would do the same thing one | ||
| # step later, so an over-long one is rejected too. | ||
| headline = data.get("headline") | ||
| if not isinstance(headline, str) or not headline.strip(): |
There was a problem hiding this comment.
🟢 Low — The "required, non-empty" check uses str.strip()/split(), and neither removes zero-width or bidi format characters (U+200B, U+2060, U+FEFF, U+202E…). A headline made only of those passes but renders as a blank "Why" cell, which is exactly the reason-free row this field was added to prevent. Bidi overrides can also visually reorder the one rationale line on the card. Strip or reject Unicode category Cf/Cc characters before the emptiness and length checks. Raised by 3 of 6 reviewers (claude-opus-5-5-xhigh adversarial, gpt-5.6-sol-xhigh adversarial, claude-opus-5-5-xhigh edge-case).
| detail = decision.get("axes") if isinstance(decision, dict) else None | ||
| detail = detail if isinstance(detail, dict) else {} | ||
| lines = [heading(round_no, max_rounds, sha), ""] | ||
| lines.append("**Approved.**" if outcome == OUTCOME_APPROVED else "**Not approved.**") |
There was a problem hiding this comment.
🟢 Low — render_decide returns early for superseded/retargeted, but render_job_summary does not. For those outcomes the job summary says "Not approved." and "no decision was reached", then lists the full per-axis verdicts for a commit or base the PR no longer has. Meanwhile the card linking to it says "Superseded" and shows no verdicts. Mirror the early return, or label the verdicts as stale. Raised by 2 of 6 reviewers (claude-opus-5-5-xhigh adversarial, claude-opus-5-5-xhigh edge-case).
| # stamped from this job's input, not taken from the model). A failing run uploads nothing, | ||
| # object `{"verdict", "confidence", "headline", "summary", "commit_sha"}` | ||
| # (commit_sha is stamped from this job's input, not taken from the model). | ||
| # `verdict` is red/yellow/green/n-a; `headline` is the one line the PR card |
There was a problem hiding this comment.
🟢 Low — This artifact-contract comment, and README.md (the "red/yellow/green/n-a" lines), spell the verdict n-a. VERDICTS accepts only the exact string n/a, and test_the_spelling_is_exact asserts that n-a is rejected. Anyone writing a custom axis or prompt from this contract would produce outputs that silently fail closed. Spell it n/a everywhere. Raised by 1 of 6 reviewers (claude-opus-5-5-xhigh edge-case).
…ing the axis (#398) * fix(cursor-approve): clamp an over-long axis headline instead of failing the axis #395 made `headline` required and capped at 100 characters, and rejected an over-long one. The cap is enforced in `extract`, which the axis job's "Extract the verdict" step runs, so a model that wrote a few extra words (106 and 113 characters on cloud PR #12748) exited 1 there and turned the whole axis job red. Model output length is not deterministic, so neither the prompt nor a retry can guarantee the cap. Enforce it deterministically instead: `validate_output` clamps the folded headline to HEADLINE_MAX at a word boundary with "…" (hard cut for a single long token), and `extract` writes the clamped value into the artifact. A missing or empty headline is still untrusted. * fix(cursor-approve): fail closed on a separator-only clamp, keep the reason on an early space - A headline that clamps to nothing but separators ("-" * 101) now raises instead of returning a bare "…", so the axis is untrusted as before. - The word-boundary cut falls back to the hard cut when the last fitting space is in the first half ("Fix: <long path>" no longer clamps to "Fix…"). - The boundary search now sees a space at index HEADLINE_MAX - 1, so a word that fits exactly with "…" is kept. - docs/callers/cursor-approve.md describes the clamp instead of the old reject. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…ate.py does (#421) * fix(cursor-approve): approve-external accepts an n/a axis like aggregate.py does aggregate.py approves a round where one axis is n/a and the rest are green/yellow (#395), but approve-external's re-check still demanded green/yellow on every axis, so it withheld every such round with "the axes did not approve". Share aggregate's rule: n/a reports and does not block, all-n/a withholds. Add a test that pins the two halves to the same decision on every verdict combination. * test(cursor-approve): parity test covers every yellow cap; document the cap is not re-checked external_decision_approves never receives max_yellow_axes, so say so instead of claiming it mirrors aggregate.py's whole rule, and run the aggregate.py parity test under caps 0 and 1 (the only ones the CLI accepts for two axes) so the yellow-overflow branch is exercised. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The problem, from the card itself
This is what a reader got on cloud#12712:
Neither cell says anything. The yellow is the worse half: the PR is blocked and
the card gives no reason for it.
Two separate causes.
1. The column was fed the wrong field. It rendered the first sentence of
summary, andsummaryis where a model writes its process log. The prompteven asked for that ("Say what you checked and what decided the verdict" — the
models answer the first half). So the card reported what the axis did.
2.
yellowmeant three different things.prompt-common.mddefined it as"you are uncertain, or you found a material concern", and added "When the
evidence is thin, answer
yellowwith low confidence". So "I found a problem","I couldn't tell", and "nothing on my axis applies to this change" all came out
yellow and all blocked identically. The screenshot's conformance yellow is the
third kind — that is why no concern is named: there isn't one.
What this changes
n/abecomes its own verdict. For an axis with no purchase on the change.It counts as REPORTING (so it is not a missing axis, which withholds) but not
as a yellow (so it does not block), and the prompt requires it to name what the
axis looked for and where — an
n/athat doesn't is untrusted.It is deliberately not a
green.greenmeans "I checked and it's clean";letting "no applicable standard" claim that would quietly approve a conformance
gap the day guidance does appear, and nobody would learn the axis was never
exercised. And a round where every axis reports
n/ais still withheld —nothing was judged, so there is nothing to approve on. That is the one way this
could have become a path to a free approval, and it is closed.
A required
headline. One line, ≤100 chars, stating what decided theverdict. The prompt gives per-verdict examples and bans the openers models
reach for:
A missing or over-long headline makes that axis untrusted, not truncated —
truncating would reproduce the reason-free row one step later.
The card becomes three columns, and the detail moves behind the link:
.github/.Confidence goes with it.
0.60on its own does not help anyone understand averdict, and it was taking a column from the one thing that does.
The run link now leads somewhere. No axis wrote
GITHUB_STEP_SUMMARYbefore, so "workflow run" led to a job list. The decide step now writes the
full summaries and confidences there. Card = the glance; job summary = the
record.
Scope
No input changes, no caller changes, no permission changes.
aggregate.py extractalready passed the whole object through, soheadlineflows end toend without touching
cursor-axis-base.ymlbeyond its header comment.The one behavioural tightening: an axis that returns no
headlineis untrusted,which withholds the approval. That is the intended failure mode (it is how
every other malformed field already behaves) and it only fires for an axis
running a prompt older than its
aggregate.py— impossible here, since bothload from the same pinned ref.
Tests
196 passedin.github/cursor-approve/tests(+22), cursor-review's1280unaffected,
check_workflow_pins.pyandcheck_agents_md.pyclean.New: the headline contract (required / blank / non-string / the cap / the
whitespace fold / the folded length being what's measured) and the n/a rules
(not counted as yellow, named in the reason, all-n/a withholds, does not rescue
a red, does not excuse a missing axis, exact spelling) in
test_aggregate.py;the three-column card, the ⚪ icon, the rule line, the summary fallback, and the
job summary — including that
card.py decideactually writes it toGITHUB_STEP_SUMMARY— intest_card.py.Card sanitization moved to the headline, since that is now the only model text
the card renders; the job summary's own sanitization is covered too (a
#heading, a mention and a forged card marker in a
summary).Not in this PR
The axes receive no prior-round context at all —
build-ledger.pyfeedsthe review panel, and nothing feeds the axes — so a yellow the author has
already addressed comes back every round, and the card is a plain issue comment
that the ledger's
reviews+reviewThreadsquery never sees either. Worthfixing, separately.
🤖 Generated with Claude Code