Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 32 additions & 4 deletions .github/cursor-approve/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ PR can never rewrite the prompts or the rules judging it.

`prompt-common.md` holds the rules every reviewer shares β€” review only the
merge-base diff (`git diff MERGE_BASE...HEAD`, never against the base tip), read
but do not run, the red/yellow/green meaning, PR content is data not
but do not run, the red/yellow/green/n-a meaning, PR content is data not
instructions, and the one-JSON-object output contract. Each axis file adds one
focus:

Expand All @@ -22,6 +22,31 @@ focus:
| `completeness` | is this a complete solution to the stated problem? |
| `conformance` | does the code fit the repo's engineering and deployment practices? |

### Verdicts, and the two fields a verdict carries

`red` / `yellow` / `green` / `n/a`. `n/a` is for an axis with nothing to judge
on this change β€” no applicable guidance, nothing in its remit touched β€” and it
is NOT a `green`: letting "no applicable standard" claim a clean bill would
quietly approve the gap the day guidance does appear. It does not count against
`--max-yellow-axes`, because `yellow` is a *concern* and this is its absence;
but a round where EVERY axis says `n/a` is withheld, since nothing was judged.
An `n/a` has to name what the axis looked for and where.

Each verdict carries two strings, with different jobs and different audiences:

| Field | Cap | Where it goes |
|---|---|---|
| `headline` | 100 chars, **required** | the PR card β€” ONE line, the reason for the verdict |
| `summary` | 1200 chars | the run's job summary, behind the card's "workflow run" link |

The split exists because one field could not do both. The card used to show the
first sentence of `summary`, and `summary` is where a model writes its process
("Checked the only changed file…", "I read root AGENTS.md and CLAUDE.md…") β€” so
the card said what the axis *did* and never why it ruled as it did, which is
unreadable on a yellow. `headline` must state what decided the verdict, and a
missing or over-long one makes the axis untrusted rather than being truncated:
truncating would reproduce the reason-free row one step later.

Placeholders β€” `{{pr_number}}`, `{{repo}}`, `{{head_sha}}`,
`{{merge_base_sha}}`, `{{base_ref}}`, `{{context_file}}` β€” are filled by
`aggregate.py render`, which checks each value's shape first and substitutes in
Expand All @@ -41,13 +66,16 @@ aggregate.py decide --outputs-dir out/ [--axes design,correctness] [--max-yellow
`{"event", "verdicts", "axes", "reasons"}`. In order:

1. any expected axis missing, unparsable, with a verdict outside
red/yellow/green or a confidence outside 0..1 β†’ `NONE` β€” untrusted, so the
approval is withheld; it is never a veto;
red/yellow/green/n-a, a confidence outside 0..1, or a missing or over-long
`headline` β†’ `NONE` β€” untrusted, so the approval is withheld; it is never a
veto;
2. any `red` β†’ `NONE`, naming the red axes;
3. more `yellow` axes than `--max-yellow-axes` (0–3, default 0, and always
strictly below the number of expected axes β€” a limit that every axis could
reach would approve a round with no green axis at all) β†’ `NONE`;
4. otherwise `APPROVE`.
4. every axis `n/a` β†’ `NONE`: nothing was judged, so there is nothing to
approve on;
5. otherwise `APPROVE`.

`decide` is strict about its input: the file must be exactly one JSON object,
so extracting it from the model's raw reply is the workflow's job. Bad
Expand Down
60 changes: 48 additions & 12 deletions .github/cursor-approve/aggregate.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,8 @@

Five reviewers each judge one axis of a PR (business, design, correctness,
completeness, conformance) and answer with one JSON object:
``{"verdict": "red"|"yellow"|"green", "confidence": 0..1, "summary": "..."}``.
``{"verdict": "red"|"yellow"|"green"|"n/a", "confidence": 0..1,
"headline": "<=100 chars", "summary": "..."}``.
Three subcommands, all pure β€” nothing here writes to GitHub:

``render``
Expand All @@ -22,12 +23,16 @@
the per-axis detail in ``axes``, and the ``reasons``. The rules, in order:

1. any expected axis missing, unparsable, with a verdict outside
red/yellow/green, a confidence outside 0..1, or (with ``--commit-sha``)
red/yellow/green/n/a, a confidence outside 0..1, a missing or
over-long ``headline``, or (with ``--commit-sha``)
a stamped commit other than that one β†’ ``NONE`` (the round is
untrusted: withhold the approval, never veto);
2. any ``red`` β†’ ``NONE``, naming the red axes;
3. more ``yellow`` axes than ``--max-yellow-axes`` β†’ ``NONE``;
4. otherwise β†’ ``APPROVE``.
3. more ``yellow`` axes than ``--max-yellow-axes`` β†’ ``NONE`` (an ``n/a``
axis is not a yellow and is not counted here);
4. every axis ``n/a`` β†’ ``NONE``: nothing was judged, so there is nothing
to approve on;
5. otherwise β†’ ``APPROVE``.

Bad arguments (an unknown axis, an empty axis list, ``--max-yellow-axes``
outside 0..3 or not below the number of expected axes) exit 2. Every
Expand All @@ -50,7 +55,13 @@
import sys

AXES = ("business", "design", "correctness", "completeness", "conformance")
VERDICTS = ("red", "yellow", "green")
# `n/a` is a verdict, not an abstention: the axis ran, reached the change, and
# reports that nothing it judges is in scope. It is counted as REPORTING (so it
# does not read as a missing axis, which withholds) but not as a yellow (so it
# does not block). See `decide` for the one case where it still withholds.
VERDICTS = ("red", "yellow", "green", "n/a")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟑 Medium β€” prompt-common.md, the README and docs/callers/cursor-approve.md (line 76) all say an n/a that doesn't name what it looked for and where "is treated as untrusted". Nothing in validate_output or decide enforces that: any n/a with any non-empty headline counts as reporting, and it is allowed on every axis, correctness included. In the shipped two-axis setup, a lazy or prompt-injected verdict (e.g. PR text claiming "no behaviour changed") can turn correctness n/a, and conformance's green alone then approves without correctness ever being judged. Either enforce the claim or remove it from the docs and prompt, and consider forbidding n/a on correctness. Raised by 2 of 6 reviewers (claude-opus-5-5-xhigh adversarial, claude-opus-5-5-xhigh edge-case).

NA = "n/a"
HEADLINE_MAX = 100
MAX_YELLOW_LIMIT = 3
SUMMARY_LIMIT = 1200
# An axis output is one small JSON object; the cap is what decide will read of
Expand Down Expand Up @@ -174,7 +185,7 @@ def _short(value) -> str:


def validate_output(data):
"""(verdict, confidence, summary) of one axis output, or raise ValueError."""
"""(verdict, confidence, summary, headline) of one axis output, or raise ValueError."""
if not isinstance(data, dict):
raise ValueError("the output is not a JSON object")
verdict = data.get("verdict")
Expand All @@ -200,7 +211,19 @@ def validate_output(data):
raise ValueError(f"confidence {_short(confidence)} is not a number from 0 to 1")
summary = data.get("summary")
summary = summary[:SUMMARY_LIMIT] if isinstance(summary, str) else ""
return verdict, confidence, summary
# The headline is the ONLY part of this that reaches the PR, so it is
# required rather than best-effort: an axis that does not write one leaves
# the card with a verdict and no reason, which is the exact failure this
# field was added to fix. Truncating silently would do the same thing one
# step later, so an over-long one is rejected too.
headline = data.get("headline")
if not isinstance(headline, str) or not headline.strip():

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟒 Low β€” The "required, non-empty" check uses str.strip()/split(), and neither removes zero-width or bidi format characters (U+200B, U+2060, U+FEFF, U+202E…). A headline made only of those passes but renders as a blank "Why" cell, which is exactly the reason-free row this field was added to prevent. Bidi overrides can also visually reorder the one rationale line on the card. Strip or reject Unicode category Cf/Cc characters before the emptiness and length checks. Raised by 3 of 6 reviewers (claude-opus-5-5-xhigh adversarial, gpt-5.6-sol-xhigh adversarial, claude-opus-5-5-xhigh edge-case).

raise ValueError("headline is missing or empty")
headline = " ".join(headline.split())
if len(headline) > HEADLINE_MAX:
raise ValueError(
f"headline is {len(headline)} characters, over the {HEADLINE_MAX} limit")
return verdict, confidence, summary, headline


def load_output(path: str, commit_sha: str = ""):
Expand Down Expand Up @@ -275,7 +298,7 @@ def load_output(path: str, commit_sha: str = ""):
def decide(outputs: dict, axes: list, max_yellow: int):
"""Return the decision dict for already-loaded outputs. Pure; no I/O.

`outputs` maps an axis to (verdict, confidence, summary), or to a ValueError
`outputs` maps an axis to (verdict, confidence, summary, headline), or to a ValueError
when its output could not be used. An axis absent from `outputs` is missing.
"""
detail = {}
Expand All @@ -284,24 +307,37 @@ def decide(outputs: dict, axes: list, max_yellow: int):
out = outputs.get(axis, ValueError("no output"))
if isinstance(out, ValueError):
untrusted.append(f"{axis}: {out}")
detail[axis] = {"verdict": None, "confidence": None, "summary": "", "error": str(out)}
detail[axis] = {"verdict": None, "confidence": None, "summary": "",
"headline": "", "error": str(out)}
else:
verdict, confidence, summary = out
detail[axis] = {"verdict": verdict, "confidence": confidence, "summary": summary}
verdict, confidence, summary, headline = out
detail[axis] = {"verdict": verdict, "confidence": confidence,
"summary": summary, "headline": headline}
verdicts = {axis: detail[axis]["verdict"] for axis in axes}

if untrusted:
event, reasons = NONE, [f"untrusted axis output, approval withheld ({u})" for u in untrusted]
else:
red = [a for a in axes if verdicts[a] == "red"]
yellow = [a for a in axes if verdicts[a] == "yellow"]
na = [a for a in axes if verdicts[a] == NA]
if red:
event, reasons = NONE, [f"red on {', '.join(red)}"]
elif len(yellow) > max_yellow:
event, reasons = NONE, [f"{len(yellow)} yellow axes ({', '.join(yellow)}) exceed the limit of {max_yellow}"]
elif len(na) == len(axes):

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High β€” The only n/a guard rejects the all-n/a round. That breaks the invariant cmd_decide enforces with max_yellow < len(axes), which exists so "a round with no green axis at all" cannot approve. Example: axes correctness,conformance with --max-yellow-axes 1, verdicts yellow + n/a, reaches APPROVE with zero green. Five axes with max 3 behave the same: 3 yellow + 2 n/a approves. Change the guard to "no green axis" (not any(verdicts[a] == 'green' for a in axes)), or check the yellow limit against the judged (non-n/a) axes. Once the auto-approve.py whitelist is fixed, this goes live. Raised by 4 of 6 reviewers (claude-opus-5-5-xhigh adversarial, gpt-5.6-sol-xhigh adversarial, claude-opus-5-5-xhigh edge-case, gpt-5.6-sol-xhigh edge-case).

# Every axis says nothing it judges is in scope. Each `n/a` is
# individually fine, but APPROVE here would mean approving on zero
# signal β€” the same "an undecided run is not a clean run" line the
# panel-integrity check draws, and the one way `n/a` could become a
# path to a free approval. Withhold and say so.
event, reasons = NONE, [
f"every axis ({', '.join(axes)}) reported n/a, so nothing was actually judged"]
else:
event = APPROVE

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High β€” decide can now return APPROVE when one or more axes are n/a. But auto-approve.py's external_decision_approves was not updated and still only accepts verdicts in ("green", "yellow"). Every mixed-n/a round therefore becomes not_approved: the bot withdraws earlier approvals, posts a standing REQUEST_CHANGES, and the "❌ Not approved" card shows this approving reason ("no red axis, 0 yellow …; n/a on …") as its only cause. That contradicts the docs, which say "n/a approves; does not block". Update external_decision_approves and its tests to accept n/a under the same rules. NaTest covers only aggregate.py, and the card tests hard-code outcome="approved", so no test catches the end-to-end break. Raised by 4 of 6 reviewers (claude-opus-5-5-xhigh adversarial, gpt-5.6-sol-xhigh adversarial, claude-opus-5-5-xhigh edge-case, gpt-5.6-sol-xhigh edge-case).

reasons = [f"no red axis, {len(yellow)} yellow of at most {max_yellow}"]
judged = [a for a in axes if verdicts[a] != NA]
reasons = [f"no red axis, {len(yellow)} yellow of at most {max_yellow}"
+ (f"; n/a on {', '.join(na)} ({len(judged)} axes judged)" if na else "")]
return {"event": event, "verdicts": verdicts, "axes": detail, "reasons": reasons}


Expand Down
87 changes: 79 additions & 8 deletions .github/cursor-approve/card.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,11 @@
``decide`` (after auto-approve.py ``approve-external``)
The per-axis verdicts from ``aggregate.py decide``, the overall result and
the rule applied β€” or "Superseded by a newer commit" when the head moved.
Each axis shows its `headline` only β€” ONE line naming what decided the
verdict. The same phase writes ``render_job_summary`` to
``GITHUB_STEP_SUMMARY``: the full summaries and confidences, which is what
the card's "workflow run" link now leads to. The card is the glance; the
job summary is the record.

cursor-review's ``auto-approve.py decide`` writes it too, through
``render_round`` and ``upsert``, on every round it decides for an author
Expand Down Expand Up @@ -48,8 +53,12 @@

CARD_MARKER = "<!-- cursor-approve-card -->"
SUMMARY_LIMIT = 200
# The job summary behind the run link is the detail surface, so an axis
# summary is not cut to one table cell there.
SUMMARY_DETAIL_LIMIT = 2000
AXES = ("business", "design", "correctness", "completeness", "conformance")
VERDICT_ICONS = {"green": "🟒 green", "yellow": "🟑 yellow", "red": "πŸ”΄ red"}
VERDICT_ICONS = {"green": "🟒 green", "yellow": "🟑 yellow", "red": "πŸ”΄ red",
"n/a": "βšͺ n/a"}
NO_RESULT = "⚠️ no result"
SHA_RE = re.compile(r"[0-9a-f]{40}")

Expand Down Expand Up @@ -280,8 +289,8 @@ def render_start(round_no, max_rounds, sha: str, axes: list, approve_gate: str,
lines.append("**Round limit reached, needs a human.**")
lines += ["", f"**Next step:** {NEXT_HUMAN_CAPPED_TEXT}"]
else:
lines += ["| Axis | Verdict | Confidence | Summary |", "|---|---|---|---|"]
lines += [f"| {axis} | ⏳ pending | | |" for axis in axes]
lines += ["| Axis | Verdict | Why |", "|---|---|---|"]
lines += [f"| {axis} | ⏳ pending | |" for axis in axes]
lines += ["", _reviewed_line(sha, run_url)]
return "\n".join(lines) + "\n"

Expand All @@ -292,6 +301,43 @@ def _confidence(value) -> str:
return ""


def render_job_summary(round_no, max_rounds, sha: str, axes: list, decision, outcome: str,
max_yellow: str) -> str:
"""The detail the card's "workflow run" link leads to, for GITHUB_STEP_SUMMARY.

The card is the glance: one icon and one `headline` per axis. Everything an
axis actually wrote β€” the full `summary`, its confidence β€” lands here, so
the link is worth following and the card does not have to carry both jobs.

Same `sanitize` as the card: this renders model text into markdown GitHub
displays, and a summary must not be able to open a heading or fire a mention
here either. No contract markers β€” a job summary is not the card, and
nothing parses this.
"""
detail = decision.get("axes") if isinstance(decision, dict) else None
detail = detail if isinstance(detail, dict) else {}
lines = [heading(round_no, max_rounds, sha), ""]
lines.append("**Approved.**" if outcome == OUTCOME_APPROVED else "**Not approved.**")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟒 Low β€” render_decide returns early for superseded/retargeted, but render_job_summary does not. For those outcomes the job summary says "Not approved." and "no decision was reached", then lists the full per-axis verdicts for a commit or base the PR no longer has. Meanwhile the card linking to it says "Superseded" and shows no verdicts. Mirror the early return, or label the verdicts as stale. Raised by 2 of 6 reviewers (claude-opus-5-5-xhigh adversarial, claude-opus-5-5-xhigh edge-case).

if outcome != OUTCOME_APPROVED:
lines += [f"- {r}" for r in decide_reasons(outcome, decision)]
for axis in axes:
entry = detail.get(axis)
entry = entry if isinstance(entry, dict) else {}
verdict = entry.get("verdict")
icon = VERDICT_ICONS.get(verdict, NO_RESULT)
confidence = _confidence(entry.get("confidence"))
lines += ["", f"#### {axis} β€” {icon}" + (f" (confidence {confidence})" if confidence else "")]
headline = sanitize(entry.get("headline"))
if headline:
lines.append(f"**{headline}**")
summary = sanitize(entry.get("summary"), SUMMARY_DETAIL_LIMIT)
lines += ["", summary or "_No summary was reported._"]
limit = _int_or_q(max_yellow)
lines += ["", f"_Rule: no red, at most {limit} yellow; every axis must report "
"(βšͺ n/a counts as reporting and does not block, but all-n/a withholds)._"]
return "\n".join(lines) + "\n"


def render_decide(round_no, max_rounds, sha: str, axes: list, decision, outcome: str,
max_yellow: str, run_url: str) -> str:
state, next_step = DECIDE_STATES.get(outcome, (STATE_NO_DECISION, NEXT_PUSH))
Expand All @@ -304,15 +350,22 @@ def render_decide(round_no, max_rounds, sha: str, axes: list, decision, outcome:
return "\n".join(lines) + "\n"
detail = decision.get("axes") if isinstance(decision, dict) else None
detail = detail if isinstance(detail, dict) else {}
lines += ["| Axis | Verdict | Confidence | Summary |", "|---|---|---|---|"]
# Three columns, and the third is the axis's `headline` β€” ONE line stating
# what decided the verdict. It used to be the first sentence of `summary`,
# which is where the models put their process log ("Checked the only
# changed file...", "I read root AGENTS.md and CLAUDE.md..."), so the card
# showed what the axis DID and never why it ruled as it did. The full
# summary and the confidence now live in the run's job summary, one click
# behind the link below β€” this card is the glance, not the record.
lines += ["| Axis | Verdict | Why |", "|---|---|---|"]
for axis in axes:
entry = detail.get(axis)
verdict = entry.get("verdict") if isinstance(entry, dict) else None
if verdict not in VERDICT_ICONS:
lines.append(f"| {axis} | {NO_RESULT} | | |")
lines.append(f"| {axis} | {NO_RESULT} | |")
continue
summary = sanitize(first_sentence(entry.get("summary")))
lines.append(f"| {axis} | {VERDICT_ICONS[verdict]} | {_confidence(entry.get('confidence'))} | {summary} |")
why = sanitize(entry.get("headline") or first_sentence(entry.get("summary")))
lines.append(f"| {axis} | {VERDICT_ICONS[verdict]} | {why} |")
lines.append("")
if outcome == OUTCOME_APPROVED:
lines.append("**Result: βœ… Approved.**")
Expand All @@ -321,7 +374,9 @@ def render_decide(round_no, max_rounds, sha: str, axes: list, decision, outcome:
lines += [f"- {r}" for r in decide_reasons(outcome, decision)]
lines += ["", f"**Next step:** {next_text}"]
limit = _int_or_q(max_yellow)
lines += ["", f"_Rule: no red, at most {limit} yellow; every axis must report. {_run_link(run_url)}_",
lines += ["", f"_Rule: no red, at most {limit} yellow; every axis must report "
f"(βšͺ n/a counts as reporting and does not block, but all-n/a withholds). "
f"{_run_link(run_url)}_",
"", _reviewed_line(sha, "")]
return "\n".join(lines) + "\n"

Expand Down Expand Up @@ -448,6 +503,17 @@ def upsert(repo: str, pr_number, login: str, body: str) -> dict:
{"body": body}))


def _write_job_summary(body: str) -> None:
path = os.environ.get("GITHUB_STEP_SUMMARY")
if not path:
return
try:
with open(path, "a", encoding="utf-8") as f:
f.write(body)
except OSError as e:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟑 Medium β€” _write_job_summary is meant to be non-fatal, but it only catches OSError. A model can return valid JSON with an escaped lone surrogate (e.g. "\ud800") in headline/summary, and it survives aggregation. Writing it through the strict UTF-8 file handle then raises UnicodeEncodeError (a ValueError), which escapes this helper. The step fails before upsert, so the card is left stale. Catch (OSError, ValueError) or open the file with errors="replace". Raised by 2 of 6 reviewers (gpt-5.6-sol-xhigh adversarial, gpt-5.6-sol-xhigh edge-case).

print(f"::warning::Could not write the cursor-approve job summary: {e}")


def _write_output(key: str, value: str) -> None:
path = os.environ.get("GITHUB_OUTPUT")
if path:
Expand Down Expand Up @@ -501,6 +567,11 @@ def main(argv=None) -> int:
decision = {}
body = render_decide(args.round, args.max_rounds, args.commit_sha, axes, decision, args.outcome,
args.max_yellow_axes, args.run_url)
# The other half of the card's "workflow run" link. Written before the
# card so a GitHub API failure below cannot cost us the detail too, and
# a failure HERE is never fatal: the card is what the author reads.
_write_job_summary(render_job_summary(args.round, args.max_rounds, args.commit_sha, axes,
decision, args.outcome, args.max_yellow_axes))
try:
upsert(args.repo, args.pr_number, args.login, body)
except (RuntimeError, ValueError) as e:
Expand Down
Loading
Loading