Repository navigation
feat(cursor-approve): make the card say WHY, and split "nothing to judge" out of yellow #395
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. Weβll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -3,7 +3,8 @@ | |
|
|
||
| Five reviewers each judge one axis of a PR (business, design, correctness, | ||
| completeness, conformance) and answer with one JSON object: | ||
| ``{"verdict": "red"|"yellow"|"green", "confidence": 0..1, "summary": "..."}``. | ||
| ``{"verdict": "red"|"yellow"|"green"|"n/a", "confidence": 0..1, | ||
| "headline": "<=100 chars", "summary": "..."}``. | ||
| Three subcommands, all pure β nothing here writes to GitHub: | ||
|
|
||
| ``render`` | ||
|
|
@@ -22,12 +23,16 @@ | |
| the per-axis detail in ``axes``, and the ``reasons``. The rules, in order: | ||
|
|
||
| 1. any expected axis missing, unparsable, with a verdict outside | ||
| red/yellow/green, a confidence outside 0..1, or (with ``--commit-sha``) | ||
| red/yellow/green/n/a, a confidence outside 0..1, a missing or | ||
| over-long ``headline``, or (with ``--commit-sha``) | ||
| a stamped commit other than that one β ``NONE`` (the round is | ||
| untrusted: withhold the approval, never veto); | ||
| 2. any ``red`` β ``NONE``, naming the red axes; | ||
| 3. more ``yellow`` axes than ``--max-yellow-axes`` β ``NONE``; | ||
| 4. otherwise β ``APPROVE``. | ||
| 3. more ``yellow`` axes than ``--max-yellow-axes`` β ``NONE`` (an ``n/a`` | ||
| axis is not a yellow and is not counted here); | ||
| 4. every axis ``n/a`` β ``NONE``: nothing was judged, so there is nothing | ||
| to approve on; | ||
| 5. otherwise β ``APPROVE``. | ||
|
|
||
| Bad arguments (an unknown axis, an empty axis list, ``--max-yellow-axes`` | ||
| outside 0..3 or not below the number of expected axes) exit 2. Every | ||
|
|
@@ -50,7 +55,13 @@ | |
| import sys | ||
|
|
||
| AXES = ("business", "design", "correctness", "completeness", "conformance") | ||
| VERDICTS = ("red", "yellow", "green") | ||
| # `n/a` is a verdict, not an abstention: the axis ran, reached the change, and | ||
| # reports that nothing it judges is in scope. It is counted as REPORTING (so it | ||
| # does not read as a missing axis, which withholds) but not as a yellow (so it | ||
| # does not block). See `decide` for the one case where it still withholds. | ||
| VERDICTS = ("red", "yellow", "green", "n/a") | ||
| NA = "n/a" | ||
| HEADLINE_MAX = 100 | ||
| MAX_YELLOW_LIMIT = 3 | ||
| SUMMARY_LIMIT = 1200 | ||
| # An axis output is one small JSON object; the cap is what decide will read of | ||
|
|
@@ -174,7 +185,7 @@ def _short(value) -> str: | |
|
|
||
|
|
||
| def validate_output(data): | ||
| """(verdict, confidence, summary) of one axis output, or raise ValueError.""" | ||
| """(verdict, confidence, summary, headline) of one axis output, or raise ValueError.""" | ||
| if not isinstance(data, dict): | ||
| raise ValueError("the output is not a JSON object") | ||
| verdict = data.get("verdict") | ||
|
|
@@ -200,7 +211,19 @@ def validate_output(data): | |
| raise ValueError(f"confidence {_short(confidence)} is not a number from 0 to 1") | ||
| summary = data.get("summary") | ||
| summary = summary[:SUMMARY_LIMIT] if isinstance(summary, str) else "" | ||
| return verdict, confidence, summary | ||
| # The headline is the ONLY part of this that reaches the PR, so it is | ||
| # required rather than best-effort: an axis that does not write one leaves | ||
| # the card with a verdict and no reason, which is the exact failure this | ||
| # field was added to fix. Truncating silently would do the same thing one | ||
| # step later, so an over-long one is rejected too. | ||
| headline = data.get("headline") | ||
| if not isinstance(headline, str) or not headline.strip(): | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. π’ Low β The "required, non-empty" check uses |
||
| raise ValueError("headline is missing or empty") | ||
| headline = " ".join(headline.split()) | ||
| if len(headline) > HEADLINE_MAX: | ||
| raise ValueError( | ||
| f"headline is {len(headline)} characters, over the {HEADLINE_MAX} limit") | ||
| return verdict, confidence, summary, headline | ||
|
|
||
|
|
||
| def load_output(path: str, commit_sha: str = ""): | ||
|
|
@@ -275,7 +298,7 @@ def load_output(path: str, commit_sha: str = ""): | |
| def decide(outputs: dict, axes: list, max_yellow: int): | ||
| """Return the decision dict for already-loaded outputs. Pure; no I/O. | ||
|
|
||
| `outputs` maps an axis to (verdict, confidence, summary), or to a ValueError | ||
| `outputs` maps an axis to (verdict, confidence, summary, headline), or to a ValueError | ||
| when its output could not be used. An axis absent from `outputs` is missing. | ||
| """ | ||
| detail = {} | ||
|
|
@@ -284,24 +307,37 @@ def decide(outputs: dict, axes: list, max_yellow: int): | |
| out = outputs.get(axis, ValueError("no output")) | ||
| if isinstance(out, ValueError): | ||
| untrusted.append(f"{axis}: {out}") | ||
| detail[axis] = {"verdict": None, "confidence": None, "summary": "", "error": str(out)} | ||
| detail[axis] = {"verdict": None, "confidence": None, "summary": "", | ||
| "headline": "", "error": str(out)} | ||
| else: | ||
| verdict, confidence, summary = out | ||
| detail[axis] = {"verdict": verdict, "confidence": confidence, "summary": summary} | ||
| verdict, confidence, summary, headline = out | ||
| detail[axis] = {"verdict": verdict, "confidence": confidence, | ||
| "summary": summary, "headline": headline} | ||
| verdicts = {axis: detail[axis]["verdict"] for axis in axes} | ||
|
|
||
| if untrusted: | ||
| event, reasons = NONE, [f"untrusted axis output, approval withheld ({u})" for u in untrusted] | ||
| else: | ||
| red = [a for a in axes if verdicts[a] == "red"] | ||
| yellow = [a for a in axes if verdicts[a] == "yellow"] | ||
| na = [a for a in axes if verdicts[a] == NA] | ||
| if red: | ||
| event, reasons = NONE, [f"red on {', '.join(red)}"] | ||
| elif len(yellow) > max_yellow: | ||
| event, reasons = NONE, [f"{len(yellow)} yellow axes ({', '.join(yellow)}) exceed the limit of {max_yellow}"] | ||
| elif len(na) == len(axes): | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. π High β The only |
||
| # Every axis says nothing it judges is in scope. Each `n/a` is | ||
| # individually fine, but APPROVE here would mean approving on zero | ||
| # signal β the same "an undecided run is not a clean run" line the | ||
| # panel-integrity check draws, and the one way `n/a` could become a | ||
| # path to a free approval. Withhold and say so. | ||
| event, reasons = NONE, [ | ||
| f"every axis ({', '.join(axes)}) reported n/a, so nothing was actually judged"] | ||
| else: | ||
| event = APPROVE | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. π High β |
||
| reasons = [f"no red axis, {len(yellow)} yellow of at most {max_yellow}"] | ||
| judged = [a for a in axes if verdicts[a] != NA] | ||
| reasons = [f"no red axis, {len(yellow)} yellow of at most {max_yellow}" | ||
| + (f"; n/a on {', '.join(na)} ({len(judged)} axes judged)" if na else "")] | ||
| return {"event": event, "verdicts": verdicts, "axes": detail, "reasons": reasons} | ||
|
|
||
|
|
||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -18,6 +18,11 @@ | |
| ``decide`` (after auto-approve.py ``approve-external``) | ||
| The per-axis verdicts from ``aggregate.py decide``, the overall result and | ||
| the rule applied β or "Superseded by a newer commit" when the head moved. | ||
| Each axis shows its `headline` only β ONE line naming what decided the | ||
| verdict. The same phase writes ``render_job_summary`` to | ||
| ``GITHUB_STEP_SUMMARY``: the full summaries and confidences, which is what | ||
| the card's "workflow run" link now leads to. The card is the glance; the | ||
| job summary is the record. | ||
|
|
||
| cursor-review's ``auto-approve.py decide`` writes it too, through | ||
| ``render_round`` and ``upsert``, on every round it decides for an author | ||
|
|
@@ -48,8 +53,12 @@ | |
|
|
||
| CARD_MARKER = "<!-- cursor-approve-card -->" | ||
| SUMMARY_LIMIT = 200 | ||
| # The job summary behind the run link is the detail surface, so an axis | ||
| # summary is not cut to one table cell there. | ||
| SUMMARY_DETAIL_LIMIT = 2000 | ||
| AXES = ("business", "design", "correctness", "completeness", "conformance") | ||
| VERDICT_ICONS = {"green": "π’ green", "yellow": "π‘ yellow", "red": "π΄ red"} | ||
| VERDICT_ICONS = {"green": "π’ green", "yellow": "π‘ yellow", "red": "π΄ red", | ||
| "n/a": "βͺ n/a"} | ||
| NO_RESULT = "β οΈ no result" | ||
| SHA_RE = re.compile(r"[0-9a-f]{40}") | ||
|
|
||
|
|
@@ -280,8 +289,8 @@ def render_start(round_no, max_rounds, sha: str, axes: list, approve_gate: str, | |
| lines.append("**Round limit reached, needs a human.**") | ||
| lines += ["", f"**Next step:** {NEXT_HUMAN_CAPPED_TEXT}"] | ||
| else: | ||
| lines += ["| Axis | Verdict | Confidence | Summary |", "|---|---|---|---|"] | ||
| lines += [f"| {axis} | β³ pending | | |" for axis in axes] | ||
| lines += ["| Axis | Verdict | Why |", "|---|---|---|"] | ||
| lines += [f"| {axis} | β³ pending | |" for axis in axes] | ||
| lines += ["", _reviewed_line(sha, run_url)] | ||
| return "\n".join(lines) + "\n" | ||
|
|
||
|
|
@@ -292,6 +301,43 @@ def _confidence(value) -> str: | |
| return "" | ||
|
|
||
|
|
||
| def render_job_summary(round_no, max_rounds, sha: str, axes: list, decision, outcome: str, | ||
| max_yellow: str) -> str: | ||
| """The detail the card's "workflow run" link leads to, for GITHUB_STEP_SUMMARY. | ||
|
|
||
| The card is the glance: one icon and one `headline` per axis. Everything an | ||
| axis actually wrote β the full `summary`, its confidence β lands here, so | ||
| the link is worth following and the card does not have to carry both jobs. | ||
|
|
||
| Same `sanitize` as the card: this renders model text into markdown GitHub | ||
| displays, and a summary must not be able to open a heading or fire a mention | ||
| here either. No contract markers β a job summary is not the card, and | ||
| nothing parses this. | ||
| """ | ||
| detail = decision.get("axes") if isinstance(decision, dict) else None | ||
| detail = detail if isinstance(detail, dict) else {} | ||
| lines = [heading(round_no, max_rounds, sha), ""] | ||
| lines.append("**Approved.**" if outcome == OUTCOME_APPROVED else "**Not approved.**") | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. π’ Low β |
||
| if outcome != OUTCOME_APPROVED: | ||
| lines += [f"- {r}" for r in decide_reasons(outcome, decision)] | ||
| for axis in axes: | ||
| entry = detail.get(axis) | ||
| entry = entry if isinstance(entry, dict) else {} | ||
| verdict = entry.get("verdict") | ||
| icon = VERDICT_ICONS.get(verdict, NO_RESULT) | ||
| confidence = _confidence(entry.get("confidence")) | ||
| lines += ["", f"#### {axis} β {icon}" + (f" (confidence {confidence})" if confidence else "")] | ||
| headline = sanitize(entry.get("headline")) | ||
| if headline: | ||
| lines.append(f"**{headline}**") | ||
| summary = sanitize(entry.get("summary"), SUMMARY_DETAIL_LIMIT) | ||
| lines += ["", summary or "_No summary was reported._"] | ||
| limit = _int_or_q(max_yellow) | ||
| lines += ["", f"_Rule: no red, at most {limit} yellow; every axis must report " | ||
| "(βͺ n/a counts as reporting and does not block, but all-n/a withholds)._"] | ||
| return "\n".join(lines) + "\n" | ||
|
|
||
|
|
||
| def render_decide(round_no, max_rounds, sha: str, axes: list, decision, outcome: str, | ||
| max_yellow: str, run_url: str) -> str: | ||
| state, next_step = DECIDE_STATES.get(outcome, (STATE_NO_DECISION, NEXT_PUSH)) | ||
|
|
@@ -304,15 +350,22 @@ def render_decide(round_no, max_rounds, sha: str, axes: list, decision, outcome: | |
| return "\n".join(lines) + "\n" | ||
| detail = decision.get("axes") if isinstance(decision, dict) else None | ||
| detail = detail if isinstance(detail, dict) else {} | ||
| lines += ["| Axis | Verdict | Confidence | Summary |", "|---|---|---|---|"] | ||
| # Three columns, and the third is the axis's `headline` β ONE line stating | ||
| # what decided the verdict. It used to be the first sentence of `summary`, | ||
| # which is where the models put their process log ("Checked the only | ||
| # changed file...", "I read root AGENTS.md and CLAUDE.md..."), so the card | ||
| # showed what the axis DID and never why it ruled as it did. The full | ||
| # summary and the confidence now live in the run's job summary, one click | ||
| # behind the link below β this card is the glance, not the record. | ||
| lines += ["| Axis | Verdict | Why |", "|---|---|---|"] | ||
| for axis in axes: | ||
| entry = detail.get(axis) | ||
| verdict = entry.get("verdict") if isinstance(entry, dict) else None | ||
| if verdict not in VERDICT_ICONS: | ||
| lines.append(f"| {axis} | {NO_RESULT} | | |") | ||
| lines.append(f"| {axis} | {NO_RESULT} | |") | ||
| continue | ||
| summary = sanitize(first_sentence(entry.get("summary"))) | ||
| lines.append(f"| {axis} | {VERDICT_ICONS[verdict]} | {_confidence(entry.get('confidence'))} | {summary} |") | ||
| why = sanitize(entry.get("headline") or first_sentence(entry.get("summary"))) | ||
| lines.append(f"| {axis} | {VERDICT_ICONS[verdict]} | {why} |") | ||
| lines.append("") | ||
| if outcome == OUTCOME_APPROVED: | ||
| lines.append("**Result: β Approved.**") | ||
|
|
@@ -321,7 +374,9 @@ def render_decide(round_no, max_rounds, sha: str, axes: list, decision, outcome: | |
| lines += [f"- {r}" for r in decide_reasons(outcome, decision)] | ||
| lines += ["", f"**Next step:** {next_text}"] | ||
| limit = _int_or_q(max_yellow) | ||
| lines += ["", f"_Rule: no red, at most {limit} yellow; every axis must report. {_run_link(run_url)}_", | ||
| lines += ["", f"_Rule: no red, at most {limit} yellow; every axis must report " | ||
| f"(βͺ n/a counts as reporting and does not block, but all-n/a withholds). " | ||
| f"{_run_link(run_url)}_", | ||
| "", _reviewed_line(sha, "")] | ||
| return "\n".join(lines) + "\n" | ||
|
|
||
|
|
@@ -448,6 +503,17 @@ def upsert(repo: str, pr_number, login: str, body: str) -> dict: | |
| {"body": body})) | ||
|
|
||
|
|
||
| def _write_job_summary(body: str) -> None: | ||
| path = os.environ.get("GITHUB_STEP_SUMMARY") | ||
| if not path: | ||
| return | ||
| try: | ||
| with open(path, "a", encoding="utf-8") as f: | ||
| f.write(body) | ||
| except OSError as e: | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. π‘ Medium β |
||
| print(f"::warning::Could not write the cursor-approve job summary: {e}") | ||
|
|
||
|
|
||
| def _write_output(key: str, value: str) -> None: | ||
| path = os.environ.get("GITHUB_OUTPUT") | ||
| if path: | ||
|
|
@@ -501,6 +567,11 @@ def main(argv=None) -> int: | |
| decision = {} | ||
| body = render_decide(args.round, args.max_rounds, args.commit_sha, axes, decision, args.outcome, | ||
| args.max_yellow_axes, args.run_url) | ||
| # The other half of the card's "workflow run" link. Written before the | ||
| # card so a GitHub API failure below cannot cost us the detail too, and | ||
| # a failure HERE is never fatal: the card is what the author reads. | ||
| _write_job_summary(render_job_summary(args.round, args.max_rounds, args.commit_sha, axes, | ||
| decision, args.outcome, args.max_yellow_axes)) | ||
| try: | ||
| upsert(args.repo, args.pr_number, args.login, body) | ||
| except (RuntimeError, ValueError) as e: | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
π‘ Medium β prompt-common.md, the README and docs/callers/cursor-approve.md (line 76) all say an
n/athat doesn't name what it looked for and where "is treated as untrusted". Nothing invalidate_outputordecideenforces that: anyn/awith any non-empty headline counts as reporting, and it is allowed on every axis,correctnessincluded. In the shipped two-axis setup, a lazy or prompt-injected verdict (e.g. PR text claiming "no behaviour changed") can turn correctnessn/a, and conformance's green alone then approves without correctness ever being judged. Either enforce the claim or remove it from the docs and prompt, and consider forbiddingn/aoncorrectness. Raised by 2 of 6 reviewers (claude-opus-5-5-xhigh adversarial, claude-opus-5-5-xhigh edge-case).