Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 12 additions & 2 deletions .github/cursor-review/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -127,7 +127,17 @@ leg's "did it submit" verdict is the `status` its own `--trust` agent wrote into
`findings.json`, so a prompt-injected cell can green itself; this is an
availability signal against stalls, crashes and step caps — which is what every
observed failure has been — and a forgery-resistant count would need the
submission recorded outside the cell's own writable job. Otherwise it prints one
submission recorded outside the cell's own writable job. One forgery it does
catch: a cell that uploads an `ok` record under ANOTHER cell's artifact name
makes that cell's own upload fail, so its leg goes red while every artifact
reads `ok`. `aggregate-panel.py` flags that combination (`panel_inconsistent`),
and a record whose `model`/`review_type` disagree with its artifact name, or
that names no cell the panel runs, counts as `status: mismatch` (which also
sets `panel_inconsistent`). The leg check is per matrix, so any other non-`ok`
cell in it — a step-cap timeout, or the forger's own — explains the red leg
away (tolerated under `approve_max_failed_reviewers` > 0), and a forger able to
delete the honest upload and re-upload leaves no red leg; see the
aggregate-panel.py docstring. Otherwise it prints one
`::notice::Panel integrity: <ok>/<total> cells, <n> anchored finding(s), 0
unanchored.` It gates no other job — a short panel must not also cost the PR the
findings it did produce — so blocking on it is the caller's call, exactly like
Expand Down Expand Up @@ -217,7 +227,7 @@ the consolidated review's panel table. See [Panel integrity](#panel-integrity).

| File | Role |
|---|---|
| [`aggregate-panel.py`](aggregate-panel.py) | Run by `consolidate`'s **Aggregate panel findings** step. Combines the cells' `findings-*` artifacts into the `panel.json` the judge reads, adds a `status=error` record for every expected cell whose artifact never arrived, and writes the `ok_count` / `total` that `Panel integrity` reads. The direct-OpenAI cells (`findings-direct-*`) count like Cursor cells when they replace the Cursor OpenAI lane (`openai_direct_replaces_cursor_openai: true`, so a failed one also counts against `approve_max_failed_reviewers`) and stay advisory side by side. Only this script sets the `direct_api` / `advisory` markers, so a cell's own record cannot excuse itself from the count. |
| [`aggregate-panel.py`](aggregate-panel.py) | Run by `consolidate`'s **Aggregate panel findings** step. Combines the cells' `findings-*` artifacts into the `panel.json` the judge reads, adds a `status=error` record for every expected cell whose artifact never arrived, and writes the `ok_count` / `total` / `panel_inconsistent` that `Panel integrity` and auto-approve read (`panel_inconsistent`: a reviewer matrix did not succeed although every counted cell artifact reads `ok`, or a counted record is `status: mismatch` — it disagrees with its artifact name or names no cell the panel runs). The direct-OpenAI cells (`findings-direct-*`) count like Cursor cells when they replace the Cursor OpenAI lane (`openai_direct_replaces_cursor_openai: true`, so a failed one also counts against `approve_max_failed_reviewers`) and stay advisory side by side. Only this script sets the `direct_api` / `advisory` markers, so a cell's own record cannot excuse itself from the count. |
| [`prompt-adversarial.md`](prompt-adversarial.md) | Prompt for the security/reliability review pass. |
| [`prompt-edge-case.md`](prompt-edge-case.md) | Prompt for the correctness/logic review pass. |
| [`prompt-judge.md`](prompt-judge.md) | Prompt the judge model uses to consolidate panel findings into one review. |
Expand Down
138 changes: 130 additions & 8 deletions .github/cursor-review/aggregate-panel.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,9 +22,46 @@
(this cell came through the direct backend) and `advisory` (keep it out of the
panel metadata auto-approve counts failures from).

Two cross-checks against what a cell's record claims, both because a `review`
cell runs its agent `--trust` with shell access over PR-authored text and can
upload an artifact under ANOTHER cell's name before that cell does (the honest
leg's own upload then 409s and goes red):

- name vs record: a cell's identity is its artifact NAME,
`findings-<review_type>-<model>` / `findings-direct-<review_type>-<model>`,
which the workflow chose. A record whose own `model` / `review_type` disagree
with it (or sits under a name that does not parse, or names no cell the
matrix runs) gets `status: "mismatch"` — not `ok`, and not the `error`
auto-approve may tolerate — so it counts as a failed reviewer. Its
`model` / `review_type` are rewritten to the name's (the claimed pair kept
as `claimed_model` / `claimed_review_type`), so it cannot occupy, or count
twice in, a slot it was not uploaded for.
- leg result vs records (`--review-result`, `--direct-result`, the
`needs.<job>.result` of each matrix): `Fail the leg when the cell did not
submit` reds every leg whose cell is not `ok`, so a matrix that did not
succeed while EVERY counted cell of it reads `ok` is inconsistent — some leg
failed after its agent ran, by an upload conflict (the forgery signature) or
a post-review infrastructure failure. Either way the count cannot be
trusted. The direct matrix is checked only when its cells count; advisory,
its legs absorb their own failures.

Either cross-check failing on a counted cell writes `panel_inconsistent=true`
to GITHUB_OUTPUT for auto-approve and `Panel integrity` to withhold on — a
step output, not a field in panel.json: the judge runs `--trust` in the same
job afterwards and can rewrite the file, but not an earlier step's outputs.
No finding is dropped either way, so the judge still reads every one.

The leg-result check is per matrix, so ANY non-ok cell in it explains every
red leg away: a forged `ok` slips through whenever another cell of the same
matrix errored (a step-cap timeout, or the forger's own), and a forger able to
delete the honest upload and re-upload under its name leaves no red
leg at all. Telling which leg failed needs per-job conclusions, which this
job has no `actions: read` to see.

Stdlib only. Usage:
aggregate-panel.py --panel-dir /tmp/panel --models '<json list>' \
[--direct-model <id> --direct-counts true|false] \
--review-result <result> [--direct-result <result>] \
--out /tmp/panel.json [--github-output "$GITHUB_OUTPUT"]
"""

Expand All @@ -38,6 +75,25 @@
REVIEW_TYPES = ("adversarial", "edge-case")
DIRECT_PREFIX = "findings-direct-"
MARKERS = ("direct_api", "advisory")
# The status a record gets when it disagrees with the artifact name it sits
# under. Deliberately not auto-approve's tolerable `error`.
MISMATCH_STATUS = "mismatch"
# The review_type a mismatched record gets when its artifact name does not parse.
UNKNOWN_TYPE = "unknown"


def identity_from_name(artifact):
"""(review_type, model) the workflow named this artifact for, or (None, None)."""
for prefix in (DIRECT_PREFIX, "findings-"):
if artifact.startswith(prefix):
rest = artifact[len(prefix):]
break
else:
return None, None
for review_type in REVIEW_TYPES:
if rest.startswith(review_type + "-") and len(rest) > len(review_type) + 1:
return review_type, rest[len(review_type) + 1:]
return None, None


def load_cells(panel_dir):
Expand All @@ -53,13 +109,49 @@ def load_cells(panel_dir):
continue
for marker in MARKERS:
cell.pop(marker, None)
if os.path.basename(os.path.dirname(path)).startswith(DIRECT_PREFIX):
artifact = os.path.basename(os.path.dirname(path))
review_type, model = identity_from_name(artifact)
if (cell.get("review_type"), cell.get("model")) != (review_type, model) or review_type is None:
Comment thread
mattmillerai marked this conversation as resolved.
Resolved
print(
f"::warning::Findings artifact {_log_safe(artifact, 100)} holds a record for "
f"{_log_safe(cell.get('review_type'))}/{_log_safe(cell.get('model'))}, not the cell "
f"it is named for; counted as status={MISMATCH_STATUS}."
)
_mark_mismatch(cell, review_type or UNKNOWN_TYPE, model or artifact)
if artifact.startswith(DIRECT_PREFIX):
direct.append(cell)
else:
cursor.append(cell)
return cursor, direct


def _mark_mismatch(cell, review_type, model):
"""Key `cell` to the slot its artifact name gives it, keeping its claim aside."""
cell["claimed_review_type"] = cell.get("review_type")
cell["claimed_model"] = cell.get("model")
cell["review_type"] = review_type
cell["model"] = model
cell["status"] = MISMATCH_STATUS


def flag_unexpected(cells, models):
"""Mismatch every cell keyed to no (model, type) the matrix runs.

A cell can upload under a made-up name as well as a victim's: a phantom
`error` would otherwise explain a red leg away (and be tolerated), and a
phantom `ok` keep a review type "completed".
"""
expected = {(m, t) for m in models for t in REVIEW_TYPES}
for c in cells:
if (c.get("model"), c.get("review_type")) not in expected and c.get("status") != MISMATCH_STATUS:
print(
f"::warning::Findings artifact for {_log_safe(c.get('review_type'))}/"
f"{_log_safe(c.get('model'))} is not a cell this panel runs; "
f"counted as status={MISMATCH_STATUS}."
)
c["status"] = MISMATCH_STATUS


def fill_missing(cells, models, error):
"""Append a status=error record for every expected (model, type) with no cell."""
present = {(c.get("model"), c.get("review_type")) for c in cells}
Expand All @@ -74,12 +166,23 @@ def fill_missing(cells, models, error):
})


def aggregate(panel_dir, models, direct_model="", direct_counts=False):
"""Return (cells, ok_count, total). `cells` is everything the judge reads."""
def _all_ok(cells):
return all(c.get("status") == "ok" for c in cells)


def aggregate(panel_dir, models, direct_model="", direct_counts=False,
review_result="success", direct_result="skipped"):
"""Return (cells, ok_count, total, inconsistent).

`cells` is everything the judge reads; `inconsistent` is either
cross-check in the module docstring failing on a counted cell.
"""
panel, direct = load_cells(panel_dir)
flag_unexpected(panel, models)
fill_missing(panel, models, "Cell findings artifact was not uploaded.")
counts = bool(direct_counts and direct_model)
if counts:
flag_unexpected(direct, [direct_model])
# Same contract as a Cursor cell: absent means failed, never "not run".
fill_missing(direct, [direct_model], "Direct-API cell findings artifact was not uploaded.")
for c in direct:
Expand All @@ -88,12 +191,17 @@ def aggregate(panel_dir, models, direct_model="", direct_counts=False):
c["advisory"] = True
counted = panel + direct if counts else panel
ok = sum(1 for c in counted if c.get("status") == "ok")
return panel + direct, ok, len(counted)
inconsistent = (
(review_result != "success" and _all_ok(panel))
or (counts and direct_result not in ("success", "skipped") and _all_ok(direct))
or any(c.get("status") == MISMATCH_STATUS for c in counted)
)
return panel + direct, ok, len(counted), inconsistent


def _log_safe(value):
def _log_safe(value, limit=32):
"""A cell-written value made safe for a ::workflow command::-parsing log line."""
return re.sub(r"[^A-Za-z0-9_.-]", "_", str(value))[:32] or "?"
return re.sub(r"[^A-Za-z0-9_.-]", "_", str(value))[:limit] or "?"


def main(argv=None):
Expand All @@ -102,15 +210,28 @@ def main(argv=None):
p.add_argument("--models", required=True, help="JSON list of Cursor panel model ids")
p.add_argument("--direct-model", default="")
p.add_argument("--direct-counts", default="false")
# Required, so a dropped flag fails the step rather than reading as success.
p.add_argument("--review-result", required=True, help="needs.review.result")
p.add_argument("--direct-result", default="skipped", help="needs.review-openai-direct.result")
p.add_argument("--out", required=True)
p.add_argument("--github-output", default="")
args = p.parse_args(argv)

counts = args.direct_counts.strip().lower() == "true"
cells, ok, total = aggregate(
args.panel_dir, json.loads(args.models), args.direct_model.strip(), counts
review_result = args.review_result.strip()
direct_result = args.direct_result.strip()
cells, ok, total, inconsistent = aggregate(
args.panel_dir, json.loads(args.models), args.direct_model.strip(), counts,
review_result, direct_result,
)
print(f"Panel: {ok}/{total} cells contributed findings.")
if inconsistent:
print(
f"::warning::A counted findings artifact may not be its own cell's: a reviewer matrix "
f"did not succeed (review={_log_safe(review_result)}, direct={_log_safe(direct_result)}) "
f"although every counted cell artifact of it reads ok, or a record is "
f"status={MISMATCH_STATUS}. The panel count is untrusted (panel_inconsistent=true)."
)
for c in cells:
if c.get("direct_api"):
role = "advisory, not counted" if c.get("advisory") else "counted"
Expand All @@ -125,6 +246,7 @@ def main(argv=None):
with open(args.github_output, "a", encoding="utf-8") as g:
g.write(f"ok_count={ok}\n")
g.write(f"total={total}\n")
g.write(f"panel_inconsistent={'true' if inconsistent else 'false'}\n")
return 0


Expand Down
38 changes: 38 additions & 0 deletions .github/cursor-review/auto-approve.py
Original file line number Diff line number Diff line change
Expand Up @@ -111,6 +111,14 @@
while every review type (``PANEL_REVIEW_TYPES``) still has an ``ok`` cell. A
non-dict cell, a missing or unknown status, or an empty panel always
withholds. Tolerated cells are named in the decision's reason;
* a counted cell's findings artifact may not be its own: a reviewer leg did
not succeed although every panel cell reports ``ok``, or a record disagrees
with its artifact name (``--panel-inconsistent``, aggregate-panel.py's step
output, written before the judge ran; the consolidated file's
``panel_inconsistent`` is OR-ed in). A cell can upload a forged ``ok`` under
another cell's artifact name, and the honest leg's own upload then fails.
Not re-run automatically — a fresh round runs the same PR content, so a
forger forges again; the card asks a human to read the red legs;
* the review did not land as resolvable threads (`delivered` is not ``true``,
or any finding reached the review body only — ``ungated_findings`` > 0 — where
the open-thread check below cannot see it);
Expand Down Expand Up @@ -627,6 +635,7 @@ def decide(
human_review: bool = False,
scope=None,
max_failed_reviewers: int = 0,
panel_inconsistent: bool = False,
):
"""Return (event, reasons, blocking_findings). Pure; no I/O.

Expand All @@ -639,6 +648,7 @@ def decide(
threshold, findings, panel, judge_status, delivered, reviewed_sha,
live_head_sha, open_thread_severities, ungated, human_review,
reviewed_diff_empty, reviewed_base, live_base, scope, max_failed_reviewers,
panel_inconsistent,
)
return event, reasons, blocking

Expand All @@ -650,10 +660,19 @@ def decide(
REASON_UNGATED_TAIL = "finding(s) reached the review body only, not as resolvable threads"
REASON_EMPTY_DIFF = ("the reviewed diff is empty — every changed path was excluded from review, "
"so nothing a reviewer saw can earn an approval")
# Structural too, though not for the reason the others are: a fresh round
# CAN clear it when a leg merely flaked after its review, but it runs the same
# PR content, so a forger forges again — auto-retry cannot tell the two apart,
# and a human reading the red legs can. Hence its own card suffix below.
REASON_PANEL_INCONSISTENT = ("a counted panel cell's findings artifact may not be its own (a reviewer "
"leg did not succeed although every cell reports ok, or a record "
"disagrees with its artifact name)")
PANEL_INCONSISTENT_CAUSE = "a panel cell's findings artifact may not be its own (see the red leg checks)"
STRUCTURAL_CAUSES = (
(REASON_NOT_DELIVERED, "the findings did not land as resolvable threads"),
(REASON_UNGATED_TAIL, "some findings reached the review body only, not as resolvable threads"),
(REASON_EMPTY_DIFF, "the reviewed diff is empty (every changed path was excluded from review)"),
(REASON_PANEL_INCONSISTENT, PANEL_INCONSISTENT_CAUSE),
)

# The two causes a fresh round on the LIVE head actually fixes, and the only
Expand Down Expand Up @@ -684,6 +703,7 @@ def decide_gate(
live_base: str = "",
scope=None,
max_failed_reviewers: int = 0,
panel_inconsistent: bool = False,
):
"""decide(), plus the approve_gate value: (event, gate, reasons, blocking).

Expand All @@ -704,6 +724,8 @@ def decide_gate(
panel_reason, tolerated = panel_gate(panel, max_failed_reviewers)
if panel_reason:
reasons.append(panel_reason)
if panel_inconsistent:
reasons.append(REASON_PANEL_INCONSISTENT)
if not delivered:
reasons.append(REASON_NOT_DELIVERED)
elif ungated:
Expand Down Expand Up @@ -1025,6 +1047,11 @@ def no_decision_next(reasons: list, label: str = ""):
causes = [cause for needle, cause in STRUCTURAL_CAUSES
if any(isinstance(r, str) and needle in r for r in reasons or [])]
if causes:
if causes == [PANEL_INCONSISTENT_CAUSE]:
return card.NEXT_HUMAN, (
f"A human is needed: {PANEL_INCONSISTENT_CAUSE}. A re-run clears a leg that "
"failed after its review, not a forged artifact: the same PR content runs again."
)
return card.NEXT_HUMAN, f"A human is needed: {'; '.join(causes)}. A re-run would land the same way."
if any(isinstance(r, str) and r == REASON_HEAD_MOVED for r in reasons or []):
# The head moved, so the next round reads a commit with no review on it
Expand Down Expand Up @@ -1708,6 +1735,14 @@ def cmd_decide(args) -> int:
data = json.load(f)
findings = data.get("findings") or []
panel = data.get("panel") or []
# The flag is the authority (a job output written before the judge ran);
# the file's copy is only OR-ed in, so neither can clear the other.
flag = (getattr(args, "panel_inconsistent", "") or "").strip().lower()
if flag not in ("", "true", "false"):
print(f"::error::--panel-inconsistent must be 'true', 'false' or empty, got {args.panel_inconsistent!r}")
withdraw_own_approvals(args)
return 2
panel_inconsistent = flag == "true" or data.get("panel_inconsistent") is True
try:
ungated = int(args.ungated or 0)
except ValueError:
Expand Down Expand Up @@ -1744,6 +1779,7 @@ def cmd_decide(args) -> int:
live_base,
scope,
max_failed,
panel_inconsistent,
)
set_output("approve_gate", gate)
if event != NONE:
Expand Down Expand Up @@ -3150,6 +3186,8 @@ def main() -> int:
d.add_argument("--open-anchors", default="", help="post-review's --open-anchors-out snapshot")
# approve_max_failed_reviewers: errored panel cells to tolerate (see panel_gate).
d.add_argument("--max-failed-reviewers", default="0")
# consolidate's panel_inconsistent output (aggregate-panel.py): `true` withholds.
d.add_argument("--panel-inconsistent", default="")
# approve_authors, as resolved by the workflow's `gate` job: `false` = the PR
# author is not listed, so auto-approve is off for this PR. Anything else
# (including the default empty) = decide as usual.
Expand Down
Loading
Loading