diff --git a/.github/cursor-review/README.md b/.github/cursor-review/README.md index cbb4c898..6350cd32 100644 --- a/.github/cursor-review/README.md +++ b/.github/cursor-review/README.md @@ -127,7 +127,17 @@ leg's "did it submit" verdict is the `status` its own `--trust` agent wrote into `findings.json`, so a prompt-injected cell can green itself; this is an availability signal against stalls, crashes and step caps — which is what every observed failure has been — and a forgery-resistant count would need the -submission recorded outside the cell's own writable job. Otherwise it prints one +submission recorded outside the cell's own writable job. One forgery it does +catch: a cell that uploads an `ok` record under ANOTHER cell's artifact name +makes that cell's own upload fail, so its leg goes red while every artifact +reads `ok`. `aggregate-panel.py` flags that combination (`panel_inconsistent`), +and a record whose `model`/`review_type` disagree with its artifact name, or +that names no cell the panel runs, counts as `status: mismatch` (which also +sets `panel_inconsistent`). The leg check is per matrix, so any other non-`ok` +cell in it — a step-cap timeout, or the forger's own — explains the red leg +away (tolerated under `approve_max_failed_reviewers` > 0), and a forger able to +delete the honest upload and re-upload leaves no red leg; see the +aggregate-panel.py docstring. Otherwise it prints one `::notice::Panel integrity: / cells, anchored finding(s), 0 unanchored.` It gates no other job — a short panel must not also cost the PR the findings it did produce — so blocking on it is the caller's call, exactly like @@ -217,7 +227,7 @@ the consolidated review's panel table. See [Panel integrity](#panel-integrity). | File | Role | |---|---| -| [`aggregate-panel.py`](aggregate-panel.py) | Run by `consolidate`'s **Aggregate panel findings** step. Combines the cells' `findings-*` artifacts into the `panel.json` the judge reads, adds a `status=error` record for every expected cell whose artifact never arrived, and writes the `ok_count` / `total` that `Panel integrity` reads. The direct-OpenAI cells (`findings-direct-*`) count like Cursor cells when they replace the Cursor OpenAI lane (`openai_direct_replaces_cursor_openai: true`, so a failed one also counts against `approve_max_failed_reviewers`) and stay advisory side by side. Only this script sets the `direct_api` / `advisory` markers, so a cell's own record cannot excuse itself from the count. | +| [`aggregate-panel.py`](aggregate-panel.py) | Run by `consolidate`'s **Aggregate panel findings** step. Combines the cells' `findings-*` artifacts into the `panel.json` the judge reads, adds a `status=error` record for every expected cell whose artifact never arrived, and writes the `ok_count` / `total` / `panel_inconsistent` that `Panel integrity` and auto-approve read (`panel_inconsistent`: a reviewer matrix did not succeed although every counted cell artifact reads `ok`, or a counted record is `status: mismatch` — it disagrees with its artifact name or names no cell the panel runs). The direct-OpenAI cells (`findings-direct-*`) count like Cursor cells when they replace the Cursor OpenAI lane (`openai_direct_replaces_cursor_openai: true`, so a failed one also counts against `approve_max_failed_reviewers`) and stay advisory side by side. Only this script sets the `direct_api` / `advisory` markers, so a cell's own record cannot excuse itself from the count. | | [`prompt-adversarial.md`](prompt-adversarial.md) | Prompt for the security/reliability review pass. | | [`prompt-edge-case.md`](prompt-edge-case.md) | Prompt for the correctness/logic review pass. | | [`prompt-judge.md`](prompt-judge.md) | Prompt the judge model uses to consolidate panel findings into one review. | diff --git a/.github/cursor-review/aggregate-panel.py b/.github/cursor-review/aggregate-panel.py index 0e7d7815..f71e2c7c 100644 --- a/.github/cursor-review/aggregate-panel.py +++ b/.github/cursor-review/aggregate-panel.py @@ -22,9 +22,46 @@ (this cell came through the direct backend) and `advisory` (keep it out of the panel metadata auto-approve counts failures from). +Two cross-checks against what a cell's record claims, both because a `review` +cell runs its agent `--trust` with shell access over PR-authored text and can +upload an artifact under ANOTHER cell's name before that cell does (the honest +leg's own upload then 409s and goes red): + +- name vs record: a cell's identity is its artifact NAME, + `findings--` / `findings-direct--`, + which the workflow chose. A record whose own `model` / `review_type` disagree + with it (or sits under a name that does not parse, or names no cell the + matrix runs) gets `status: "mismatch"` — not `ok`, and not the `error` + auto-approve may tolerate — so it counts as a failed reviewer. Its + `model` / `review_type` are rewritten to the name's (the claimed pair kept + as `claimed_model` / `claimed_review_type`), so it cannot occupy, or count + twice in, a slot it was not uploaded for. +- leg result vs records (`--review-result`, `--direct-result`, the + `needs..result` of each matrix): `Fail the leg when the cell did not + submit` reds every leg whose cell is not `ok`, so a matrix that did not + succeed while EVERY counted cell of it reads `ok` is inconsistent — some leg + failed after its agent ran, by an upload conflict (the forgery signature) or + a post-review infrastructure failure. Either way the count cannot be + trusted. The direct matrix is checked only when its cells count; advisory, + its legs absorb their own failures. + +Either cross-check failing on a counted cell writes `panel_inconsistent=true` +to GITHUB_OUTPUT for auto-approve and `Panel integrity` to withhold on — a +step output, not a field in panel.json: the judge runs `--trust` in the same +job afterwards and can rewrite the file, but not an earlier step's outputs. +No finding is dropped either way, so the judge still reads every one. + +The leg-result check is per matrix, so ANY non-ok cell in it explains every +red leg away: a forged `ok` slips through whenever another cell of the same +matrix errored (a step-cap timeout, or the forger's own), and a forger able to +delete the honest upload and re-upload under its name leaves no red +leg at all. Telling which leg failed needs per-job conclusions, which this +job has no `actions: read` to see. + Stdlib only. Usage: aggregate-panel.py --panel-dir /tmp/panel --models '' \ [--direct-model --direct-counts true|false] \ + --review-result [--direct-result ] \ --out /tmp/panel.json [--github-output "$GITHUB_OUTPUT"] """ @@ -38,6 +75,25 @@ REVIEW_TYPES = ("adversarial", "edge-case") DIRECT_PREFIX = "findings-direct-" MARKERS = ("direct_api", "advisory") +# The status a record gets when it disagrees with the artifact name it sits +# under. Deliberately not auto-approve's tolerable `error`. +MISMATCH_STATUS = "mismatch" +# The review_type a mismatched record gets when its artifact name does not parse. +UNKNOWN_TYPE = "unknown" + + +def identity_from_name(artifact): + """(review_type, model) the workflow named this artifact for, or (None, None).""" + for prefix in (DIRECT_PREFIX, "findings-"): + if artifact.startswith(prefix): + rest = artifact[len(prefix):] + break + else: + return None, None + for review_type in REVIEW_TYPES: + if rest.startswith(review_type + "-") and len(rest) > len(review_type) + 1: + return review_type, rest[len(review_type) + 1:] + return None, None def load_cells(panel_dir): @@ -53,13 +109,49 @@ def load_cells(panel_dir): continue for marker in MARKERS: cell.pop(marker, None) - if os.path.basename(os.path.dirname(path)).startswith(DIRECT_PREFIX): + artifact = os.path.basename(os.path.dirname(path)) + review_type, model = identity_from_name(artifact) + if (cell.get("review_type"), cell.get("model")) != (review_type, model) or review_type is None: + print( + f"::warning::Findings artifact {_log_safe(artifact, 100)} holds a record for " + f"{_log_safe(cell.get('review_type'))}/{_log_safe(cell.get('model'))}, not the cell " + f"it is named for; counted as status={MISMATCH_STATUS}." + ) + _mark_mismatch(cell, review_type or UNKNOWN_TYPE, model or artifact) + if artifact.startswith(DIRECT_PREFIX): direct.append(cell) else: cursor.append(cell) return cursor, direct +def _mark_mismatch(cell, review_type, model): + """Key `cell` to the slot its artifact name gives it, keeping its claim aside.""" + cell["claimed_review_type"] = cell.get("review_type") + cell["claimed_model"] = cell.get("model") + cell["review_type"] = review_type + cell["model"] = model + cell["status"] = MISMATCH_STATUS + + +def flag_unexpected(cells, models): + """Mismatch every cell keyed to no (model, type) the matrix runs. + + A cell can upload under a made-up name as well as a victim's: a phantom + `error` would otherwise explain a red leg away (and be tolerated), and a + phantom `ok` keep a review type "completed". + """ + expected = {(m, t) for m in models for t in REVIEW_TYPES} + for c in cells: + if (c.get("model"), c.get("review_type")) not in expected and c.get("status") != MISMATCH_STATUS: + print( + f"::warning::Findings artifact for {_log_safe(c.get('review_type'))}/" + f"{_log_safe(c.get('model'))} is not a cell this panel runs; " + f"counted as status={MISMATCH_STATUS}." + ) + c["status"] = MISMATCH_STATUS + + def fill_missing(cells, models, error): """Append a status=error record for every expected (model, type) with no cell.""" present = {(c.get("model"), c.get("review_type")) for c in cells} @@ -74,12 +166,23 @@ def fill_missing(cells, models, error): }) -def aggregate(panel_dir, models, direct_model="", direct_counts=False): - """Return (cells, ok_count, total). `cells` is everything the judge reads.""" +def _all_ok(cells): + return all(c.get("status") == "ok" for c in cells) + + +def aggregate(panel_dir, models, direct_model="", direct_counts=False, + review_result="success", direct_result="skipped"): + """Return (cells, ok_count, total, inconsistent). + + `cells` is everything the judge reads; `inconsistent` is either + cross-check in the module docstring failing on a counted cell. + """ panel, direct = load_cells(panel_dir) + flag_unexpected(panel, models) fill_missing(panel, models, "Cell findings artifact was not uploaded.") counts = bool(direct_counts and direct_model) if counts: + flag_unexpected(direct, [direct_model]) # Same contract as a Cursor cell: absent means failed, never "not run". fill_missing(direct, [direct_model], "Direct-API cell findings artifact was not uploaded.") for c in direct: @@ -88,12 +191,17 @@ def aggregate(panel_dir, models, direct_model="", direct_counts=False): c["advisory"] = True counted = panel + direct if counts else panel ok = sum(1 for c in counted if c.get("status") == "ok") - return panel + direct, ok, len(counted) + inconsistent = ( + (review_result != "success" and _all_ok(panel)) + or (counts and direct_result not in ("success", "skipped") and _all_ok(direct)) + or any(c.get("status") == MISMATCH_STATUS for c in counted) + ) + return panel + direct, ok, len(counted), inconsistent -def _log_safe(value): +def _log_safe(value, limit=32): """A cell-written value made safe for a ::workflow command::-parsing log line.""" - return re.sub(r"[^A-Za-z0-9_.-]", "_", str(value))[:32] or "?" + return re.sub(r"[^A-Za-z0-9_.-]", "_", str(value))[:limit] or "?" def main(argv=None): @@ -102,15 +210,28 @@ def main(argv=None): p.add_argument("--models", required=True, help="JSON list of Cursor panel model ids") p.add_argument("--direct-model", default="") p.add_argument("--direct-counts", default="false") + # Required, so a dropped flag fails the step rather than reading as success. + p.add_argument("--review-result", required=True, help="needs.review.result") + p.add_argument("--direct-result", default="skipped", help="needs.review-openai-direct.result") p.add_argument("--out", required=True) p.add_argument("--github-output", default="") args = p.parse_args(argv) counts = args.direct_counts.strip().lower() == "true" - cells, ok, total = aggregate( - args.panel_dir, json.loads(args.models), args.direct_model.strip(), counts + review_result = args.review_result.strip() + direct_result = args.direct_result.strip() + cells, ok, total, inconsistent = aggregate( + args.panel_dir, json.loads(args.models), args.direct_model.strip(), counts, + review_result, direct_result, ) print(f"Panel: {ok}/{total} cells contributed findings.") + if inconsistent: + print( + f"::warning::A counted findings artifact may not be its own cell's: a reviewer matrix " + f"did not succeed (review={_log_safe(review_result)}, direct={_log_safe(direct_result)}) " + f"although every counted cell artifact of it reads ok, or a record is " + f"status={MISMATCH_STATUS}. The panel count is untrusted (panel_inconsistent=true)." + ) for c in cells: if c.get("direct_api"): role = "advisory, not counted" if c.get("advisory") else "counted" @@ -125,6 +246,7 @@ def main(argv=None): with open(args.github_output, "a", encoding="utf-8") as g: g.write(f"ok_count={ok}\n") g.write(f"total={total}\n") + g.write(f"panel_inconsistent={'true' if inconsistent else 'false'}\n") return 0 diff --git a/.github/cursor-review/auto-approve.py b/.github/cursor-review/auto-approve.py index 5d201827..d5b11502 100644 --- a/.github/cursor-review/auto-approve.py +++ b/.github/cursor-review/auto-approve.py @@ -111,6 +111,14 @@ while every review type (``PANEL_REVIEW_TYPES``) still has an ``ok`` cell. A non-dict cell, a missing or unknown status, or an empty panel always withholds. Tolerated cells are named in the decision's reason; +* a counted cell's findings artifact may not be its own: a reviewer leg did + not succeed although every panel cell reports ``ok``, or a record disagrees + with its artifact name (``--panel-inconsistent``, aggregate-panel.py's step + output, written before the judge ran; the consolidated file's + ``panel_inconsistent`` is OR-ed in). A cell can upload a forged ``ok`` under + another cell's artifact name, and the honest leg's own upload then fails. + Not re-run automatically — a fresh round runs the same PR content, so a + forger forges again; the card asks a human to read the red legs; * the review did not land as resolvable threads (`delivered` is not ``true``, or any finding reached the review body only — ``ungated_findings`` > 0 — where the open-thread check below cannot see it); @@ -627,6 +635,7 @@ def decide( human_review: bool = False, scope=None, max_failed_reviewers: int = 0, + panel_inconsistent: bool = False, ): """Return (event, reasons, blocking_findings). Pure; no I/O. @@ -639,6 +648,7 @@ def decide( threshold, findings, panel, judge_status, delivered, reviewed_sha, live_head_sha, open_thread_severities, ungated, human_review, reviewed_diff_empty, reviewed_base, live_base, scope, max_failed_reviewers, + panel_inconsistent, ) return event, reasons, blocking @@ -650,10 +660,19 @@ def decide( REASON_UNGATED_TAIL = "finding(s) reached the review body only, not as resolvable threads" REASON_EMPTY_DIFF = ("the reviewed diff is empty — every changed path was excluded from review, " "so nothing a reviewer saw can earn an approval") +# Structural too, though not for the reason the others are: a fresh round +# CAN clear it when a leg merely flaked after its review, but it runs the same +# PR content, so a forger forges again — auto-retry cannot tell the two apart, +# and a human reading the red legs can. Hence its own card suffix below. +REASON_PANEL_INCONSISTENT = ("a counted panel cell's findings artifact may not be its own (a reviewer " + "leg did not succeed although every cell reports ok, or a record " + "disagrees with its artifact name)") +PANEL_INCONSISTENT_CAUSE = "a panel cell's findings artifact may not be its own (see the red leg checks)" STRUCTURAL_CAUSES = ( (REASON_NOT_DELIVERED, "the findings did not land as resolvable threads"), (REASON_UNGATED_TAIL, "some findings reached the review body only, not as resolvable threads"), (REASON_EMPTY_DIFF, "the reviewed diff is empty (every changed path was excluded from review)"), + (REASON_PANEL_INCONSISTENT, PANEL_INCONSISTENT_CAUSE), ) # The two causes a fresh round on the LIVE head actually fixes, and the only @@ -684,6 +703,7 @@ def decide_gate( live_base: str = "", scope=None, max_failed_reviewers: int = 0, + panel_inconsistent: bool = False, ): """decide(), plus the approve_gate value: (event, gate, reasons, blocking). @@ -704,6 +724,8 @@ def decide_gate( panel_reason, tolerated = panel_gate(panel, max_failed_reviewers) if panel_reason: reasons.append(panel_reason) + if panel_inconsistent: + reasons.append(REASON_PANEL_INCONSISTENT) if not delivered: reasons.append(REASON_NOT_DELIVERED) elif ungated: @@ -1025,6 +1047,11 @@ def no_decision_next(reasons: list, label: str = ""): causes = [cause for needle, cause in STRUCTURAL_CAUSES if any(isinstance(r, str) and needle in r for r in reasons or [])] if causes: + if causes == [PANEL_INCONSISTENT_CAUSE]: + return card.NEXT_HUMAN, ( + f"A human is needed: {PANEL_INCONSISTENT_CAUSE}. A re-run clears a leg that " + "failed after its review, not a forged artifact: the same PR content runs again." + ) return card.NEXT_HUMAN, f"A human is needed: {'; '.join(causes)}. A re-run would land the same way." if any(isinstance(r, str) and r == REASON_HEAD_MOVED for r in reasons or []): # The head moved, so the next round reads a commit with no review on it @@ -1708,6 +1735,14 @@ def cmd_decide(args) -> int: data = json.load(f) findings = data.get("findings") or [] panel = data.get("panel") or [] + # The flag is the authority (a job output written before the judge ran); + # the file's copy is only OR-ed in, so neither can clear the other. + flag = (getattr(args, "panel_inconsistent", "") or "").strip().lower() + if flag not in ("", "true", "false"): + print(f"::error::--panel-inconsistent must be 'true', 'false' or empty, got {args.panel_inconsistent!r}") + withdraw_own_approvals(args) + return 2 + panel_inconsistent = flag == "true" or data.get("panel_inconsistent") is True try: ungated = int(args.ungated or 0) except ValueError: @@ -1744,6 +1779,7 @@ def cmd_decide(args) -> int: live_base, scope, max_failed, + panel_inconsistent, ) set_output("approve_gate", gate) if event != NONE: @@ -3150,6 +3186,8 @@ def main() -> int: d.add_argument("--open-anchors", default="", help="post-review's --open-anchors-out snapshot") # approve_max_failed_reviewers: errored panel cells to tolerate (see panel_gate). d.add_argument("--max-failed-reviewers", default="0") + # consolidate's panel_inconsistent output (aggregate-panel.py): `true` withholds. + d.add_argument("--panel-inconsistent", default="") # approve_authors, as resolved by the workflow's `gate` job: `false` = the PR # author is not listed, so auto-approve is off for this PR. Anything else # (including the default empty) = decide as usual. diff --git a/.github/cursor-review/tests/test_aggregate_panel.py b/.github/cursor-review/tests/test_aggregate_panel.py index 1cc6c9b8..f1aad745 100644 --- a/.github/cursor-review/tests/test_aggregate_panel.py +++ b/.github/cursor-review/tests/test_aggregate_panel.py @@ -57,7 +57,7 @@ def tearDown(self): self._tmp.cleanup() def write(self, artifact, record): - os.makedirs(os.path.join(self.dir, artifact)) + os.makedirs(os.path.join(self.dir, artifact), exist_ok=True) with open(os.path.join(self.dir, artifact, "findings.json"), "w", encoding="utf-8") as f: json.dump(record, f) @@ -76,7 +76,7 @@ def test_replace_true_both_direct_ok_are_counted(self): self.full_cursor_panel() self.direct("adversarial") self.direct("edge-case") - cells, ok, total = self.run_agg(True) + cells, ok, total, _ = self.run_agg(True) self.assertEqual((ok, total), (6, 6)) direct = [c for c in cells if c.get("direct_api")] self.assertEqual(sorted(c["review_type"] for c in direct), ["adversarial", "edge-case"]) @@ -86,13 +86,13 @@ def test_replace_true_one_direct_error_counts_as_failed(self): self.full_cursor_panel() self.direct("adversarial") self.direct("edge-case", status="error") - _, ok, total = self.run_agg(True) + _, ok, total, _ = self.run_agg(True) self.assertEqual((ok, total), (5, 6)) def test_replace_true_missing_direct_artifact_is_synthesised_as_error(self): self.full_cursor_panel() self.direct("adversarial") - cells, ok, total = self.run_agg(True) + cells, ok, total, _ = self.run_agg(True) self.assertEqual((ok, total), (5, 6)) missing = [c for c in cells if c.get("direct_api") and c["review_type"] == "edge-case"] self.assertEqual(len(missing), 1) @@ -103,7 +103,7 @@ def test_replace_false_direct_cells_are_advisory_and_uncounted(self): self.full_cursor_panel() self.direct("adversarial", status="error") self.direct("edge-case") - cells, ok, total = self.run_agg(False) + cells, ok, total, _ = self.run_agg(False) self.assertEqual((ok, total), (4, 4)) direct = [c for c in cells if c.get("direct_api")] self.assertEqual(len(direct), 2) @@ -111,7 +111,7 @@ def test_replace_false_direct_cells_are_advisory_and_uncounted(self): def test_replace_false_missing_direct_cell_is_not_synthesised(self): self.full_cursor_panel() - cells, ok, total = self.run_agg(False) + cells, ok, total, _ = self.run_agg(False) self.assertEqual((ok, total), (4, 4)) self.assertFalse(any(c.get("direct_api") for c in cells)) @@ -120,13 +120,13 @@ def test_advisory_direct_cell_cannot_mask_a_missing_cursor_cell(self): # record must not stand in for a Cursor cell that never uploaded. self.write("findings-adversarial-claude-opus", cell("claude-opus", "adversarial")) self.write(f"findings-direct-edge-case-{DIRECT}", cell("claude-opus", "edge-case")) - cells, ok, total = AGG.aggregate(self.dir, ["claude-opus"], DIRECT, False) + cells, ok, total, _ = AGG.aggregate(self.dir, ["claude-opus"], DIRECT, False) self.assertEqual((ok, total), (1, 2)) def test_a_cell_cannot_set_its_own_markers(self): self.write("findings-adversarial-kimi", cell("kimi", "adversarial", "error", advisory=True, direct_api=True)) self.direct("adversarial", status="error", advisory=True) - cells, ok, total = AGG.aggregate(self.dir, ["kimi"], DIRECT, True) + cells, ok, total, _ = AGG.aggregate(self.dir, ["kimi"], DIRECT, True) cursor = [c for c in cells if c["model"] == "kimi" and c["review_type"] == "adversarial"] self.assertNotIn("advisory", cursor[0]) self.assertNotIn("direct_api", cursor[0]) @@ -136,14 +136,14 @@ def test_a_cell_cannot_set_its_own_markers(self): def test_counts_without_a_model_falls_back_to_advisory(self): self.full_cursor_panel() self.direct("adversarial") - _, ok, total = AGG.aggregate(self.dir, CURSOR_MODELS, "", True) + _, ok, total, _ = AGG.aggregate(self.dir, CURSOR_MODELS, "", True) self.assertEqual((ok, total), (4, 4)) def test_unreadable_artifacts_are_skipped(self): os.makedirs(os.path.join(self.dir, "findings-adversarial-kimi")) with open(os.path.join(self.dir, "findings-adversarial-kimi", "findings.json"), "wb") as f: f.write(b"\xff\xfe not json") - cells, ok, total = AGG.aggregate(self.dir, ["kimi"], "", False) + cells, ok, total, _ = AGG.aggregate(self.dir, ["kimi"], "", False) self.assertEqual((ok, total), (0, 2)) def test_cli_writes_panel_and_outputs(self): @@ -154,17 +154,221 @@ def test_cli_writes_panel_and_outputs(self): gh_out = os.path.join(self.dir, "gh-output") result = subprocess.run( [sys.executable, SCRIPT, "--panel-dir", self.dir, "--models", json.dumps(CURSOR_MODELS), - "--direct-model", DIRECT, "--direct-counts", "true", "--out", out, "--github-output", gh_out], + "--direct-model", DIRECT, "--direct-counts", "true", "--review-result", "success", + "--out", out, "--github-output", gh_out], capture_output=True, text=True, check=True, ) self.assertIn("Panel: 5/6 cells contributed findings.", result.stdout) self.assertIn("Direct-API cell edge-case (counted): status=error.", result.stdout) with open(gh_out, encoding="utf-8") as f: - self.assertEqual(f.read(), "ok_count=5\ntotal=6\n") + self.assertEqual(f.read(), "ok_count=5\ntotal=6\npanel_inconsistent=false\n") with open(out, encoding="utf-8") as f: self.assertEqual(len(json.load(f)), 6) +class ConsistencyTest(AggregateTest): + """Leg results vs artifacts, and artifact name vs record. + + `Fail the leg when the cell did not submit` reds every non-ok cell, so a + red matrix whose every counted artifact reads ok means some leg failed + after its agent ran — an upload 409 behind a forged artifact, or a + post-review infra failure. Neither leaves the count trustworthy. + """ + + def consistent(self, counts=False, review_result="success", direct_result="skipped"): + return AGG.aggregate(self.dir, CURSOR_MODELS, DIRECT, counts, review_result, direct_result)[3] + + def test_all_ok_under_a_failed_review_matrix_is_inconsistent(self): + self.full_cursor_panel() + self.assertTrue(self.consistent(review_result="failure")) + + def test_all_ok_under_a_successful_matrix_is_consistent(self): + self.full_cursor_panel() + self.assertFalse(self.consistent(review_result="success")) + + def test_defaults_are_consistent(self): + self.full_cursor_panel() + self.assertFalse(AGG.aggregate(self.dir, CURSOR_MODELS)[3]) + + def test_an_error_cell_explains_the_red_matrix(self): + self.full_cursor_panel() + self.write("findings-edge-case-kimi", cell("kimi", "edge-case", "error")) + self.assertFalse(self.consistent(review_result="failure")) + + def test_a_missing_artifact_explains_the_red_matrix(self): + self.full_cursor_panel() + os.remove(os.path.join(self.dir, "findings-edge-case-kimi", "findings.json")) + self.assertFalse(self.consistent(review_result="failure")) + + def test_cancelled_matrix_with_all_ok_is_inconsistent(self): + self.full_cursor_panel() + self.assertTrue(self.consistent(review_result="cancelled")) + + def test_counted_direct_lane_failed_with_all_ok_is_inconsistent(self): + self.full_cursor_panel() + self.direct("adversarial") + self.direct("edge-case") + self.assertTrue(self.consistent(counts=True, direct_result="failure")) + self.assertFalse(self.consistent(counts=True, direct_result="success")) + self.assertFalse(self.consistent(counts=True, direct_result="skipped")) + + def test_counted_direct_lane_failed_with_an_error_cell_is_consistent(self): + self.full_cursor_panel() + self.direct("adversarial") + self.direct("edge-case", status="error") + self.assertFalse(self.consistent(counts=True, direct_result="failure")) + + def test_counted_direct_lane_failed_with_its_artifact_missing_is_consistent(self): + self.full_cursor_panel() + self.direct("adversarial") + self.assertFalse(self.consistent(counts=True, direct_result="failure")) + + def test_advisory_direct_lane_result_is_ignored(self): + self.full_cursor_panel() + self.direct("adversarial") + self.direct("edge-case") + self.assertFalse(self.consistent(counts=False, direct_result="failure")) + + def test_a_direct_error_does_not_explain_a_red_review_matrix(self): + # The two lanes are checked separately: an errored direct cell is no + # reason the Cursor matrix went red. + self.full_cursor_panel() + self.direct("adversarial") + self.direct("edge-case", status="error") + self.assertTrue(self.consistent(counts=True, review_result="failure", direct_result="failure")) + + def test_forged_ok_under_another_cells_name_with_its_leg_red(self): + # The forger's own cell is honest-looking; the cell it impersonated + # lost the upload race (409) and went red, so only the result shows it. + self.full_cursor_panel() + self.write("findings-adversarial-kimi", cell("kimi", "adversarial", "ok", findings=[])) + cells, ok, total, inconsistent = AGG.aggregate(self.dir, CURSOR_MODELS, DIRECT, False, "failure") + self.assertTrue(inconsistent) + # The records are left alone: the judge still reads every finding. + self.assertEqual((ok, total), (4, 4)) + + def test_record_naming_another_cell_is_a_mismatch(self): + self.write("findings-adversarial-kimi", cell("claude-opus", "adversarial")) + cells, ok, total, inconsistent = AGG.aggregate(self.dir, ["kimi"], "", False) + mismatched = [c for c in cells if c.get("status") == AGG.MISMATCH_STATUS] + self.assertEqual(len(mismatched), 1) + # Keyed to the slot its artifact was uploaded for; the claim kept aside. + self.assertEqual((mismatched[0]["model"], mismatched[0]["review_type"]), ("kimi", "adversarial")) + self.assertEqual(mismatched[0]["claimed_model"], "claude-opus") + self.assertEqual(ok, 0) + # One slot per artifact: no phantom error is synthesised for the slot + # the artifact exists for, and the claimed slot is not counted twice. + self.assertEqual(total, 2) + self.assertTrue(inconsistent) + # Not the tolerable `error`: auto-approve never excuses it. + self.assertNotEqual(AGG.MISMATCH_STATUS, AA.TOLERABLE_CELL_STATUS) + reason, tolerated = AA.panel_gate( + [{k: c[k] for k in ("model", "review_type", "status")} for c in cells], max_failed=5 + ) + self.assertIsNotNone(reason) + self.assertEqual(tolerated, []) + + def test_record_naming_another_review_type_is_a_mismatch(self): + self.write("findings-edge-case-kimi", cell("kimi", "adversarial")) + cells, _, _, _ = AGG.aggregate(self.dir, ["kimi"], "", False) + self.assertIn(AGG.MISMATCH_STATUS, [c["status"] for c in cells]) + + def test_direct_record_is_checked_against_its_direct_name(self): + self.write(f"findings-direct-edge-case-{DIRECT}", cell("kimi", "edge-case")) + cells, _, _, _ = AGG.aggregate(self.dir, ["kimi"], DIRECT, True) + direct = [c for c in cells if c.get("direct_api") and c.get("claimed_model") == "kimi"] + self.assertEqual(direct[0]["status"], AGG.MISMATCH_STATUS) + self.assertEqual((direct[0]["model"], direct[0]["review_type"]), (DIRECT, "edge-case")) + + def test_unparseable_artifact_name_is_a_mismatch(self): + self.write("findings-other-kimi", cell("kimi", "adversarial")) + cells, _, total, inconsistent = AGG.aggregate(self.dir, ["kimi"], "", False) + self.assertIn(AGG.MISMATCH_STATUS, [c["status"] for c in cells]) + # It cannot occupy kimi/adversarial: that slot is synthesised as error. + synthesised = [c for c in cells if (c["model"], c["review_type"]) == ("kimi", "adversarial")] + self.assertEqual([c["status"] for c in synthesised], ["error"]) + self.assertEqual(total, 3) + self.assertTrue(inconsistent) + + def test_phantom_error_under_a_made_up_name_does_not_explain_a_red_leg(self): + # A forged ok under the victim's name plus a self-consistent `error` + # under a cell the matrix never ran: the phantom must not be the + # tolerable error that explains the victim's red leg away. + self.full_cursor_panel() + self.write("findings-adversarial-phantom", cell("phantom", "adversarial", "error")) + cells, ok, total, inconsistent = AGG.aggregate(self.dir, CURSOR_MODELS, DIRECT, False, "failure") + phantom = [c for c in cells if c["model"] == "phantom"] + self.assertEqual(phantom[0]["status"], AGG.MISMATCH_STATUS) + self.assertTrue(inconsistent) + reason, tolerated = AA.panel_gate( + [{k: c[k] for k in ("model", "review_type", "status")} for c in cells], max_failed=5 + ) + self.assertIsNotNone(reason) + self.assertEqual(tolerated, []) + + def test_phantom_ok_cannot_complete_a_review_type(self): + self.write("findings-edge-case-kimi", cell("kimi", "edge-case")) + self.write("findings-adversarial-phantom", cell("phantom", "adversarial")) + cells, ok, _, inconsistent = AGG.aggregate(self.dir, ["kimi"], "", False) + self.assertEqual(ok, 1) + self.assertTrue(inconsistent) + + def test_unexpected_direct_cell_is_a_mismatch_only_when_counted(self): + self.full_cursor_panel() + self.write("findings-direct-adversarial-other", cell("other", "adversarial")) + cells, _, _, inconsistent = AGG.aggregate(self.dir, CURSOR_MODELS, DIRECT, True) + self.assertIn(AGG.MISMATCH_STATUS, [c["status"] for c in cells if c["model"] == "other"]) + self.assertTrue(inconsistent) + cells, _, _, inconsistent = AGG.aggregate(self.dir, CURSOR_MODELS, DIRECT, False) + self.assertEqual([c["status"] for c in cells if c["model"] == "other"], ["ok"]) + self.assertFalse(inconsistent) + + def test_cli_requires_the_review_result(self): + # Fail closed: a dropped flag must not read as a successful matrix. + result = subprocess.run( + [sys.executable, SCRIPT, "--panel-dir", self.dir, "--models", json.dumps(["kimi"]), + "--out", os.path.join(self.dir, "panel.json")], + capture_output=True, text=True, + ) + self.assertNotEqual(result.returncode, 0) + self.assertIn("--review-result", result.stderr) + + def test_matching_record_with_hyphenated_model_keeps_its_status(self): + self.write("findings-edge-case-gpt-5.6-sol-xhigh", cell("gpt-5.6-sol-xhigh", "edge-case")) + self.write("findings-adversarial-gpt-5.6-sol-xhigh", cell("gpt-5.6-sol-xhigh", "adversarial")) + _, ok, total, _ = AGG.aggregate(self.dir, ["gpt-5.6-sol-xhigh"], "", False) + self.assertEqual((ok, total), (2, 2)) + + def test_identity_from_name(self): + self.assertEqual(AGG.identity_from_name("findings-edge-case-kimi"), ("edge-case", "kimi")) + self.assertEqual(AGG.identity_from_name(f"findings-direct-adversarial-{DIRECT}"), ("adversarial", DIRECT)) + self.assertEqual(AGG.identity_from_name("findings-adversarial-"), (None, None)) + self.assertEqual(AGG.identity_from_name("pr-diff"), (None, None)) + + def test_cli_writes_panel_inconsistent(self): + self.full_cursor_panel() + gh_out = os.path.join(self.dir, "gh-output") + result = subprocess.run( + [sys.executable, SCRIPT, "--panel-dir", self.dir, "--models", json.dumps(CURSOR_MODELS), + "--review-result", "failure", "--direct-result", "skipped", + "--out", os.path.join(self.dir, "panel.json"), "--github-output", gh_out], + capture_output=True, text=True, check=True, + ) + self.assertIn("a reviewer matrix did not succeed (review=failure", result.stdout) + with open(gh_out, encoding="utf-8") as f: + self.assertIn("panel_inconsistent=true\n", f.read()) + + def test_cli_log_line_cannot_forge_a_command(self): + self.write("findings-adversarial-kimi", cell("x\n::error::forged", "adversarial")) + result = subprocess.run( + [sys.executable, SCRIPT, "--panel-dir", self.dir, "--models", json.dumps(["kimi"]), + "--review-result", "success", "--out", os.path.join(self.dir, "panel.json")], + capture_output=True, text=True, check=True, + ) + self.assertNotIn("\n::error::forged", result.stdout) + self.assertIn("counted as status=mismatch", result.stdout) + + class DownstreamTest(unittest.TestCase): def test_summary_names_a_failed_direct_cell_with_its_backend(self): panel = [ diff --git a/.github/cursor-review/tests/test_auto_approve.py b/.github/cursor-review/tests/test_auto_approve.py index 77f0ddd3..6a90631e 100644 --- a/.github/cursor-review/tests/test_auto_approve.py +++ b/.github/cursor-review/tests/test_auto_approve.py @@ -1368,7 +1368,7 @@ class CmdDecideGateOutputTest(unittest.TestCase): def run_decide(self, findings=(), labels=(), heads=(SHA, SHA), threshold="medium", post_error=None, labels_after=None, reviews=(), put_error=None, panel=PANEL_OK, max_failed=None, - approve_scope=None, **extra): + approve_scope=None, consolidated=None, **extra): heads = iter(heads) reads = [] writes = [] @@ -1398,7 +1398,7 @@ def fake_gh(args, payload=None): out = os.path.join(d, "out") open(out, "w").close() with open(fpath, "w") as f: - json.dump({"findings": list(findings), "panel": list(panel)}, f) + json.dump({"findings": list(findings), "panel": list(panel), **(consolidated or {})}, f) with open(dpath, "w") as f: f.write(DIFF) args = argparse.Namespace(threshold=threshold, findings=fpath, repo="o/r", pr_number="1", @@ -1422,6 +1422,29 @@ def fake_gh(args, payload=None): self.outputs = read_outputs(out) return rc, self.outputs.get("approve_gate") + def test_an_inconsistent_panel_is_untrusted(self): + rc, gate = self.run_decide(consolidated={"panel_inconsistent": True}) + self.assertEqual(gate, "untrusted") + self.assertEqual([p["event"] for p in self.posted], ["REQUEST_CHANGES"]) + self.assertIn(AA.REASON_PANEL_INCONSISTENT, self.posted[0]["body"]) + + def test_the_flag_withholds_even_when_the_file_says_consistent(self): + # The flag is the pre-judge job output; the file is built after the + # `--trust` judge ran and cannot clear it. + rc, gate = self.run_decide(consolidated={"panel_inconsistent": False}, panel_inconsistent="true") + self.assertEqual(gate, "untrusted") + self.assertIn(AA.REASON_PANEL_INCONSISTENT, self.posted[0]["body"]) + + def test_an_unrecognised_flag_fails_closed(self): + rc, _ = self.run_decide(panel_inconsistent="yes") + self.assertEqual(rc, 2) + self.assertTrue(any("--panel-inconsistent must be" in p for p in self.printed)) + + def test_panel_inconsistent_absent_or_false_still_approves(self): + for consolidated in (None, {"panel_inconsistent": False}): + with self.subTest(consolidated=consolidated): + self.assertEqual(self.run_decide(consolidated=consolidated), (0, "pass")) + def test_label_applied_during_the_post_withdraws_the_approval(self): self.assertEqual(self.run_decide(labels_after=["needs-human-review"]), (0, "capped")) self.assertEqual(self.writes, ["repos/o/r/pulls/1/reviews/99/dismissals"]) @@ -2776,6 +2799,51 @@ def test_decides_card_constants_match_card_py(self): AA.CARD_NEXT_HUMAN), CARD.NEXTS) +class PanelInconsistentTest(unittest.TestCase): + """A reviewer leg failed although every panel cell reports ok: a forged + artifact under another cell's name, or an upload that failed after the + review. The count cannot be trusted, and a re-run cannot clear it.""" + + def test_it_is_untrusted_however_clean_the_round(self): + event, gate, reasons, blocking = AA.decide_gate( + "low", [], PANEL_OK, "ok", True, SHA, SHA, [], panel_inconsistent=True) + self.assertEqual((event, gate, blocking), (AA.NONE, AA.GATE_UNTRUSTED, [])) + self.assertEqual(reasons, [AA.REASON_PANEL_INCONSISTENT]) + self.assertEqual( + AA.REASON_PANEL_INCONSISTENT, + "a counted panel cell's findings artifact may not be its own (a reviewer " + "leg did not succeed although every cell reports ok, or a record " + "disagrees with its artifact name)", + ) + + def test_tolerated_failures_do_not_excuse_it(self): + _, gate, _, _ = AA.decide_gate( + "low", [], PANEL_OK, "ok", True, SHA, SHA, [], max_failed_reviewers=5, panel_inconsistent=True) + self.assertEqual(gate, AA.GATE_UNTRUSTED) + + def test_default_is_unchanged(self): + self.assertEqual(AA.decide_gate("low", [], PANEL_OK, "ok", True, SHA, SHA, [])[1], AA.GATE_PASS) + self.assertEqual(AA.decide("low", [], PANEL_OK, "ok", True, SHA, SHA, [], panel_inconsistent=True)[0], + AA.NONE) + + def test_it_is_not_retryable(self): + # A fresh round runs the same PR content, so a forger forges again; + # auto-retry cannot tell that from a flake, a human reading the legs can. + self.assertFalse(AA.auto_retry_eligible([AA.REASON_HEAD_MOVED, AA.REASON_PANEL_INCONSISTENT])) + self.assertNotIn(AA.REASON_PANEL_INCONSISTENT, AA.RETRYABLE_CAUSES) + nxt, text = AA.no_decision_next([AA.REASON_HEAD_MOVED, AA.REASON_PANEL_INCONSISTENT]) + self.assertEqual(nxt, CARD.NEXT_HUMAN) + self.assertTrue(text.startswith("A human is needed: ")) + # Not the other structural causes' "a re-run would land the same way": + # a leg that flaked after its review IS cleared by a re-run. + self.assertNotIn("A re-run would land the same way", text) + self.assertIn("A re-run clears a leg that failed after its review, not a forged artifact", text) + + def test_alongside_another_structural_cause_the_card_keeps_its_suffix(self): + _, text = AA.no_decision_next([AA.REASON_PANEL_INCONSISTENT, AA.REASON_EMPTY_DIFF]) + self.assertIn("A re-run would land the same way", text) + + class NoDecisionNextTest(unittest.TestCase): def test_a_moved_head_relabels(self): # The only transient cause a relabel alone clears: the new round reads a diff --git a/.github/cursor-review/tests/test_workflow_panel_integrity.py b/.github/cursor-review/tests/test_workflow_panel_integrity.py index a0e42308..3278869b 100644 --- a/.github/cursor-review/tests/test_workflow_panel_integrity.py +++ b/.github/cursor-review/tests/test_workflow_panel_integrity.py @@ -109,6 +109,7 @@ "needs.consolidate.outputs.ok_count", "needs.consolidate.outputs.total", "needs.consolidate.outputs.degraded", + "needs.consolidate.outputs.panel_inconsistent", "needs.post-review.outputs.delivered", "needs.post-review.outputs.ungated_findings", "needs.diff-size.outputs.incremental_subset", @@ -289,6 +290,7 @@ def test_the_panel_counts_are_job_outputs(self): for key, source in ( ("ok_count", "steps.aggregate.outputs.ok_count"), ("total", "steps.aggregate.outputs.total"), + ("panel_inconsistent", "steps.aggregate.outputs.panel_inconsistent"), ("degraded", "steps.consolidated.outputs.degraded"), ("judge_status", "steps.consolidated.outputs.judge_status"), ): @@ -306,9 +308,33 @@ def test_the_aggregate_step_still_writes_both_counts(self): script = os.path.join(os.path.dirname(WORKFLOW), "..", "cursor-review", "aggregate-panel.py") with open(script, encoding="utf-8") as f: source = f.read() - for written in ('g.write(f"ok_count={ok}\\n")', 'g.write(f"total={total}\\n")'): + for written in ('g.write(f"ok_count={ok}\\n")', 'g.write(f"total={total}\\n")', + 'g.write(f"panel_inconsistent='): self.assertIn(written, source) + def test_the_aggregate_step_reads_the_matrix_results(self): + # The leg-result cross-check: without these the script sees its + # defaults (`success` / `skipped`) and can never flag a forgery. + step = "\n".join(code_lines(step_named(self.jobs["consolidate"], "Aggregate panel findings"))) + self.assertIn("REVIEW_RESULT: ${{ needs.review.result }}", step) + self.assertIn("DIRECT_RESULT: ${{ needs.review-openai-direct.result }}", step) + self.assertIn('--review-result "$REVIEW_RESULT"', step) + self.assertIn('--direct-result "${DIRECT_RESULT:-skipped}"', step) + + def test_the_consolidated_file_carries_panel_inconsistent(self): + # From the step output, written before the judge ran in this job. + step = "\n".join(code_lines(step_named(self.jobs["consolidate"], "Build consolidated findings file"))) + self.assertIn("PANEL_INCONSISTENT: ${{ steps.aggregate.outputs.panel_inconsistent }}", step) + self.assertIn('"panel_inconsistent": panel_inconsistent', step) + + def test_auto_approve_reads_the_job_output_not_only_the_file(self): + # consolidated.json is built after the `--trust` judge ran in the same + # job, so the decision takes the pre-judge step output directly. + post = self.jobs["post-review"] + step = "\n".join(code_lines(step_named(post, "Auto-approve decision"))) + self.assertIn("PANEL_INCONSISTENT: ${{ needs.consolidate.outputs.panel_inconsistent }}", step) + self.assertIn('--panel-inconsistent "$PANEL_INCONSISTENT"', step) + class PanelIntegrityJobTest(unittest.TestCase): def setUp(self): @@ -374,6 +400,14 @@ def test_it_reads_every_cause(self): f"`{PANEL_JOB}` no longer reads `{read}`", ) + def test_an_inconsistent_panel_is_a_failing_cause(self): + self.assertIn("PANEL_INCONSISTENT: ${{ needs.consolidate.outputs.panel_inconsistent }}", self.body) + self.assertRegex( + self.body, + r'if \[ "\$PANEL_INCONSISTENT" = "true" \]; then\n\s+echo "::error::A findings artifact may not be its own' + r'[^\n]*"\n\s+causes=\$\(\(causes \+ 1\)\)', + ) + def test_a_discarded_incremental_block_is_not_a_failing_cause(self): # `incremental_subset` is `false` only when the incremental block was # built and then FAILED its byte-for-byte subset check — at which point diff --git a/.github/workflows/cursor-review.yml b/.github/workflows/cursor-review.yml index e9a954fc..f6467082 100644 --- a/.github/workflows/cursor-review.yml +++ b/.github/workflows/cursor-review.yml @@ -3199,10 +3199,12 @@ jobs: # "Credential isolation" note for why the judge must not share a job with # a write-scoped token. name: Consolidate panel - # `review-openai-direct` is here only for ORDERING — its artifacts must land - # before `Download panel findings` runs. Its result is never read: when it + # `review-openai-direct` is here for ORDERING — its artifacts must land + # before `Download panel findings` runs. It never gates this job: when it # is skipped (the default) or fails, `!cancelled()` below still runs, and - # a failed cell is counted from its artifact (or its absence) instead. + # a failed cell is counted from its artifact (or its absence) instead. Its + # result is read once, by `Aggregate panel findings`, as a consistency + # check on those artifacts (see aggregate-panel.py). needs: [gate, diff-size, preflight, review, review-openai-direct, ledger] # `!cancelled()`, not `always()`: the judge is the most expensive call in # the workflow, and the only job that does anything with its payload, @@ -3250,6 +3252,12 @@ jobs: outputs: ok_count: ${{ steps.aggregate.outputs.ok_count }} total: ${{ steps.aggregate.outputs.total }} + # 'true' when a counted cell artifact may not be its own cell's: a + # reviewer matrix did not succeed although every counted cell of it + # reads `ok`, or a record is `mismatch`. Written by aggregate-panel.py + # BEFORE the judge runs in this job; `Auto-approve decision` and `Panel + # integrity` read it from here, never from consolidated.json. + panel_inconsistent: ${{ steps.aggregate.outputs.panel_inconsistent }} degraded: ${{ steps.consolidated.outputs.degraded }} # Untrusted: this is whatever string the judge agent's own tool wrote as # `status` in /tmp/judge-findings.json, in a job running `cursor-agent @@ -3413,15 +3421,25 @@ jobs: # lane; side by side they stay advisory. See aggregate-panel.py. DIRECT_MODEL: ${{ needs.preflight.outputs.openai_direct == 'true' && inputs.openai_direct_model || '' }} DIRECT_COUNTS: ${{ needs.preflight.outputs.openai_direct_counts }} + # The matrices' RESULTS, not their artifacts: a cell can upload under + # another cell's name, but it cannot turn the honest leg's 409 green. + # `Fail the leg when the cell did not submit` reds every non-ok cell, + # so a red matrix whose artifacts all read ok is a forgery or a + # post-review failure — either way, panel_inconsistent=true. + REVIEW_RESULT: ${{ needs.review.result }} + DIRECT_RESULT: ${{ needs.review-openai-direct.result }} run: | # Each artifact directory contains one findings.json. Combine into # a single panel.json array preserving cell metadata, and emit the - # ok_count/total `Panel integrity` and auto-approve gate on. + # ok_count/total/panel_inconsistent `Panel integrity` and auto-approve + # gate on. python3 "$CURSOR_REVIEW_ASSETS/aggregate-panel.py" \ --panel-dir /tmp/panel \ --models "$PANEL_MODELS" \ --direct-model "$DIRECT_MODEL" \ --direct-counts "${DIRECT_COUNTS:-false}" \ + --review-result "$REVIEW_RESULT" \ + --direct-result "${DIRECT_RESULT:-skipped}" \ --out /tmp/panel.json \ --github-output "$GITHUB_OUTPUT" @@ -3519,6 +3537,9 @@ jobs: id: consolidated env: OK_COUNT: ${{ steps.aggregate.outputs.ok_count }} + # From the step output, not /tmp/panel.json: the judge ran in this job + # since `Aggregate panel findings` wrote it, and can rewrite the file. + PANEL_INCONSISTENT: ${{ steps.aggregate.outputs.panel_inconsistent }} run: | # If no cells completed, short-circuit — post-review.py will render # a "no contributors" review with the panel summary. Otherwise the @@ -3579,7 +3600,10 @@ jobs: seen.add(key) findings.append(fnd) - consolidated = {"findings": findings, "panel": panel_meta} + # auto-approve withholds on it (a reviewer leg did not succeed + # although every panel cell reports ok). + panel_inconsistent = os.environ.get("PANEL_INCONSISTENT") == "true" + consolidated = {"findings": findings, "panel": panel_meta, "panel_inconsistent": panel_inconsistent} with open("/tmp/consolidated.json", "w", encoding="utf-8") as f: json.dump(consolidated, f) @@ -4167,6 +4191,10 @@ jobs: APPROVE_SCOPE: ${{ inputs.approve_scope }} INCREMENTAL_STATE: ${{ needs.diff-size.outputs.incremental_state }} MAX_FAILED_REVIEWERS: ${{ inputs.approve_max_failed_reviewers }} + # aggregate-panel.py's step output, written before the judge ran — + # not consolidated.json, which `consolidate` builds after the judge + # (`--trust`, same job) could have rewritten its inputs. + PANEL_INCONSISTENT: ${{ needs.consolidate.outputs.panel_inconsistent }} # approve_authors, resolved by `gate`: `false` → decide posts nothing # and reports `off` for this author. AUTHOR_ENABLED: ${{ needs.gate.outputs.approve_author_ok }} @@ -4223,6 +4251,7 @@ jobs: --ledger /tmp/ledger/ledger.json \ --open-anchors /tmp/open-anchors.json \ --max-failed-reviewers "${MAX_FAILED_REVIEWERS:-0}" \ + --panel-inconsistent "$PANEL_INCONSISTENT" \ --author-enabled "$AUTHOR_ENABLED" \ --pr-author "$PR_AUTHOR" \ --card true \ @@ -4562,6 +4591,7 @@ jobs: env: OK_COUNT: ${{ needs.consolidate.outputs.ok_count }} TOTAL: ${{ needs.consolidate.outputs.total }} + PANEL_INCONSISTENT: ${{ needs.consolidate.outputs.panel_inconsistent }} DEGRADED: ${{ needs.consolidate.outputs.degraded }} JUDGE_STATUS: ${{ needs.consolidate.outputs.judge_status }} DELIVERED: ${{ needs.post-review.outputs.delivered }} @@ -4654,6 +4684,13 @@ jobs: causes=$((causes + 1)) fi + # Separate from the counts: here they can MATCH, and that is the + # problem. See aggregate-panel.py's two cross-checks. + if [ "$PANEL_INCONSISTENT" = "true" ]; then + echo "::error::A findings artifact may not be its own cell's: a reviewer leg did not succeed although every findings artifact reads ok, or a record disagrees with the artifact name it sits under (see the Aggregate panel findings warnings). Something other than that cell uploaded it, or an upload failed after the review. The panel count is untrusted; see the red leg checks." + causes=$((causes + 1)) + fi + # Non-empty AND greater than zero: an EMPTY value means post-review # never got as far as deciding, which `DELIVERED` below reports on its # own terms rather than as a phantom unanchored finding. diff --git a/docs/callers/cursor-approve.md b/docs/callers/cursor-approve.md index 7828e87f..be041983 100644 --- a/docs/callers/cursor-approve.md +++ b/docs/callers/cursor-approve.md @@ -350,9 +350,19 @@ cell's real upload fails and its leg goes red while consolidate reads the forgery. A bare `needs:` used to skip the axes on that red leg. With `!cancelled()` the round goes on to the axes, which still review the change independently, and decide still withholds on any red axis (or more yellow -than `max_yellow_axes`). Treat the panel count -as tamper-evident only in the rollup (the red leg stays visible), not in the -gate. +than `max_yellow_axes`). The gate now reads that red leg too: when a reviewer +leg did not succeed although every counted cell artifact reads `ok`, or a +record disagrees with its artifact name (or names no cell the panel runs), the +panel is marked inconsistent and `approve_gate` is `untrusted` — not re-run +automatically, since a fresh round runs the same PR content and a forger +forges again — and Panel integrity fails naming it. The check is per matrix, +not per leg, so it is narrow: ANY non-`ok` cell in the matrix explains the red +leg away — an ordinary step-cap timeout as much as the forger erroring its own +cell — and under `approve_max_failed_reviewers` > 0 that error is tolerated. +A forger able to delete the honest upload and re-upload under its name leaves +no red leg at all. In those cases the count stays tamper-evident only in the +rollup, if at all; telling which leg failed needs per-job conclusions +(`actions: read`), which the consolidate job does not have. ## Inputs diff --git a/docs/callers/cursor-review.md b/docs/callers/cursor-review.md index ed161444..60d58c1a 100644 --- a/docs/callers/cursor-review.md +++ b/docs/callers/cursor-review.md @@ -343,6 +343,7 @@ the job log: | Cause | What it means | |---|---| | Panel incomplete | Fewer cells submitted findings than ran. The consolidated review was adjudicated over a short panel. The individual leg checks (`adversarial ()` / `edge-case ()`) are red for exactly the cells that did not submit — almost always the 15-minute agent cap. | +| Panel inconsistent | A reviewer leg is red although every counted cell's findings artifact reads `ok`, or a record disagrees with the artifact name it sits under (or names no cell the panel runs). A leg reds only when its cell is not `ok`, so this means an artifact under a cell's name was uploaded by something other than that cell (a prompt-injected cell claiming another's name; the honest upload then fails) or the upload failed after the review. Auto-approve withholds (`untrusted`) and does not re-run on it. The leg check is per matrix: any other non-`ok` cell in the same matrix explains a red leg away, so it does not catch every forgery. | | Unanchored findings | Findings the review could not anchor to a line of the reviewed diff, so they were demoted to the review **body** and have no thread. The Blocking gate cannot see them; read the body. | | Nothing delivered | No review carrying resolvable finding threads reached the PR — a read-only token, a rejected inline payload, or a post that could not be confirmed. The findings are in the `Post review` job summary. | | Judge degraded | The judge model never adjudicated; the review is the raw union of the cells' findings, so duplicates and false positives were not filtered out. | @@ -613,7 +614,8 @@ After `Post review` lands, `auto-approve.py decide` submits one of: severity; - **no decision** when the round can't be trusted: the judge did not adjudicate, a panel reviewer did not complete (beyond what `approve_max_failed_reviewers` - tolerates — see below), the review did not land as threads (or some + tolerates — see below), a panel cell's artifact may not be its own + (Panel inconsistent, above), the review did not land as threads (or some finding reached the review body only), the head moved or the base was retargeted mid-run, the PR state could not be read, an earlier round's thread above the threshold is still @@ -880,7 +882,7 @@ jobs: |---|---| | `pass` | The auto-approve decision was APPROVE (posted, unless `defer_approval` left it to cursor-approve). | | `fail` | REQUEST_CHANGES, or an earlier round's open thread above the threshold withheld approval. | -| `untrusted` | The judge was degraded, a panel cell failed (beyond what `approve_max_failed_reviewers` tolerates), the review was not delivered, or the head moved — and also any run that delivered no round at all (an unrelated event, an already-reviewed head, an over-cap diff). | +| `untrusted` | The judge was degraded, a panel cell failed (beyond what `approve_max_failed_reviewers` tolerates), a reviewer leg failed although every panel cell reports `ok` (a cell's artifact may not be its own — see Panel integrity), the review was not delivered, or the head moved — and also any run that delivered no round at all (an unrelated event, an already-reviewed head, an over-cap diff). | | `capped` | The round cap was hit by this run, or the PR carries `needs-human-review` (while `max_rounds` or `approve_max_severity` is set). Wins over `off`. | | `off` | `approve_max_severity` is empty, or `approve_authors` does not list the PR's author (and the cap was not hit). |