From 0db11aa91419b68fcb856b5f36b6bc8d08305b48 Mon Sep 17 00:00:00 2001 From: Matt Miller Date: Fri, 9 Oct 2026 18:20:26 -0700 Subject: [PATCH 1/3] feat(cursor-review): say why a direct-API cell failed instead of discarding it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Both direct lanes send their agent's structured output to a file that is never echoed — `codex exec --json` to `codex-events.jsonl`, `claude --output-format json` to `claude-result.json` — because that file also holds the agent's prose, which is model output steered by PR-authored text. But `codex` reports API-level failures as events on STDOUT, i.e. into that same file. So an outage left the public log showing `Reading prompt from stdin...`, a non-zero exit, and nothing else, and the lane's failures had to be inferred from job DURATION. That is not hypothetical: an OpenAI `insufficient_quota` / `credit_balance_exhausted` — an error that named its own cause and linked the billing page — was discarded exactly this way. `agent-error.py` reads the file and prints, from objects that look like errors, an ALLOWLIST of short provider metadata plus a census of event types. The allowlist is the safety property, not a convenience: `result`, `content`, `text`, `arguments` and `reasoning` are not in it, so assistant prose cannot reach a public log however the provider nests it, and an allowlisted `message` holding content blocks is skipped rather than flattened. Values are sanitized for a `::workflow command::` line by REPLACING disallowed characters, never deleting them, so a crafted value cannot collapse into different text. An `error` KEY is read on its own terms rather than by heuristic. A provider's error object need not describe ITSELF as an error — OpenAI's says `insufficient_quota` / `credit_balance_exhausted`, neither of which contains the string "error" — and the first draft of this script parsed that payload and still printed only the envelope's `type=error`, missing the one message worth having. `NestedEnvelopeTest` pins that shape. The raw file is deliberately NOT uploaded as an artifact. The Anthropic lane screens `findings.json` for a leaked API key before publishing it; shipping the unscreened transcript alongside would route the agent's own prose straight past that guard. When the allowlist plus the census is not enough, the fix is to widen the allowlist, not to publish the file. It runs inside `Report cell outcome` — where a reader already looks — and only on the non-ok branch, and it exits 0 on anything (missing, truncated, non-JSON, invalid UTF-8, a directory), because it runs after the cell has already failed and must not become a second failure. Co-Authored-By: Claude Opus 5 --- .github/cursor-review/README.md | 1 + .github/cursor-review/agent-error.py | 232 ++++++++++ .../cursor-review/tests/test_agent_error.py | 397 ++++++++++++++++++ .github/workflows/cursor-review.yml | 54 ++- docs/callers/cursor-review.md | 10 + 5 files changed, 684 insertions(+), 10 deletions(-) create mode 100644 .github/cursor-review/agent-error.py create mode 100644 .github/cursor-review/tests/test_agent_error.py diff --git a/.github/cursor-review/README.md b/.github/cursor-review/README.md index 5031f8f3..24096272 100644 --- a/.github/cursor-review/README.md +++ b/.github/cursor-review/README.md @@ -256,6 +256,7 @@ the consolidated review's panel table. See [Panel integrity](#panel-integrity). | [`build-ledger.py`](build-ledger.py) | Builds the **prior-review ledger** — what earlier rounds raised on this PR and how the author answered — and splices it into the panel/judge prompts. Untrusted prose is defanged twice over: fence-opening lines are rewritten, and continuation lines of quoted prose are prefixed ` |` so no imported text can sit at a field's indent and forge one. Also the prompt splicer, so the no-ledger path is byte-identical to the pre-ledger prompt. | | [`fence-diff.py`](fence-diff.py) | Wraps the reviewed diff (plus the incremental hunks, and the judge's panel-findings block) in `=== BEGIN/END DIFF ===` fences. The diff is attacker-authored PR bytes, so static literal fences are not a control; the nonce is what a PR cannot forge. Each prompt-build step mints its OWN nonce (`mint`), into a shell variable rather than a step `env:` or job output — Actions dumps a step's env map into the public run log, and a per-prompt value means a leak in one job cannot forge a fence in another. Copies the body through **byte for byte** — it never defangs or normalizes the payload. | | [`incremental-diff.py`](incremental-diff.py) | Builds the incremental "hunks new since the last reviewed round" block as the difference between two PR patches (`build`), and verifies the result really is a subset of the reviewed diff (`check`) so the workflow can discard it if it is not. Never a commit range: `git diff LAST_REVIEWED...HEAD` carries every commit a merge of the base branch brought in — see [the ledger section](#the-prior-review-ledger-and-the-repeat-policy). | +| [`agent-error.py`](agent-error.py) | Run by each **direct-API** cell's **Report cell outcome** step, and only when that cell did not submit: prints WHY it failed. Both direct lanes send their agent's structured output to a file (`codex-events.jsonl`, `claude-result.json`) that is never echoed, because it also holds the agent's prose — so an API-level failure, which `codex exec --json` reports as an event on **stdout**, used to leave the log showing a non-zero exit and nothing else, and the lane's failures had to be inferred from job DURATION (a real `insufficient_quota` / `credit_balance_exhausted` outage was discarded that way). It walks the file and prints, from objects that look like errors, an **allowlist** of short provider metadata (`type`, `subtype`, `code`, `param`, `status`, `message`) plus a census of event types. The allowlist is the safety property — `result`, `content`, `text`, `arguments` and `reasoning` are not in it, so assistant prose cannot reach the public log however the provider nests it, and a `message` holding content blocks is skipped rather than flattened. An `error` KEY is read on its own terms: a provider's error object need not describe itself as an error (OpenAI's says `insufficient_quota`), so the nested shape is handled explicitly rather than by looking for the substring "error". Values are sanitized for a `::workflow command::` line by REPLACING disallowed characters, never deleting them. The raw file is deliberately **not** uploaded: the Anthropic lane screens `findings.json` for a leaked API key before publishing it, and shipping the unscreened transcript alongside would route the agent's prose past that guard. It exits **0 on anything** — missing, truncated, non-JSON or invalid-UTF-8 — since it runs after the cell already failed and must not become a second failure. | | [`catalog-drift.py`](catalog-drift.py) | Backs the weekly catalog-drift check. Extracts the pins from `cursor-review.yml`, diffs them against raw `cursor-agent models` output, and renders the sticky issue title + body (delisted pins, pins marked NO-ZDR, unpinned same-lab ids, catalog ids from unpinned families, stale audit date). Reports only — it never edits a pin. | ## Adopt it in your repo diff --git a/.github/cursor-review/agent-error.py b/.github/cursor-review/agent-error.py new file mode 100644 index 00000000..062a3684 --- /dev/null +++ b/.github/cursor-review/agent-error.py @@ -0,0 +1,232 @@ +#!/usr/bin/env python3 +"""Why a direct-API reviewer cell failed, in a form a PUBLIC run log can carry. + +Both direct lanes write their agent's output to a file and, until this script, +threw it away: the OpenAI lane redirects `codex exec --json` to +`codex-events.jsonl` and the Anthropic lane redirects `claude --output-format +json` to `claude-result.json`. Neither file was echoed or uploaded, and codex +reports API failures on STDOUT as events — so the log showed `Reading prompt +from stdin...`, `exit code 1`, and nothing else. A real outage +(`insufficient_quota` / `credit_balance_exhausted`, naming its own cause and +linking the billing page) was discarded, and the lane's failures had to be +inferred from job DURATION. That is what this fixes. + +WHAT IT PRINTS, AND WHY SO LITTLE. These files hold the agent's reasoning, its +tool calls and its findings — model output steered by PR-authored text, which +`Report cell outcome` already refuses to echo for that reason. So this does not +dump them. It walks the JSON and emits, from objects that look like ERRORS, an +ALLOWLIST of short metadata fields (``ERROR_FIELDS``) plus a census of event +types. The allowlist is the safety property: `result`, `content`, `text`, +`arguments`, `reasoning` and friends are not in it, so assistant prose cannot +reach the log through here no matter how the provider nests it. The raw file is +deliberately NOT uploaded as an artifact either: the Anthropic lane screens +`findings.json` for a leaked API key before publishing it, and shipping the +unscreened transcript alongside would route the agent's own prose straight past +that guard. When the allowlist plus the event census is not enough to explain a +failure, widen the allowlist — do not publish the file. + +Every emitted value is sanitized for a ``::workflow command::`` line the same +way the rest of this workflow does it — disallowed characters are REPLACED, not +dropped, because deleting would let a crafted "e r r o r" collapse into a value +that reads as something else. + +DIAGNOSTIC ONLY: it exits 0 on anything, including a missing, truncated, +invalid-UTF-8 or non-JSON file. The step that runs it has already captured the +agent's real exit code; a traceback here would replace the diagnosis with a +crash, which is the failure this script exists to remove. + +Usage: + agent-error.py --events /tmp/codex-events.jsonl --label adversarial/gpt-5.6-sol +""" + +import argparse +import json +import re +import sys + +# Short, provider-authored metadata. Deliberately NOT `result`, `content`, +# `text`, `input`, `output`, `arguments`, `reasoning`, `delta` or `thinking`: +# those carry model output. Adding a key here widens what a public log can show, +# so it is a decision, not a convenience. +ERROR_FIELDS = ("type", "subtype", "code", "param", "status", "status_code", "http_status", "message") + +# Bounds on what one invocation can print. A failing agent can emit a very +# large file, and this output goes to a log a human reads. +MAX_VALUE = 300 +MAX_LINES = 40 +MAX_BYTES = 8000 +# Bound the walk: these files are agent-written, so depth and breadth are not +# ours to trust. A findings payload nests ~4 deep; 12 is slack, not a limit +# anyone legitimately reaches. +MAX_DEPTH = 12 +MAX_NODES = 20000 + +_SAFE = re.compile(r"[^A-Za-z0-9 .,:;_/@+()\[\]{}='\"!?#$%^&*|~`<>-]") + + +def sanitize(value, limit: int = MAX_VALUE) -> str: + """One line, safe to drop into a run log. Replaces, never deletes.""" + text = str(value) + # Newlines first: a single value must not become two log lines, and must + # not be able to open a `::workflow command::` of its own. + text = text.replace("\r", " ").replace("\n", " ") + text = _SAFE.sub("?", text) + # `::` is what starts a workflow command; break the pair rather than drop a + # character, so the text still reads. + text = text.replace("::", ": :") + if len(text) > limit: + text = text[: limit - 1].rstrip() + "…" + return text + + +def _looks_like_error(obj: dict) -> bool: + """True when this object is reporting a failure rather than describing work.""" + if obj.get("is_error") is True: + return True + for key in ("type", "subtype", "code"): + value = obj.get(key) + if isinstance(value, str) and "error" in value.lower(): + return True + for key in ("status", "status_code", "http_status"): + value = obj.get(key) + if isinstance(value, bool): + continue + if isinstance(value, int) and value >= 400: + return True + return False + + +def _fields(obj: dict) -> str: + """The allowlisted fields of one error object, as `k=v` pairs.""" + parts = [] + for key in ERROR_FIELDS: + if key not in obj: + continue + value = obj[key] + # Only scalars. A dict or list under an allowlisted name (a `message` + # holding content blocks, say) is NOT flattened — that is how model + # prose would get in. + if isinstance(value, (dict, list)): + continue + if value is None or value == "": + continue + parts.append(f"{key}={sanitize(value)}") + return " ".join(parts) + + +def walk(node, out: list, types: dict, state: dict, depth: int = 0) -> None: + """Collect error objects and a census of `type` values. Bounded.""" + if state["nodes"] >= MAX_NODES or depth > MAX_DEPTH: + state["truncated"] = True + return + state["nodes"] += 1 + if isinstance(node, dict): + kind = node.get("type") + if isinstance(kind, str): + key = sanitize(kind, 60) + types[key] = types.get(key, 0) + 1 + if _looks_like_error(node): + rendered = _fields(node) + if rendered and rendered not in out: + out.append(rendered) + # An `error` KEY is the canonical envelope and must be read on its own + # terms. The provider's error object does not have to describe ITSELF as + # an error — OpenAI's is `{"error": {"type": "insufficient_quota", + # "code": "credit_balance_exhausted", ...}}`, and neither field contains + # the string "error", so `_looks_like_error` alone walked straight past + # the one message worth printing and reported only the envelope's own + # `type=error`. That is the exact failure this script was written for, + # so it is handled explicitly rather than by heuristic. + err = node.get("error") + if isinstance(err, str) and err.strip(): + rendered = f"error={sanitize(err)}" + if rendered not in out: + out.append(rendered) + elif isinstance(err, dict): + rendered = _fields(err) + if rendered and rendered not in out: + out.append(rendered) + for value in node.values(): + walk(value, out, types, state, depth + 1) + elif isinstance(node, list): + for value in node: + walk(value, out, types, state, depth + 1) + + +def parse(text: str) -> list: + """Whole-file JSON (claude) or newline-delimited events (codex), or []. + + Partial output is the normal case for a killed agent, so a file that does + not parse as a whole is read line by line and the unparseable lines are + skipped rather than failing the read. + """ + stripped = text.strip() + if not stripped: + return [] + try: + return [json.loads(stripped)] + except ValueError: + pass + nodes = [] + for line in stripped.splitlines(): + line = line.strip() + if not line or line[0] not in "{[": + continue + try: + nodes.append(json.loads(line)) + except ValueError: + continue + return nodes + + +def summarize(text: str) -> list: + nodes = parse(text) + out, types, state = [], {}, {"nodes": 0, "truncated": False} + for node in nodes: + walk(node, out, types, state) + lines = [] + if types: + census = ", ".join(f"{k}×{v}" for k, v in sorted(types.items(), key=lambda kv: (-kv[1], kv[0]))[:10]) + lines.append(f"events: {len(nodes)} ({census})") + elif nodes: + lines.append(f"events: {len(nodes)}") + for item in out[:MAX_LINES]: + lines.append(f"error: {item}") + if len(out) > MAX_LINES: + lines.append(f"… and {len(out) - MAX_LINES} more error object(s) (output bounded)") + if state["truncated"]: + lines.append("(walk bounded: the agent's output is larger than this summary reads)") + if not lines: + lines.append("no JSON events and no error object in the agent's output") + elif not out: + lines.append("no error object in the agent's output") + return lines + + +def main(argv=None) -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--events", required=True) + parser.add_argument("--label", default="") + args = parser.parse_args(argv) + + label = sanitize(args.label, 80) if args.label else "" + head = f"=== why the cell failed ({label}) ===" if label else "=== why the cell failed ===" + print(head) + try: + # Bounded read, errors="replace": these bytes are agent-written, so + # invalid UTF-8 is reachable and must not raise. + with open(args.events, "rb") as f: + raw = f.read(4_000_000) + text = raw.decode("utf-8", "replace") + except OSError as e: + print(f"could not read {sanitize(args.events, 120)}: {sanitize(type(e).__name__, 40)}") + return 0 + body = "\n".join(summarize(text)) + if len(body.encode("utf-8")) > MAX_BYTES: + body = body.encode("utf-8")[:MAX_BYTES].decode("utf-8", "ignore") + "\n… (bounded)" + print(body) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/.github/cursor-review/tests/test_agent_error.py b/.github/cursor-review/tests/test_agent_error.py new file mode 100644 index 00000000..fcf119b6 --- /dev/null +++ b/.github/cursor-review/tests/test_agent_error.py @@ -0,0 +1,397 @@ +#!/usr/bin/env python3 +"""Tests for agent-error.py, which makes a direct-API cell's failure legible. + +Two things are being protected, and they pull against each other. + +LEGIBILITY. The motivating outage was an OpenAI `insufficient_quota` / +`credit_balance_exhausted`, reported by `codex exec --json` as an event on +STDOUT — i.e. into the events file, which is never echoed. The log showed a +non-zero exit and nothing else, so the lane's failures had to be inferred from +job DURATION. `NestedEnvelopeTest` pins that exact shape, because the first +draft of this script DID parse it and still printed only `type=error`: the +provider's error object describes itself as `insufficient_quota`, and the +heuristic looking for the substring "error" in `type`/`subtype`/`code` walked +straight past the one message worth printing. + +SILENCE ABOUT MODEL OUTPUT. These files also hold the agent's reasoning, tool +calls and findings — model output steered by PR-authored text, which the +workflow's own `Report cell outcome` step refuses to echo for that reason. The +allowlist is the safety property, so `AllowlistTest` asserts the absence of +prose rather than the presence of fields: a future key added to ERROR_FIELDS +that happens to carry content would fail there. + +Everything else is about not becoming a second failure. The script runs in an +`always()` step after the agent already failed, so a truncated, empty, +non-JSON, invalid-UTF-8 or missing file must still exit 0 (`DegenerateTest`), +and an agent-written file must not be able to exhaust memory or the recursion +limit (`BoundsTest`). + +Run: python3 -m unittest discover -s .github/cursor-review/tests -p 'test_*.py' -v +""" + +import importlib.util +import io +import json +import os +import re +import subprocess +import sys +import tempfile +import unittest + +SCRIPT = os.path.normpath( + os.path.join(os.path.dirname(__file__), "..", "agent-error.py") +) +SPEC = importlib.util.spec_from_file_location("agent_error", SCRIPT) +agent_error = importlib.util.module_from_spec(SPEC) +SPEC.loader.exec_module(agent_error) + +WORKFLOW = os.path.normpath( + os.path.join( + os.path.dirname(__file__), "..", "..", "workflows", "cursor-review.yml" + ) +) + +# Marker strings. If one of these reaches the output, prose reached a public log. +PROSE = "PROSE_MUST_NOT_APPEAR" + + +def summary(obj_or_text): + """The script's output lines for a payload, as the workflow would get them.""" + text = obj_or_text + if not isinstance(text, str): + text = json.dumps(obj_or_text) + return agent_error.summarize(text) + + +def joined(obj_or_text): + return "\n".join(summary(obj_or_text)) + + +class NestedEnvelopeTest(unittest.TestCase): + """The outage this script exists for, pinned shape by shape.""" + + # The literal body OpenAI returned, with the surrounding codex events. + QUOTA = ( + '{"type":"session.created","session_id":"abc"}\n' + '{"type":"item.completed","item":{"type":"reasoning","text":"%s"}}\n' + '{"type":"error","error":{"type":"insufficient_quota",' + '"code":"credit_balance_exhausted","message":"You have no credits ' + 'remaining. Add credits to continue using the API at ' + 'https://platform.openai.com/settings/organization/billing/."}}\n' + ) % PROSE + + def test_the_provider_error_is_named(self): + out = joined(self.QUOTA) + # Each of these is independently the thing a reader needs. + self.assertIn("insufficient_quota", out) + self.assertIn("credit_balance_exhausted", out) + self.assertIn("no credits remaining", out) + + def test_the_billing_link_survives(self): + # The provider's message names the fix; truncating it away would leave + # the reader with a code and no next step. + self.assertIn("platform.openai.com", joined(self.QUOTA)) + + def test_the_reasoning_event_does_not(self): + self.assertNotIn(PROSE, joined(self.QUOTA)) + + def test_a_nested_error_needs_no_self_description(self): + # The regression directly: nothing here contains the string "error" + # except the envelope key, so a substring heuristic alone finds nothing. + out = joined({"error": {"type": "overloaded", "code": "slow_down"}}) + self.assertIn("overloaded", out) + self.assertIn("slow_down", out) + + def test_a_string_error_is_still_read(self): + self.assertIn("boom", joined({"error": "boom"})) + + def test_a_blank_string_error_is_not_reported(self): + out = joined({"type": "turn.completed", "error": " "}) + self.assertNotIn("error:", out) + + +class AllowlistTest(unittest.TestCase): + """What may NOT appear. Asserted as absence, so a widened allowlist fails.""" + + PROSE_KEYS = ( + "result", "content", "text", "input", "output", "arguments", + "reasoning", "delta", "thinking", "summary", "findings", + ) + + def test_no_prose_key_is_printed_from_an_error_object(self): + for key in self.PROSE_KEYS: + with self.subTest(key=key): + payload = {"type": "error", "is_error": True, key: PROSE} + self.assertNotIn(PROSE, joined(payload)) + + def test_the_allowlist_holds_no_prose_key(self): + for key in self.PROSE_KEYS: + self.assertNotIn(key, agent_error.ERROR_FIELDS) + + def test_a_structured_message_is_not_flattened(self): + # `message` IS allowlisted, so a provider that nests content blocks + # under it must be skipped rather than stringified. + payload = {"type": "error", "message": [{"type": "text", "text": PROSE}]} + self.assertNotIn(PROSE, joined(payload)) + + def test_a_dict_message_is_not_flattened(self): + payload = {"type": "error", "message": {"body": PROSE}} + self.assertNotIn(PROSE, joined(payload)) + + def test_claude_result_prose_is_withheld_but_the_failure_is_named(self): + payload = { + "type": "result", + "subtype": "error_during_execution", + "is_error": True, + "result": PROSE, + "usage": {"input_tokens": 10}, + } + out = joined(payload) + self.assertNotIn(PROSE, out) + self.assertIn("error_during_execution", out) + + +class DetectionTest(unittest.TestCase): + def test_is_error_true(self): + self.assertTrue(agent_error._looks_like_error({"is_error": True})) + + def test_is_error_false_is_not_an_error(self): + self.assertFalse(agent_error._looks_like_error({"is_error": False})) + + def test_an_http_status_at_or_over_400(self): + self.assertTrue(agent_error._looks_like_error({"status": 429})) + self.assertTrue(agent_error._looks_like_error({"status_code": 500})) + self.assertTrue(agent_error._looks_like_error({"http_status": 400})) + + def test_a_successful_status_is_not_an_error(self): + self.assertFalse(agent_error._looks_like_error({"status": 200})) + + def test_a_boolean_status_is_not_a_status_code(self): + # Pins the BEHAVIOUR, which today holds by arithmetic rather than by + # the `isinstance(value, bool)` guard above it: `bool` is an `int` + # subclass, but `True >= 400` is already false, so deleting that guard + # changes nothing and this test cannot catch it. The guard stays as a + # statement of intent for anyone who lowers the threshold. + self.assertFalse(agent_error._looks_like_error({"status": True})) + self.assertFalse(agent_error._looks_like_error({"status": False})) + + def test_a_clean_run_reports_no_error(self): + text = ( + '{"type":"session.created"}\n' + '{"type":"item.completed","item":{"type":"tool_call"}}\n' + '{"type":"turn.completed"}\n' + ) + out = joined(text) + self.assertIn("no error object", out) + self.assertNotIn("error: ", out) + + def test_the_event_census_is_printed(self): + # The census is the fallback diagnosis when there is no error object: + # it says how far the agent got. + out = joined('{"type":"session.created"}\n{"type":"turn.completed"}\n') + self.assertIn("events: 2", out) + self.assertIn("session.created", out) + + +class SanitizationTest(unittest.TestCase): + def test_a_workflow_command_cannot_be_forged(self): + payload = {"error": {"message": "x\n::error::forged\n::set-output name=a::b"}} + out = joined(payload) + self.assertNotIn("::", out) + self.assertIn("forged", out) # broken, not deleted + + def test_a_value_cannot_become_two_log_lines(self): + out = summary({"error": {"message": "first\nsecond\rthird"}}) + self.assertEqual( + 1, sum(1 for line in out if "first" in line and "third" in line) + ) + + def test_disallowed_characters_are_replaced_not_dropped(self): + # Deleting would let a crafted value collapse into different text. + self.assertEqual("a?b", agent_error.sanitize("a\u0000b")) + + def test_a_long_value_is_truncated(self): + out = agent_error.sanitize("x" * 5000) + self.assertLessEqual(len(out), agent_error.MAX_VALUE) + self.assertTrue(out.endswith("…")) + + def test_the_label_is_sanitized(self): + with tempfile.TemporaryDirectory() as d: + path = os.path.join(d, "events.jsonl") + with open(path, "w", encoding="utf-8") as f: + f.write("{}\n") + out = run_script(path, label="::error::forged") + self.assertNotIn("::error::", out) + + +class BoundsTest(unittest.TestCase): + def test_deep_nesting_does_not_recurse_without_limit(self): + node = {"type": "error", "code": "deep"} + for _ in range(5000): + node = {"wrap": node} + out = joined(node) # must not raise RecursionError + self.assertIn("walk bounded", out) + + def test_wide_breadth_is_bounded(self): + payload = {"items": [{"type": f"t{i}"} for i in range(50000)]} + out = joined(payload) + self.assertIn("walk bounded", out) + + def test_many_error_objects_are_capped(self): + text = "\n".join( + json.dumps({"type": "error", "code": f"c{i}"}) for i in range(200) + ) + out = summary(text) + self.assertLessEqual( + sum(1 for line in out if line.startswith("error: ")), + agent_error.MAX_LINES, + ) + self.assertTrue(any("more error object(s)" in line for line in out)) + + def test_duplicate_error_objects_collapse(self): + one = json.dumps({"type": "error", "code": "same"}) + out = summary("\n".join([one] * 20)) + self.assertEqual(1, sum(1 for line in out if line.startswith("error: "))) + + def test_the_census_is_capped(self): + text = "\n".join(json.dumps({"type": f"t{i}"}) for i in range(100)) + census = [line for line in summary(text) if line.startswith("events: ")] + self.assertEqual(1, len(census)) + self.assertLessEqual(census[0].count("×"), 10) + + +def run_script(path, label="adversarial/gpt-5.6-sol"): + """The script as the workflow runs it: a subprocess, whose code must be 0.""" + proc = subprocess.run( + [sys.executable, SCRIPT, "--events", path, "--label", label], + capture_output=True, text=True, + ) + assert proc.returncode == 0, f"exit {proc.returncode}: {proc.stderr}" + return proc.stdout + + +class DegenerateTest(unittest.TestCase): + """It runs after the agent already failed; it must never fail in turn.""" + + def setUp(self): + self.dir = tempfile.TemporaryDirectory() + self.addCleanup(self.dir.cleanup) + + def write(self, data, name="events.jsonl"): + path = os.path.join(self.dir.name, name) + mode = "wb" if isinstance(data, bytes) else "w" + with open(path, mode) as f: + f.write(data) + return path + + def test_a_missing_file_exits_zero(self): + out = run_script(os.path.join(self.dir.name, "nope.jsonl")) + self.assertIn("could not read", out) + + def test_an_empty_file_exits_zero(self): + self.assertIn("no JSON events", run_script(self.write(""))) + + def test_plain_text_exits_zero(self): + out = run_script(self.write("Reading prompt from stdin...\nnot json\n")) + self.assertIn("no JSON events", out) + + def test_invalid_utf8_exits_zero(self): + path = self.write(b'{"error":{"message":"caf\xc3\x28 bytes"}}\n') + self.assertIn("bytes", run_script(path)) + + def test_a_truncated_last_event_still_reads_the_rest(self): + # The normal shape for a killed agent: complete events, then half of one. + path = self.write( + '{"type":"error","error":{"code":"first_error"}}\n' + '{"type":"item.completed","item":{"te' + ) + self.assertIn("first_error", run_script(path)) + + def test_a_directory_exits_zero(self): + self.assertIn("could not read", run_script(self.dir.name)) + + def test_the_header_names_the_cell(self): + out = run_script(self.write("{}\n"), label="edge-case/claude-opus-5-5") + self.assertIn("edge-case/claude-opus-5-5", out) + + def test_output_is_bounded_overall(self): + text = "\n".join( + json.dumps({"type": "error", "message": "m" * 400, "code": f"c{i}"}) + for i in range(500) + ) + out = run_script(self.write(text)) + self.assertLessEqual(len(out.encode("utf-8")), agent_error.MAX_BYTES + 512) + + +class WorkflowWiringTest(unittest.TestCase): + """Both direct lanes must actually call it, on their own output file. + + Parsed WITHOUT PyYAML, like its sibling suites: this repo is stdlib-only + and CI installs no requirements for these tests. + """ + + @classmethod + def setUpClass(cls): + with io.open(WORKFLOW, encoding="utf-8") as f: + cls.text = f.read() + cls.blocks = cls._steps(cls.text) + + @staticmethod + def _steps(text): + """The `Report cell outcome` step bodies, keyed in file order.""" + out, lines, current = [], text.splitlines(), None + for line in lines: + if re.match(r"^ - name: ", line): + if current is not None: + out.append("\n".join(current)) + current = [line] if "Report cell outcome" in line else None + elif current is not None: + current.append(line) + if current is not None: + out.append("\n".join(current)) + return out + + def test_both_direct_lanes_report_the_failure(self): + calling = [b for b in self.blocks if "agent-error.py" in b] + self.assertEqual(2, len(calling), "expected exactly the two direct lanes") + + def test_each_lane_reads_its_own_output_file(self): + files = sorted( + re.search(r"--events (\S+)", b).group(1) + for b in self.blocks + if "agent-error.py" in b + ) + self.assertEqual(["/tmp/claude-result.json", "/tmp/codex-events.jsonl"], files) + + def test_the_file_each_lane_reads_is_the_file_its_agent_writes(self): + # A rename on one side only would leave the diagnosis reading a file + # nobody writes, and `could not read` is a plausible-looking output. + for path in ("/tmp/codex-events.jsonl", "/tmp/claude-result.json"): + with self.subTest(path=path): + self.assertIn(f"> {path}", self.text) + + def test_the_diagnosis_runs_only_when_the_cell_failed(self): + # In the `ok` branch it would print "why the cell failed" about a cell + # that succeeded. + for block in self.blocks: + if "agent-error.py" not in block: + continue + ok_at = block.index("submitted its findings.") + self.assertLess(ok_at, block.index("agent-error.py")) + self.assertIn("else", block[ok_at:block.index("agent-error.py")]) + + def test_the_script_is_loaded_from_the_pinned_assets(self): + # Never from the caller's checkout: a PR must not be able to rewrite + # the code that reports on it. + for block in self.blocks: + if "agent-error.py" in block: + self.assertIn('"$CURSOR_REVIEW_ASSETS/agent-error.py"', block) + + def test_the_cell_outcome_step_still_withholds_the_findings(self): + for block in self.blocks: + self.assertNotIn("cat /tmp/findings-out/findings.json", block) + + +if __name__ == "__main__": + unittest.main(verbosity=2) diff --git a/.github/workflows/cursor-review.yml b/.github/workflows/cursor-review.yml index dd48c49a..714a1495 100644 --- a/.github/workflows/cursor-review.yml +++ b/.github/workflows/cursor-review.yml @@ -3454,12 +3454,29 @@ jobs: echo "Direct-API cell: status=${status}, ${count} finding(s)." if [ "$status" = "ok" ]; then echo "Direct-API cell ${REVIEW_TYPE}/${MODEL} submitted its findings." - elif [ "$COUNTS" = "true" ]; then - # Failed AFTER the upload below, by `Fail the leg when the cell did - # not submit`, so the errored record still reaches the judge. - echo "Direct-API cell ${REVIEW_TYPE}/${MODEL} did not submit (status=$status); it replaces the Cursor OpenAI lane, so it counts as a failed reviewer." else - echo "::warning::Direct-API cell ${REVIEW_TYPE}/${MODEL} did not submit (status=$status). Its record still reaches the judge; this leg stays green because the cell is advisory." + # WHY it failed, not just THAT it did. The agent reports an + # API-level failure in its STRUCTURED OUTPUT, which is redirected + # to a file and — unlike the Cursor cells, which echo 500 chars of + # raw output and all of stderr — never echoed, because that file + # also holds the agent's prose. So a real outage + # (`insufficient_quota` / `credit_balance_exhausted`, naming its + # own cause) was discarded and this log showed only a non-zero + # exit, leaving the lane's failures to be inferred from job + # DURATION. agent-error.py prints an ALLOWLIST of provider error + # metadata and never the prose, and exits 0 on anything — a + # truncated or non-JSON file must not turn the diagnosis into a + # second failure. + python3 "$CURSOR_REVIEW_ASSETS/agent-error.py" \ + --events /tmp/codex-events.jsonl \ + --label "${REVIEW_TYPE}/${MODEL}" + if [ "$COUNTS" = "true" ]; then + # Failed AFTER the upload below, by `Fail the leg when the cell did + # not submit`, so the errored record still reaches the judge. + echo "Direct-API cell ${REVIEW_TYPE}/${MODEL} did not submit (status=$status); it replaces the Cursor OpenAI lane, so it counts as a failed reviewer." + else + echo "::warning::Direct-API cell ${REVIEW_TYPE}/${MODEL} did not submit (status=$status). Its record still reaches the judge; this leg stays green because the cell is advisory." + fi fi - name: Upload findings artifact @@ -3909,12 +3926,29 @@ jobs: echo "Direct-API cell: status=${status}, ${count} finding(s)." if [ "$status" = "ok" ]; then echo "Direct-API cell ${REVIEW_TYPE}/${MODEL} submitted its findings." - elif [ "$COUNTS" = "true" ]; then - # Failed AFTER the upload below, by `Fail the leg when the cell did - # not submit`, so the errored record still reaches the judge. - echo "Direct-API cell ${REVIEW_TYPE}/${MODEL} did not submit (status=$status); it replaces the Cursor Anthropic lane, so it counts as a failed reviewer." else - echo "::warning::Direct-API cell ${REVIEW_TYPE}/${MODEL} did not submit (status=$status). Its record still reaches the judge; this leg stays green because the cell is advisory." + # WHY it failed, not just THAT it did. The agent reports an + # API-level failure in its STRUCTURED OUTPUT, which is redirected + # to a file and — unlike the Cursor cells, which echo 500 chars of + # raw output and all of stderr — never echoed, because that file + # also holds the agent's prose. So a real outage + # (`insufficient_quota` / `credit_balance_exhausted`, naming its + # own cause) was discarded and this log showed only a non-zero + # exit, leaving the lane's failures to be inferred from job + # DURATION. agent-error.py prints an ALLOWLIST of provider error + # metadata and never the prose, and exits 0 on anything — a + # truncated or non-JSON file must not turn the diagnosis into a + # second failure. + python3 "$CURSOR_REVIEW_ASSETS/agent-error.py" \ + --events /tmp/claude-result.json \ + --label "${REVIEW_TYPE}/${MODEL}" + if [ "$COUNTS" = "true" ]; then + # Failed AFTER the upload below, by `Fail the leg when the cell did + # not submit`, so the errored record still reaches the judge. + echo "Direct-API cell ${REVIEW_TYPE}/${MODEL} did not submit (status=$status); it replaces the Cursor Anthropic lane, so it counts as a failed reviewer." + else + echo "::warning::Direct-API cell ${REVIEW_TYPE}/${MODEL} did not submit (status=$status). Its record still reaches the judge; this leg stays green because the cell is advisory." + fi fi - name: Upload findings artifact diff --git a/docs/callers/cursor-review.md b/docs/callers/cursor-review.md index ecdfd89d..f7a28664 100644 --- a/docs/callers/cursor-review.md +++ b/docs/callers/cursor-review.md @@ -229,6 +229,16 @@ another workflow using the default token silently fails to start a review. That is exactly what [`cursor-review-auto-label.yml`](cursor-review-auto-label.md) exists to handle. +**A direct-API cell that fails says why in its own job log.** The direct +lanes never echo their agent's output — it carries model prose steered by the +PR — so when a cell does not submit, its **Report cell outcome** step prints an +allowlisted summary of the provider's error instead (`type`, `code`, `status`, +`message`) plus a census of the events it got that far. A quota or rate-limit +failure names itself there; a cell that ran but submitted nothing shows up as a +census with no error object. Nothing else is printed and the raw output is not +uploaded, so that step is the whole diagnosis — if it is not enough, the fix is +to widen the allowlist in `agent-error.py`, not to publish the transcript. + **Applying the label does not guarantee a run.** If the event was swallowed, remove the label, confirm it is gone, then re-add it. From fef9377b041a0c80bc656463cbd6be7350ec6f1f Mon Sep 17 00:00:00 2001 From: Matt Miller Date: Fri, 9 Oct 2026 18:24:02 -0700 Subject: [PATCH 2/3] test(cursor-review): execute the cell-outcome step, not just lint it `WorkflowWiringTest` reads the step statically, which cannot tell whether the branch the diagnosis sits in is ever reached, nor whether the step as a whole keeps the agent's prose out of a public log. So the `run:` body is now extracted and EXECUTED against fixtures, with its hardcoded /tmp paths rewritten into a sandbox. The leak assertion is the reason this exists: it covers the WHOLE step, so a `cat` of the events file added beside the diagnosis fails here, where a test of the script alone would still pass. Mutation-checked rather than trusted: echoing the events file next to the diagnosis, pointing one lane at the other lane's file, and deleting the diagnosis from the Anthropic lane are each caught. Co-Authored-By: Claude Opus 5 --- .../cursor-review/tests/test_agent_error.py | 136 ++++++++++++++++++ 1 file changed, 136 insertions(+) diff --git a/.github/cursor-review/tests/test_agent_error.py b/.github/cursor-review/tests/test_agent_error.py index fcf119b6..39af5023 100644 --- a/.github/cursor-review/tests/test_agent_error.py +++ b/.github/cursor-review/tests/test_agent_error.py @@ -393,5 +393,141 @@ def test_the_cell_outcome_step_still_withholds_the_findings(self): self.assertNotIn("cat /tmp/findings-out/findings.json", block) +class StepBehaviourTest(unittest.TestCase): + """Runs the workflow's REAL `Report cell outcome` body, not just the script. + + `WorkflowWiringTest` reads the step statically, which cannot tell whether + the branch it sits in is reached or whether the step as a whole keeps the + agent's prose out. So the body is extracted and EXECUTED against fixtures, + with its hardcoded `/tmp` paths rewritten into a sandbox. The leak + assertion is the point: it covers the whole step, so a future `cat` of the + events file added beside the diagnosis would fail here. + """ + + @classmethod + def setUpClass(cls): + with io.open(WORKFLOW, encoding="utf-8") as f: + text = f.read() + cls.bodies = cls._bodies(text) + # Both lanes, or the extraction below has drifted from the workflow. + assert len(cls.bodies) == 2, f"extracted {len(cls.bodies)} wired bodies" + + @staticmethod + def _bodies(text): + """The `run:` script of every `Report cell outcome` step that calls us.""" + out, cur, lines = [], None, text.splitlines() + for line in lines: + if re.match(r"^ - name: ", line): + if cur: + out.append("\n".join(cur)) + cur = [] if "Report cell outcome" in line else None + elif cur is not None: + cur.append(line) + if cur: + out.append("\n".join(cur)) + bodies = [] + for step in out: + m = re.search(r"^ run: \|\n(.*)", step, re.S | re.M) + if not m: + continue + body = [] + for ln in m.group(1).splitlines(): + if ln.strip() and not ln.startswith(" "): + break + body.append(ln[10:]) + body = "\n".join(body) + if "agent-error.py" in body: + bodies.append(body) + return bodies + + EVENTS = ( + '{"type":"item.completed","item":{"type":"reasoning","text":"%s"}}\n' + '{"type":"error","error":{"type":"insufficient_quota",' + '"code":"credit_balance_exhausted"}}\n' + ) % PROSE + + def run_step(self, body, record, counts="true", events=EVENTS): + d = tempfile.mkdtemp() + self.addCleanup(lambda: __import__("shutil").rmtree(d, ignore_errors=True)) + findings = os.path.join(d, "findings.json") + with open(findings, "w", encoding="utf-8") as f: + f.write(record if isinstance(record, str) else json.dumps(record)) + evfile = os.path.join(d, "events") + if events is not None: + with open(evfile, "w", encoding="utf-8") as f: + f.write(events) + script = body + for placeholder in ("/tmp/findings-out/findings.json",): + script = script.replace(placeholder, findings) + for placeholder in ("/tmp/codex-events.jsonl", "/tmp/claude-result.json"): + script = script.replace(placeholder, evfile) + proc = subprocess.run( + ["bash", "-e", "-c", script], capture_output=True, text=True, + env={ + **os.environ, + "CURSOR_REVIEW_ASSETS": os.path.dirname(SCRIPT), + "MODEL": "gpt-5.6-sol", + "REVIEW_TYPE": "adversarial", + "COUNTS": counts, + }, + ) + self.assertEqual(0, proc.returncode, proc.stderr) + return proc.stdout + + def test_a_failed_cell_names_the_provider_error(self): + for i, body in enumerate(self.bodies): + with self.subTest(lane=i): + out = self.run_step(body, {"status": "error", "findings": []}) + self.assertIn("why the cell failed", out) + self.assertIn("credit_balance_exhausted", out) + + def test_the_step_never_prints_the_agents_prose(self): + # The whole step, not just the script: a `cat` of the events file added + # next to the diagnosis would fail here. + for i, body in enumerate(self.bodies): + for counts in ("true", "false"): + with self.subTest(lane=i, counts=counts): + out = self.run_step( + body, {"status": "error", "findings": []}, counts=counts + ) + self.assertNotIn(PROSE, out) + + def test_a_submitted_cell_gets_no_failure_diagnosis(self): + for i, body in enumerate(self.bodies): + with self.subTest(lane=i): + out = self.run_step(body, {"status": "ok", "findings": [{"a": 1}]}) + self.assertNotIn("why the cell failed", out) + self.assertIn("submitted its findings", out) + + def test_a_missing_output_file_does_not_fail_the_step(self): + # The step is `always()`; a cell killed before its agent wrote anything + # must still report, not error. + for i, body in enumerate(self.bodies): + with self.subTest(lane=i): + out = self.run_step( + body, {"status": "error", "findings": []}, events=None + ) + self.assertIn("could not read", out) + + def test_an_unparseable_record_still_reports_a_failure(self): + for i, body in enumerate(self.bodies): + with self.subTest(lane=i): + out = self.run_step(body, "not json at all") + self.assertIn("status=error", out) + + def test_the_gating_and_advisory_wording_both_survive(self): + for i, body in enumerate(self.bodies): + with self.subTest(lane=i): + gating = self.run_step( + body, {"status": "error", "findings": []}, counts="true" + ) + self.assertIn("counts as a failed reviewer", gating) + advisory = self.run_step( + body, {"status": "error", "findings": []}, counts="false" + ) + self.assertIn("::warning::", advisory) + self.assertIn("advisory", advisory) + + if __name__ == "__main__": unittest.main(verbosity=2) From b537a269647d503ae88aa50e257359e2c1a0a9bd Mon Sep 17 00:00:00 2001 From: Matt Miller Date: Fri, 9 Oct 2026 18:41:47 -0700 Subject: [PATCH 3/3] fix(cursor-review): close the review's six findings on the error reporter MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The panel raised 6 and CodeRabbit 2 (overlapping). All were real; three were serious. Each is now pinned by a test that fails without the fix. MODEL-AUTHORED SUBTREES WERE BEING READ (4/6 reviewers, and CodeRabbit). Restricting the FIELDS was never sufficient, because the agent writes part of this file: it chooses the `arguments` of its own MCP tool calls. A prompt-injected one could plant `{"error": {"message": "…"}}` there and the walk, which descended into every value, printed it — up to 40×300 chars of attacker-chosen text into the log, flatly contradicting the claim I had made that prose cannot reach it "however the provider nests it". It leaked through the pre-existing `{"type":"error"}` shape too, so this was not only the new envelope branch. DESCENT is now an allowlist of envelope keys a CLI owns (`ENVELOPE_KEYS`), so the model's writable subtrees are never reached rather than denied one at a time. `json.loads` RAISES RecursionError (6/6 reviewers). It is a RuntimeError, not a ValueError, so neither `except ValueError` caught it, and `summarize` ran outside `main`'s `try` — the script tracebacked and exited 1, which under the step's `bash -e` aborts `Report cell outcome` before its status line and turns an advisory leg RED. The exact opposite of the contract. I could not reproduce it at first: `json.loads("[" * 100000)` gives a JSONDecodeError, and only WELL-FORMED nesting (~200k deep, 400 KB, inside the 4 MB read) raises. The reviewers were right and my first probe was too shallow. Caught at both decode sites plus a broad guard in `main`. THE BOUNDS KEPT THE WRONG END OF THE FILE (2/6). The event that explains a codex failure is the LAST one written, and the read started at byte 0 with no truncation notice, so a long run that failed late lost exactly the event this script exists to print. Now reads the TAIL and walks events newest-first, and says when the head was cut. CLAUDE'S CAUSE IS IN `result` (2/6). Claude Code reports an API failure as `{"type":"result","subtype":"success","is_error":true,"result":""}` — so the lane logged `type=result subtype=success`, which reads as SUCCESS and names nothing. `is_error` is now allowlisted so the line cannot read as success. The cause itself stays withheld: `result` is the field the review prose uses and is attacker-steerable. That makes the caller doc's "a quota failure names itself" claim false for this lane, so it is now stated per-lane with the trade-off, rather than left to mislead. `##[command]` AND ODD COLON RUNS (2/6). The runner also honours the legacy `##[...]` form anywhere in a line, and `_SAFE` permits `#`, `[`, `]`. Separately `str.replace` does not rescan its own output, so `":::"` came back as `": ::"` — the no-`::` guarantee failed for odd-length runs. Both forms are now broken with single-pass lookahead regexes. NO `|| true` ON THE CALL (1/6). The script's exit-0 contract cannot cover a script that never STARTED: a skipped assets checkout makes `python3` exit 2, and `bash -e` would lose the status and `::warning::` lines. Two of the bounds tests were, as one reviewer noted, vacuous — they nested through arbitrary keys, which the descent allowlist no longer visits, and never reached the decoder. Reworked to nest through envelope keys. All 9 mutations of the new guards are caught, including the one that previously was not: `main`'s broad `except` is unreachable through any constructible input, so it is now exercised directly by stubbing `walk`. Co-Authored-By: Claude Opus 5 --- .github/cursor-review/README.md | 2 +- .github/cursor-review/agent-error.py | 234 ++++++++++++------ .../cursor-review/tests/test_agent_error.py | 208 +++++++++++++++- .github/workflows/cursor-review.yml | 12 +- docs/callers/cursor-review.md | 28 ++- 5 files changed, 395 insertions(+), 89 deletions(-) diff --git a/.github/cursor-review/README.md b/.github/cursor-review/README.md index 24096272..a03be6c4 100644 --- a/.github/cursor-review/README.md +++ b/.github/cursor-review/README.md @@ -256,7 +256,7 @@ the consolidated review's panel table. See [Panel integrity](#panel-integrity). | [`build-ledger.py`](build-ledger.py) | Builds the **prior-review ledger** — what earlier rounds raised on this PR and how the author answered — and splices it into the panel/judge prompts. Untrusted prose is defanged twice over: fence-opening lines are rewritten, and continuation lines of quoted prose are prefixed ` |` so no imported text can sit at a field's indent and forge one. Also the prompt splicer, so the no-ledger path is byte-identical to the pre-ledger prompt. | | [`fence-diff.py`](fence-diff.py) | Wraps the reviewed diff (plus the incremental hunks, and the judge's panel-findings block) in `=== BEGIN/END DIFF ===` fences. The diff is attacker-authored PR bytes, so static literal fences are not a control; the nonce is what a PR cannot forge. Each prompt-build step mints its OWN nonce (`mint`), into a shell variable rather than a step `env:` or job output — Actions dumps a step's env map into the public run log, and a per-prompt value means a leak in one job cannot forge a fence in another. Copies the body through **byte for byte** — it never defangs or normalizes the payload. | | [`incremental-diff.py`](incremental-diff.py) | Builds the incremental "hunks new since the last reviewed round" block as the difference between two PR patches (`build`), and verifies the result really is a subset of the reviewed diff (`check`) so the workflow can discard it if it is not. Never a commit range: `git diff LAST_REVIEWED...HEAD` carries every commit a merge of the base branch brought in — see [the ledger section](#the-prior-review-ledger-and-the-repeat-policy). | -| [`agent-error.py`](agent-error.py) | Run by each **direct-API** cell's **Report cell outcome** step, and only when that cell did not submit: prints WHY it failed. Both direct lanes send their agent's structured output to a file (`codex-events.jsonl`, `claude-result.json`) that is never echoed, because it also holds the agent's prose — so an API-level failure, which `codex exec --json` reports as an event on **stdout**, used to leave the log showing a non-zero exit and nothing else, and the lane's failures had to be inferred from job DURATION (a real `insufficient_quota` / `credit_balance_exhausted` outage was discarded that way). It walks the file and prints, from objects that look like errors, an **allowlist** of short provider metadata (`type`, `subtype`, `code`, `param`, `status`, `message`) plus a census of event types. The allowlist is the safety property — `result`, `content`, `text`, `arguments` and `reasoning` are not in it, so assistant prose cannot reach the public log however the provider nests it, and a `message` holding content blocks is skipped rather than flattened. An `error` KEY is read on its own terms: a provider's error object need not describe itself as an error (OpenAI's says `insufficient_quota`), so the nested shape is handled explicitly rather than by looking for the substring "error". Values are sanitized for a `::workflow command::` line by REPLACING disallowed characters, never deleting them. The raw file is deliberately **not** uploaded: the Anthropic lane screens `findings.json` for a leaked API key before publishing it, and shipping the unscreened transcript alongside would route the agent's prose past that guard. It exits **0 on anything** — missing, truncated, non-JSON or invalid-UTF-8 — since it runs after the cell already failed and must not become a second failure. | +| [`agent-error.py`](agent-error.py) | Run by each **direct-API** cell's **Report cell outcome** step, and only when that cell did not submit: prints WHY it failed. Both direct lanes send their agent's structured output to a file (`codex-events.jsonl`, `claude-result.json`) that is never echoed, because it also holds the agent's prose — so an API-level failure, which `codex exec --json` reports as an event on **stdout**, used to leave the log showing a non-zero exit and nothing else, and the lane's failures had to be inferred from job DURATION (a real `insufficient_quota` / `credit_balance_exhausted` outage was discarded that way). It prints an allowlist of short provider metadata (`type`, `subtype`, `is_error`, `code`, `param`, `status`, `message`) from objects that look like errors, plus a census of event types. **Two** allowlists, and the second is the load-bearing one: restricting the FIELDS is not enough, because the agent writes part of this file — it chooses the `arguments` of its own MCP tool calls — so a prompt-injected one could plant `{"error": {"message": "…"}}` there and a walk that descended into every value would print it. **Descent** is therefore an allowlist too (`ENVELOPE_KEYS`: `error`, `last_error`, `item`, `response`, `data`, `detail`), so the model's writable subtrees are never reached rather than denied one by one; widen it only to a key a CLI owns. An `error` KEY is read on its own terms, since a provider's error object need not describe itself as an error (OpenAI's says `insufficient_quota`). Values are sanitized by REPLACING disallowed characters, never deleting them, and both the `::` and legacy `##[command]` forms are broken with a single-pass regex (`str.replace` does not rescan its own output, so `a:::b` defeated it). Reads the **tail** of the file, not the head: the event that explains a failure is the last thing an agent writes. Exits **0 on anything** — missing, truncated, non-JSON, invalid UTF-8, or nested deeply enough that `json.loads` raises `RecursionError` (a RuntimeError, not a ValueError; 400 KB of `[` suffices) — because it runs after the cell already failed and an abort under the step's `bash -e` would cost the status line. **Known gap:** Claude Code puts an API failure's cause in `result`, the same field its review prose uses, so the Anthropic lane reports `is_error=true` and the census but not the cause — a deliberate trade, since that field is attacker-steerable. The raw file is not uploaded either: the Anthropic lane screens `findings.json` for a leaked API key before publishing it, and shipping the unscreened transcript alongside would route the agent's prose past that guard. | | [`catalog-drift.py`](catalog-drift.py) | Backs the weekly catalog-drift check. Extracts the pins from `cursor-review.yml`, diffs them against raw `cursor-agent models` output, and renders the sticky issue title + body (delisted pins, pins marked NO-ZDR, unpinned same-lab ids, catalog ids from unpinned families, stale audit date). Reports only — it never edits a pin. | ## Adopt it in your repo diff --git a/.github/cursor-review/agent-error.py b/.github/cursor-review/agent-error.py index 062a3684..4d1a679e 100644 --- a/.github/cursor-review/agent-error.py +++ b/.github/cursor-review/agent-error.py @@ -11,29 +11,42 @@ linking the billing page) was discarded, and the lane's failures had to be inferred from job DURATION. That is what this fixes. -WHAT IT PRINTS, AND WHY SO LITTLE. These files hold the agent's reasoning, its -tool calls and its findings — model output steered by PR-authored text, which -`Report cell outcome` already refuses to echo for that reason. So this does not -dump them. It walks the JSON and emits, from objects that look like ERRORS, an -ALLOWLIST of short metadata fields (``ERROR_FIELDS``) plus a census of event -types. The allowlist is the safety property: `result`, `content`, `text`, -`arguments`, `reasoning` and friends are not in it, so assistant prose cannot -reach the log through here no matter how the provider nests it. The raw file is -deliberately NOT uploaded as an artifact either: the Anthropic lane screens -`findings.json` for a leaked API key before publishing it, and shipping the -unscreened transcript alongside would route the agent's own prose straight past -that guard. When the allowlist plus the event census is not enough to explain a -failure, widen the allowlist — do not publish the file. - -Every emitted value is sanitized for a ``::workflow command::`` line the same -way the rest of this workflow does it — disallowed characters are REPLACED, not -dropped, because deleting would let a crafted "e r r o r" collapse into a value -that reads as something else. +WHAT IT PRINTS, AND WHY SO LITTLE. These files also hold the agent's reasoning, +its tool calls and its findings — model output steered by PR-authored text, +which `Report cell outcome` already refuses to echo for that reason. So this +does not dump them. It prints an ALLOWLIST of short metadata fields +(``ERROR_FIELDS``) from objects that look like errors, plus a census of event +types. + +TWO allowlists, and the second one is the load-bearing one. Restricting the +FIELDS is not enough on its own, because a reviewer agent writes part of this +file: it chooses the `arguments` of its own MCP tool calls. A prompt-injected +agent could therefore plant `{"error": {"message": ""}}` in a tool +call, and a walk that descended into every value would dutifully print it — +putting attacker-chosen text in the log and contradicting the whole point of +withholding the file. So DESCENT is an allowlist too (``ENVELOPE_KEYS``): from +an event object this visits only the keys a CLI writes around the model's +output, never into `arguments`, `result`, `content` or `text`. The model's +writable surfaces are not denied one by one — they are simply never reached. +That is deliberately narrower than "find any error anywhere": a provider shape +nesting its error somewhere unanticipated reports only the event census, which +is the safe direction to be wrong in. Widen ``ENVELOPE_KEYS`` to a key a CLI +owns, never to one the model fills. + +Every emitted value is sanitized for a run-log line: disallowed characters are +REPLACED, not dropped, because deleting would let a crafted "e r r o r" +collapse into a value that reads as something else. Both the modern `::` and +the legacy `##[command]` forms are broken, each with a single-pass regex that +provably leaves no `::` or `##[` behind (`"a:::b"` defeats a lone +`str.replace`, which does not rescan what it just wrote). DIAGNOSTIC ONLY: it exits 0 on anything, including a missing, truncated, -invalid-UTF-8 or non-JSON file. The step that runs it has already captured the -agent's real exit code; a traceback here would replace the diagnosis with a -crash, which is the failure this script exists to remove. +invalid-UTF-8 or non-JSON file, and including a payload that makes the JSON +decoder raise — `json.loads` raises `RecursionError` (a RuntimeError, NOT a +ValueError) on deeply nested input, and 400 KB of `[` is enough. The step that +runs it has already captured the agent's real exit code; a traceback here would +replace the diagnosis with a crash, and under the step's `bash -e` it would +abort `Report cell outcome` before the status line, turning an advisory leg red. Usage: agent-error.py --events /tmp/codex-events.jsonl --label adversarial/gpt-5.6-sol @@ -41,6 +54,7 @@ import argparse import json +import os import re import sys @@ -48,13 +62,25 @@ # `text`, `input`, `output`, `arguments`, `reasoning`, `delta` or `thinking`: # those carry model output. Adding a key here widens what a public log can show, # so it is a decision, not a convenience. -ERROR_FIELDS = ("type", "subtype", "code", "param", "status", "status_code", "http_status", "message") +ERROR_FIELDS = ( + "type", "subtype", "is_error", "code", "param", + "status", "status_code", "http_status", "message", +) + +# The ONLY keys this descends through — the envelope a CLI writes around the +# model's output. `item` carries codex's per-event payload (its `type` is what +# the census counts); `error`, `last_error`, `response`, `data` and `detail` are +# where the known providers put a failure. Every key a model fills is absent by +# construction, which is the point: see the module docstring. +ENVELOPE_KEYS = ("error", "last_error", "item", "response", "data", "detail") # Bounds on what one invocation can print. A failing agent can emit a very # large file, and this output goes to a log a human reads. MAX_VALUE = 300 MAX_LINES = 40 MAX_BYTES = 8000 +# Read at most this much, from the END of the file (see `read_tail`). +MAX_READ = 4_000_000 # Bound the walk: these files are agent-written, so depth and breadth are not # ours to trust. A findings payload nests ~4 deep; 12 is slack, not a limit # anyone legitimately reaches. @@ -62,18 +88,24 @@ MAX_NODES = 20000 _SAFE = re.compile(r"[^A-Za-z0-9 .,:;_/@+()\[\]{}='\"!?#$%^&*|~`<>-]") +# A colon followed by a colon, and a `#` that begins a `##[`. Lookahead, so one +# pass breaks every run however long: `":::"` -> `": : :"`, `"###["` -> `"## #["`. +_COLON_RUN = re.compile(r":(?=:)") +_LEGACY_CMD = re.compile(r"#(?=#\[)") def sanitize(value, limit: int = MAX_VALUE) -> str: """One line, safe to drop into a run log. Replaces, never deletes.""" text = str(value) # Newlines first: a single value must not become two log lines, and must - # not be able to open a `::workflow command::` of its own. + # not be able to open a workflow command of its own. text = text.replace("\r", " ").replace("\n", " ") text = _SAFE.sub("?", text) - # `::` is what starts a workflow command; break the pair rather than drop a - # character, so the text still reads. - text = text.replace("::", ": :") + # Break both command forms rather than drop a character, so the text still + # reads. `str.replace` alone is not enough — it does not rescan its own + # output, so `":::"` would come back as `": ::"`. + text = _COLON_RUN.sub(": ", text) + text = _LEGACY_CMD.sub("# ", text) if len(text) > limit: text = text[: limit - 1].rstrip() + "…" return text @@ -115,78 +147,121 @@ def _fields(obj: dict) -> str: def walk(node, out: list, types: dict, state: dict, depth: int = 0) -> None: - """Collect error objects and a census of `type` values. Bounded.""" + """Collect error objects and a census of `type` values. Bounded. + + Descends only through ``ENVELOPE_KEYS``, so the model's own writable + subtrees are never visited. + """ if state["nodes"] >= MAX_NODES or depth > MAX_DEPTH: state["truncated"] = True return state["nodes"] += 1 - if isinstance(node, dict): - kind = node.get("type") - if isinstance(kind, str): - key = sanitize(kind, 60) - types[key] = types.get(key, 0) + 1 - if _looks_like_error(node): - rendered = _fields(node) - if rendered and rendered not in out: - out.append(rendered) - # An `error` KEY is the canonical envelope and must be read on its own - # terms. The provider's error object does not have to describe ITSELF as - # an error — OpenAI's is `{"error": {"type": "insufficient_quota", - # "code": "credit_balance_exhausted", ...}}`, and neither field contains - # the string "error", so `_looks_like_error` alone walked straight past - # the one message worth printing and reported only the envelope's own - # `type=error`. That is the exact failure this script was written for, - # so it is handled explicitly rather than by heuristic. - err = node.get("error") - if isinstance(err, str) and err.strip(): - rendered = f"error={sanitize(err)}" - if rendered not in out: - out.append(rendered) - elif isinstance(err, dict): - rendered = _fields(err) - if rendered and rendered not in out: - out.append(rendered) - for value in node.values(): - walk(value, out, types, state, depth + 1) - elif isinstance(node, list): + if isinstance(node, list): for value in node: walk(value, out, types, state, depth + 1) + return + if not isinstance(node, dict): + return + + kind = node.get("type") + if isinstance(kind, str): + key = sanitize(kind, 60) + types[key] = types.get(key, 0) + 1 + if _looks_like_error(node): + rendered = _fields(node) + if rendered and rendered not in out: + out.append(rendered) + # An `error` KEY is the canonical envelope and must be read on its own + # terms. The provider's error object does not have to describe ITSELF as an + # error — OpenAI's is `{"error": {"type": "insufficient_quota", "code": + # "credit_balance_exhausted", ...}}`, and neither field contains the string + # "error", so `_looks_like_error` alone walked straight past the one + # message worth printing and reported only the envelope's own `type=error`. + err = node.get("error") + if isinstance(err, str) and err.strip(): + rendered = f"error={sanitize(err)}" + if rendered not in out: + out.append(rendered) + elif isinstance(err, dict): + rendered = _fields(err) + if rendered and rendered not in out: + out.append(rendered) + for key in ENVELOPE_KEYS: + if key in node: + walk(node[key], out, types, state, depth + 1) -def parse(text: str) -> list: + +def _loads(text): + """`json.loads` that cannot raise. Deep nesting raises RecursionError.""" + try: + return True, json.loads(text) + except (ValueError, RecursionError): + return False, None + + +def parse(text: str, drop_first_line: bool = False) -> list: """Whole-file JSON (claude) or newline-delimited events (codex), or []. Partial output is the normal case for a killed agent, so a file that does not parse as a whole is read line by line and the unparseable lines are - skipped rather than failing the read. + skipped rather than failing the read. `drop_first_line` discards a line the + tail read cut in half. """ stripped = text.strip() if not stripped: return [] - try: - return [json.loads(stripped)] - except ValueError: - pass + ok, value = _loads(stripped) + if ok: + return [value] + lines = stripped.splitlines() + if drop_first_line and lines: + lines = lines[1:] nodes = [] - for line in stripped.splitlines(): + for line in lines: line = line.strip() if not line or line[0] not in "{[": continue - try: - nodes.append(json.loads(line)) - except ValueError: - continue + ok, value = _loads(line) + if ok: + nodes.append(value) return nodes -def summarize(text: str) -> list: - nodes = parse(text) +def read_tail(path: str): + """The LAST ``MAX_READ`` bytes, as text, plus whether the head was cut. + + The tail, not the head: the event that explains a failure (a terminal + `error`, a `turn.failed`) is the LAST thing an agent writes, so reading + forward from byte 0 of a long run drops exactly the event worth having. + """ + with open(path, "rb") as f: + f.seek(0, os.SEEK_END) + size = f.tell() + start = max(0, size - MAX_READ) + f.seek(start) + raw = f.read(MAX_READ) + # Agent-written bytes, so invalid UTF-8 is reachable and must not raise. + return raw.decode("utf-8", "replace"), start > 0 + + +def summarize(text: str, head_cut: bool = False) -> list: + nodes = parse(text, drop_first_line=head_cut) out, types, state = [], {}, {"nodes": 0, "truncated": False} - for node in nodes: + # Newest first: if the node budget runs out, spend it on the events nearest + # the failure rather than on the session's opening handshake. + for node in reversed(nodes): walk(node, out, types, state) lines = [] + if head_cut: + lines.append( + f"(read the last {MAX_READ // 1_000_000} MB of a larger file)" + ) if types: - census = ", ".join(f"{k}×{v}" for k, v in sorted(types.items(), key=lambda kv: (-kv[1], kv[0]))[:10]) + census = ", ".join( + f"{k}×{v}" + for k, v in sorted(types.items(), key=lambda kv: (-kv[1], kv[0]))[:10] + ) lines.append(f"events: {len(nodes)} ({census})") elif nodes: lines.append(f"events: {len(nodes)}") @@ -196,7 +271,7 @@ def summarize(text: str) -> list: lines.append(f"… and {len(out) - MAX_LINES} more error object(s) (output bounded)") if state["truncated"]: lines.append("(walk bounded: the agent's output is larger than this summary reads)") - if not lines: + if not lines or (head_cut and len(lines) == 1): lines.append("no JSON events and no error object in the agent's output") elif not out: lines.append("no error object in the agent's output") @@ -213,15 +288,18 @@ def main(argv=None) -> int: head = f"=== why the cell failed ({label}) ===" if label else "=== why the cell failed ===" print(head) try: - # Bounded read, errors="replace": these bytes are agent-written, so - # invalid UTF-8 is reachable and must not raise. - with open(args.events, "rb") as f: - raw = f.read(4_000_000) - text = raw.decode("utf-8", "replace") + text, head_cut = read_tail(args.events) except OSError as e: print(f"could not read {sanitize(args.events, 120)}: {sanitize(type(e).__name__, 40)}") return 0 - body = "\n".join(summarize(text)) + try: + body = "\n".join(summarize(text, head_cut=head_cut)) + except Exception as e: # noqa: BLE001 + # Belt and braces over the bounds above. This runs after the cell has + # already failed; whatever goes wrong in here, the step must still + # reach its status line. + print(f"could not summarize the agent's output: {sanitize(type(e).__name__, 40)}") + return 0 if len(body.encode("utf-8")) > MAX_BYTES: body = body.encode("utf-8")[:MAX_BYTES].decode("utf-8", "ignore") + "\n… (bounded)" print(body) diff --git a/.github/cursor-review/tests/test_agent_error.py b/.github/cursor-review/tests/test_agent_error.py index 39af5023..d807a728 100644 --- a/.github/cursor-review/tests/test_agent_error.py +++ b/.github/cursor-review/tests/test_agent_error.py @@ -29,6 +29,7 @@ Run: python3 -m unittest discover -s .github/cursor-review/tests -p 'test_*.py' -v """ +import contextlib import importlib.util import io import json @@ -211,6 +212,29 @@ def test_disallowed_characters_are_replaced_not_dropped(self): # Deleting would let a crafted value collapse into different text. self.assertEqual("a?b", agent_error.sanitize("a\u0000b")) + def test_an_odd_run_of_colons_leaves_no_pair(self): + # `str.replace` does not rescan its own output: `":::"` came back as + # `": ::"`, so the no-`::` guarantee failed for odd-length runs. + for value in (":::", "::::", "a:::b", ":" * 9): + with self.subTest(value=value): + self.assertNotIn("::", agent_error.sanitize(value)) + + def test_the_legacy_command_form_is_broken_too(self): + # The runner also recognises `##[command]` anywhere in a line, and + # `_SAFE` allows `#`, `[` and `]`. + for value in ("##[error]forged", "##[add-mask]x", "###[error]x"): + with self.subTest(value=value): + out = agent_error.sanitize(value) + self.assertNotIn("##[", out) + self.assertIn("forged" if "forged" in value else "x", out) + + def test_no_emitted_line_can_carry_either_command_form(self): + payload = {"error": {"type": ":::", "code": "##[add-mask]", + "message": "x:::y ##[error]z"}} + for line in summary(payload): + self.assertNotIn("::", line) + self.assertNotIn("##[", line) + def test_a_long_value_is_truncated(self): out = agent_error.sanitize("x" * 5000) self.assertLessEqual(len(out), agent_error.MAX_VALUE) @@ -227,14 +251,18 @@ def test_the_label_is_sanitized(self): class BoundsTest(unittest.TestCase): def test_deep_nesting_does_not_recurse_without_limit(self): + # Nested through an ENVELOPE key, since that is the only path the walk + # descends at all: an earlier version of this test wrapped the payload + # in an arbitrary `wrap` key and so stopped proving anything once + # descent became an allowlist. node = {"type": "error", "code": "deep"} for _ in range(5000): - node = {"wrap": node} + node = {"error": node} out = joined(node) # must not raise RecursionError self.assertIn("walk bounded", out) def test_wide_breadth_is_bounded(self): - payload = {"items": [{"type": f"t{i}"} for i in range(50000)]} + payload = {"error": [{"type": "error", "code": f"c{i}"} for i in range(50000)]} out = joined(payload) self.assertIn("walk bounded", out) @@ -261,6 +289,173 @@ def test_the_census_is_capped(self): self.assertLessEqual(census[0].count("×"), 10) +class ModelAuthoredSubtreeTest(unittest.TestCase): + """The agent writes part of this file. Those parts must never be read. + + A reviewer agent chooses the `arguments` of its own MCP tool calls, so a + prompt-injected one can plant an error-shaped object there and — if the + walk descended into every value — get attacker-chosen text into the run + log. That is not hypothetical nesting: it is the one subtree in these files + whose contents the model dictates. + """ + + SHAPES = ( + # codex: an MCP tool call whose arguments the model wrote. + {"type": "item.completed", + "item": {"type": "mcp_tool_call", "name": "cursor_review_record_finding", + "arguments": {"severity": "high", "error": {"message": PROSE}}}}, + # The shape that needed no `error` key at all. + {"item": {"arguments": {"type": "error", "message": PROSE}}}, + # A tool result's content. + {"type": "item.completed", + "item": {"type": "tool_result", "content": [{"is_error": True, "message": PROSE}]}}, + # Prose hidden under an allowlisted field name one level down. + {"type": "result", "is_error": True, "result": {"error": {"message": PROSE}}}, + # A model-chosen `type`, which would otherwise reach the census. + {"item": {"arguments": {"type": PROSE}}}, + ) + + def test_no_model_authored_subtree_reaches_the_log(self): + for i, shape in enumerate(self.SHAPES): + with self.subTest(shape=i): + self.assertNotIn(PROSE, joined(shape)) + + def test_the_descent_allowlist_holds_no_model_written_key(self): + # The guard that matters when someone widens ENVELOPE_KEYS later. + for key in ("arguments", "input", "tool_input", "result", "content", + "text", "output", "reasoning", "thinking", "delta", + "parameters", "params", "messages", "findings"): + self.assertNotIn(key, agent_error.ENVELOPE_KEYS) + + def test_a_provider_error_beside_the_arguments_is_still_read(self): + # Blocking the model's subtree must not cost the CLI's own error. + payload = {"type": "item.completed", + "item": {"type": "mcp_tool_call", + "arguments": {"error": {"message": PROSE}}, + "error": {"code": "tool_rejected"}}} + out = joined(payload) + self.assertIn("tool_rejected", out) + self.assertNotIn(PROSE, out) + + +class ClaudeResultShapeTest(unittest.TestCase): + """Claude Code reports an API failure as a result object. + + The cause sits in `result` — the same field its review prose uses — so it + is NOT printed. `is_error` is, so the line cannot read as a success. This + pins the documented gap: see the per-lane note in + `docs/callers/cursor-review.md`. + """ + + FAILURE = { + "type": "result", + "subtype": "success", + "is_error": True, + "result": "API Error: 400 credit balance is too low", + } + + def test_the_failure_is_not_reported_as_a_success(self): + out = joined(self.FAILURE) + self.assertIn("is_error=True", out) + + def test_the_cause_in_result_is_withheld(self): + self.assertNotIn("credit balance", joined(self.FAILURE)) + + def test_is_error_is_allowlisted(self): + self.assertIn("is_error", agent_error.ERROR_FIELDS) + + +class TailReadTest(unittest.TestCase): + """The explaining event is the LAST thing an agent writes.""" + + def setUp(self): + self.dir = tempfile.TemporaryDirectory() + self.addCleanup(self.dir.cleanup) + + def test_a_late_error_in_an_oversized_file_is_still_found(self): + # Reading forward from byte 0 would spend the whole budget on filler + # and lose the terminal event — the one the script exists to print. + path = os.path.join(self.dir.name, "events.jsonl") + filler = json.dumps({"type": "item.completed", "item": {"type": "reasoning"}}) + with open(path, "w", encoding="utf-8") as f: + while f.tell() < agent_error.MAX_READ + 500_000: + f.write(filler + "\n") + f.write(json.dumps( + {"type": "error", "error": {"code": "terminal_failure"}}) + "\n") + out = run_script(path) + self.assertIn("terminal_failure", out) + + def test_the_truncated_read_is_declared(self): + path = os.path.join(self.dir.name, "big.jsonl") + with open(path, "w", encoding="utf-8") as f: + f.write("x" * (agent_error.MAX_READ + 10)) + self.assertIn("larger file", run_script(path)) + + def test_the_newest_error_is_reported_first(self): + text = ( + json.dumps({"type": "error", "error": {"code": "first_failure"}}) + "\n" + + json.dumps({"type": "error", "error": {"code": "last_failure"}}) + "\n" + ) + out = "\n".join(summary(text)) + self.assertLess(out.index("last_failure"), out.index("first_failure")) + + +class DecoderHostileInputTest(unittest.TestCase): + """`json.loads` raises RecursionError — a RuntimeError, not a ValueError. + + 400 KB of `[` is enough, well inside the 4 MB read. An uncaught one would + traceback, exit 1, and under the step's `bash -e` abort `Report cell + outcome` before its status line — turning an advisory leg red. + """ + + def setUp(self): + self.dir = tempfile.TemporaryDirectory() + self.addCleanup(self.dir.cleanup) + + def write(self, text): + path = os.path.join(self.dir.name, "events.jsonl") + with open(path, "w", encoding="utf-8") as f: + f.write(text) + return path + + def test_deeply_nested_json_exits_zero(self): + depth = 200_000 + run_script(self.write("[" * depth + "]" * depth)) # asserts exit 0 + + def test_deeply_nested_json_on_one_ndjson_line_exits_zero(self): + depth = 200_000 + run_script(self.write( + '{"type":"session.created"}\n' + "[" * depth + "]" * depth + "\n")) + + def test_an_unexpected_failure_in_the_walk_still_exits_zero(self): + # `main`'s broad `except` is unreachable through any input the rest of + # this suite can construct — `_loads` already absorbs the decoder — so + # the guard is exercised directly instead of left as untested + # defensive code. Whatever future change makes `summarize` raise, the + # step must still reach its status line. + path = self.write('{"type":"error","code":"x"}\n') + original = agent_error.walk + + def explode(*_args, **_kwargs): + raise RuntimeError("boom") + + agent_error.walk = explode + try: + buf = io.StringIO() + with contextlib.redirect_stdout(buf): + code = agent_error.main(["--events", path]) + finally: + agent_error.walk = original + self.assertEqual(0, code) + self.assertIn("could not summarize", buf.getvalue()) + self.assertNotIn("boom", buf.getvalue()) # the message is not echoed + + def test_the_loads_helper_never_raises(self): + for payload in ("[" * 200_000, "{" * 200_000, "nonsense", ""): + ok, _ = agent_error._loads(payload) + self.assertFalse(ok) + + def run_script(path, label="adversarial/gpt-5.6-sol"): """The script as the workflow runs it: a subprocess, whose code must be 0.""" proc = subprocess.run( @@ -388,6 +583,15 @@ def test_the_script_is_loaded_from_the_pinned_assets(self): if "agent-error.py" in block: self.assertIn('"$CURSOR_REVIEW_ASSETS/agent-error.py"', block) + def test_the_diagnosis_cannot_abort_the_step(self): + # The step runs under `bash -e`, and the script's own exit-0 contract + # cannot cover a script that never STARTED — a skipped assets checkout + # makes `python3` exit 2, which would lose the status and `::warning::` + # lines this step exists to print. + for block in self.blocks: + if "agent-error.py" in block: + self.assertIn('--label "${REVIEW_TYPE}/${MODEL}" || true', block) + def test_the_cell_outcome_step_still_withholds_the_findings(self): for block in self.blocks: self.assertNotIn("cat /tmp/findings-out/findings.json", block) diff --git a/.github/workflows/cursor-review.yml b/.github/workflows/cursor-review.yml index 714a1495..208b2bc5 100644 --- a/.github/workflows/cursor-review.yml +++ b/.github/workflows/cursor-review.yml @@ -3467,9 +3467,13 @@ jobs: # metadata and never the prose, and exits 0 on anything — a # truncated or non-JSON file must not turn the diagnosis into a # second failure. + # `|| true` because this step runs under `bash -e`: the script's + # own exit-0 contract cannot cover a script that never STARTED + # (a skipped assets checkout, so `python3` exits 2), and aborting + # here would lose the status and `::warning::` lines below. python3 "$CURSOR_REVIEW_ASSETS/agent-error.py" \ --events /tmp/codex-events.jsonl \ - --label "${REVIEW_TYPE}/${MODEL}" + --label "${REVIEW_TYPE}/${MODEL}" || true if [ "$COUNTS" = "true" ]; then # Failed AFTER the upload below, by `Fail the leg when the cell did # not submit`, so the errored record still reaches the judge. @@ -3939,9 +3943,13 @@ jobs: # metadata and never the prose, and exits 0 on anything — a # truncated or non-JSON file must not turn the diagnosis into a # second failure. + # `|| true` because this step runs under `bash -e`: the script's + # own exit-0 contract cannot cover a script that never STARTED + # (a skipped assets checkout, so `python3` exits 2), and aborting + # here would lose the status and `::warning::` lines below. python3 "$CURSOR_REVIEW_ASSETS/agent-error.py" \ --events /tmp/claude-result.json \ - --label "${REVIEW_TYPE}/${MODEL}" + --label "${REVIEW_TYPE}/${MODEL}" || true if [ "$COUNTS" = "true" ]; then # Failed AFTER the upload below, by `Fail the leg when the cell did # not submit`, so the errored record still reaches the judge. diff --git a/docs/callers/cursor-review.md b/docs/callers/cursor-review.md index f7a28664..e449d860 100644 --- a/docs/callers/cursor-review.md +++ b/docs/callers/cursor-review.md @@ -232,12 +232,28 @@ exists to handle. **A direct-API cell that fails says why in its own job log.** The direct lanes never echo their agent's output — it carries model prose steered by the PR — so when a cell does not submit, its **Report cell outcome** step prints an -allowlisted summary of the provider's error instead (`type`, `code`, `status`, -`message`) plus a census of the events it got that far. A quota or rate-limit -failure names itself there; a cell that ran but submitted nothing shows up as a -census with no error object. Nothing else is printed and the raw output is not -uploaded, so that step is the whole diagnosis — if it is not enough, the fix is -to widen the allowlist in `agent-error.py`, not to publish the transcript. +allowlisted summary of the provider's error instead (`type`, `subtype`, +`is_error`, `code`, `param`, `status`, `message`) plus a census of the events it +got that far. + +How much that names depends on the lane, because the two CLIs report failures +differently. **OpenAI/codex** puts the cause in a structured error event, so a +quota or rate-limit failure names itself — `type=insufficient_quota +code=credit_balance_exhausted` and the provider's own message. +**Anthropic/claude** reports an API-level failure as a result object whose +cause sits in `result`, which is the same field its review prose uses: that +field is not allowlisted and is not printed, so the Anthropic lane tells you +`is_error=true` and the event census but **not** the cause. Reading it means +re-running the cell, or checking the provider's status/billing directly. + +That is a deliberate trade, not an oversight: `result` is attacker-steerable +(the agent reads PR-authored text), so printing it would hand a prompt-injected +agent a channel into the run log, which is the exact thing withholding these +files prevents. A cell that ran but submitted nothing shows up as a census with +no error object. Nothing else is printed and the raw output is not uploaded, so +that step is the whole diagnosis — if it is not enough, the fix is to widen the +allowlist in `agent-error.py` to a field the CLI owns, not to publish the +transcript. **Applying the label does not guarantee a run.** If the event was swallowed, remove the label, confirm it is gone, then re-add it.