diff --git a/.github/cursor-review/README.md b/.github/cursor-review/README.md index 5031f8f3..a03be6c4 100644 --- a/.github/cursor-review/README.md +++ b/.github/cursor-review/README.md @@ -256,6 +256,7 @@ the consolidated review's panel table. See [Panel integrity](#panel-integrity). | [`build-ledger.py`](build-ledger.py) | Builds the **prior-review ledger** — what earlier rounds raised on this PR and how the author answered — and splices it into the panel/judge prompts. Untrusted prose is defanged twice over: fence-opening lines are rewritten, and continuation lines of quoted prose are prefixed ` |` so no imported text can sit at a field's indent and forge one. Also the prompt splicer, so the no-ledger path is byte-identical to the pre-ledger prompt. | | [`fence-diff.py`](fence-diff.py) | Wraps the reviewed diff (plus the incremental hunks, and the judge's panel-findings block) in `=== BEGIN/END DIFF ===` fences. The diff is attacker-authored PR bytes, so static literal fences are not a control; the nonce is what a PR cannot forge. Each prompt-build step mints its OWN nonce (`mint`), into a shell variable rather than a step `env:` or job output — Actions dumps a step's env map into the public run log, and a per-prompt value means a leak in one job cannot forge a fence in another. Copies the body through **byte for byte** — it never defangs or normalizes the payload. | | [`incremental-diff.py`](incremental-diff.py) | Builds the incremental "hunks new since the last reviewed round" block as the difference between two PR patches (`build`), and verifies the result really is a subset of the reviewed diff (`check`) so the workflow can discard it if it is not. Never a commit range: `git diff LAST_REVIEWED...HEAD` carries every commit a merge of the base branch brought in — see [the ledger section](#the-prior-review-ledger-and-the-repeat-policy). | +| [`agent-error.py`](agent-error.py) | Run by each **direct-API** cell's **Report cell outcome** step, and only when that cell did not submit: prints WHY it failed. Both direct lanes send their agent's structured output to a file (`codex-events.jsonl`, `claude-result.json`) that is never echoed, because it also holds the agent's prose — so an API-level failure, which `codex exec --json` reports as an event on **stdout**, used to leave the log showing a non-zero exit and nothing else, and the lane's failures had to be inferred from job DURATION (a real `insufficient_quota` / `credit_balance_exhausted` outage was discarded that way). It prints an allowlist of short provider metadata (`type`, `subtype`, `is_error`, `code`, `param`, `status`, `message`) from objects that look like errors, plus a census of event types. **Two** allowlists, and the second is the load-bearing one: restricting the FIELDS is not enough, because the agent writes part of this file — it chooses the `arguments` of its own MCP tool calls — so a prompt-injected one could plant `{"error": {"message": "…"}}` there and a walk that descended into every value would print it. **Descent** is therefore an allowlist too (`ENVELOPE_KEYS`: `error`, `last_error`, `item`, `response`, `data`, `detail`), so the model's writable subtrees are never reached rather than denied one by one; widen it only to a key a CLI owns. An `error` KEY is read on its own terms, since a provider's error object need not describe itself as an error (OpenAI's says `insufficient_quota`). Values are sanitized by REPLACING disallowed characters, never deleting them, and both the `::` and legacy `##[command]` forms are broken with a single-pass regex (`str.replace` does not rescan its own output, so `a:::b` defeated it). Reads the **tail** of the file, not the head: the event that explains a failure is the last thing an agent writes. Exits **0 on anything** — missing, truncated, non-JSON, invalid UTF-8, or nested deeply enough that `json.loads` raises `RecursionError` (a RuntimeError, not a ValueError; 400 KB of `[` suffices) — because it runs after the cell already failed and an abort under the step's `bash -e` would cost the status line. **Known gap:** Claude Code puts an API failure's cause in `result`, the same field its review prose uses, so the Anthropic lane reports `is_error=true` and the census but not the cause — a deliberate trade, since that field is attacker-steerable. The raw file is not uploaded either: the Anthropic lane screens `findings.json` for a leaked API key before publishing it, and shipping the unscreened transcript alongside would route the agent's prose past that guard. | | [`catalog-drift.py`](catalog-drift.py) | Backs the weekly catalog-drift check. Extracts the pins from `cursor-review.yml`, diffs them against raw `cursor-agent models` output, and renders the sticky issue title + body (delisted pins, pins marked NO-ZDR, unpinned same-lab ids, catalog ids from unpinned families, stale audit date). Reports only — it never edits a pin. | ## Adopt it in your repo diff --git a/.github/cursor-review/agent-error.py b/.github/cursor-review/agent-error.py new file mode 100644 index 00000000..4d1a679e --- /dev/null +++ b/.github/cursor-review/agent-error.py @@ -0,0 +1,310 @@ +#!/usr/bin/env python3 +"""Why a direct-API reviewer cell failed, in a form a PUBLIC run log can carry. + +Both direct lanes write their agent's output to a file and, until this script, +threw it away: the OpenAI lane redirects `codex exec --json` to +`codex-events.jsonl` and the Anthropic lane redirects `claude --output-format +json` to `claude-result.json`. Neither file was echoed or uploaded, and codex +reports API failures on STDOUT as events — so the log showed `Reading prompt +from stdin...`, `exit code 1`, and nothing else. A real outage +(`insufficient_quota` / `credit_balance_exhausted`, naming its own cause and +linking the billing page) was discarded, and the lane's failures had to be +inferred from job DURATION. That is what this fixes. + +WHAT IT PRINTS, AND WHY SO LITTLE. These files also hold the agent's reasoning, +its tool calls and its findings — model output steered by PR-authored text, +which `Report cell outcome` already refuses to echo for that reason. So this +does not dump them. It prints an ALLOWLIST of short metadata fields +(``ERROR_FIELDS``) from objects that look like errors, plus a census of event +types. + +TWO allowlists, and the second one is the load-bearing one. Restricting the +FIELDS is not enough on its own, because a reviewer agent writes part of this +file: it chooses the `arguments` of its own MCP tool calls. A prompt-injected +agent could therefore plant `{"error": {"message": ""}}` in a tool +call, and a walk that descended into every value would dutifully print it — +putting attacker-chosen text in the log and contradicting the whole point of +withholding the file. So DESCENT is an allowlist too (``ENVELOPE_KEYS``): from +an event object this visits only the keys a CLI writes around the model's +output, never into `arguments`, `result`, `content` or `text`. The model's +writable surfaces are not denied one by one — they are simply never reached. +That is deliberately narrower than "find any error anywhere": a provider shape +nesting its error somewhere unanticipated reports only the event census, which +is the safe direction to be wrong in. Widen ``ENVELOPE_KEYS`` to a key a CLI +owns, never to one the model fills. + +Every emitted value is sanitized for a run-log line: disallowed characters are +REPLACED, not dropped, because deleting would let a crafted "e r r o r" +collapse into a value that reads as something else. Both the modern `::` and +the legacy `##[command]` forms are broken, each with a single-pass regex that +provably leaves no `::` or `##[` behind (`"a:::b"` defeats a lone +`str.replace`, which does not rescan what it just wrote). + +DIAGNOSTIC ONLY: it exits 0 on anything, including a missing, truncated, +invalid-UTF-8 or non-JSON file, and including a payload that makes the JSON +decoder raise — `json.loads` raises `RecursionError` (a RuntimeError, NOT a +ValueError) on deeply nested input, and 400 KB of `[` is enough. The step that +runs it has already captured the agent's real exit code; a traceback here would +replace the diagnosis with a crash, and under the step's `bash -e` it would +abort `Report cell outcome` before the status line, turning an advisory leg red. + +Usage: + agent-error.py --events /tmp/codex-events.jsonl --label adversarial/gpt-5.6-sol +""" + +import argparse +import json +import os +import re +import sys + +# Short, provider-authored metadata. Deliberately NOT `result`, `content`, +# `text`, `input`, `output`, `arguments`, `reasoning`, `delta` or `thinking`: +# those carry model output. Adding a key here widens what a public log can show, +# so it is a decision, not a convenience. +ERROR_FIELDS = ( + "type", "subtype", "is_error", "code", "param", + "status", "status_code", "http_status", "message", +) + +# The ONLY keys this descends through — the envelope a CLI writes around the +# model's output. `item` carries codex's per-event payload (its `type` is what +# the census counts); `error`, `last_error`, `response`, `data` and `detail` are +# where the known providers put a failure. Every key a model fills is absent by +# construction, which is the point: see the module docstring. +ENVELOPE_KEYS = ("error", "last_error", "item", "response", "data", "detail") + +# Bounds on what one invocation can print. A failing agent can emit a very +# large file, and this output goes to a log a human reads. +MAX_VALUE = 300 +MAX_LINES = 40 +MAX_BYTES = 8000 +# Read at most this much, from the END of the file (see `read_tail`). +MAX_READ = 4_000_000 +# Bound the walk: these files are agent-written, so depth and breadth are not +# ours to trust. A findings payload nests ~4 deep; 12 is slack, not a limit +# anyone legitimately reaches. +MAX_DEPTH = 12 +MAX_NODES = 20000 + +_SAFE = re.compile(r"[^A-Za-z0-9 .,:;_/@+()\[\]{}='\"!?#$%^&*|~`<>-]") +# A colon followed by a colon, and a `#` that begins a `##[`. Lookahead, so one +# pass breaks every run however long: `":::"` -> `": : :"`, `"###["` -> `"## #["`. +_COLON_RUN = re.compile(r":(?=:)") +_LEGACY_CMD = re.compile(r"#(?=#\[)") + + +def sanitize(value, limit: int = MAX_VALUE) -> str: + """One line, safe to drop into a run log. Replaces, never deletes.""" + text = str(value) + # Newlines first: a single value must not become two log lines, and must + # not be able to open a workflow command of its own. + text = text.replace("\r", " ").replace("\n", " ") + text = _SAFE.sub("?", text) + # Break both command forms rather than drop a character, so the text still + # reads. `str.replace` alone is not enough — it does not rescan its own + # output, so `":::"` would come back as `": ::"`. + text = _COLON_RUN.sub(": ", text) + text = _LEGACY_CMD.sub("# ", text) + if len(text) > limit: + text = text[: limit - 1].rstrip() + "…" + return text + + +def _looks_like_error(obj: dict) -> bool: + """True when this object is reporting a failure rather than describing work.""" + if obj.get("is_error") is True: + return True + for key in ("type", "subtype", "code"): + value = obj.get(key) + if isinstance(value, str) and "error" in value.lower(): + return True + for key in ("status", "status_code", "http_status"): + value = obj.get(key) + if isinstance(value, bool): + continue + if isinstance(value, int) and value >= 400: + return True + return False + + +def _fields(obj: dict) -> str: + """The allowlisted fields of one error object, as `k=v` pairs.""" + parts = [] + for key in ERROR_FIELDS: + if key not in obj: + continue + value = obj[key] + # Only scalars. A dict or list under an allowlisted name (a `message` + # holding content blocks, say) is NOT flattened — that is how model + # prose would get in. + if isinstance(value, (dict, list)): + continue + if value is None or value == "": + continue + parts.append(f"{key}={sanitize(value)}") + return " ".join(parts) + + +def walk(node, out: list, types: dict, state: dict, depth: int = 0) -> None: + """Collect error objects and a census of `type` values. Bounded. + + Descends only through ``ENVELOPE_KEYS``, so the model's own writable + subtrees are never visited. + """ + if state["nodes"] >= MAX_NODES or depth > MAX_DEPTH: + state["truncated"] = True + return + state["nodes"] += 1 + if isinstance(node, list): + for value in node: + walk(value, out, types, state, depth + 1) + return + if not isinstance(node, dict): + return + + kind = node.get("type") + if isinstance(kind, str): + key = sanitize(kind, 60) + types[key] = types.get(key, 0) + 1 + if _looks_like_error(node): + rendered = _fields(node) + if rendered and rendered not in out: + out.append(rendered) + # An `error` KEY is the canonical envelope and must be read on its own + # terms. The provider's error object does not have to describe ITSELF as an + # error — OpenAI's is `{"error": {"type": "insufficient_quota", "code": + # "credit_balance_exhausted", ...}}`, and neither field contains the string + # "error", so `_looks_like_error` alone walked straight past the one + # message worth printing and reported only the envelope's own `type=error`. + err = node.get("error") + if isinstance(err, str) and err.strip(): + rendered = f"error={sanitize(err)}" + if rendered not in out: + out.append(rendered) + elif isinstance(err, dict): + rendered = _fields(err) + if rendered and rendered not in out: + out.append(rendered) + + for key in ENVELOPE_KEYS: + if key in node: + walk(node[key], out, types, state, depth + 1) + + +def _loads(text): + """`json.loads` that cannot raise. Deep nesting raises RecursionError.""" + try: + return True, json.loads(text) + except (ValueError, RecursionError): + return False, None + + +def parse(text: str, drop_first_line: bool = False) -> list: + """Whole-file JSON (claude) or newline-delimited events (codex), or []. + + Partial output is the normal case for a killed agent, so a file that does + not parse as a whole is read line by line and the unparseable lines are + skipped rather than failing the read. `drop_first_line` discards a line the + tail read cut in half. + """ + stripped = text.strip() + if not stripped: + return [] + ok, value = _loads(stripped) + if ok: + return [value] + lines = stripped.splitlines() + if drop_first_line and lines: + lines = lines[1:] + nodes = [] + for line in lines: + line = line.strip() + if not line or line[0] not in "{[": + continue + ok, value = _loads(line) + if ok: + nodes.append(value) + return nodes + + +def read_tail(path: str): + """The LAST ``MAX_READ`` bytes, as text, plus whether the head was cut. + + The tail, not the head: the event that explains a failure (a terminal + `error`, a `turn.failed`) is the LAST thing an agent writes, so reading + forward from byte 0 of a long run drops exactly the event worth having. + """ + with open(path, "rb") as f: + f.seek(0, os.SEEK_END) + size = f.tell() + start = max(0, size - MAX_READ) + f.seek(start) + raw = f.read(MAX_READ) + # Agent-written bytes, so invalid UTF-8 is reachable and must not raise. + return raw.decode("utf-8", "replace"), start > 0 + + +def summarize(text: str, head_cut: bool = False) -> list: + nodes = parse(text, drop_first_line=head_cut) + out, types, state = [], {}, {"nodes": 0, "truncated": False} + # Newest first: if the node budget runs out, spend it on the events nearest + # the failure rather than on the session's opening handshake. + for node in reversed(nodes): + walk(node, out, types, state) + lines = [] + if head_cut: + lines.append( + f"(read the last {MAX_READ // 1_000_000} MB of a larger file)" + ) + if types: + census = ", ".join( + f"{k}×{v}" + for k, v in sorted(types.items(), key=lambda kv: (-kv[1], kv[0]))[:10] + ) + lines.append(f"events: {len(nodes)} ({census})") + elif nodes: + lines.append(f"events: {len(nodes)}") + for item in out[:MAX_LINES]: + lines.append(f"error: {item}") + if len(out) > MAX_LINES: + lines.append(f"… and {len(out) - MAX_LINES} more error object(s) (output bounded)") + if state["truncated"]: + lines.append("(walk bounded: the agent's output is larger than this summary reads)") + if not lines or (head_cut and len(lines) == 1): + lines.append("no JSON events and no error object in the agent's output") + elif not out: + lines.append("no error object in the agent's output") + return lines + + +def main(argv=None) -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--events", required=True) + parser.add_argument("--label", default="") + args = parser.parse_args(argv) + + label = sanitize(args.label, 80) if args.label else "" + head = f"=== why the cell failed ({label}) ===" if label else "=== why the cell failed ===" + print(head) + try: + text, head_cut = read_tail(args.events) + except OSError as e: + print(f"could not read {sanitize(args.events, 120)}: {sanitize(type(e).__name__, 40)}") + return 0 + try: + body = "\n".join(summarize(text, head_cut=head_cut)) + except Exception as e: # noqa: BLE001 + # Belt and braces over the bounds above. This runs after the cell has + # already failed; whatever goes wrong in here, the step must still + # reach its status line. + print(f"could not summarize the agent's output: {sanitize(type(e).__name__, 40)}") + return 0 + if len(body.encode("utf-8")) > MAX_BYTES: + body = body.encode("utf-8")[:MAX_BYTES].decode("utf-8", "ignore") + "\n… (bounded)" + print(body) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/.github/cursor-review/tests/test_agent_error.py b/.github/cursor-review/tests/test_agent_error.py new file mode 100644 index 00000000..d807a728 --- /dev/null +++ b/.github/cursor-review/tests/test_agent_error.py @@ -0,0 +1,737 @@ +#!/usr/bin/env python3 +"""Tests for agent-error.py, which makes a direct-API cell's failure legible. + +Two things are being protected, and they pull against each other. + +LEGIBILITY. The motivating outage was an OpenAI `insufficient_quota` / +`credit_balance_exhausted`, reported by `codex exec --json` as an event on +STDOUT — i.e. into the events file, which is never echoed. The log showed a +non-zero exit and nothing else, so the lane's failures had to be inferred from +job DURATION. `NestedEnvelopeTest` pins that exact shape, because the first +draft of this script DID parse it and still printed only `type=error`: the +provider's error object describes itself as `insufficient_quota`, and the +heuristic looking for the substring "error" in `type`/`subtype`/`code` walked +straight past the one message worth printing. + +SILENCE ABOUT MODEL OUTPUT. These files also hold the agent's reasoning, tool +calls and findings — model output steered by PR-authored text, which the +workflow's own `Report cell outcome` step refuses to echo for that reason. The +allowlist is the safety property, so `AllowlistTest` asserts the absence of +prose rather than the presence of fields: a future key added to ERROR_FIELDS +that happens to carry content would fail there. + +Everything else is about not becoming a second failure. The script runs in an +`always()` step after the agent already failed, so a truncated, empty, +non-JSON, invalid-UTF-8 or missing file must still exit 0 (`DegenerateTest`), +and an agent-written file must not be able to exhaust memory or the recursion +limit (`BoundsTest`). + +Run: python3 -m unittest discover -s .github/cursor-review/tests -p 'test_*.py' -v +""" + +import contextlib +import importlib.util +import io +import json +import os +import re +import subprocess +import sys +import tempfile +import unittest + +SCRIPT = os.path.normpath( + os.path.join(os.path.dirname(__file__), "..", "agent-error.py") +) +SPEC = importlib.util.spec_from_file_location("agent_error", SCRIPT) +agent_error = importlib.util.module_from_spec(SPEC) +SPEC.loader.exec_module(agent_error) + +WORKFLOW = os.path.normpath( + os.path.join( + os.path.dirname(__file__), "..", "..", "workflows", "cursor-review.yml" + ) +) + +# Marker strings. If one of these reaches the output, prose reached a public log. +PROSE = "PROSE_MUST_NOT_APPEAR" + + +def summary(obj_or_text): + """The script's output lines for a payload, as the workflow would get them.""" + text = obj_or_text + if not isinstance(text, str): + text = json.dumps(obj_or_text) + return agent_error.summarize(text) + + +def joined(obj_or_text): + return "\n".join(summary(obj_or_text)) + + +class NestedEnvelopeTest(unittest.TestCase): + """The outage this script exists for, pinned shape by shape.""" + + # The literal body OpenAI returned, with the surrounding codex events. + QUOTA = ( + '{"type":"session.created","session_id":"abc"}\n' + '{"type":"item.completed","item":{"type":"reasoning","text":"%s"}}\n' + '{"type":"error","error":{"type":"insufficient_quota",' + '"code":"credit_balance_exhausted","message":"You have no credits ' + 'remaining. Add credits to continue using the API at ' + 'https://platform.openai.com/settings/organization/billing/."}}\n' + ) % PROSE + + def test_the_provider_error_is_named(self): + out = joined(self.QUOTA) + # Each of these is independently the thing a reader needs. + self.assertIn("insufficient_quota", out) + self.assertIn("credit_balance_exhausted", out) + self.assertIn("no credits remaining", out) + + def test_the_billing_link_survives(self): + # The provider's message names the fix; truncating it away would leave + # the reader with a code and no next step. + self.assertIn("platform.openai.com", joined(self.QUOTA)) + + def test_the_reasoning_event_does_not(self): + self.assertNotIn(PROSE, joined(self.QUOTA)) + + def test_a_nested_error_needs_no_self_description(self): + # The regression directly: nothing here contains the string "error" + # except the envelope key, so a substring heuristic alone finds nothing. + out = joined({"error": {"type": "overloaded", "code": "slow_down"}}) + self.assertIn("overloaded", out) + self.assertIn("slow_down", out) + + def test_a_string_error_is_still_read(self): + self.assertIn("boom", joined({"error": "boom"})) + + def test_a_blank_string_error_is_not_reported(self): + out = joined({"type": "turn.completed", "error": " "}) + self.assertNotIn("error:", out) + + +class AllowlistTest(unittest.TestCase): + """What may NOT appear. Asserted as absence, so a widened allowlist fails.""" + + PROSE_KEYS = ( + "result", "content", "text", "input", "output", "arguments", + "reasoning", "delta", "thinking", "summary", "findings", + ) + + def test_no_prose_key_is_printed_from_an_error_object(self): + for key in self.PROSE_KEYS: + with self.subTest(key=key): + payload = {"type": "error", "is_error": True, key: PROSE} + self.assertNotIn(PROSE, joined(payload)) + + def test_the_allowlist_holds_no_prose_key(self): + for key in self.PROSE_KEYS: + self.assertNotIn(key, agent_error.ERROR_FIELDS) + + def test_a_structured_message_is_not_flattened(self): + # `message` IS allowlisted, so a provider that nests content blocks + # under it must be skipped rather than stringified. + payload = {"type": "error", "message": [{"type": "text", "text": PROSE}]} + self.assertNotIn(PROSE, joined(payload)) + + def test_a_dict_message_is_not_flattened(self): + payload = {"type": "error", "message": {"body": PROSE}} + self.assertNotIn(PROSE, joined(payload)) + + def test_claude_result_prose_is_withheld_but_the_failure_is_named(self): + payload = { + "type": "result", + "subtype": "error_during_execution", + "is_error": True, + "result": PROSE, + "usage": {"input_tokens": 10}, + } + out = joined(payload) + self.assertNotIn(PROSE, out) + self.assertIn("error_during_execution", out) + + +class DetectionTest(unittest.TestCase): + def test_is_error_true(self): + self.assertTrue(agent_error._looks_like_error({"is_error": True})) + + def test_is_error_false_is_not_an_error(self): + self.assertFalse(agent_error._looks_like_error({"is_error": False})) + + def test_an_http_status_at_or_over_400(self): + self.assertTrue(agent_error._looks_like_error({"status": 429})) + self.assertTrue(agent_error._looks_like_error({"status_code": 500})) + self.assertTrue(agent_error._looks_like_error({"http_status": 400})) + + def test_a_successful_status_is_not_an_error(self): + self.assertFalse(agent_error._looks_like_error({"status": 200})) + + def test_a_boolean_status_is_not_a_status_code(self): + # Pins the BEHAVIOUR, which today holds by arithmetic rather than by + # the `isinstance(value, bool)` guard above it: `bool` is an `int` + # subclass, but `True >= 400` is already false, so deleting that guard + # changes nothing and this test cannot catch it. The guard stays as a + # statement of intent for anyone who lowers the threshold. + self.assertFalse(agent_error._looks_like_error({"status": True})) + self.assertFalse(agent_error._looks_like_error({"status": False})) + + def test_a_clean_run_reports_no_error(self): + text = ( + '{"type":"session.created"}\n' + '{"type":"item.completed","item":{"type":"tool_call"}}\n' + '{"type":"turn.completed"}\n' + ) + out = joined(text) + self.assertIn("no error object", out) + self.assertNotIn("error: ", out) + + def test_the_event_census_is_printed(self): + # The census is the fallback diagnosis when there is no error object: + # it says how far the agent got. + out = joined('{"type":"session.created"}\n{"type":"turn.completed"}\n') + self.assertIn("events: 2", out) + self.assertIn("session.created", out) + + +class SanitizationTest(unittest.TestCase): + def test_a_workflow_command_cannot_be_forged(self): + payload = {"error": {"message": "x\n::error::forged\n::set-output name=a::b"}} + out = joined(payload) + self.assertNotIn("::", out) + self.assertIn("forged", out) # broken, not deleted + + def test_a_value_cannot_become_two_log_lines(self): + out = summary({"error": {"message": "first\nsecond\rthird"}}) + self.assertEqual( + 1, sum(1 for line in out if "first" in line and "third" in line) + ) + + def test_disallowed_characters_are_replaced_not_dropped(self): + # Deleting would let a crafted value collapse into different text. + self.assertEqual("a?b", agent_error.sanitize("a\u0000b")) + + def test_an_odd_run_of_colons_leaves_no_pair(self): + # `str.replace` does not rescan its own output: `":::"` came back as + # `": ::"`, so the no-`::` guarantee failed for odd-length runs. + for value in (":::", "::::", "a:::b", ":" * 9): + with self.subTest(value=value): + self.assertNotIn("::", agent_error.sanitize(value)) + + def test_the_legacy_command_form_is_broken_too(self): + # The runner also recognises `##[command]` anywhere in a line, and + # `_SAFE` allows `#`, `[` and `]`. + for value in ("##[error]forged", "##[add-mask]x", "###[error]x"): + with self.subTest(value=value): + out = agent_error.sanitize(value) + self.assertNotIn("##[", out) + self.assertIn("forged" if "forged" in value else "x", out) + + def test_no_emitted_line_can_carry_either_command_form(self): + payload = {"error": {"type": ":::", "code": "##[add-mask]", + "message": "x:::y ##[error]z"}} + for line in summary(payload): + self.assertNotIn("::", line) + self.assertNotIn("##[", line) + + def test_a_long_value_is_truncated(self): + out = agent_error.sanitize("x" * 5000) + self.assertLessEqual(len(out), agent_error.MAX_VALUE) + self.assertTrue(out.endswith("…")) + + def test_the_label_is_sanitized(self): + with tempfile.TemporaryDirectory() as d: + path = os.path.join(d, "events.jsonl") + with open(path, "w", encoding="utf-8") as f: + f.write("{}\n") + out = run_script(path, label="::error::forged") + self.assertNotIn("::error::", out) + + +class BoundsTest(unittest.TestCase): + def test_deep_nesting_does_not_recurse_without_limit(self): + # Nested through an ENVELOPE key, since that is the only path the walk + # descends at all: an earlier version of this test wrapped the payload + # in an arbitrary `wrap` key and so stopped proving anything once + # descent became an allowlist. + node = {"type": "error", "code": "deep"} + for _ in range(5000): + node = {"error": node} + out = joined(node) # must not raise RecursionError + self.assertIn("walk bounded", out) + + def test_wide_breadth_is_bounded(self): + payload = {"error": [{"type": "error", "code": f"c{i}"} for i in range(50000)]} + out = joined(payload) + self.assertIn("walk bounded", out) + + def test_many_error_objects_are_capped(self): + text = "\n".join( + json.dumps({"type": "error", "code": f"c{i}"}) for i in range(200) + ) + out = summary(text) + self.assertLessEqual( + sum(1 for line in out if line.startswith("error: ")), + agent_error.MAX_LINES, + ) + self.assertTrue(any("more error object(s)" in line for line in out)) + + def test_duplicate_error_objects_collapse(self): + one = json.dumps({"type": "error", "code": "same"}) + out = summary("\n".join([one] * 20)) + self.assertEqual(1, sum(1 for line in out if line.startswith("error: "))) + + def test_the_census_is_capped(self): + text = "\n".join(json.dumps({"type": f"t{i}"}) for i in range(100)) + census = [line for line in summary(text) if line.startswith("events: ")] + self.assertEqual(1, len(census)) + self.assertLessEqual(census[0].count("×"), 10) + + +class ModelAuthoredSubtreeTest(unittest.TestCase): + """The agent writes part of this file. Those parts must never be read. + + A reviewer agent chooses the `arguments` of its own MCP tool calls, so a + prompt-injected one can plant an error-shaped object there and — if the + walk descended into every value — get attacker-chosen text into the run + log. That is not hypothetical nesting: it is the one subtree in these files + whose contents the model dictates. + """ + + SHAPES = ( + # codex: an MCP tool call whose arguments the model wrote. + {"type": "item.completed", + "item": {"type": "mcp_tool_call", "name": "cursor_review_record_finding", + "arguments": {"severity": "high", "error": {"message": PROSE}}}}, + # The shape that needed no `error` key at all. + {"item": {"arguments": {"type": "error", "message": PROSE}}}, + # A tool result's content. + {"type": "item.completed", + "item": {"type": "tool_result", "content": [{"is_error": True, "message": PROSE}]}}, + # Prose hidden under an allowlisted field name one level down. + {"type": "result", "is_error": True, "result": {"error": {"message": PROSE}}}, + # A model-chosen `type`, which would otherwise reach the census. + {"item": {"arguments": {"type": PROSE}}}, + ) + + def test_no_model_authored_subtree_reaches_the_log(self): + for i, shape in enumerate(self.SHAPES): + with self.subTest(shape=i): + self.assertNotIn(PROSE, joined(shape)) + + def test_the_descent_allowlist_holds_no_model_written_key(self): + # The guard that matters when someone widens ENVELOPE_KEYS later. + for key in ("arguments", "input", "tool_input", "result", "content", + "text", "output", "reasoning", "thinking", "delta", + "parameters", "params", "messages", "findings"): + self.assertNotIn(key, agent_error.ENVELOPE_KEYS) + + def test_a_provider_error_beside_the_arguments_is_still_read(self): + # Blocking the model's subtree must not cost the CLI's own error. + payload = {"type": "item.completed", + "item": {"type": "mcp_tool_call", + "arguments": {"error": {"message": PROSE}}, + "error": {"code": "tool_rejected"}}} + out = joined(payload) + self.assertIn("tool_rejected", out) + self.assertNotIn(PROSE, out) + + +class ClaudeResultShapeTest(unittest.TestCase): + """Claude Code reports an API failure as a result object. + + The cause sits in `result` — the same field its review prose uses — so it + is NOT printed. `is_error` is, so the line cannot read as a success. This + pins the documented gap: see the per-lane note in + `docs/callers/cursor-review.md`. + """ + + FAILURE = { + "type": "result", + "subtype": "success", + "is_error": True, + "result": "API Error: 400 credit balance is too low", + } + + def test_the_failure_is_not_reported_as_a_success(self): + out = joined(self.FAILURE) + self.assertIn("is_error=True", out) + + def test_the_cause_in_result_is_withheld(self): + self.assertNotIn("credit balance", joined(self.FAILURE)) + + def test_is_error_is_allowlisted(self): + self.assertIn("is_error", agent_error.ERROR_FIELDS) + + +class TailReadTest(unittest.TestCase): + """The explaining event is the LAST thing an agent writes.""" + + def setUp(self): + self.dir = tempfile.TemporaryDirectory() + self.addCleanup(self.dir.cleanup) + + def test_a_late_error_in_an_oversized_file_is_still_found(self): + # Reading forward from byte 0 would spend the whole budget on filler + # and lose the terminal event — the one the script exists to print. + path = os.path.join(self.dir.name, "events.jsonl") + filler = json.dumps({"type": "item.completed", "item": {"type": "reasoning"}}) + with open(path, "w", encoding="utf-8") as f: + while f.tell() < agent_error.MAX_READ + 500_000: + f.write(filler + "\n") + f.write(json.dumps( + {"type": "error", "error": {"code": "terminal_failure"}}) + "\n") + out = run_script(path) + self.assertIn("terminal_failure", out) + + def test_the_truncated_read_is_declared(self): + path = os.path.join(self.dir.name, "big.jsonl") + with open(path, "w", encoding="utf-8") as f: + f.write("x" * (agent_error.MAX_READ + 10)) + self.assertIn("larger file", run_script(path)) + + def test_the_newest_error_is_reported_first(self): + text = ( + json.dumps({"type": "error", "error": {"code": "first_failure"}}) + "\n" + + json.dumps({"type": "error", "error": {"code": "last_failure"}}) + "\n" + ) + out = "\n".join(summary(text)) + self.assertLess(out.index("last_failure"), out.index("first_failure")) + + +class DecoderHostileInputTest(unittest.TestCase): + """`json.loads` raises RecursionError — a RuntimeError, not a ValueError. + + 400 KB of `[` is enough, well inside the 4 MB read. An uncaught one would + traceback, exit 1, and under the step's `bash -e` abort `Report cell + outcome` before its status line — turning an advisory leg red. + """ + + def setUp(self): + self.dir = tempfile.TemporaryDirectory() + self.addCleanup(self.dir.cleanup) + + def write(self, text): + path = os.path.join(self.dir.name, "events.jsonl") + with open(path, "w", encoding="utf-8") as f: + f.write(text) + return path + + def test_deeply_nested_json_exits_zero(self): + depth = 200_000 + run_script(self.write("[" * depth + "]" * depth)) # asserts exit 0 + + def test_deeply_nested_json_on_one_ndjson_line_exits_zero(self): + depth = 200_000 + run_script(self.write( + '{"type":"session.created"}\n' + "[" * depth + "]" * depth + "\n")) + + def test_an_unexpected_failure_in_the_walk_still_exits_zero(self): + # `main`'s broad `except` is unreachable through any input the rest of + # this suite can construct — `_loads` already absorbs the decoder — so + # the guard is exercised directly instead of left as untested + # defensive code. Whatever future change makes `summarize` raise, the + # step must still reach its status line. + path = self.write('{"type":"error","code":"x"}\n') + original = agent_error.walk + + def explode(*_args, **_kwargs): + raise RuntimeError("boom") + + agent_error.walk = explode + try: + buf = io.StringIO() + with contextlib.redirect_stdout(buf): + code = agent_error.main(["--events", path]) + finally: + agent_error.walk = original + self.assertEqual(0, code) + self.assertIn("could not summarize", buf.getvalue()) + self.assertNotIn("boom", buf.getvalue()) # the message is not echoed + + def test_the_loads_helper_never_raises(self): + for payload in ("[" * 200_000, "{" * 200_000, "nonsense", ""): + ok, _ = agent_error._loads(payload) + self.assertFalse(ok) + + +def run_script(path, label="adversarial/gpt-5.6-sol"): + """The script as the workflow runs it: a subprocess, whose code must be 0.""" + proc = subprocess.run( + [sys.executable, SCRIPT, "--events", path, "--label", label], + capture_output=True, text=True, + ) + assert proc.returncode == 0, f"exit {proc.returncode}: {proc.stderr}" + return proc.stdout + + +class DegenerateTest(unittest.TestCase): + """It runs after the agent already failed; it must never fail in turn.""" + + def setUp(self): + self.dir = tempfile.TemporaryDirectory() + self.addCleanup(self.dir.cleanup) + + def write(self, data, name="events.jsonl"): + path = os.path.join(self.dir.name, name) + mode = "wb" if isinstance(data, bytes) else "w" + with open(path, mode) as f: + f.write(data) + return path + + def test_a_missing_file_exits_zero(self): + out = run_script(os.path.join(self.dir.name, "nope.jsonl")) + self.assertIn("could not read", out) + + def test_an_empty_file_exits_zero(self): + self.assertIn("no JSON events", run_script(self.write(""))) + + def test_plain_text_exits_zero(self): + out = run_script(self.write("Reading prompt from stdin...\nnot json\n")) + self.assertIn("no JSON events", out) + + def test_invalid_utf8_exits_zero(self): + path = self.write(b'{"error":{"message":"caf\xc3\x28 bytes"}}\n') + self.assertIn("bytes", run_script(path)) + + def test_a_truncated_last_event_still_reads_the_rest(self): + # The normal shape for a killed agent: complete events, then half of one. + path = self.write( + '{"type":"error","error":{"code":"first_error"}}\n' + '{"type":"item.completed","item":{"te' + ) + self.assertIn("first_error", run_script(path)) + + def test_a_directory_exits_zero(self): + self.assertIn("could not read", run_script(self.dir.name)) + + def test_the_header_names_the_cell(self): + out = run_script(self.write("{}\n"), label="edge-case/claude-opus-5-5") + self.assertIn("edge-case/claude-opus-5-5", out) + + def test_output_is_bounded_overall(self): + text = "\n".join( + json.dumps({"type": "error", "message": "m" * 400, "code": f"c{i}"}) + for i in range(500) + ) + out = run_script(self.write(text)) + self.assertLessEqual(len(out.encode("utf-8")), agent_error.MAX_BYTES + 512) + + +class WorkflowWiringTest(unittest.TestCase): + """Both direct lanes must actually call it, on their own output file. + + Parsed WITHOUT PyYAML, like its sibling suites: this repo is stdlib-only + and CI installs no requirements for these tests. + """ + + @classmethod + def setUpClass(cls): + with io.open(WORKFLOW, encoding="utf-8") as f: + cls.text = f.read() + cls.blocks = cls._steps(cls.text) + + @staticmethod + def _steps(text): + """The `Report cell outcome` step bodies, keyed in file order.""" + out, lines, current = [], text.splitlines(), None + for line in lines: + if re.match(r"^ - name: ", line): + if current is not None: + out.append("\n".join(current)) + current = [line] if "Report cell outcome" in line else None + elif current is not None: + current.append(line) + if current is not None: + out.append("\n".join(current)) + return out + + def test_both_direct_lanes_report_the_failure(self): + calling = [b for b in self.blocks if "agent-error.py" in b] + self.assertEqual(2, len(calling), "expected exactly the two direct lanes") + + def test_each_lane_reads_its_own_output_file(self): + files = sorted( + re.search(r"--events (\S+)", b).group(1) + for b in self.blocks + if "agent-error.py" in b + ) + self.assertEqual(["/tmp/claude-result.json", "/tmp/codex-events.jsonl"], files) + + def test_the_file_each_lane_reads_is_the_file_its_agent_writes(self): + # A rename on one side only would leave the diagnosis reading a file + # nobody writes, and `could not read` is a plausible-looking output. + for path in ("/tmp/codex-events.jsonl", "/tmp/claude-result.json"): + with self.subTest(path=path): + self.assertIn(f"> {path}", self.text) + + def test_the_diagnosis_runs_only_when_the_cell_failed(self): + # In the `ok` branch it would print "why the cell failed" about a cell + # that succeeded. + for block in self.blocks: + if "agent-error.py" not in block: + continue + ok_at = block.index("submitted its findings.") + self.assertLess(ok_at, block.index("agent-error.py")) + self.assertIn("else", block[ok_at:block.index("agent-error.py")]) + + def test_the_script_is_loaded_from_the_pinned_assets(self): + # Never from the caller's checkout: a PR must not be able to rewrite + # the code that reports on it. + for block in self.blocks: + if "agent-error.py" in block: + self.assertIn('"$CURSOR_REVIEW_ASSETS/agent-error.py"', block) + + def test_the_diagnosis_cannot_abort_the_step(self): + # The step runs under `bash -e`, and the script's own exit-0 contract + # cannot cover a script that never STARTED — a skipped assets checkout + # makes `python3` exit 2, which would lose the status and `::warning::` + # lines this step exists to print. + for block in self.blocks: + if "agent-error.py" in block: + self.assertIn('--label "${REVIEW_TYPE}/${MODEL}" || true', block) + + def test_the_cell_outcome_step_still_withholds_the_findings(self): + for block in self.blocks: + self.assertNotIn("cat /tmp/findings-out/findings.json", block) + + +class StepBehaviourTest(unittest.TestCase): + """Runs the workflow's REAL `Report cell outcome` body, not just the script. + + `WorkflowWiringTest` reads the step statically, which cannot tell whether + the branch it sits in is reached or whether the step as a whole keeps the + agent's prose out. So the body is extracted and EXECUTED against fixtures, + with its hardcoded `/tmp` paths rewritten into a sandbox. The leak + assertion is the point: it covers the whole step, so a future `cat` of the + events file added beside the diagnosis would fail here. + """ + + @classmethod + def setUpClass(cls): + with io.open(WORKFLOW, encoding="utf-8") as f: + text = f.read() + cls.bodies = cls._bodies(text) + # Both lanes, or the extraction below has drifted from the workflow. + assert len(cls.bodies) == 2, f"extracted {len(cls.bodies)} wired bodies" + + @staticmethod + def _bodies(text): + """The `run:` script of every `Report cell outcome` step that calls us.""" + out, cur, lines = [], None, text.splitlines() + for line in lines: + if re.match(r"^ - name: ", line): + if cur: + out.append("\n".join(cur)) + cur = [] if "Report cell outcome" in line else None + elif cur is not None: + cur.append(line) + if cur: + out.append("\n".join(cur)) + bodies = [] + for step in out: + m = re.search(r"^ run: \|\n(.*)", step, re.S | re.M) + if not m: + continue + body = [] + for ln in m.group(1).splitlines(): + if ln.strip() and not ln.startswith(" "): + break + body.append(ln[10:]) + body = "\n".join(body) + if "agent-error.py" in body: + bodies.append(body) + return bodies + + EVENTS = ( + '{"type":"item.completed","item":{"type":"reasoning","text":"%s"}}\n' + '{"type":"error","error":{"type":"insufficient_quota",' + '"code":"credit_balance_exhausted"}}\n' + ) % PROSE + + def run_step(self, body, record, counts="true", events=EVENTS): + d = tempfile.mkdtemp() + self.addCleanup(lambda: __import__("shutil").rmtree(d, ignore_errors=True)) + findings = os.path.join(d, "findings.json") + with open(findings, "w", encoding="utf-8") as f: + f.write(record if isinstance(record, str) else json.dumps(record)) + evfile = os.path.join(d, "events") + if events is not None: + with open(evfile, "w", encoding="utf-8") as f: + f.write(events) + script = body + for placeholder in ("/tmp/findings-out/findings.json",): + script = script.replace(placeholder, findings) + for placeholder in ("/tmp/codex-events.jsonl", "/tmp/claude-result.json"): + script = script.replace(placeholder, evfile) + proc = subprocess.run( + ["bash", "-e", "-c", script], capture_output=True, text=True, + env={ + **os.environ, + "CURSOR_REVIEW_ASSETS": os.path.dirname(SCRIPT), + "MODEL": "gpt-5.6-sol", + "REVIEW_TYPE": "adversarial", + "COUNTS": counts, + }, + ) + self.assertEqual(0, proc.returncode, proc.stderr) + return proc.stdout + + def test_a_failed_cell_names_the_provider_error(self): + for i, body in enumerate(self.bodies): + with self.subTest(lane=i): + out = self.run_step(body, {"status": "error", "findings": []}) + self.assertIn("why the cell failed", out) + self.assertIn("credit_balance_exhausted", out) + + def test_the_step_never_prints_the_agents_prose(self): + # The whole step, not just the script: a `cat` of the events file added + # next to the diagnosis would fail here. + for i, body in enumerate(self.bodies): + for counts in ("true", "false"): + with self.subTest(lane=i, counts=counts): + out = self.run_step( + body, {"status": "error", "findings": []}, counts=counts + ) + self.assertNotIn(PROSE, out) + + def test_a_submitted_cell_gets_no_failure_diagnosis(self): + for i, body in enumerate(self.bodies): + with self.subTest(lane=i): + out = self.run_step(body, {"status": "ok", "findings": [{"a": 1}]}) + self.assertNotIn("why the cell failed", out) + self.assertIn("submitted its findings", out) + + def test_a_missing_output_file_does_not_fail_the_step(self): + # The step is `always()`; a cell killed before its agent wrote anything + # must still report, not error. + for i, body in enumerate(self.bodies): + with self.subTest(lane=i): + out = self.run_step( + body, {"status": "error", "findings": []}, events=None + ) + self.assertIn("could not read", out) + + def test_an_unparseable_record_still_reports_a_failure(self): + for i, body in enumerate(self.bodies): + with self.subTest(lane=i): + out = self.run_step(body, "not json at all") + self.assertIn("status=error", out) + + def test_the_gating_and_advisory_wording_both_survive(self): + for i, body in enumerate(self.bodies): + with self.subTest(lane=i): + gating = self.run_step( + body, {"status": "error", "findings": []}, counts="true" + ) + self.assertIn("counts as a failed reviewer", gating) + advisory = self.run_step( + body, {"status": "error", "findings": []}, counts="false" + ) + self.assertIn("::warning::", advisory) + self.assertIn("advisory", advisory) + + +if __name__ == "__main__": + unittest.main(verbosity=2) diff --git a/.github/workflows/cursor-review.yml b/.github/workflows/cursor-review.yml index dd48c49a..208b2bc5 100644 --- a/.github/workflows/cursor-review.yml +++ b/.github/workflows/cursor-review.yml @@ -3454,12 +3454,33 @@ jobs: echo "Direct-API cell: status=${status}, ${count} finding(s)." if [ "$status" = "ok" ]; then echo "Direct-API cell ${REVIEW_TYPE}/${MODEL} submitted its findings." - elif [ "$COUNTS" = "true" ]; then - # Failed AFTER the upload below, by `Fail the leg when the cell did - # not submit`, so the errored record still reaches the judge. - echo "Direct-API cell ${REVIEW_TYPE}/${MODEL} did not submit (status=$status); it replaces the Cursor OpenAI lane, so it counts as a failed reviewer." else - echo "::warning::Direct-API cell ${REVIEW_TYPE}/${MODEL} did not submit (status=$status). Its record still reaches the judge; this leg stays green because the cell is advisory." + # WHY it failed, not just THAT it did. The agent reports an + # API-level failure in its STRUCTURED OUTPUT, which is redirected + # to a file and — unlike the Cursor cells, which echo 500 chars of + # raw output and all of stderr — never echoed, because that file + # also holds the agent's prose. So a real outage + # (`insufficient_quota` / `credit_balance_exhausted`, naming its + # own cause) was discarded and this log showed only a non-zero + # exit, leaving the lane's failures to be inferred from job + # DURATION. agent-error.py prints an ALLOWLIST of provider error + # metadata and never the prose, and exits 0 on anything — a + # truncated or non-JSON file must not turn the diagnosis into a + # second failure. + # `|| true` because this step runs under `bash -e`: the script's + # own exit-0 contract cannot cover a script that never STARTED + # (a skipped assets checkout, so `python3` exits 2), and aborting + # here would lose the status and `::warning::` lines below. + python3 "$CURSOR_REVIEW_ASSETS/agent-error.py" \ + --events /tmp/codex-events.jsonl \ + --label "${REVIEW_TYPE}/${MODEL}" || true + if [ "$COUNTS" = "true" ]; then + # Failed AFTER the upload below, by `Fail the leg when the cell did + # not submit`, so the errored record still reaches the judge. + echo "Direct-API cell ${REVIEW_TYPE}/${MODEL} did not submit (status=$status); it replaces the Cursor OpenAI lane, so it counts as a failed reviewer." + else + echo "::warning::Direct-API cell ${REVIEW_TYPE}/${MODEL} did not submit (status=$status). Its record still reaches the judge; this leg stays green because the cell is advisory." + fi fi - name: Upload findings artifact @@ -3909,12 +3930,33 @@ jobs: echo "Direct-API cell: status=${status}, ${count} finding(s)." if [ "$status" = "ok" ]; then echo "Direct-API cell ${REVIEW_TYPE}/${MODEL} submitted its findings." - elif [ "$COUNTS" = "true" ]; then - # Failed AFTER the upload below, by `Fail the leg when the cell did - # not submit`, so the errored record still reaches the judge. - echo "Direct-API cell ${REVIEW_TYPE}/${MODEL} did not submit (status=$status); it replaces the Cursor Anthropic lane, so it counts as a failed reviewer." else - echo "::warning::Direct-API cell ${REVIEW_TYPE}/${MODEL} did not submit (status=$status). Its record still reaches the judge; this leg stays green because the cell is advisory." + # WHY it failed, not just THAT it did. The agent reports an + # API-level failure in its STRUCTURED OUTPUT, which is redirected + # to a file and — unlike the Cursor cells, which echo 500 chars of + # raw output and all of stderr — never echoed, because that file + # also holds the agent's prose. So a real outage + # (`insufficient_quota` / `credit_balance_exhausted`, naming its + # own cause) was discarded and this log showed only a non-zero + # exit, leaving the lane's failures to be inferred from job + # DURATION. agent-error.py prints an ALLOWLIST of provider error + # metadata and never the prose, and exits 0 on anything — a + # truncated or non-JSON file must not turn the diagnosis into a + # second failure. + # `|| true` because this step runs under `bash -e`: the script's + # own exit-0 contract cannot cover a script that never STARTED + # (a skipped assets checkout, so `python3` exits 2), and aborting + # here would lose the status and `::warning::` lines below. + python3 "$CURSOR_REVIEW_ASSETS/agent-error.py" \ + --events /tmp/claude-result.json \ + --label "${REVIEW_TYPE}/${MODEL}" || true + if [ "$COUNTS" = "true" ]; then + # Failed AFTER the upload below, by `Fail the leg when the cell did + # not submit`, so the errored record still reaches the judge. + echo "Direct-API cell ${REVIEW_TYPE}/${MODEL} did not submit (status=$status); it replaces the Cursor Anthropic lane, so it counts as a failed reviewer." + else + echo "::warning::Direct-API cell ${REVIEW_TYPE}/${MODEL} did not submit (status=$status). Its record still reaches the judge; this leg stays green because the cell is advisory." + fi fi - name: Upload findings artifact diff --git a/docs/callers/cursor-review.md b/docs/callers/cursor-review.md index ecdfd89d..e449d860 100644 --- a/docs/callers/cursor-review.md +++ b/docs/callers/cursor-review.md @@ -229,6 +229,32 @@ another workflow using the default token silently fails to start a review. That is exactly what [`cursor-review-auto-label.yml`](cursor-review-auto-label.md) exists to handle. +**A direct-API cell that fails says why in its own job log.** The direct +lanes never echo their agent's output — it carries model prose steered by the +PR — so when a cell does not submit, its **Report cell outcome** step prints an +allowlisted summary of the provider's error instead (`type`, `subtype`, +`is_error`, `code`, `param`, `status`, `message`) plus a census of the events it +got that far. + +How much that names depends on the lane, because the two CLIs report failures +differently. **OpenAI/codex** puts the cause in a structured error event, so a +quota or rate-limit failure names itself — `type=insufficient_quota +code=credit_balance_exhausted` and the provider's own message. +**Anthropic/claude** reports an API-level failure as a result object whose +cause sits in `result`, which is the same field its review prose uses: that +field is not allowlisted and is not printed, so the Anthropic lane tells you +`is_error=true` and the event census but **not** the cause. Reading it means +re-running the cell, or checking the provider's status/billing directly. + +That is a deliberate trade, not an oversight: `result` is attacker-steerable +(the agent reads PR-authored text), so printing it would hand a prompt-injected +agent a channel into the run log, which is the exact thing withholding these +files prevents. A cell that ran but submitted nothing shows up as a census with +no error object. Nothing else is printed and the raw output is not uploaded, so +that step is the whole diagnosis — if it is not enough, the fix is to widen the +allowlist in `agent-error.py` to a field the CLI owns, not to publish the +transcript. + **Applying the label does not guarantee a run.** If the event was swallowed, remove the label, confirm it is gone, then re-add it.