diff --git a/.github/cursor-review/README.md b/.github/cursor-review/README.md index a03be6c4..9586dfe9 100644 --- a/.github/cursor-review/README.md +++ b/.github/cursor-review/README.md @@ -256,7 +256,7 @@ the consolidated review's panel table. See [Panel integrity](#panel-integrity). | [`build-ledger.py`](build-ledger.py) | Builds the **prior-review ledger** — what earlier rounds raised on this PR and how the author answered — and splices it into the panel/judge prompts. Untrusted prose is defanged twice over: fence-opening lines are rewritten, and continuation lines of quoted prose are prefixed ` |` so no imported text can sit at a field's indent and forge one. Also the prompt splicer, so the no-ledger path is byte-identical to the pre-ledger prompt. | | [`fence-diff.py`](fence-diff.py) | Wraps the reviewed diff (plus the incremental hunks, and the judge's panel-findings block) in `=== BEGIN/END DIFF ===` fences. The diff is attacker-authored PR bytes, so static literal fences are not a control; the nonce is what a PR cannot forge. Each prompt-build step mints its OWN nonce (`mint`), into a shell variable rather than a step `env:` or job output — Actions dumps a step's env map into the public run log, and a per-prompt value means a leak in one job cannot forge a fence in another. Copies the body through **byte for byte** — it never defangs or normalizes the payload. | | [`incremental-diff.py`](incremental-diff.py) | Builds the incremental "hunks new since the last reviewed round" block as the difference between two PR patches (`build`), and verifies the result really is a subset of the reviewed diff (`check`) so the workflow can discard it if it is not. Never a commit range: `git diff LAST_REVIEWED...HEAD` carries every commit a merge of the base branch brought in — see [the ledger section](#the-prior-review-ledger-and-the-repeat-policy). | -| [`agent-error.py`](agent-error.py) | Run by each **direct-API** cell's **Report cell outcome** step, and only when that cell did not submit: prints WHY it failed. Both direct lanes send their agent's structured output to a file (`codex-events.jsonl`, `claude-result.json`) that is never echoed, because it also holds the agent's prose — so an API-level failure, which `codex exec --json` reports as an event on **stdout**, used to leave the log showing a non-zero exit and nothing else, and the lane's failures had to be inferred from job DURATION (a real `insufficient_quota` / `credit_balance_exhausted` outage was discarded that way). It prints an allowlist of short provider metadata (`type`, `subtype`, `is_error`, `code`, `param`, `status`, `message`) from objects that look like errors, plus a census of event types. **Two** allowlists, and the second is the load-bearing one: restricting the FIELDS is not enough, because the agent writes part of this file — it chooses the `arguments` of its own MCP tool calls — so a prompt-injected one could plant `{"error": {"message": "…"}}` there and a walk that descended into every value would print it. **Descent** is therefore an allowlist too (`ENVELOPE_KEYS`: `error`, `last_error`, `item`, `response`, `data`, `detail`), so the model's writable subtrees are never reached rather than denied one by one; widen it only to a key a CLI owns. An `error` KEY is read on its own terms, since a provider's error object need not describe itself as an error (OpenAI's says `insufficient_quota`). Values are sanitized by REPLACING disallowed characters, never deleting them, and both the `::` and legacy `##[command]` forms are broken with a single-pass regex (`str.replace` does not rescan its own output, so `a:::b` defeated it). Reads the **tail** of the file, not the head: the event that explains a failure is the last thing an agent writes. Exits **0 on anything** — missing, truncated, non-JSON, invalid UTF-8, or nested deeply enough that `json.loads` raises `RecursionError` (a RuntimeError, not a ValueError; 400 KB of `[` suffices) — because it runs after the cell already failed and an abort under the step's `bash -e` would cost the status line. **Known gap:** Claude Code puts an API failure's cause in `result`, the same field its review prose uses, so the Anthropic lane reports `is_error=true` and the census but not the cause — a deliberate trade, since that field is attacker-steerable. The raw file is not uploaded either: the Anthropic lane screens `findings.json` for a leaked API key before publishing it, and shipping the unscreened transcript alongside would route the agent's prose past that guard. | +| [`agent-error.py`](agent-error.py) | Run by each **direct-API** cell's **Report cell outcome** step, and only when that cell did not submit: prints WHY it failed. Both direct lanes send their agent's structured output to a file (`codex-events.jsonl`, `claude-result.json`) that is never echoed, because it also holds the agent's prose — so an API-level failure, which `codex exec --json` reports as an event on **stdout**, used to leave the log showing a non-zero exit and nothing else, and the lane's failures had to be inferred from job DURATION (a real `insufficient_quota` / `credit_balance_exhausted` outage was discarded that way). It prints an allowlist of short provider metadata (`type`, `subtype`, `is_error`, `code`, `param`, `status`, `message`, and Claude Code's `api_error_status` / `terminal_reason`) from objects that look like errors, plus a census of event types. **Two** allowlists, and the second is the load-bearing one: restricting the FIELDS is not enough, because the agent writes part of this file — it chooses the `arguments` of its own MCP tool calls — so a prompt-injected one could plant `{"error": {"message": "…"}}` there and a walk that descended into every value would print it. **Descent** is therefore an allowlist too (`ENVELOPE_KEYS`: `error`, `last_error`, `item`, `response`, `data`, `detail`), so the model's writable subtrees are never reached rather than denied one by one; widen it only to a key a CLI owns. An `error` KEY is read on its own terms, since a provider's error object need not describe itself as an error (OpenAI's says `insufficient_quota`). Values are sanitized by REPLACING disallowed characters, never deleting them, and both the `::` and legacy `##[command]` forms are broken with a single-pass regex (`str.replace` does not rescan its own output, so `a:::b` defeated it). Reads the **tail** of the file, not the head: the event that explains a failure is the last thing an agent writes. Exits **0 on anything** — missing, truncated, non-JSON, invalid UTF-8, or nested deeply enough that `json.loads` raises `RecursionError` (a RuntimeError, not a ValueError; 400 KB of `[` suffices) — because it runs after the cell already failed and an abort under the step's `bash -e` would cost the status line. **One stated exception** to withholding model-written fields: Claude Code puts an API failure's cause in `result` — the same field its review prose uses — and reports `subtype=success` on a failed call, so the cause is printed under gates: only from the top-level `{"type":"result"}` object with `is_error` true; only when the CLI marked it an API failure (a numeric `api_error_status` — the pinned CLI's real 401 is `Invalid API key · Fix external API key`, with no preamble — or an `API Error: ` preamble); never when it holds the `sk-ant-` prefix or any 16-char slice of the step's own `ANTHROPIC_API_KEY` (the screen `findings.json` gets, which a 120-char cut would otherwise slip past Actions' exact-value mask); and cut to 120 sanitized chars. A prompt-injected agent can forge the preamble, so it is a bounded, documented channel, not a safe one; it was taken knowingly because the cause appears nowhere else and the file is not uploaded. A failure with neither marker prints nothing rather than a guess. The raw file is not uploaded either: the Anthropic lane screens `findings.json` for a leaked API key before publishing it, and shipping the unscreened transcript alongside would route the agent's prose past that guard. | | [`catalog-drift.py`](catalog-drift.py) | Backs the weekly catalog-drift check. Extracts the pins from `cursor-review.yml`, diffs them against raw `cursor-agent models` output, and renders the sticky issue title + body (delisted pins, pins marked NO-ZDR, unpinned same-lab ids, catalog ids from unpinned families, stale audit date). Reports only — it never edits a pin. | ## Adopt it in your repo diff --git a/.github/cursor-review/agent-error.py b/.github/cursor-review/agent-error.py index 4d1a679e..f467ed03 100644 --- a/.github/cursor-review/agent-error.py +++ b/.github/cursor-review/agent-error.py @@ -18,6 +18,11 @@ (``ERROR_FIELDS``) from objects that look like errors, plus a census of event types. +ONE field is read out of model-written territory — Claude Code's API-failure +cause, which sits in `result` and nowhere else — under the gates in +``_api_error_cause``. Everything below describes the rule that field is the +stated exception to. + TWO allowlists, and the second one is the load-bearing one. Restricting the FIELDS is not enough on its own, because a reviewer agent writes part of this file: it chooses the `arguments` of its own MCP tool calls. A prompt-injected @@ -65,6 +70,9 @@ ERROR_FIELDS = ( "type", "subtype", "is_error", "code", "param", "status", "status_code", "http_status", "message", + # Claude Code's own scalars on a failed result object, beside the + # model-written `result`: the HTTP status and `api_error` respectively. + "api_error_status", "terminal_reason", ) # The ONLY keys this descends through — the envelope a CLI writes around the @@ -87,6 +95,20 @@ MAX_DEPTH = 12 MAX_NODES = 20000 +# Claude Code's API-failure cause sits in `result`, a field the model also +# writes. Printing it is THE one exception to "no model-written field reaches +# the log" — see `_api_error_cause`, which gates it. +CAUSE_FIELD = "result" +CAUSE_LIMIT = 120 +# `[0-9]`, not `\d`: in a str pattern `\d` matches any Unicode digit. +_API_ERROR = re.compile(r"^API Error: [0-9]{3} ") +# An Anthropic key's prefix, and the window of the REAL key (from the step's +# env) that a cause may not contain. A window rather than the whole key: the +# cut to CAUSE_LIMIT leaves a truncated key, which Actions' exact-value mask +# no longer matches. +_KEY_PREFIX = "sk-ant-" +KEY_WINDOW = 16 + _SAFE = re.compile(r"[^A-Za-z0-9 .,:;_/@+()\[\]{}='\"!?#$%^&*|~`<>-]") # A colon followed by a colon, and a `#` that begins a `##[`. Lookahead, so one # pass breaks every run however long: `":::"` -> `": : :"`, `"###["` -> `"## #["`. @@ -146,6 +168,70 @@ def _fields(obj: dict) -> str: return " ".join(parts) +def _api_error_cause(obj: dict, top: bool = False) -> str: + """Claude Code's API-failure cause, read out of a model-written field. + + THE one exception to the rule that no model-written field reaches the log, + and it is deliberate rather than an oversight. Claude Code reports an + API-level failure as `{"type":"result","subtype":"success","is_error":true, + "result":"API Error: 400 ..."}` — note `subtype`, which reads as SUCCESS — + and the cause appears nowhere else in the file. Since the file is not + uploaded either, an Anthropic-lane outage was otherwise undiagnosable + without re-running the cell. + + So it is gated: + + 1. the object is the CLI's own top-level `{"type":"result"}` object with + `is_error` true — never a nested or codex-lane object (``walk`` passes + ``top``), where the shape is not the one this exception covers; + 2. the CLI marked it an API failure: a numeric `api_error_status` (a CLI + scalar the model cannot write — `Invalid API key · Fix external API + key` with status 401 is a real 2.1.286 failure, and carries no + preamble), or a string opening with `API Error: `; + 3. it holds no Anthropic key: not the `sk-ant-` prefix and no + ``KEY_WINDOW``-char slice of the step's own ``ANTHROPIC_API_KEY`` — + the screen `findings.json` gets before publishing, which this path + would otherwise skip; + 4. it is cut to ``CAUSE_LIMIT`` and sanitized like every other value. + + A prompt-injected agent CAN forge the preamble, so this is a BOUNDED and + documented channel, not a safe one — roughly 120 sanitized characters + that cannot carry a workflow command or the key, but can carry other + text. That trade was taken knowingly (#402); the alternative was a lane + that reports `subtype=success` and names nothing. + + A failure with neither marker yields nothing rather than a guess, which is + the safe direction: widen the gate only against a real message, never + speculatively. + """ + if not top or obj.get("type") != "result" or obj.get("is_error") is not True: + return "" + value = obj.get(CAUSE_FIELD) + if not isinstance(value, str): + return "" + status = obj.get("api_error_status") + cli_marked = isinstance(status, int) and not isinstance(status, bool) + if not cli_marked and not _API_ERROR.match(value): + return "" + if _holds_key(value): + return f"{CAUSE_FIELD}=(withheld: it contains an Anthropic API key or part of one)" + return f"{CAUSE_FIELD}={sanitize(value, CAUSE_LIMIT)}" + + +def _holds_key(value: str) -> bool: + """True when ``value`` carries the key prefix or a slice of the real key.""" + if _KEY_PREFIX in value: + return True + key = os.environ.get("ANTHROPIC_API_KEY", "") + if not key: + return False + if len(key) <= KEY_WINDOW: + return key in value + return any( + key[i : i + KEY_WINDOW] in value for i in range(len(key) - KEY_WINDOW + 1) + ) + + def walk(node, out: list, types: dict, state: dict, depth: int = 0) -> None: """Collect error objects and a census of `type` values. Bounded. @@ -171,6 +257,9 @@ def walk(node, out: list, types: dict, state: dict, depth: int = 0) -> None: rendered = _fields(node) if rendered and rendered not in out: out.append(rendered) + cause = _api_error_cause(node, top=depth == 0) + if cause and cause not in out: + out.append(cause) # An `error` KEY is the canonical envelope and must be read on its own # terms. The provider's error object does not have to describe ITSELF as an # error — OpenAI's is `{"error": {"type": "insufficient_quota", "code": diff --git a/.github/cursor-review/tests/test_agent_error.py b/.github/cursor-review/tests/test_agent_error.py index d807a728..f8440bdf 100644 --- a/.github/cursor-review/tests/test_agent_error.py +++ b/.github/cursor-review/tests/test_agent_error.py @@ -39,6 +39,7 @@ import sys import tempfile import unittest +from unittest import mock SCRIPT = os.path.normpath( os.path.join(os.path.dirname(__file__), "..", "agent-error.py") @@ -121,6 +122,9 @@ class AllowlistTest(unittest.TestCase): ) def test_no_prose_key_is_printed_from_an_error_object(self): + # `result` is covered too: bare prose on a non-`result` object does + # not pass `_api_error_cause`'s gates. The one value that DOES pass is + # pinned in ClaudeResultShapeTest, deliberately in one place. for key in self.PROSE_KEYS: with self.subTest(key=key): payload = {"type": "error", "is_error": True, key: PROSE} @@ -313,6 +317,15 @@ class ModelAuthoredSubtreeTest(unittest.TestCase): {"type": "result", "is_error": True, "result": {"error": {"message": PROSE}}}, # A model-chosen `type`, which would otherwise reach the census. {"item": {"arguments": {"type": PROSE}}}, + # The Claude result shape, but NOT at the top level: the cause + # exception covers only the CLI's own top-level object. + {"type": "item.completed", + "item": {"type": "result", "is_error": True, "api_error_status": 400, + "result": f"API Error: 400 {PROSE}"}}, + {"error": {"type": "result", "is_error": True, "api_error_status": 400, + "result": f"API Error: 400 {PROSE}"}}, + [{"type": "result", "is_error": True, "api_error_status": 400, + "result": f"API Error: 400 {PROSE}"}], ) def test_no_model_authored_subtree_reaches_the_log(self): @@ -341,10 +354,13 @@ def test_a_provider_error_beside_the_arguments_is_still_read(self): class ClaudeResultShapeTest(unittest.TestCase): """Claude Code reports an API failure as a result object. - The cause sits in `result` — the same field its review prose uses — so it - is NOT printed. `is_error` is, so the line cannot read as a success. This - pins the documented gap: see the per-lane note in - `docs/callers/cursor-review.md`. + `subtype` reads `success` on a failed call, so without `is_error` the line + would read as a success that names nothing. The cause sits in `result` — + a field the model also writes — and is printed under the three gates in + `_api_error_cause`. That is the ONE exception to withholding model-written + fields, approved deliberately on #402, so each gate is pinned separately: + a gate that quietly stopped holding would widen the channel without + changing any other test. """ FAILURE = { @@ -355,15 +371,135 @@ class ClaudeResultShapeTest(unittest.TestCase): } def test_the_failure_is_not_reported_as_a_success(self): - out = joined(self.FAILURE) - self.assertIn("is_error=True", out) + self.assertIn("is_error=True", joined(self.FAILURE)) - def test_the_cause_in_result_is_withheld(self): - self.assertNotIn("credit balance", joined(self.FAILURE)) + def test_the_cause_is_named(self): + out = joined(self.FAILURE) + self.assertIn("credit balance is too low", out) def test_is_error_is_allowlisted(self): self.assertIn("is_error", agent_error.ERROR_FIELDS) + # --- the gates --- + + def test_review_prose_without_the_preamble_is_withheld(self): + # The normal case by far: `result` holds the agent's review. + payload = dict(self.FAILURE, result=f"I reviewed the diff. {PROSE}") + self.assertNotIn(PROSE, joined(payload)) + + def test_a_message_without_a_status_code_yields_nothing(self): + # Fails safe rather than guessing; widen only against a real message. + payload = dict(self.FAILURE, result=f"API Error: {PROSE}") + self.assertNotIn(PROSE, joined(payload)) + + def test_the_cause_needs_is_error(self): + # The object must still look like an error by ANOTHER signal, or the + # extractor is never reached and the test proves nothing about the + # gate. `subtype` carrying "error" and a 5xx status are those signals; + # `type` stays `result` so only `is_error` is under test. + for other_signal in ({"subtype": "error_max_turns"}, {"status": 500}): + payload = dict(self.FAILURE, **other_signal) + payload["is_error"] = False + payload["result"] = f"API Error: 400 {PROSE}" + with self.subTest(signal=other_signal): + out = joined(payload) + self.assertNotIn(PROSE, out) + # ...and the object WAS recognised as an error, so the only + # thing stopping the cause is the gate under test. + self.assertIn("error: ", out) + + def test_a_structured_result_is_never_flattened(self): + payload = dict(self.FAILURE, result={"blocks": [{"text": PROSE}]}) + self.assertNotIn(PROSE, joined(payload)) + + def test_the_cause_is_capped(self): + payload = dict(self.FAILURE, result="API Error: 500 " + "z" * 4000) + line = [l for l in summary(payload) if l.startswith("error: result=")] + self.assertEqual(1, len(line)) + self.assertLessEqual(len(line[0]), len("error: result=") + agent_error.CAUSE_LIMIT) + + def test_the_cause_cannot_carry_a_workflow_command(self): + payload = dict(self.FAILURE, + result="API Error: 429 x::error::forged ##[add-mask]y") + for line in summary(payload): + self.assertNotIn("::", line) + self.assertNotIn("##[", line) + + def test_the_cause_needs_the_top_level_result_type(self): + payload = dict(self.FAILURE, type="error", + result=f"API Error: 400 {PROSE}") + out = joined(payload) + self.assertNotIn(PROSE, out) + self.assertIn("error: ", out) + + def test_only_an_ascii_status_passes_the_preamble(self): + # `\d` would accept Arabic-Indic digits. + payload = dict(self.FAILURE, result=f"API Error: \u0664\u0660\u0660 {PROSE}") + self.assertNotIn(PROSE, joined(payload)) + + def test_a_boolean_api_error_status_is_not_the_cli_marker(self): + payload = dict(self.FAILURE, api_error_status=True, result=PROSE) + self.assertNotIn(PROSE, joined(payload)) + + # --- the key screen --- + + def test_a_key_prefixed_cause_is_withheld(self): + payload = dict(self.FAILURE, result="API Error: 401 sk-ant-api03-" + "A" * 90) + out = joined(payload) + self.assertNotIn("sk-ant", out) + self.assertIn("result=(withheld", out) + + def test_a_slice_of_the_real_key_is_withheld(self): + # Truncated by the cap, and offset past its prefix: Actions' mask + # matches neither, so the screen must. + key = "sk-ant-api03-" + "Kq7xZ2pLm9Vw" * 8 + cases = (key, key[20:], key[30:60]) + with mock.patch.dict(os.environ, {"ANTHROPIC_API_KEY": key}): + for leaked in cases: + with self.subTest(leaked=leaked[:12]): + payload = dict(self.FAILURE, result=f"API Error: 401 {leaked}") + out = joined(payload) + self.assertNotIn(key[30:46], out) + self.assertIn("result=(withheld", out) + + def test_the_screen_passes_an_ordinary_cause_with_a_key_set(self): + with mock.patch.dict(os.environ, {"ANTHROPIC_API_KEY": "sk-ant-api03-" + "Z" * 90}): + self.assertIn("credit balance is too low", joined(self.FAILURE)) + + def test_the_exception_is_one_field_only(self): + # Guards against the gate being reused for another prose field later. + self.assertEqual("result", agent_error.CAUSE_FIELD) + + +class ClaudeRealFailureTest(unittest.TestCase): + """A real failure, not a hand-written one. + + Captured from `@anthropic-ai/claude-code@2.1.286` (the pinned CLI) run with + an invalid ANTHROPIC_API_KEY and `--output-format json`, trimmed to the + fields this reads. Its `result` carries NO `API Error: ` preamble + — a gate on the preamble alone missed exactly the outage it was for — and + the status sits in the CLI-written `api_error_status`. + """ + + CAPTURED = { + "type": "result", "subtype": "success", "is_error": True, + "api_error_status": 401, "terminal_reason": "api_error", + "num_turns": 1, "duration_api_ms": 0, "total_cost_usd": 0, + "result": "Invalid API key \u00b7 Fix external API key", + } + + def test_status_and_reason_are_printed(self): + out = joined(self.CAPTURED) + self.assertIn("api_error_status=401", out) + self.assertIn("terminal_reason=api_error", out) + + def test_the_cause_is_named(self): + self.assertIn("result=Invalid API key ? Fix external API key", joined(self.CAPTURED)) + + def test_the_cli_scalars_are_allowlisted(self): + for key in ("api_error_status", "terminal_reason"): + self.assertIn(key, agent_error.ERROR_FIELDS) + class TailReadTest(unittest.TestCase): """The explaining event is the LAST thing an agent writes.""" diff --git a/docs/callers/cursor-review.md b/docs/callers/cursor-review.md index e449d860..33035a47 100644 --- a/docs/callers/cursor-review.md +++ b/docs/callers/cursor-review.md @@ -241,19 +241,31 @@ differently. **OpenAI/codex** puts the cause in a structured error event, so a quota or rate-limit failure names itself — `type=insufficient_quota code=credit_balance_exhausted` and the provider's own message. **Anthropic/claude** reports an API-level failure as a result object whose -cause sits in `result`, which is the same field its review prose uses: that -field is not allowlisted and is not printed, so the Anthropic lane tells you -`is_error=true` and the event census but **not** the cause. Reading it means -re-running the cell, or checking the provider's status/billing directly. - -That is a deliberate trade, not an oversight: `result` is attacker-steerable -(the agent reads PR-authored text), so printing it would hand a prompt-injected -agent a channel into the run log, which is the exact thing withholding these -files prevents. A cell that ran but submitted nothing shows up as a census with -no error object. Nothing else is printed and the raw output is not uploaded, so -that step is the whole diagnosis — if it is not enough, the fix is to widen the -allowlist in `agent-error.py` to a field the CLI owns, not to publish the -transcript. +`subtype` reads `success` — so `is_error=true` is what tells you it failed — +and whose cause sits in `result`, the same field its review prose uses. The +CLI's own `api_error_status` and `terminal_reason` are printed beside it, and +the cause itself **is** printed, under gates: only from the top-level result +object with `is_error` true; only when the CLI marked it an API failure (a +numeric `api_error_status`, or an `API Error: ` preamble); never when +it contains an Anthropic key or any slice of the step's own; and cut to 120 +sanitized characters. A bad key, for example, prints +`api_error_status=401 terminal_reason=api_error` and +`result=Invalid API key ? Fix external API key`. + +This is the one place a model-written field reaches the log, and it is a +knowing trade rather than an oversight. A prompt-injected agent *can* forge +the preamble, so treat it as a bounded, documented channel — ~120 sanitized +characters that cannot carry a workflow command or the API key, but can carry +other text — not a safe one. The alternative was the lane reporting +`subtype=success` and naming nothing, since the cause appears nowhere else in +the file and the file is not uploaded. A failure with neither marker prints no +cause rather than a guess, so widen the gate only against a real message. + +A cell that ran but submitted nothing prints whatever error objects (and, +on the Anthropic lane, the gated cause) it recognized, plus the census; when +no error object is found, the census is all you get. Nothing else is printed +and the raw output is not uploaded, so that step is the whole diagnosis — if it is not enough, prefer widening the +allowlist to a field the CLI owns over publishing the transcript. **Applying the label does not guarantee a run.** If the event was swallowed, remove the label, confirm it is gone, then re-add it.