From 16df226eb84cd45cff12282fffb9fa37691d393c Mon Sep 17 00:00:00 2001 From: Matt Miller Date: Fri, 9 Oct 2026 19:42:02 -0700 Subject: [PATCH 1/3] feat(cursor-review): name the Anthropic lane's failure cause, under three gates MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The panel's finding stands: Claude Code reports an API-level failure as `{"type":"result","subtype":"success","is_error":true,"result":"API Error: 400 ..."}`. `subtype` reads SUCCESS on a failed call, and the cause sits only in `result` — a field the model also writes. b537a26 took the safe half (print `is_error`, document the gap); this takes the other half, as the reviewer's first option and on the maintainer's call. So this is the ONE place a model-written field reaches the log, and it is gated three ways: `is_error` must be true, the value must open with the CLI's own `API Error: ` preamble, and it is cut to 120 sanitized characters. A prompt-injected agent CAN forge that preamble, so it is a bounded, documented channel — ~120 sanitized characters that cannot carry a workflow command — not a safe one. The alternative was a lane that reports `subtype=success` and names nothing, since the cause appears nowhere else in the file and the file is not uploaded. A message carrying no status code prints no cause rather than a guess. Widen that pattern against a real message, never speculatively. Each gate is pinned separately, because a gate that quietly stopped holding would widen the channel without changing any other test. All six mutations of them are caught — including one that first passed for the wrong reason: with `is_error` false the fixture no longer looked like an error at all, so the extractor was never reached and the test proved nothing. It now carries an independent error signal. The docstring's claim that no model-written field reaches the log, the README row and the caller page all stated the opposite of this behaviour and are corrected rather than left to mislead — the same overstatement the review caught the first time. Co-Authored-By: Claude Opus 5 --- .github/cursor-review/README.md | 2 +- .github/cursor-review/agent-error.py | 47 +++++++++++++ .../cursor-review/tests/test_agent_error.py | 70 ++++++++++++++++--- docs/callers/cursor-review.md | 32 +++++---- 4 files changed, 129 insertions(+), 22 deletions(-) diff --git a/.github/cursor-review/README.md b/.github/cursor-review/README.md index a03be6c4..c3b848d4 100644 --- a/.github/cursor-review/README.md +++ b/.github/cursor-review/README.md @@ -256,7 +256,7 @@ the consolidated review's panel table. See [Panel integrity](#panel-integrity). | [`build-ledger.py`](build-ledger.py) | Builds the **prior-review ledger** — what earlier rounds raised on this PR and how the author answered — and splices it into the panel/judge prompts. Untrusted prose is defanged twice over: fence-opening lines are rewritten, and continuation lines of quoted prose are prefixed ` |` so no imported text can sit at a field's indent and forge one. Also the prompt splicer, so the no-ledger path is byte-identical to the pre-ledger prompt. | | [`fence-diff.py`](fence-diff.py) | Wraps the reviewed diff (plus the incremental hunks, and the judge's panel-findings block) in `=== BEGIN/END DIFF ===` fences. The diff is attacker-authored PR bytes, so static literal fences are not a control; the nonce is what a PR cannot forge. Each prompt-build step mints its OWN nonce (`mint`), into a shell variable rather than a step `env:` or job output — Actions dumps a step's env map into the public run log, and a per-prompt value means a leak in one job cannot forge a fence in another. Copies the body through **byte for byte** — it never defangs or normalizes the payload. | | [`incremental-diff.py`](incremental-diff.py) | Builds the incremental "hunks new since the last reviewed round" block as the difference between two PR patches (`build`), and verifies the result really is a subset of the reviewed diff (`check`) so the workflow can discard it if it is not. Never a commit range: `git diff LAST_REVIEWED...HEAD` carries every commit a merge of the base branch brought in — see [the ledger section](#the-prior-review-ledger-and-the-repeat-policy). | -| [`agent-error.py`](agent-error.py) | Run by each **direct-API** cell's **Report cell outcome** step, and only when that cell did not submit: prints WHY it failed. Both direct lanes send their agent's structured output to a file (`codex-events.jsonl`, `claude-result.json`) that is never echoed, because it also holds the agent's prose — so an API-level failure, which `codex exec --json` reports as an event on **stdout**, used to leave the log showing a non-zero exit and nothing else, and the lane's failures had to be inferred from job DURATION (a real `insufficient_quota` / `credit_balance_exhausted` outage was discarded that way). It prints an allowlist of short provider metadata (`type`, `subtype`, `is_error`, `code`, `param`, `status`, `message`) from objects that look like errors, plus a census of event types. **Two** allowlists, and the second is the load-bearing one: restricting the FIELDS is not enough, because the agent writes part of this file — it chooses the `arguments` of its own MCP tool calls — so a prompt-injected one could plant `{"error": {"message": "…"}}` there and a walk that descended into every value would print it. **Descent** is therefore an allowlist too (`ENVELOPE_KEYS`: `error`, `last_error`, `item`, `response`, `data`, `detail`), so the model's writable subtrees are never reached rather than denied one by one; widen it only to a key a CLI owns. An `error` KEY is read on its own terms, since a provider's error object need not describe itself as an error (OpenAI's says `insufficient_quota`). Values are sanitized by REPLACING disallowed characters, never deleting them, and both the `::` and legacy `##[command]` forms are broken with a single-pass regex (`str.replace` does not rescan its own output, so `a:::b` defeated it). Reads the **tail** of the file, not the head: the event that explains a failure is the last thing an agent writes. Exits **0 on anything** — missing, truncated, non-JSON, invalid UTF-8, or nested deeply enough that `json.loads` raises `RecursionError` (a RuntimeError, not a ValueError; 400 KB of `[` suffices) — because it runs after the cell already failed and an abort under the step's `bash -e` would cost the status line. **Known gap:** Claude Code puts an API failure's cause in `result`, the same field its review prose uses, so the Anthropic lane reports `is_error=true` and the census but not the cause — a deliberate trade, since that field is attacker-steerable. The raw file is not uploaded either: the Anthropic lane screens `findings.json` for a leaked API key before publishing it, and shipping the unscreened transcript alongside would route the agent's prose past that guard. | +| [`agent-error.py`](agent-error.py) | Run by each **direct-API** cell's **Report cell outcome** step, and only when that cell did not submit: prints WHY it failed. Both direct lanes send their agent's structured output to a file (`codex-events.jsonl`, `claude-result.json`) that is never echoed, because it also holds the agent's prose — so an API-level failure, which `codex exec --json` reports as an event on **stdout**, used to leave the log showing a non-zero exit and nothing else, and the lane's failures had to be inferred from job DURATION (a real `insufficient_quota` / `credit_balance_exhausted` outage was discarded that way). It prints an allowlist of short provider metadata (`type`, `subtype`, `is_error`, `code`, `param`, `status`, `message`) from objects that look like errors, plus a census of event types. **Two** allowlists, and the second is the load-bearing one: restricting the FIELDS is not enough, because the agent writes part of this file — it chooses the `arguments` of its own MCP tool calls — so a prompt-injected one could plant `{"error": {"message": "…"}}` there and a walk that descended into every value would print it. **Descent** is therefore an allowlist too (`ENVELOPE_KEYS`: `error`, `last_error`, `item`, `response`, `data`, `detail`), so the model's writable subtrees are never reached rather than denied one by one; widen it only to a key a CLI owns. An `error` KEY is read on its own terms, since a provider's error object need not describe itself as an error (OpenAI's says `insufficient_quota`). Values are sanitized by REPLACING disallowed characters, never deleting them, and both the `::` and legacy `##[command]` forms are broken with a single-pass regex (`str.replace` does not rescan its own output, so `a:::b` defeated it). Reads the **tail** of the file, not the head: the event that explains a failure is the last thing an agent writes. Exits **0 on anything** — missing, truncated, non-JSON, invalid UTF-8, or nested deeply enough that `json.loads` raises `RecursionError` (a RuntimeError, not a ValueError; 400 KB of `[` suffices) — because it runs after the cell already failed and an abort under the step's `bash -e` would cost the status line. **One stated exception** to withholding model-written fields: Claude Code puts an API failure's cause in `result` — the same field its review prose uses — and reports `subtype=success` on a failed call, so the cause is printed under three gates (`is_error` true, the CLI's own `API Error: ` preamble, 120 sanitized chars). A prompt-injected agent can forge that preamble, so it is a bounded, documented channel, not a safe one; it was taken knowingly because the cause appears nowhere else and the file is not uploaded. A message without a status code prints nothing rather than a guess. The raw file is not uploaded either: the Anthropic lane screens `findings.json` for a leaked API key before publishing it, and shipping the unscreened transcript alongside would route the agent's prose past that guard. | | [`catalog-drift.py`](catalog-drift.py) | Backs the weekly catalog-drift check. Extracts the pins from `cursor-review.yml`, diffs them against raw `cursor-agent models` output, and renders the sticky issue title + body (delisted pins, pins marked NO-ZDR, unpinned same-lab ids, catalog ids from unpinned families, stale audit date). Reports only — it never edits a pin. | ## Adopt it in your repo diff --git a/.github/cursor-review/agent-error.py b/.github/cursor-review/agent-error.py index 4d1a679e..c1184fbc 100644 --- a/.github/cursor-review/agent-error.py +++ b/.github/cursor-review/agent-error.py @@ -18,6 +18,11 @@ (``ERROR_FIELDS``) from objects that look like errors, plus a census of event types. +ONE field is read out of model-written territory — Claude Code's API-failure +cause, which sits in `result` and nowhere else — under the three gates in +``_api_error_cause``. Everything below describes the rule that field is the +stated exception to. + TWO allowlists, and the second one is the load-bearing one. Restricting the FIELDS is not enough on its own, because a reviewer agent writes part of this file: it chooses the `arguments` of its own MCP tool calls. A prompt-injected @@ -87,6 +92,13 @@ MAX_DEPTH = 12 MAX_NODES = 20000 +# Claude Code's API-failure cause sits in `result`, a field the model also +# writes. Printing it is THE one exception to "no model-written field reaches +# the log" — see `_api_error_cause`, which gates it three ways. +CAUSE_FIELD = "result" +CAUSE_LIMIT = 120 +_API_ERROR = re.compile(r"^API Error: \d{3} ") + _SAFE = re.compile(r"[^A-Za-z0-9 .,:;_/@+()\[\]{}='\"!?#$%^&*|~`<>-]") # A colon followed by a colon, and a `#` that begins a `##[`. Lookahead, so one # pass breaks every run however long: `":::"` -> `": : :"`, `"###["` -> `"## #["`. @@ -146,6 +158,38 @@ def _fields(obj: dict) -> str: return " ".join(parts) +def _api_error_cause(obj: dict) -> str: + """Claude Code's API-failure cause, read out of a model-written field. + + THE one exception to the rule that no model-written field reaches the log, + and it is deliberate rather than an oversight. Claude Code reports an + API-level failure as `{"type":"result","subtype":"success","is_error":true, + "result":"API Error: 400 ..."}` — note `subtype`, which reads as SUCCESS — + and the cause appears nowhere else in the file. Since the file is not + uploaded either, an Anthropic-lane outage was otherwise undiagnosable + without re-running the cell. + + So it is gated three ways: the object must declare `is_error` true, the + value must be a string opening with the CLI's own `API Error: ` + preamble, and it is cut to ``CAUSE_LIMIT`` and sanitized like every other + value. A prompt-injected agent CAN forge that preamble, so this is a + BOUNDED and documented channel, not a safe one — roughly 120 sanitized + characters that cannot carry a workflow command. That trade was taken + knowingly (#402); the alternative was a lane that reports `subtype=success` + and names nothing. + + A failure whose message does not carry a status code yields nothing rather + than a guess, which is the safe direction: widen the pattern only against + a real message, never speculatively. + """ + if obj.get("is_error") is not True: + return "" + value = obj.get(CAUSE_FIELD) + if not isinstance(value, str) or not _API_ERROR.match(value): + return "" + return f"{CAUSE_FIELD}={sanitize(value, CAUSE_LIMIT)}" + + def walk(node, out: list, types: dict, state: dict, depth: int = 0) -> None: """Collect error objects and a census of `type` values. Bounded. @@ -171,6 +215,9 @@ def walk(node, out: list, types: dict, state: dict, depth: int = 0) -> None: rendered = _fields(node) if rendered and rendered not in out: out.append(rendered) + cause = _api_error_cause(node) + if cause and cause not in out: + out.append(cause) # An `error` KEY is the canonical envelope and must be read on its own # terms. The provider's error object does not have to describe ITSELF as an # error — OpenAI's is `{"error": {"type": "insufficient_quota", "code": diff --git a/.github/cursor-review/tests/test_agent_error.py b/.github/cursor-review/tests/test_agent_error.py index d807a728..54048549 100644 --- a/.github/cursor-review/tests/test_agent_error.py +++ b/.github/cursor-review/tests/test_agent_error.py @@ -121,6 +121,9 @@ class AllowlistTest(unittest.TestCase): ) def test_no_prose_key_is_printed_from_an_error_object(self): + # `result` is covered too: bare prose does not pass + # `_api_error_cause`'s preamble gate. The one value that DOES pass is + # pinned in ClaudeResultShapeTest, deliberately in one place. for key in self.PROSE_KEYS: with self.subTest(key=key): payload = {"type": "error", "is_error": True, key: PROSE} @@ -341,10 +344,13 @@ def test_a_provider_error_beside_the_arguments_is_still_read(self): class ClaudeResultShapeTest(unittest.TestCase): """Claude Code reports an API failure as a result object. - The cause sits in `result` — the same field its review prose uses — so it - is NOT printed. `is_error` is, so the line cannot read as a success. This - pins the documented gap: see the per-lane note in - `docs/callers/cursor-review.md`. + `subtype` reads `success` on a failed call, so without `is_error` the line + would read as a success that names nothing. The cause sits in `result` — + a field the model also writes — and is printed under the three gates in + `_api_error_cause`. That is the ONE exception to withholding model-written + fields, approved deliberately on #402, so each gate is pinned separately: + a gate that quietly stopped holding would widen the channel without + changing any other test. """ FAILURE = { @@ -355,15 +361,63 @@ class ClaudeResultShapeTest(unittest.TestCase): } def test_the_failure_is_not_reported_as_a_success(self): - out = joined(self.FAILURE) - self.assertIn("is_error=True", out) + self.assertIn("is_error=True", joined(self.FAILURE)) - def test_the_cause_in_result_is_withheld(self): - self.assertNotIn("credit balance", joined(self.FAILURE)) + def test_the_cause_is_named(self): + out = joined(self.FAILURE) + self.assertIn("credit balance is too low", out) def test_is_error_is_allowlisted(self): self.assertIn("is_error", agent_error.ERROR_FIELDS) + # --- the gates --- + + def test_review_prose_without_the_preamble_is_withheld(self): + # The normal case by far: `result` holds the agent's review. + payload = dict(self.FAILURE, result=f"I reviewed the diff. {PROSE}") + self.assertNotIn(PROSE, joined(payload)) + + def test_a_message_without_a_status_code_yields_nothing(self): + # Fails safe rather than guessing; widen only against a real message. + payload = dict(self.FAILURE, result=f"API Error: {PROSE}") + self.assertNotIn(PROSE, joined(payload)) + + def test_the_cause_needs_is_error(self): + # The object must still look like an error by ANOTHER signal, or the + # extractor is never reached and the test proves nothing about the + # gate. `type` carrying "error" and a 4xx status are those signals. + for other_signal in ({"type": "error"}, {"type": "x", "status": 500}): + payload = dict(self.FAILURE, **other_signal) + payload["is_error"] = False + payload["result"] = f"API Error: 400 {PROSE}" + with self.subTest(signal=other_signal): + out = joined(payload) + self.assertNotIn(PROSE, out) + # ...and the object WAS recognised as an error, so the only + # thing stopping the cause is the gate under test. + self.assertIn("error: ", out) + + def test_a_structured_result_is_never_flattened(self): + payload = dict(self.FAILURE, result={"blocks": [{"text": PROSE}]}) + self.assertNotIn(PROSE, joined(payload)) + + def test_the_cause_is_capped(self): + payload = dict(self.FAILURE, result="API Error: 500 " + "z" * 4000) + line = [l for l in summary(payload) if l.startswith("error: result=")] + self.assertEqual(1, len(line)) + self.assertLessEqual(len(line[0]), len("error: result=") + agent_error.CAUSE_LIMIT) + + def test_the_cause_cannot_carry_a_workflow_command(self): + payload = dict(self.FAILURE, + result="API Error: 429 x::error::forged ##[add-mask]y") + for line in summary(payload): + self.assertNotIn("::", line) + self.assertNotIn("##[", line) + + def test_the_exception_is_one_field_only(self): + # Guards against the gate being reused for another prose field later. + self.assertEqual("result", agent_error.CAUSE_FIELD) + class TailReadTest(unittest.TestCase): """The explaining event is the LAST thing an agent writes.""" diff --git a/docs/callers/cursor-review.md b/docs/callers/cursor-review.md index e449d860..36d30702 100644 --- a/docs/callers/cursor-review.md +++ b/docs/callers/cursor-review.md @@ -241,19 +241,25 @@ differently. **OpenAI/codex** puts the cause in a structured error event, so a quota or rate-limit failure names itself — `type=insufficient_quota code=credit_balance_exhausted` and the provider's own message. **Anthropic/claude** reports an API-level failure as a result object whose -cause sits in `result`, which is the same field its review prose uses: that -field is not allowlisted and is not printed, so the Anthropic lane tells you -`is_error=true` and the event census but **not** the cause. Reading it means -re-running the cell, or checking the provider's status/billing directly. - -That is a deliberate trade, not an oversight: `result` is attacker-steerable -(the agent reads PR-authored text), so printing it would hand a prompt-injected -agent a channel into the run log, which is the exact thing withholding these -files prevents. A cell that ran but submitted nothing shows up as a census with -no error object. Nothing else is printed and the raw output is not uploaded, so -that step is the whole diagnosis — if it is not enough, the fix is to widen the -allowlist in `agent-error.py` to a field the CLI owns, not to publish the -transcript. +`subtype` reads `success` — so `is_error=true` is what tells you it failed — +and whose cause sits in `result`, the same field its review prose uses. That +cause **is** printed, under three gates: `is_error` must be true, the value +must open with the CLI's own `API Error: ` preamble, and it is cut to +120 sanitized characters. + +This is the one place a model-written field reaches the log, and it is a +knowing trade rather than an oversight. A prompt-injected agent *can* forge +that preamble, so treat it as a bounded, documented channel — ~120 sanitized +characters that cannot carry a workflow command — not a safe one. The +alternative was the lane reporting `subtype=success` and naming nothing, since +the cause appears nowhere else in the file and the file is not uploaded. A +failure whose message carries no status code prints no cause rather than a +guess, so widen that pattern only against a real message. + +A cell that ran but submitted nothing shows up as a census with no error +object. Nothing else is printed and the raw output is not uploaded, so that +step is the whole diagnosis — if it is not enough, prefer widening the +allowlist to a field the CLI owns over publishing the transcript. **Applying the label does not guarantee a run.** If the event was swallowed, remove the label, confirm it is gone, then re-add it. From ba75547c2375a22c838858da82ee8030bcc1d07b Mon Sep 17 00:00:00 2001 From: Matt Miller Date: Sat, 10 Oct 2026 03:03:44 +0000 Subject: [PATCH 2/3] docs(cursor-review): a no-submit cell can still print error objects; census-only is the no-error-object case Co-Authored-By: Claude Opus 5.5 --- docs/callers/cursor-review.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/callers/cursor-review.md b/docs/callers/cursor-review.md index 36d30702..180bd5a5 100644 --- a/docs/callers/cursor-review.md +++ b/docs/callers/cursor-review.md @@ -256,9 +256,10 @@ the cause appears nowhere else in the file and the file is not uploaded. A failure whose message carries no status code prints no cause rather than a guess, so widen that pattern only against a real message. -A cell that ran but submitted nothing shows up as a census with no error -object. Nothing else is printed and the raw output is not uploaded, so that -step is the whole diagnosis — if it is not enough, prefer widening the +A cell that ran but submitted nothing prints whatever error objects (and, +on the Anthropic lane, the gated cause) it recognized, plus the census; when +no error object is found, the census is all you get. Nothing else is printed +and the raw output is not uploaded, so that step is the whole diagnosis — if it is not enough, prefer widening the allowlist to a field the CLI owns over publishing the transcript. **Applying the label does not guarantee a run.** If the event was swallowed, From 8457fe72bb912ea4eb14a974119005b3bb442747 Mon Sep 17 00:00:00 2001 From: Matt Miller Date: Sat, 10 Oct 2026 03:11:14 +0000 Subject: [PATCH 3/3] fix(cursor-review): gate the Anthropic cause on the real CLI shape, and screen it for the API key MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Addresses the panel round on 16df226: - A real failure from the pinned CLI (claude-code 2.1.286, invalid key, --output-format json) is `result: "Invalid API key · Fix external API key"` with NO `API Error: ` preamble — the status sits in the CLI-written `api_error_status`. The preamble-only gate missed exactly the outage it was for. Accept a numeric `api_error_status` as the CLI's marker, allowlist `api_error_status` / `terminal_reason`, and pin the captured object as a fixture. - Withhold a cause holding the `sk-ant-` prefix or any 16-char slice of the step's own ANTHROPIC_API_KEY — the screen findings.json gets, which a 120-char cut would otherwise slip past Actions' exact-value mask. - Only the top-level `{"type":"result"}` object yields a cause; nested and codex-lane objects never do. - `[0-9]{3}`, not `\d{3}` (Unicode digits). Co-Authored-By: Claude Opus 5.5 --- .github/cursor-review/README.md | 2 +- .github/cursor-review/agent-error.py | 80 +++++++++++++---- .../cursor-review/tests/test_agent_error.py | 90 ++++++++++++++++++- docs/callers/cursor-review.md | 25 +++--- 4 files changed, 163 insertions(+), 34 deletions(-) diff --git a/.github/cursor-review/README.md b/.github/cursor-review/README.md index c3b848d4..9586dfe9 100644 --- a/.github/cursor-review/README.md +++ b/.github/cursor-review/README.md @@ -256,7 +256,7 @@ the consolidated review's panel table. See [Panel integrity](#panel-integrity). | [`build-ledger.py`](build-ledger.py) | Builds the **prior-review ledger** — what earlier rounds raised on this PR and how the author answered — and splices it into the panel/judge prompts. Untrusted prose is defanged twice over: fence-opening lines are rewritten, and continuation lines of quoted prose are prefixed ` |` so no imported text can sit at a field's indent and forge one. Also the prompt splicer, so the no-ledger path is byte-identical to the pre-ledger prompt. | | [`fence-diff.py`](fence-diff.py) | Wraps the reviewed diff (plus the incremental hunks, and the judge's panel-findings block) in `=== BEGIN/END DIFF ===` fences. The diff is attacker-authored PR bytes, so static literal fences are not a control; the nonce is what a PR cannot forge. Each prompt-build step mints its OWN nonce (`mint`), into a shell variable rather than a step `env:` or job output — Actions dumps a step's env map into the public run log, and a per-prompt value means a leak in one job cannot forge a fence in another. Copies the body through **byte for byte** — it never defangs or normalizes the payload. | | [`incremental-diff.py`](incremental-diff.py) | Builds the incremental "hunks new since the last reviewed round" block as the difference between two PR patches (`build`), and verifies the result really is a subset of the reviewed diff (`check`) so the workflow can discard it if it is not. Never a commit range: `git diff LAST_REVIEWED...HEAD` carries every commit a merge of the base branch brought in — see [the ledger section](#the-prior-review-ledger-and-the-repeat-policy). | -| [`agent-error.py`](agent-error.py) | Run by each **direct-API** cell's **Report cell outcome** step, and only when that cell did not submit: prints WHY it failed. Both direct lanes send their agent's structured output to a file (`codex-events.jsonl`, `claude-result.json`) that is never echoed, because it also holds the agent's prose — so an API-level failure, which `codex exec --json` reports as an event on **stdout**, used to leave the log showing a non-zero exit and nothing else, and the lane's failures had to be inferred from job DURATION (a real `insufficient_quota` / `credit_balance_exhausted` outage was discarded that way). It prints an allowlist of short provider metadata (`type`, `subtype`, `is_error`, `code`, `param`, `status`, `message`) from objects that look like errors, plus a census of event types. **Two** allowlists, and the second is the load-bearing one: restricting the FIELDS is not enough, because the agent writes part of this file — it chooses the `arguments` of its own MCP tool calls — so a prompt-injected one could plant `{"error": {"message": "…"}}` there and a walk that descended into every value would print it. **Descent** is therefore an allowlist too (`ENVELOPE_KEYS`: `error`, `last_error`, `item`, `response`, `data`, `detail`), so the model's writable subtrees are never reached rather than denied one by one; widen it only to a key a CLI owns. An `error` KEY is read on its own terms, since a provider's error object need not describe itself as an error (OpenAI's says `insufficient_quota`). Values are sanitized by REPLACING disallowed characters, never deleting them, and both the `::` and legacy `##[command]` forms are broken with a single-pass regex (`str.replace` does not rescan its own output, so `a:::b` defeated it). Reads the **tail** of the file, not the head: the event that explains a failure is the last thing an agent writes. Exits **0 on anything** — missing, truncated, non-JSON, invalid UTF-8, or nested deeply enough that `json.loads` raises `RecursionError` (a RuntimeError, not a ValueError; 400 KB of `[` suffices) — because it runs after the cell already failed and an abort under the step's `bash -e` would cost the status line. **One stated exception** to withholding model-written fields: Claude Code puts an API failure's cause in `result` — the same field its review prose uses — and reports `subtype=success` on a failed call, so the cause is printed under three gates (`is_error` true, the CLI's own `API Error: ` preamble, 120 sanitized chars). A prompt-injected agent can forge that preamble, so it is a bounded, documented channel, not a safe one; it was taken knowingly because the cause appears nowhere else and the file is not uploaded. A message without a status code prints nothing rather than a guess. The raw file is not uploaded either: the Anthropic lane screens `findings.json` for a leaked API key before publishing it, and shipping the unscreened transcript alongside would route the agent's prose past that guard. | +| [`agent-error.py`](agent-error.py) | Run by each **direct-API** cell's **Report cell outcome** step, and only when that cell did not submit: prints WHY it failed. Both direct lanes send their agent's structured output to a file (`codex-events.jsonl`, `claude-result.json`) that is never echoed, because it also holds the agent's prose — so an API-level failure, which `codex exec --json` reports as an event on **stdout**, used to leave the log showing a non-zero exit and nothing else, and the lane's failures had to be inferred from job DURATION (a real `insufficient_quota` / `credit_balance_exhausted` outage was discarded that way). It prints an allowlist of short provider metadata (`type`, `subtype`, `is_error`, `code`, `param`, `status`, `message`, and Claude Code's `api_error_status` / `terminal_reason`) from objects that look like errors, plus a census of event types. **Two** allowlists, and the second is the load-bearing one: restricting the FIELDS is not enough, because the agent writes part of this file — it chooses the `arguments` of its own MCP tool calls — so a prompt-injected one could plant `{"error": {"message": "…"}}` there and a walk that descended into every value would print it. **Descent** is therefore an allowlist too (`ENVELOPE_KEYS`: `error`, `last_error`, `item`, `response`, `data`, `detail`), so the model's writable subtrees are never reached rather than denied one by one; widen it only to a key a CLI owns. An `error` KEY is read on its own terms, since a provider's error object need not describe itself as an error (OpenAI's says `insufficient_quota`). Values are sanitized by REPLACING disallowed characters, never deleting them, and both the `::` and legacy `##[command]` forms are broken with a single-pass regex (`str.replace` does not rescan its own output, so `a:::b` defeated it). Reads the **tail** of the file, not the head: the event that explains a failure is the last thing an agent writes. Exits **0 on anything** — missing, truncated, non-JSON, invalid UTF-8, or nested deeply enough that `json.loads` raises `RecursionError` (a RuntimeError, not a ValueError; 400 KB of `[` suffices) — because it runs after the cell already failed and an abort under the step's `bash -e` would cost the status line. **One stated exception** to withholding model-written fields: Claude Code puts an API failure's cause in `result` — the same field its review prose uses — and reports `subtype=success` on a failed call, so the cause is printed under gates: only from the top-level `{"type":"result"}` object with `is_error` true; only when the CLI marked it an API failure (a numeric `api_error_status` — the pinned CLI's real 401 is `Invalid API key · Fix external API key`, with no preamble — or an `API Error: ` preamble); never when it holds the `sk-ant-` prefix or any 16-char slice of the step's own `ANTHROPIC_API_KEY` (the screen `findings.json` gets, which a 120-char cut would otherwise slip past Actions' exact-value mask); and cut to 120 sanitized chars. A prompt-injected agent can forge the preamble, so it is a bounded, documented channel, not a safe one; it was taken knowingly because the cause appears nowhere else and the file is not uploaded. A failure with neither marker prints nothing rather than a guess. The raw file is not uploaded either: the Anthropic lane screens `findings.json` for a leaked API key before publishing it, and shipping the unscreened transcript alongside would route the agent's prose past that guard. | | [`catalog-drift.py`](catalog-drift.py) | Backs the weekly catalog-drift check. Extracts the pins from `cursor-review.yml`, diffs them against raw `cursor-agent models` output, and renders the sticky issue title + body (delisted pins, pins marked NO-ZDR, unpinned same-lab ids, catalog ids from unpinned families, stale audit date). Reports only — it never edits a pin. | ## Adopt it in your repo diff --git a/.github/cursor-review/agent-error.py b/.github/cursor-review/agent-error.py index c1184fbc..f467ed03 100644 --- a/.github/cursor-review/agent-error.py +++ b/.github/cursor-review/agent-error.py @@ -19,7 +19,7 @@ types. ONE field is read out of model-written territory — Claude Code's API-failure -cause, which sits in `result` and nowhere else — under the three gates in +cause, which sits in `result` and nowhere else — under the gates in ``_api_error_cause``. Everything below describes the rule that field is the stated exception to. @@ -70,6 +70,9 @@ ERROR_FIELDS = ( "type", "subtype", "is_error", "code", "param", "status", "status_code", "http_status", "message", + # Claude Code's own scalars on a failed result object, beside the + # model-written `result`: the HTTP status and `api_error` respectively. + "api_error_status", "terminal_reason", ) # The ONLY keys this descends through — the envelope a CLI writes around the @@ -94,10 +97,17 @@ # Claude Code's API-failure cause sits in `result`, a field the model also # writes. Printing it is THE one exception to "no model-written field reaches -# the log" — see `_api_error_cause`, which gates it three ways. +# the log" — see `_api_error_cause`, which gates it. CAUSE_FIELD = "result" CAUSE_LIMIT = 120 -_API_ERROR = re.compile(r"^API Error: \d{3} ") +# `[0-9]`, not `\d`: in a str pattern `\d` matches any Unicode digit. +_API_ERROR = re.compile(r"^API Error: [0-9]{3} ") +# An Anthropic key's prefix, and the window of the REAL key (from the step's +# env) that a cause may not contain. A window rather than the whole key: the +# cut to CAUSE_LIMIT leaves a truncated key, which Actions' exact-value mask +# no longer matches. +_KEY_PREFIX = "sk-ant-" +KEY_WINDOW = 16 _SAFE = re.compile(r"[^A-Za-z0-9 .,:;_/@+()\[\]{}='\"!?#$%^&*|~`<>-]") # A colon followed by a colon, and a `#` that begins a `##[`. Lookahead, so one @@ -158,7 +168,7 @@ def _fields(obj: dict) -> str: return " ".join(parts) -def _api_error_cause(obj: dict) -> str: +def _api_error_cause(obj: dict, top: bool = False) -> str: """Claude Code's API-failure cause, read out of a model-written field. THE one exception to the rule that no model-written field reaches the log, @@ -169,27 +179,59 @@ def _api_error_cause(obj: dict) -> str: uploaded either, an Anthropic-lane outage was otherwise undiagnosable without re-running the cell. - So it is gated three ways: the object must declare `is_error` true, the - value must be a string opening with the CLI's own `API Error: ` - preamble, and it is cut to ``CAUSE_LIMIT`` and sanitized like every other - value. A prompt-injected agent CAN forge that preamble, so this is a - BOUNDED and documented channel, not a safe one — roughly 120 sanitized - characters that cannot carry a workflow command. That trade was taken - knowingly (#402); the alternative was a lane that reports `subtype=success` - and names nothing. - - A failure whose message does not carry a status code yields nothing rather - than a guess, which is the safe direction: widen the pattern only against - a real message, never speculatively. + So it is gated: + + 1. the object is the CLI's own top-level `{"type":"result"}` object with + `is_error` true — never a nested or codex-lane object (``walk`` passes + ``top``), where the shape is not the one this exception covers; + 2. the CLI marked it an API failure: a numeric `api_error_status` (a CLI + scalar the model cannot write — `Invalid API key · Fix external API + key` with status 401 is a real 2.1.286 failure, and carries no + preamble), or a string opening with `API Error: `; + 3. it holds no Anthropic key: not the `sk-ant-` prefix and no + ``KEY_WINDOW``-char slice of the step's own ``ANTHROPIC_API_KEY`` — + the screen `findings.json` gets before publishing, which this path + would otherwise skip; + 4. it is cut to ``CAUSE_LIMIT`` and sanitized like every other value. + + A prompt-injected agent CAN forge the preamble, so this is a BOUNDED and + documented channel, not a safe one — roughly 120 sanitized characters + that cannot carry a workflow command or the key, but can carry other + text. That trade was taken knowingly (#402); the alternative was a lane + that reports `subtype=success` and names nothing. + + A failure with neither marker yields nothing rather than a guess, which is + the safe direction: widen the gate only against a real message, never + speculatively. """ - if obj.get("is_error") is not True: + if not top or obj.get("type") != "result" or obj.get("is_error") is not True: return "" value = obj.get(CAUSE_FIELD) - if not isinstance(value, str) or not _API_ERROR.match(value): + if not isinstance(value, str): return "" + status = obj.get("api_error_status") + cli_marked = isinstance(status, int) and not isinstance(status, bool) + if not cli_marked and not _API_ERROR.match(value): + return "" + if _holds_key(value): + return f"{CAUSE_FIELD}=(withheld: it contains an Anthropic API key or part of one)" return f"{CAUSE_FIELD}={sanitize(value, CAUSE_LIMIT)}" +def _holds_key(value: str) -> bool: + """True when ``value`` carries the key prefix or a slice of the real key.""" + if _KEY_PREFIX in value: + return True + key = os.environ.get("ANTHROPIC_API_KEY", "") + if not key: + return False + if len(key) <= KEY_WINDOW: + return key in value + return any( + key[i : i + KEY_WINDOW] in value for i in range(len(key) - KEY_WINDOW + 1) + ) + + def walk(node, out: list, types: dict, state: dict, depth: int = 0) -> None: """Collect error objects and a census of `type` values. Bounded. @@ -215,7 +257,7 @@ def walk(node, out: list, types: dict, state: dict, depth: int = 0) -> None: rendered = _fields(node) if rendered and rendered not in out: out.append(rendered) - cause = _api_error_cause(node) + cause = _api_error_cause(node, top=depth == 0) if cause and cause not in out: out.append(cause) # An `error` KEY is the canonical envelope and must be read on its own diff --git a/.github/cursor-review/tests/test_agent_error.py b/.github/cursor-review/tests/test_agent_error.py index 54048549..f8440bdf 100644 --- a/.github/cursor-review/tests/test_agent_error.py +++ b/.github/cursor-review/tests/test_agent_error.py @@ -39,6 +39,7 @@ import sys import tempfile import unittest +from unittest import mock SCRIPT = os.path.normpath( os.path.join(os.path.dirname(__file__), "..", "agent-error.py") @@ -121,8 +122,8 @@ class AllowlistTest(unittest.TestCase): ) def test_no_prose_key_is_printed_from_an_error_object(self): - # `result` is covered too: bare prose does not pass - # `_api_error_cause`'s preamble gate. The one value that DOES pass is + # `result` is covered too: bare prose on a non-`result` object does + # not pass `_api_error_cause`'s gates. The one value that DOES pass is # pinned in ClaudeResultShapeTest, deliberately in one place. for key in self.PROSE_KEYS: with self.subTest(key=key): @@ -316,6 +317,15 @@ class ModelAuthoredSubtreeTest(unittest.TestCase): {"type": "result", "is_error": True, "result": {"error": {"message": PROSE}}}, # A model-chosen `type`, which would otherwise reach the census. {"item": {"arguments": {"type": PROSE}}}, + # The Claude result shape, but NOT at the top level: the cause + # exception covers only the CLI's own top-level object. + {"type": "item.completed", + "item": {"type": "result", "is_error": True, "api_error_status": 400, + "result": f"API Error: 400 {PROSE}"}}, + {"error": {"type": "result", "is_error": True, "api_error_status": 400, + "result": f"API Error: 400 {PROSE}"}}, + [{"type": "result", "is_error": True, "api_error_status": 400, + "result": f"API Error: 400 {PROSE}"}], ) def test_no_model_authored_subtree_reaches_the_log(self): @@ -385,8 +395,9 @@ def test_a_message_without_a_status_code_yields_nothing(self): def test_the_cause_needs_is_error(self): # The object must still look like an error by ANOTHER signal, or the # extractor is never reached and the test proves nothing about the - # gate. `type` carrying "error" and a 4xx status are those signals. - for other_signal in ({"type": "error"}, {"type": "x", "status": 500}): + # gate. `subtype` carrying "error" and a 5xx status are those signals; + # `type` stays `result` so only `is_error` is under test. + for other_signal in ({"subtype": "error_max_turns"}, {"status": 500}): payload = dict(self.FAILURE, **other_signal) payload["is_error"] = False payload["result"] = f"API Error: 400 {PROSE}" @@ -414,11 +425,82 @@ def test_the_cause_cannot_carry_a_workflow_command(self): self.assertNotIn("::", line) self.assertNotIn("##[", line) + def test_the_cause_needs_the_top_level_result_type(self): + payload = dict(self.FAILURE, type="error", + result=f"API Error: 400 {PROSE}") + out = joined(payload) + self.assertNotIn(PROSE, out) + self.assertIn("error: ", out) + + def test_only_an_ascii_status_passes_the_preamble(self): + # `\d` would accept Arabic-Indic digits. + payload = dict(self.FAILURE, result=f"API Error: \u0664\u0660\u0660 {PROSE}") + self.assertNotIn(PROSE, joined(payload)) + + def test_a_boolean_api_error_status_is_not_the_cli_marker(self): + payload = dict(self.FAILURE, api_error_status=True, result=PROSE) + self.assertNotIn(PROSE, joined(payload)) + + # --- the key screen --- + + def test_a_key_prefixed_cause_is_withheld(self): + payload = dict(self.FAILURE, result="API Error: 401 sk-ant-api03-" + "A" * 90) + out = joined(payload) + self.assertNotIn("sk-ant", out) + self.assertIn("result=(withheld", out) + + def test_a_slice_of_the_real_key_is_withheld(self): + # Truncated by the cap, and offset past its prefix: Actions' mask + # matches neither, so the screen must. + key = "sk-ant-api03-" + "Kq7xZ2pLm9Vw" * 8 + cases = (key, key[20:], key[30:60]) + with mock.patch.dict(os.environ, {"ANTHROPIC_API_KEY": key}): + for leaked in cases: + with self.subTest(leaked=leaked[:12]): + payload = dict(self.FAILURE, result=f"API Error: 401 {leaked}") + out = joined(payload) + self.assertNotIn(key[30:46], out) + self.assertIn("result=(withheld", out) + + def test_the_screen_passes_an_ordinary_cause_with_a_key_set(self): + with mock.patch.dict(os.environ, {"ANTHROPIC_API_KEY": "sk-ant-api03-" + "Z" * 90}): + self.assertIn("credit balance is too low", joined(self.FAILURE)) + def test_the_exception_is_one_field_only(self): # Guards against the gate being reused for another prose field later. self.assertEqual("result", agent_error.CAUSE_FIELD) +class ClaudeRealFailureTest(unittest.TestCase): + """A real failure, not a hand-written one. + + Captured from `@anthropic-ai/claude-code@2.1.286` (the pinned CLI) run with + an invalid ANTHROPIC_API_KEY and `--output-format json`, trimmed to the + fields this reads. Its `result` carries NO `API Error: ` preamble + — a gate on the preamble alone missed exactly the outage it was for — and + the status sits in the CLI-written `api_error_status`. + """ + + CAPTURED = { + "type": "result", "subtype": "success", "is_error": True, + "api_error_status": 401, "terminal_reason": "api_error", + "num_turns": 1, "duration_api_ms": 0, "total_cost_usd": 0, + "result": "Invalid API key \u00b7 Fix external API key", + } + + def test_status_and_reason_are_printed(self): + out = joined(self.CAPTURED) + self.assertIn("api_error_status=401", out) + self.assertIn("terminal_reason=api_error", out) + + def test_the_cause_is_named(self): + self.assertIn("result=Invalid API key ? Fix external API key", joined(self.CAPTURED)) + + def test_the_cli_scalars_are_allowlisted(self): + for key in ("api_error_status", "terminal_reason"): + self.assertIn(key, agent_error.ERROR_FIELDS) + + class TailReadTest(unittest.TestCase): """The explaining event is the LAST thing an agent writes.""" diff --git a/docs/callers/cursor-review.md b/docs/callers/cursor-review.md index 180bd5a5..33035a47 100644 --- a/docs/callers/cursor-review.md +++ b/docs/callers/cursor-review.md @@ -242,19 +242,24 @@ quota or rate-limit failure names itself — `type=insufficient_quota code=credit_balance_exhausted` and the provider's own message. **Anthropic/claude** reports an API-level failure as a result object whose `subtype` reads `success` — so `is_error=true` is what tells you it failed — -and whose cause sits in `result`, the same field its review prose uses. That -cause **is** printed, under three gates: `is_error` must be true, the value -must open with the CLI's own `API Error: ` preamble, and it is cut to -120 sanitized characters. +and whose cause sits in `result`, the same field its review prose uses. The +CLI's own `api_error_status` and `terminal_reason` are printed beside it, and +the cause itself **is** printed, under gates: only from the top-level result +object with `is_error` true; only when the CLI marked it an API failure (a +numeric `api_error_status`, or an `API Error: ` preamble); never when +it contains an Anthropic key or any slice of the step's own; and cut to 120 +sanitized characters. A bad key, for example, prints +`api_error_status=401 terminal_reason=api_error` and +`result=Invalid API key ? Fix external API key`. This is the one place a model-written field reaches the log, and it is a knowing trade rather than an oversight. A prompt-injected agent *can* forge -that preamble, so treat it as a bounded, documented channel — ~120 sanitized -characters that cannot carry a workflow command — not a safe one. The -alternative was the lane reporting `subtype=success` and naming nothing, since -the cause appears nowhere else in the file and the file is not uploaded. A -failure whose message carries no status code prints no cause rather than a -guess, so widen that pattern only against a real message. +the preamble, so treat it as a bounded, documented channel — ~120 sanitized +characters that cannot carry a workflow command or the API key, but can carry +other text — not a safe one. The alternative was the lane reporting +`subtype=success` and naming nothing, since the cause appears nowhere else in +the file and the file is not uploaded. A failure with neither marker prints no +cause rather than a guess, so widen the gate only against a real message. A cell that ran but submitted nothing prints whatever error objects (and, on the Anthropic lane, the gated cause) it recognized, plus the census; when