Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/cursor-review/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -256,7 +256,7 @@ the consolidated review's panel table. See [Panel integrity](#panel-integrity).
| [`build-ledger.py`](build-ledger.py) | Builds the **prior-review ledger** — what earlier rounds raised on this PR and how the author answered — and splices it into the panel/judge prompts. Untrusted prose is defanged twice over: fence-opening lines are rewritten, and continuation lines of quoted prose are prefixed ` |` so no imported text can sit at a field's indent and forge one. Also the prompt splicer, so the no-ledger path is byte-identical to the pre-ledger prompt. |
| [`fence-diff.py`](fence-diff.py) | Wraps the reviewed diff (plus the incremental hunks, and the judge's panel-findings block) in `=== BEGIN/END DIFF <nonce> ===` fences. The diff is attacker-authored PR bytes, so static literal fences are not a control; the nonce is what a PR cannot forge. Each prompt-build step mints its OWN nonce (`mint`), into a shell variable rather than a step `env:` or job output — Actions dumps a step's env map into the public run log, and a per-prompt value means a leak in one job cannot forge a fence in another. Copies the body through **byte for byte** — it never defangs or normalizes the payload. |
| [`incremental-diff.py`](incremental-diff.py) | Builds the incremental "hunks new since the last reviewed round" block as the difference between two PR patches (`build`), and verifies the result really is a subset of the reviewed diff (`check`) so the workflow can discard it if it is not. Never a commit range: `git diff LAST_REVIEWED...HEAD` carries every commit a merge of the base branch brought in — see [the ledger section](#the-prior-review-ledger-and-the-repeat-policy). |
| [`agent-error.py`](agent-error.py) | Run by each **direct-API** cell's **Report cell outcome** step, and only when that cell did not submit: prints WHY it failed. Both direct lanes send their agent's structured output to a file (`codex-events.jsonl`, `claude-result.json`) that is never echoed, because it also holds the agent's prose — so an API-level failure, which `codex exec --json` reports as an event on **stdout**, used to leave the log showing a non-zero exit and nothing else, and the lane's failures had to be inferred from job DURATION (a real `insufficient_quota` / `credit_balance_exhausted` outage was discarded that way). It prints an allowlist of short provider metadata (`type`, `subtype`, `is_error`, `code`, `param`, `status`, `message`) from objects that look like errors, plus a census of event types. **Two** allowlists, and the second is the load-bearing one: restricting the FIELDS is not enough, because the agent writes part of this file — it chooses the `arguments` of its own MCP tool calls — so a prompt-injected one could plant `{"error": {"message": "…"}}` there and a walk that descended into every value would print it. **Descent** is therefore an allowlist too (`ENVELOPE_KEYS`: `error`, `last_error`, `item`, `response`, `data`, `detail`), so the model's writable subtrees are never reached rather than denied one by one; widen it only to a key a CLI owns. An `error` KEY is read on its own terms, since a provider's error object need not describe itself as an error (OpenAI's says `insufficient_quota`). Values are sanitized by REPLACING disallowed characters, never deleting them, and both the `::` and legacy `##[command]` forms are broken with a single-pass regex (`str.replace` does not rescan its own output, so `a:::b` defeated it). Reads the **tail** of the file, not the head: the event that explains a failure is the last thing an agent writes. Exits **0 on anything** — missing, truncated, non-JSON, invalid UTF-8, or nested deeply enough that `json.loads` raises `RecursionError` (a RuntimeError, not a ValueError; 400 KB of `[` suffices) — because it runs after the cell already failed and an abort under the step's `bash -e` would cost the status line. **Known gap:** Claude Code puts an API failure's cause in `result`, the same field its review prose uses, so the Anthropic lane reports `is_error=true` and the census but not the cause — a deliberate trade, since that field is attacker-steerable. The raw file is not uploaded either: the Anthropic lane screens `findings.json` for a leaked API key before publishing it, and shipping the unscreened transcript alongside would route the agent's prose past that guard. |
| [`agent-error.py`](agent-error.py) | Run by each **direct-API** cell's **Report cell outcome** step, and only when that cell did not submit: prints WHY it failed. Both direct lanes send their agent's structured output to a file (`codex-events.jsonl`, `claude-result.json`) that is never echoed, because it also holds the agent's prose — so an API-level failure, which `codex exec --json` reports as an event on **stdout**, used to leave the log showing a non-zero exit and nothing else, and the lane's failures had to be inferred from job DURATION (a real `insufficient_quota` / `credit_balance_exhausted` outage was discarded that way). It prints an allowlist of short provider metadata (`type`, `subtype`, `is_error`, `code`, `param`, `status`, `message`, and Claude Code's `api_error_status` / `terminal_reason`) from objects that look like errors, plus a census of event types. **Two** allowlists, and the second is the load-bearing one: restricting the FIELDS is not enough, because the agent writes part of this file — it chooses the `arguments` of its own MCP tool calls — so a prompt-injected one could plant `{"error": {"message": "…"}}` there and a walk that descended into every value would print it. **Descent** is therefore an allowlist too (`ENVELOPE_KEYS`: `error`, `last_error`, `item`, `response`, `data`, `detail`), so the model's writable subtrees are never reached rather than denied one by one; widen it only to a key a CLI owns. An `error` KEY is read on its own terms, since a provider's error object need not describe itself as an error (OpenAI's says `insufficient_quota`). Values are sanitized by REPLACING disallowed characters, never deleting them, and both the `::` and legacy `##[command]` forms are broken with a single-pass regex (`str.replace` does not rescan its own output, so `a:::b` defeated it). Reads the **tail** of the file, not the head: the event that explains a failure is the last thing an agent writes. Exits **0 on anything** — missing, truncated, non-JSON, invalid UTF-8, or nested deeply enough that `json.loads` raises `RecursionError` (a RuntimeError, not a ValueError; 400 KB of `[` suffices) — because it runs after the cell already failed and an abort under the step's `bash -e` would cost the status line. **One stated exception** to withholding model-written fields: Claude Code puts an API failure's cause in `result` — the same field its review prose uses — and reports `subtype=success` on a failed call, so the cause is printed under gates: only from the top-level `{"type":"result"}` object with `is_error` true; only when the CLI marked it an API failure (a numeric `api_error_status` — the pinned CLI's real 401 is `Invalid API key · Fix external API key`, with no preamble — or an `API Error: <status> ` preamble); never when it holds the `sk-ant-` prefix or any 16-char slice of the step's own `ANTHROPIC_API_KEY` (the screen `findings.json` gets, which a 120-char cut would otherwise slip past Actions' exact-value mask); and cut to 120 sanitized chars. A prompt-injected agent can forge the preamble, so it is a bounded, documented channel, not a safe one; it was taken knowingly because the cause appears nowhere else and the file is not uploaded. A failure with neither marker prints nothing rather than a guess. The raw file is not uploaded either: the Anthropic lane screens `findings.json` for a leaked API key before publishing it, and shipping the unscreened transcript alongside would route the agent's prose past that guard. |
| [`catalog-drift.py`](catalog-drift.py) | Backs the weekly catalog-drift check. Extracts the pins from `cursor-review.yml`, diffs them against raw `cursor-agent models` output, and renders the sticky issue title + body (delisted pins, pins marked NO-ZDR, unpinned same-lab ids, catalog ids from unpinned families, stale audit date). Reports only — it never edits a pin. |

## Adopt it in your repo
Expand Down
89 changes: 89 additions & 0 deletions .github/cursor-review/agent-error.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,11 @@
(``ERROR_FIELDS``) from objects that look like errors, plus a census of event
types.

ONE field is read out of model-written territory — Claude Code's API-failure
cause, which sits in `result` and nowhere else — under the gates in
``_api_error_cause``. Everything below describes the rule that field is the
stated exception to.

TWO allowlists, and the second one is the load-bearing one. Restricting the
FIELDS is not enough on its own, because a reviewer agent writes part of this
file: it chooses the `arguments` of its own MCP tool calls. A prompt-injected
Expand Down Expand Up @@ -65,6 +70,9 @@
ERROR_FIELDS = (
"type", "subtype", "is_error", "code", "param",
"status", "status_code", "http_status", "message",
# Claude Code's own scalars on a failed result object, beside the
# model-written `result`: the HTTP status and `api_error` respectively.
"api_error_status", "terminal_reason",
)

# The ONLY keys this descends through — the envelope a CLI writes around the
Expand All @@ -87,6 +95,20 @@
MAX_DEPTH = 12
MAX_NODES = 20000

# Claude Code's API-failure cause sits in `result`, a field the model also
# writes. Printing it is THE one exception to "no model-written field reaches
# the log" — see `_api_error_cause`, which gates it.
CAUSE_FIELD = "result"
CAUSE_LIMIT = 120
# `[0-9]`, not `\d`: in a str pattern `\d` matches any Unicode digit.
_API_ERROR = re.compile(r"^API Error: [0-9]{3} ")
# An Anthropic key's prefix, and the window of the REAL key (from the step's
# env) that a cause may not contain. A window rather than the whole key: the
# cut to CAUSE_LIMIT leaves a truncated key, which Actions' exact-value mask
# no longer matches.
_KEY_PREFIX = "sk-ant-"
KEY_WINDOW = 16

Comment thread
mattmillerai marked this conversation as resolved.
Resolved
Comment thread
mattmillerai marked this conversation as resolved.
Resolved
_SAFE = re.compile(r"[^A-Za-z0-9 .,:;_/@+()\[\]{}='\"!?#$%^&*|~`<>-]")
# A colon followed by a colon, and a `#` that begins a `##[`. Lookahead, so one
# pass breaks every run however long: `":::"` -> `": : :"`, `"###["` -> `"## #["`.
Expand Down Expand Up @@ -146,6 +168,70 @@ def _fields(obj: dict) -> str:
return " ".join(parts)


def _api_error_cause(obj: dict, top: bool = False) -> str:
"""Claude Code's API-failure cause, read out of a model-written field.

THE one exception to the rule that no model-written field reaches the log,
and it is deliberate rather than an oversight. Claude Code reports an
API-level failure as `{"type":"result","subtype":"success","is_error":true,
"result":"API Error: 400 ..."}` — note `subtype`, which reads as SUCCESS —
and the cause appears nowhere else in the file. Since the file is not
uploaded either, an Anthropic-lane outage was otherwise undiagnosable
without re-running the cell.

So it is gated:

1. the object is the CLI's own top-level `{"type":"result"}` object with
`is_error` true — never a nested or codex-lane object (``walk`` passes
``top``), where the shape is not the one this exception covers;
2. the CLI marked it an API failure: a numeric `api_error_status` (a CLI
scalar the model cannot write — `Invalid API key · Fix external API
key` with status 401 is a real 2.1.286 failure, and carries no
preamble), or a string opening with `API Error: <status> `;
3. it holds no Anthropic key: not the `sk-ant-` prefix and no
``KEY_WINDOW``-char slice of the step's own ``ANTHROPIC_API_KEY`` —
the screen `findings.json` gets before publishing, which this path
would otherwise skip;
4. it is cut to ``CAUSE_LIMIT`` and sanitized like every other value.

A prompt-injected agent CAN forge the preamble, so this is a BOUNDED and
documented channel, not a safe one — roughly 120 sanitized characters
that cannot carry a workflow command or the key, but can carry other
text. That trade was taken knowingly (#402); the alternative was a lane
that reports `subtype=success` and names nothing.

A failure with neither marker yields nothing rather than a guess, which is
the safe direction: widen the gate only against a real message, never
speculatively.
"""
if not top or obj.get("type") != "result" or obj.get("is_error") is not True:
return ""
value = obj.get(CAUSE_FIELD)
if not isinstance(value, str):
return ""
status = obj.get("api_error_status")
cli_marked = isinstance(status, int) and not isinstance(status, bool)
if not cli_marked and not _API_ERROR.match(value):
return ""
if _holds_key(value):
return f"{CAUSE_FIELD}=(withheld: it contains an Anthropic API key or part of one)"
return f"{CAUSE_FIELD}={sanitize(value, CAUSE_LIMIT)}"
Comment thread
mattmillerai marked this conversation as resolved.
Resolved


def _holds_key(value: str) -> bool:
"""True when ``value`` carries the key prefix or a slice of the real key."""
if _KEY_PREFIX in value:
return True
key = os.environ.get("ANTHROPIC_API_KEY", "")
if not key:
return False
if len(key) <= KEY_WINDOW:
return key in value
return any(
key[i : i + KEY_WINDOW] in value for i in range(len(key) - KEY_WINDOW + 1)
)


def walk(node, out: list, types: dict, state: dict, depth: int = 0) -> None:
"""Collect error objects and a census of `type` values. Bounded.

Expand All @@ -171,6 +257,9 @@ def walk(node, out: list, types: dict, state: dict, depth: int = 0) -> None:
rendered = _fields(node)
if rendered and rendered not in out:
out.append(rendered)
cause = _api_error_cause(node, top=depth == 0)
if cause and cause not in out:
out.append(cause)
# An `error` KEY is the canonical envelope and must be read on its own
# terms. The provider's error object does not have to describe ITSELF as an
# error — OpenAI's is `{"error": {"type": "insufficient_quota", "code":
Expand Down
Loading
Loading