From ce872cd5261e4703e5f57e08b2db0f4888c88955 Mon Sep 17 00:00:00 2001 From: Matt Miller Date: Fri, 9 Oct 2026 18:56:28 +0000 Subject: [PATCH 1/7] feat(cursor-review): optional direct-Anthropic review cells via a pinned Claude Code CLI over a read-only PR checkout (BE-19913) Mirror of the direct-OpenAI cells for the Anthropic lane: - inputs anthropic_direct_model / anthropic_direct_effort / anthropic_direct_replaces_cursor_anthropic, optional ANTHROPIC_API_KEY - gate resolves anthropic_key_present; preflight emits anthropic_direct / anthropic_direct_counts and drops the Cursor claude-* cells when replacing (judge_model untouched) - review-anthropic-direct job: ubuntu-latest, contents: read, read-only PR checkout with symlinks removed, claude -p --bare --restricted with Read/Grep/Glob plus the shared MCP submission tool, key-exfil guard - aggregate-panel.py counts direct cells per lab (--anthropic-direct-*) - CLI pinned in .github/cursor-review/package.json (Dependabot-visible) Co-Authored-By: Claude Opus 5.5 --- .github/cursor-review/README.md | 6 +- .github/cursor-review/aggregate-panel.py | 67 ++- .github/cursor-review/package.json | 5 +- .../tests/test_aggregate_panel.py | 122 ++++ .../tests/test_workflow_job_isolation.py | 12 + .../tests/test_workflow_panel_integrity.py | 57 +- .../tests/test_workflow_ref_guard_parity.py | 2 +- .../tests/test_check_workflow_pins.py | 5 +- .github/workflows/cursor-review.yml | 538 +++++++++++++++++- docs/callers/cursor-review.md | 4 + 10 files changed, 781 insertions(+), 37 deletions(-) diff --git a/.github/cursor-review/README.md b/.github/cursor-review/README.md index 698277cf..276b4a17 100644 --- a/.github/cursor-review/README.md +++ b/.github/cursor-review/README.md @@ -242,7 +242,7 @@ the consolidated review's panel table. See [Panel integrity](#panel-integrity). | File | Role | |---|---| -| [`aggregate-panel.py`](aggregate-panel.py) | Run by `consolidate`'s **Aggregate panel findings** step. Combines the cells' `findings-*` artifacts into the `panel.json` the judge reads, adds a `status=error` record for every expected cell whose artifact never arrived, and writes the `ok_count` / `total` / `panel_inconsistent` that `Panel integrity` and auto-approve read (`panel_inconsistent`: a reviewer matrix did not succeed although every counted cell artifact reads `ok`, or a counted record is `status: mismatch` — it disagrees with its artifact name or names no cell the panel runs). The direct-OpenAI cells (`findings-direct-*`) count like Cursor cells when they replace the Cursor OpenAI lane (`openai_direct_replaces_cursor_openai: true`, so a failed one also counts against `approve_max_failed_reviewers`) and stay advisory side by side. Only this script sets the `direct_api` / `advisory` markers, so a cell's own record cannot excuse itself from the count. | +| [`aggregate-panel.py`](aggregate-panel.py) | Run by `consolidate`'s **Aggregate panel findings** step. Combines the cells' `findings-*` artifacts into the `panel.json` the judge reads, adds a `status=error` record for every expected cell whose artifact never arrived, and writes the `ok_count` / `total` / `panel_inconsistent` that `Panel integrity` and auto-approve read (`panel_inconsistent`: a reviewer matrix did not succeed although every counted cell artifact reads `ok`, or a counted record is `status: mismatch` — it disagrees with its artifact name or names no cell the panel runs). The direct-OpenAI cells (`findings-direct-*`) count like Cursor cells when they replace the Cursor OpenAI lane (`openai_direct_replaces_cursor_openai: true`, so a failed one also counts against `approve_max_failed_reviewers`) and stay advisory side by side; the direct-Anthropic cells do the same under `anthropic_direct_replaces_cursor_anthropic`, each artifact assigned to its lab by model id and counted only by that lab's flag. Only this script sets the `direct_api` / `advisory` markers, so a cell's own record cannot excuse itself from the count. | | [`prompt-adversarial.md`](prompt-adversarial.md) | Prompt for the security/reliability review pass. | | [`prompt-edge-case.md`](prompt-edge-case.md) | Prompt for the correctness/logic review pass. | | [`prompt-judge.md`](prompt-judge.md) | Prompt the judge model uses to consolidate panel findings into one review. | @@ -327,6 +327,7 @@ jobs: |---|---|---|---| | Secret | `CURSOR_API_KEY` | **yes** | Bills the panel + judge `cursor-agent` calls. | | Secret | `OPENAI_API_KEY` | no | Only with `openai_direct_model`: bills the optional direct-OpenAI cells. Use a project-scoped service-account key with a spend limit. | +| Secret | `ANTHROPIC_API_KEY` | no | Only with `anthropic_direct_model`: bills the optional direct-Anthropic cells. Use a key scoped to its own Anthropic workspace with a spend limit, never a personal key. | | Secret | `SLACK_BOT_TOKEN` | no | Enables start/complete DMs to the triggerer. | | Variable | `CURSOR_REVIEW_DM_EMAIL_MAP` | no | Maps GitHub logins → emails for Slack DM lookup. | @@ -391,6 +392,9 @@ All optional except `workflows_ref` (required, no default) — pass them under | `panel_models` | `''` (built-in list) | JSON array of Cursor model ids that **replaces** the panel list; each still runs both review types and is validated against the live catalog by preflight. For per-repo experiments (e.g. `-xhigh` vs `-max` tiers). | | `openai_direct_model` | `''` (off) | OPT-IN. An OpenAI model id reviewed through the OpenAI API directly (pinned Codex CLI, `package.json` here) as two cells (**adversarial** + **edge-case**, the Cursor cells' prompts) on GitHub-hosted runners, by default **replacing** the Cursor `gpt-*` cells (next row). Needs the `OPENAI_API_KEY` secret and a plain id (`[A-Za-z0-9._-]`); without either, preflight warns and leaves the cells off and the panel unchanged. Not part of `panel_models` (not in the Cursor catalog). Whether the cells gate depends on the next row. | | `openai_direct_replaces_cursor_openai` | `true` | When the direct cells are in effect, drop the Cursor `gpt-*` cell(s) so the OpenAI lane is reviewed once — the direct cells then **gate** like the Cursor cells they replace: counted in `Panel integrity` and the auto-approve gate, a failed one red in the rollup and named `(direct, )` in the review's "did not contribute" line. `false` runs both for a side-by-side backend comparison, with the direct cells **advisory** (findings reach the judge; never counted, never red). Preflight fails rather than leave the Cursor panel empty. | +| `anthropic_direct_model` | `''` (off) | OPT-IN. The Anthropic twin of `openai_direct_model`: an Anthropic model id (e.g. `claude-opus-5-5`) reviewed through the Anthropic API directly (pinned Claude Code CLI, `package.json` here) as two cells on GitHub-hosted runners, by default **replacing** the Cursor `claude-*` cells. Needs the `ANTHROPIC_API_KEY` secret and a plain id; without either, preflight warns and leaves the cells off and the panel unchanged. The agent also reads a **read-only PR checkout** (symlinks removed) with Read, Grep and Glob only — no Bash or code-running tool — confined to it by `--restricted`. This panel reviews private-repo diffs and the workflow cannot check the Anthropic organization's data-retention terms: callers on private repos must use a key from an organization with zero data retention. | +| `anthropic_direct_effort` | `xhigh` | `--effort` for the direct Anthropic cells: `low`, `medium`, `high`, `xhigh` or `max`; anything else warns and falls back to `xhigh`. | +| `anthropic_direct_replaces_cursor_anthropic` | `true` | As `openai_direct_replaces_cursor_openai`, for the Cursor `claude-*` cells. Each lab counts by its own flag. Never touches `judge_model`. | | `cell_timeout_minutes` | `15` | Wall-clock cap on one reviewer cell's agent step (Cursor or direct). Over it the cell exits 130, uploads `status: error` with no findings, reddens its leg and fails `Panel integrity`. Shared by the cell's in-step retries, so it bounds attempts + backoff. Derives three more caps that cannot be set alone: cell job = this + 15, `Run judge` step = max(30, this × 2), judge job = that + 10. At the default: 15 / 30 / 30 / 40, today's values exactly. Preflight bounds it 10–45. | | `skip_bot_branch_prefixes` | `ci/bump- chore/refresh- auto/refresh-` | Skip the panel when the PR author is a Bot **and** the head branch starts with one of these prefixes (machine pin bumps / catalog refreshes). `''` reviews every bot PR. | | `diff_size_cap` | `5000` | Max counted changed lines (after generated-file exclusion and comment discounting); larger PRs are skipped. | diff --git a/.github/cursor-review/aggregate-panel.py b/.github/cursor-review/aggregate-panel.py index f71e2c7c..f80b42ba 100644 --- a/.github/cursor-review/aggregate-panel.py +++ b/.github/cursor-review/aggregate-panel.py @@ -17,6 +17,12 @@ reach the judge, but they stay out of `present` (so one cannot mask a missing Cursor cell with the same id) and out of ok_count/total. +The role is chosen PER LAB: the OpenAI cells (`--direct-model`, +`--direct-counts`, `--direct-result`, the original flags) and the Anthropic +ones (`--anthropic-direct-*`) each count only by their own lab's flag. A direct +artifact is assigned to a lab by its model id; one matching neither lab is +unexpected — a mismatch when any lab counts, advisory otherwise. + Two markers are set HERE and only here — a cell's own record is model-adjacent output and must never be able to excuse itself from the count: `direct_api` (this cell came through the direct backend) and `advisory` (keep it out of the @@ -61,7 +67,9 @@ Stdlib only. Usage: aggregate-panel.py --panel-dir /tmp/panel --models '' \ [--direct-model --direct-counts true|false] \ + [--anthropic-direct-model --anthropic-direct-counts true|false] \ --review-result [--direct-result ] \ + [--anthropic-direct-result ] \ --out /tmp/panel.json [--github-output "$GITHUB_OUTPUT"] """ @@ -171,7 +179,9 @@ def _all_ok(cells): def aggregate(panel_dir, models, direct_model="", direct_counts=False, - review_result="success", direct_result="skipped"): + review_result="success", direct_result="skipped", + anthropic_direct_model="", anthropic_direct_counts=False, + anthropic_direct_result="skipped"): """Return (cells, ok_count, total, inconsistent). `cells` is everything the judge reads; `inconsistent` is either @@ -180,22 +190,43 @@ def aggregate(panel_dir, models, direct_model="", direct_counts=False, panel, direct = load_cells(panel_dir) flag_unexpected(panel, models) fill_missing(panel, models, "Cell findings artifact was not uploaded.") - counts = bool(direct_counts and direct_model) - if counts: - flag_unexpected(direct, [direct_model]) - # Same contract as a Cursor cell: absent means failed, never "not run". - fill_missing(direct, [direct_model], "Direct-API cell findings artifact was not uploaded.") + # (model, counts, matrix result) per lab; OpenAI first, the original flags. + labs = [ + (direct_model, bool(direct_counts and direct_model), direct_result), + (anthropic_direct_model, bool(anthropic_direct_counts and anthropic_direct_model), + anthropic_direct_result), + ] + by_lab = [[] for _ in labs] + unassigned = [] + for c in direct: + for i, (model, _, _) in enumerate(labs): + if model and c.get("model") == model: + by_lab[i].append(c) + break + else: + unassigned.append(c) + any_counts = any(counts for _, counts, _ in labs) + if any_counts: + # Same as a Cursor cell: a direct artifact no lab runs is a mismatch. + flag_unexpected(unassigned, []) + counted = list(panel) + inconsistent = review_result != "success" and _all_ok(panel) + for (model, counts, result), cells in zip(labs, by_lab): + if counts: + # Same contract as a Cursor cell: absent means failed, never "not run". + fill_missing(cells, [model], "Direct-API cell findings artifact was not uploaded.") + counted += cells + inconsistent = inconsistent or (result not in ("success", "skipped") and _all_ok(cells)) + if any_counts: + counted += unassigned + direct = [c for cells in by_lab for c in cells] + unassigned + counted_ids = {id(c) for c in counted} for c in direct: c["direct_api"] = True - if not counts: + if id(c) not in counted_ids: c["advisory"] = True - counted = panel + direct if counts else panel ok = sum(1 for c in counted if c.get("status") == "ok") - inconsistent = ( - (review_result != "success" and _all_ok(panel)) - or (counts and direct_result not in ("success", "skipped") and _all_ok(direct)) - or any(c.get("status") == MISMATCH_STATUS for c in counted) - ) + inconsistent = inconsistent or any(c.get("status") == MISMATCH_STATUS for c in counted) return panel + direct, ok, len(counted), inconsistent @@ -213,6 +244,10 @@ def main(argv=None): # Required, so a dropped flag fails the step rather than reading as success. p.add_argument("--review-result", required=True, help="needs.review.result") p.add_argument("--direct-result", default="skipped", help="needs.review-openai-direct.result") + p.add_argument("--anthropic-direct-model", default="") + p.add_argument("--anthropic-direct-counts", default="false") + p.add_argument("--anthropic-direct-result", default="skipped", + help="needs.review-anthropic-direct.result") p.add_argument("--out", required=True) p.add_argument("--github-output", default="") args = p.parse_args(argv) @@ -220,15 +255,19 @@ def main(argv=None): counts = args.direct_counts.strip().lower() == "true" review_result = args.review_result.strip() direct_result = args.direct_result.strip() + anthropic_counts = args.anthropic_direct_counts.strip().lower() == "true" + anthropic_result = args.anthropic_direct_result.strip() cells, ok, total, inconsistent = aggregate( args.panel_dir, json.loads(args.models), args.direct_model.strip(), counts, review_result, direct_result, + args.anthropic_direct_model.strip(), anthropic_counts, anthropic_result, ) print(f"Panel: {ok}/{total} cells contributed findings.") if inconsistent: print( f"::warning::A counted findings artifact may not be its own cell's: a reviewer matrix " - f"did not succeed (review={_log_safe(review_result)}, direct={_log_safe(direct_result)}) " + f"did not succeed (review={_log_safe(review_result)}, direct={_log_safe(direct_result)}, " + f"anthropic_direct={_log_safe(anthropic_result)}) " f"although every counted cell artifact of it reads ok, or a record is " f"status={MISMATCH_STATUS}. The panel count is untrusted (panel_inconsistent=true)." ) diff --git a/.github/cursor-review/package.json b/.github/cursor-review/package.json index 0918b43e..c2718213 100644 --- a/.github/cursor-review/package.json +++ b/.github/cursor-review/package.json @@ -1,8 +1,9 @@ { "name": "comfy-cursor-review-direct-cli", "private": true, - "description": "Manifest whose ONLY job is to be the single, Dependabot-visible source of truth for the @openai/codex CLI version that cursor-review.yml's optional direct-API review cell installs. Nothing is ever `npm install`ed from this directory: the job reads the pin out of this file and runs `npm install -g @openai/codex@`. It is a real npm manifest rather than a workflow `env:` constant purely so Dependabot's npm ecosystem can see the pin and open a bump PR — an inline `npm install -g pkg@1.2.3` inside a `run:` step is invisible to every ecosystem, which is how the groom CLI pin silently rotted (see .github/groom/package.json, the precedent this mirrors). Keep the version EXACT (no ^ or ~): the agent CLI is executable supply chain for a step that reads untrusted repo content, and the job rejects a non-exact pin at run time.", + "description": "Manifest whose ONLY job is to be the single, Dependabot-visible source of truth for the CLI versions cursor-review.yml's optional direct-API review cells install: @openai/codex (the direct OpenAI cells) and @anthropic-ai/claude-code (the direct Anthropic cells). Nothing is ever `npm install`ed from this directory: the job reads the pin out of this file and runs `npm install -g @`. It is a real npm manifest rather than a workflow `env:` constant purely so Dependabot's npm ecosystem can see the pin and open a bump PR — an inline `npm install -g pkg@1.2.3` inside a `run:` step is invisible to every ecosystem, which is how the groom CLI pin silently rotted (see .github/groom/package.json, the precedent this mirrors). Keep the version EXACT (no ^ or ~): the agent CLI is executable supply chain for a step that reads untrusted repo content, and the job rejects a non-exact pin at run time.", "dependencies": { - "@openai/codex": "0.162.0" + "@openai/codex": "0.162.0", + "@anthropic-ai/claude-code": "2.1.286" } } diff --git a/.github/cursor-review/tests/test_aggregate_panel.py b/.github/cursor-review/tests/test_aggregate_panel.py index f1aad745..4c79d70a 100644 --- a/.github/cursor-review/tests/test_aggregate_panel.py +++ b/.github/cursor-review/tests/test_aggregate_panel.py @@ -369,6 +369,128 @@ def test_cli_log_line_cannot_forge_a_command(self): self.assertIn("counted as status=mismatch", result.stdout) +ANTHROPIC = "claude-opus-5-5" + + +class PerLabTest(unittest.TestCase): + """The OpenAI and Anthropic direct cells each count by their OWN lab's + flag: one lab's role must never leak into the other's.""" + + def setUp(self): + self._tmp = tempfile.TemporaryDirectory() + self.addCleanup(self._tmp.cleanup) + self.dir = self._tmp.name + # The Cursor panel once both labs' Cursor cells are replaced. + self.models = ["kimi"] + for review_type in AGG.REVIEW_TYPES: + self.write(f"findings-{review_type}-kimi", cell("kimi", review_type)) + + def write(self, artifact, record): + os.makedirs(os.path.join(self.dir, artifact), exist_ok=True) + with open(os.path.join(self.dir, artifact, "findings.json"), "w", encoding="utf-8") as f: + json.dump(record, f) + + def direct(self, model, review_type, status="ok", **extra): + self.write(f"findings-direct-{review_type}-{model}", cell(model, review_type, status, **extra)) + + def both_labs(self, openai_status="ok", anthropic_status="ok"): + for review_type in AGG.REVIEW_TYPES: + self.direct(DIRECT, review_type, openai_status) + self.direct(ANTHROPIC, review_type, anthropic_status) + + def run_agg(self, openai_counts, anthropic_counts, **kw): + return AGG.aggregate(self.dir, self.models, DIRECT, openai_counts, + kw.get("review_result", "success"), kw.get("direct_result", "success"), + ANTHROPIC, anthropic_counts, kw.get("anthropic_result", "success")) + + def roles(self, cells): + return {(c["model"], c["review_type"]): c.get("advisory", False) + for c in cells if c.get("direct_api")} + + def test_both_labs_counting(self): + self.both_labs(anthropic_status="error") + cells, ok, total, inconsistent = self.run_agg(True, True, anthropic_result="failure") + self.assertEqual((ok, total), (4, 6)) + self.assertFalse(inconsistent) + self.assertFalse(any(self.roles(cells).values())) + + def test_both_labs_advisory(self): + self.both_labs(openai_status="error", anthropic_status="error") + cells, ok, total, inconsistent = self.run_agg(False, False) + self.assertEqual((ok, total), (2, 2)) + self.assertFalse(inconsistent) + self.assertTrue(all(self.roles(cells).values())) + + def test_openai_counting_anthropic_advisory(self): + self.both_labs(anthropic_status="error") + cells, ok, total, _ = self.run_agg(True, False) + self.assertEqual((ok, total), (4, 4), "an advisory Anthropic cell was counted") + roles = self.roles(cells) + self.assertFalse(roles[(DIRECT, "adversarial")]) + self.assertTrue(roles[(ANTHROPIC, "adversarial")]) + + def test_anthropic_counting_openai_advisory(self): + self.both_labs(openai_status="error") + cells, ok, total, _ = self.run_agg(False, True) + self.assertEqual((ok, total), (4, 4), "an advisory OpenAI cell was counted") + roles = self.roles(cells) + self.assertTrue(roles[(DIRECT, "edge-case")]) + self.assertFalse(roles[(ANTHROPIC, "edge-case")]) + + def test_missing_anthropic_artifact_is_synthesised_as_error(self): + self.direct(ANTHROPIC, "adversarial") + cells, ok, total, _ = self.run_agg(False, True) + self.assertEqual((ok, total), (3, 4)) + synthesised = [c for c in cells if c["model"] == ANTHROPIC and c["review_type"] == "edge-case"] + self.assertEqual([c["status"] for c in synthesised], ["error"]) + self.assertTrue(synthesised[0]["direct_api"]) + + def test_missing_advisory_anthropic_artifact_is_not_synthesised(self): + cells, ok, total, _ = self.run_agg(True, False) + self.assertFalse([c for c in cells if c["model"] == ANTHROPIC]) + + def test_an_anthropic_cell_cannot_set_its_own_markers(self): + self.both_labs() + self.direct(ANTHROPIC, "edge-case", "error", advisory=True, direct_api=False) + cells, ok, total, _ = self.run_agg(False, True) + self.assertEqual((ok, total), (3, 4), "a forged `advisory` excused a failed counted cell") + forged = [c for c in cells if c["model"] == ANTHROPIC and c["review_type"] == "edge-case"][0] + self.assertIs(forged["direct_api"], True) + self.assertNotIn("advisory", forged) + + def test_counted_anthropic_lane_failed_with_all_ok_is_inconsistent(self): + self.both_labs() + _, _, _, inconsistent = self.run_agg(False, True, anthropic_result="failure") + self.assertTrue(inconsistent) + _, _, _, inconsistent = self.run_agg(True, False, anthropic_result="failure") + self.assertFalse(inconsistent, "an advisory lab's red matrix was read as a forgery") + + def test_direct_artifact_matching_neither_lab_is_a_mismatch_when_any_counts(self): + self.both_labs() + self.direct("other-model", "adversarial") + cells, ok, total, inconsistent = self.run_agg(False, True) + self.assertEqual((ok, total), (4, 5)) + self.assertTrue(inconsistent) + cells, ok, total, inconsistent = self.run_agg(False, False) + self.assertEqual((ok, total), (2, 2)) + self.assertFalse(inconsistent) + + def test_cli_takes_the_anthropic_flags(self): + self.both_labs(anthropic_status="error") + out = os.path.join(self.dir, "panel.json") + gh = os.path.join(self.dir, "gh-output") + result = subprocess.run( + [sys.executable, SCRIPT, "--panel-dir", self.dir, "--models", json.dumps(self.models), + "--direct-model", DIRECT, "--direct-counts", "false", "--review-result", "success", + "--anthropic-direct-model", ANTHROPIC, "--anthropic-direct-counts", "true", + "--anthropic-direct-result", "failure", "--out", out, "--github-output", gh], + capture_output=True, text=True, check=True, + ) + self.assertIn("Panel: 2/4 cells contributed findings.", result.stdout) + with open(gh, encoding="utf-8") as f: + self.assertIn("panel_inconsistent=false", f.read()) + + class DownstreamTest(unittest.TestCase): def test_summary_names_a_failed_direct_cell_with_its_backend(self): panel = [ diff --git a/.github/cursor-review/tests/test_workflow_job_isolation.py b/.github/cursor-review/tests/test_workflow_job_isolation.py index dded1928..a22dfcd4 100644 --- a/.github/cursor-review/tests/test_workflow_job_isolation.py +++ b/.github/cursor-review/tests/test_workflow_job_isolation.py @@ -303,6 +303,18 @@ def test_the_bot_key_is_confined_to_the_checkout_free_posting_jobs(self): self.assertEqual(holders, {"over-cap-comment", "post-review", "dismiss-stale-approval", "round-cap", "auto-retry"}) + def test_the_anthropic_key_reaches_only_the_gate_and_its_direct_cells(self): + # `gate` only resolves a presence boolean; the cells are the one + # consumer, and they hold a model key beside a PR checkout, so they + # must stay GitHub-hosted. + holders = { + name for name, body in self.jobs.items() + if any("secrets.ANTHROPIC_API_KEY" in l for l in code_lines(body)) + } + self.assertEqual(holders, {"gate", "review-anthropic-direct"}) + runs_on = [l.strip() for l in self.jobs["review-anthropic-direct"] if l.startswith(" runs-on:")] + self.assertEqual(runs_on, ["runs-on: ubuntu-latest"]) + def test_only_credential_free_jobs_follow_the_runs_on_input(self): # `runs_on` hands jobs to a caller-chosen pool where a prompt-injected # model cell may have run. Anything holding the bot key, a write scope, diff --git a/.github/cursor-review/tests/test_workflow_panel_integrity.py b/.github/cursor-review/tests/test_workflow_panel_integrity.py index 3278869b..a34069cb 100644 --- a/.github/cursor-review/tests/test_workflow_panel_integrity.py +++ b/.github/cursor-review/tests/test_workflow_panel_integrity.py @@ -1045,6 +1045,13 @@ class DirectCellsGateWhenTheyReplaceTest(unittest.TestCase): """ DIRECT_MODEL = "gpt-6.1-sol" + # Per lab, so the Anthropic twin below runs the same chain. + JOB = DIRECT_JOB + MODEL_INPUT = "openai_direct_model" + REPLACE_INPUT = "openai_direct_replaces_cursor_openai" + KEY_OUTPUT = "openai_key_present" + DIRECT_OUTPUT = "openai_direct" + VENDOR_PREFIX = "gpt-" @classmethod def setUpClass(cls): @@ -1054,7 +1061,7 @@ def setUpClass(cls): jobs = split_jobs(read_workflow()) cls.preflight = jobs["preflight"] cls.review = jobs[MATRIX_JOB] - cls.direct = jobs[DIRECT_JOB] + cls.direct = jobs[cls.JOB] cls.consolidate = jobs["consolidate"] def setUp(self): @@ -1072,9 +1079,9 @@ def preflight_outputs(self, replace, key_present="true"): done, step_outputs = run_step( self.step(self.preflight, PANEL_MODELS_STEP), { - "inputs.openai_direct_model": self.DIRECT_MODEL, - "inputs.openai_direct_replaces_cursor_openai": replace, - "needs.gate.outputs.openai_key_present": key_present, + "inputs." + self.MODEL_INPUT: self.DIRECT_MODEL, + "inputs." + self.REPLACE_INPUT: replace, + "needs.gate.outputs." + self.KEY_OUTPUT: key_present, }, tempfile.mkdtemp(dir=self.tmp), ) @@ -1088,7 +1095,7 @@ def preflight_outputs(self, replace, key_present="true"): def downstream(self, replace): """The context the direct legs and `consolidate` evaluate against.""" context = self.preflight_outputs(replace) - context["inputs.openai_direct_model"] = self.DIRECT_MODEL + context["inputs." + self.MODEL_INPUT] = self.DIRECT_MODEL return context def upload(self, panel, job_lines, cell_context, model, review_type, status): @@ -1134,24 +1141,24 @@ def leg(self, context, review_type, submitted): def test_preflight_marks_replacing_cells_as_counted(self): replacing = self.preflight_outputs(replace=True) - self.assertEqual(replacing["needs.preflight.outputs.openai_direct"], "true") + self.assertEqual(replacing["needs.preflight.outputs." + self.DIRECT_OUTPUT], "true") self.assertEqual( - replacing["needs.preflight.outputs.openai_direct_counts"], + replacing["needs.preflight.outputs." + self.DIRECT_OUTPUT + "_counts"], "true", "preflight no longer marks REPLACING direct cells as counted, so every " "reader downstream treats them as advisory", ) self.assertEqual( - [m for m in json.loads(replacing["needs.preflight.outputs.models"]) if m.startswith("gpt-")], + [m for m in json.loads(replacing["needs.preflight.outputs.models"]) if m.lower().startswith(self.VENDOR_PREFIX)], [], - "the Cursor OpenAI cells were not dropped from the panel", + f"the Cursor {self.VENDOR_PREFIX}* cells were not dropped from the panel", ) for label, outputs in ( ("side by side", self.preflight_outputs(replace=False)), - ("no OPENAI_API_KEY", self.preflight_outputs(replace=True, key_present="false")), + ("no API key", self.preflight_outputs(replace=True, key_present="false")), ): with self.subTest(label): - self.assertEqual(outputs["needs.preflight.outputs.openai_direct_counts"], "false") + self.assertEqual(outputs["needs.preflight.outputs." + self.DIRECT_OUTPUT + "_counts"], "false") def test_a_failed_replacing_cell_counts_against_the_panel(self): ok, total, cursor = self.aggregate( @@ -1179,7 +1186,7 @@ def test_a_replacing_cell_that_did_not_submit_fails_its_leg(self): self.assertEqual( render(job_scalar(self.direct, "continue-on-error") or "false", context), "false", - f"`{DIRECT_JOB}` absorbs its own failure while it REPLACES the Cursor OpenAI lane", + f"`{self.JOB}` absorbs its own failure while it REPLACES the Cursor OpenAI lane", ) for review_type in REVIEW_TYPES: with self.subTest(review_type): @@ -1214,14 +1221,36 @@ def test_the_leg_check_follows_the_upload_and_spares_its_sibling(self): ) self.assertIsNone( step_scalar(self.step(self.direct, LEG_STEP), "continue-on-error"), - f"`{DIRECT_JOB}`'s `{LEG_STEP}` carries continue-on-error and can no longer turn the leg red", + f"`{self.JOB}`'s `{LEG_STEP}` carries continue-on-error and can no longer turn the leg red", ) self.assertIn( " fail-fast: false", code_lines(self.direct), - f"`{DIRECT_JOB}` lost `fail-fast: false`: one red direct leg would cancel the other", + f"`{self.JOB}` lost `fail-fast: false`: one red direct leg would cancel the other", ) + +class AnthropicDirectCellsGateWhenTheyReplaceTest(DirectCellsGateWhenTheyReplaceTest): + """The same executed chain for `review-anthropic-direct`: its own inputs, + its own preflight outputs and its own `--anthropic-direct-*` aggregate + flags, so a broken Anthropic link fails here even with the OpenAI one + intact.""" + + DIRECT_MODEL = "claude-opus-5-5" + JOB = "review-anthropic-direct" + MODEL_INPUT = "anthropic_direct_model" + REPLACE_INPUT = "anthropic_direct_replaces_cursor_anthropic" + KEY_OUTPUT = "anthropic_key_present" + DIRECT_OUTPUT = "anthropic_direct" + VENDOR_PREFIX = "claude-" + + def test_the_judge_model_is_untouched(self): + step = self.step(self.preflight, PANEL_MODELS_STEP) + self.assertFalse( + any("judge" in line.lower() for line in code_lines(step) if "anthropic" in line.lower()), + "the Anthropic replacement reaches the judge model", + ) + if __name__ == "__main__": unittest.main(verbosity=2) diff --git a/.github/cursor-review/tests/test_workflow_ref_guard_parity.py b/.github/cursor-review/tests/test_workflow_ref_guard_parity.py index ff7feb18..9cd749f9 100644 --- a/.github/cursor-review/tests/test_workflow_ref_guard_parity.py +++ b/.github/cursor-review/tests/test_workflow_ref_guard_parity.py @@ -42,7 +42,7 @@ # The guard steps. `.strip()`-compared so the assertion is about the step name, # not about the indentation of whichever job happens to hold it. GUARD_STEP_NAME = "- name: Require a pinned workflows_ref" -EXPECTED_GUARDS = 12 +EXPECTED_GUARDS = 13 # The ledger's exempt resolver, addressed by its step id rather than its name so # a comment rewrite above it cannot move the anchor. diff --git a/.github/workflow-pins/tests/test_check_workflow_pins.py b/.github/workflow-pins/tests/test_check_workflow_pins.py index 6a8f516c..cb030c5c 100644 --- a/.github/workflow-pins/tests/test_check_workflow_pins.py +++ b/.github/workflow-pins/tests/test_check_workflow_pins.py @@ -3504,7 +3504,7 @@ def test_this_repos_own_workflows_guard_every_ref_checkout(self): self.assertEqual(cwp.find_unguarded_ref_checkouts(lines), [], name) self.assertEqual( seen, - 23, + 24, "expected the 12 guarded sites BE-5546 fixed + pr-size.yml's (BE-5858) " "+ cursor-review.yml's preflight (hard guard) site picked up merging " "main + cursor-review.yml's diff-size job's check-pr-size-tool " @@ -3539,7 +3539,8 @@ def test_this_repos_own_workflows_guard_every_ref_checkout(self): "cursor-review.yml's opt-in `review-openai-direct` job (BE-19639): " "the direct-OpenAI cell loads its prompt, fence-diff.py and " "review-output-mcp.py from `workflows_ref` in its own job and " - "carries its own copy of the guard.", + "carries its own copy of the guard. The 24th is its Anthropic twin, " + "`review-anthropic-direct` (BE-19913), which carries its own copy too.", ) WORKFLOWS_DIR = os.path.normpath( diff --git a/.github/workflows/cursor-review.yml b/.github/workflows/cursor-review.yml index 62aa226a..24febcd6 100644 --- a/.github/workflows/cursor-review.yml +++ b/.github/workflows/cursor-review.yml @@ -314,6 +314,48 @@ on: type: boolean required: false default: true + anthropic_direct_model: + description: >- + OPT-IN. An Anthropic model id (e.g. `claude-opus-5-5`) reviewed + through the Anthropic API directly, via the Claude Code CLI, as two + cells — adversarial and edge-case, the same passes and prompts a + Cursor model gets. The Anthropic twin of `openai_direct_model`, with + the same rules: empty (default) = the job is skipped and nothing + changes; requires the `ANTHROPIC_API_KEY` secret and a plain id + (`[A-Za-z0-9._-]`), and set without either, preflight warns, the + cells stay off and the panel is unchanged. Unlike the OpenAI cells, + which see the diff only, the agent also reads a read-only checkout of + the PR with Read, Grep and Glob alone — no Bash or other + code-running tool — and `--restricted` confines those file tools to + that checkout. This panel reviews private-repo diffs and the workflow + cannot check the Anthropic organization's data-retention terms, so a + caller on a private repo must use a key from an organization with + zero data retention. By default the cells REPLACE the Cursor + Anthropic cells; see `anthropic_direct_replaces_cursor_anthropic`. + type: string + required: false + default: '' + anthropic_direct_effort: + description: >- + Reasoning effort for the direct Anthropic cells (`--effort`): one of + `low`, `medium`, `high`, `xhigh`, `max`. Any other value warns and + falls back to `xhigh`, the default. Ignored unless + `anthropic_direct_model` is in effect. + type: string + required: false + default: xhigh + anthropic_direct_replaces_cursor_anthropic: + description: >- + The Anthropic twin of `openai_direct_replaces_cursor_openai`, with + the same semantics: when the direct Anthropic cells are in effect, + drop the Cursor-hosted Anthropic cells (vendor prefix `claude-`, + case-insensitive) so they REPLACE them and gate exactly as they + would. Set false for a side-by-side comparison in which the direct + cells are ADVISORY. Refuses to leave the Cursor panel empty. Never + touches `judge_model`. + type: boolean + required: false + default: true skip_bot_branch_prefixes: description: >- Space-separated head-branch prefixes that skip the review when the PR @@ -533,6 +575,14 @@ on: a user key — the cell bills per call against it directly rather than against a pooled seat. required: false + ANTHROPIC_API_KEY: + description: >- + Anthropic API key for the optional direct-API Anthropic review cells. + Required only when the `anthropic_direct_model` input is set; ignored + otherwise. Use a key scoped to its own Anthropic workspace with a + spend limit, never a personal key — the cells bill per call against + it directly rather than against a pooled seat. + required: false SLACK_BOT_TOKEN: description: Slack bot token for the start/complete DM notifications. Optional — DMs are skipped if absent. required: false @@ -683,6 +733,7 @@ jobs: already_reviewed: ${{ steps.dup.outputs.already_reviewed }} approve_author_ok: ${{ steps.author.outputs.approve_author_ok }} openai_key_present: ${{ steps.openai_key.outputs.present }} + anthropic_key_present: ${{ steps.anthropic_key.outputs.present }} steps: - name: Resolve OPENAI_API_KEY presence # A boolean for `preflight`, which cannot read the secret itself: it @@ -696,6 +747,15 @@ jobs: OPENAI_KEY_PRESENT: ${{ secrets.OPENAI_API_KEY != '' }} run: echo "present=${OPENAI_KEY_PRESENT}" >> "$GITHUB_OUTPUT" + - name: Resolve ANTHROPIC_API_KEY presence + # Same as `Resolve OPENAI_API_KEY presence` above, for the direct + # Anthropic cells. + id: anthropic_key + if: inputs.anthropic_direct_model != '' + env: + ANTHROPIC_KEY_PRESENT: ${{ secrets.ANTHROPIC_API_KEY != '' }} + run: echo "present=${ANTHROPIC_KEY_PRESENT}" >> "$GITHUB_OUTPUT" + - name: Resolve the auto-approve author opt-in # approve_authors, resolved ONCE here so the workflow-level # `approve_gate` output (which must say `off` on every event, round or @@ -2031,6 +2091,11 @@ jobs: # cells (counted in ok_count/total, leg red on failure). Side by side # ('false') they stay advisory. openai_direct_counts: ${{ steps.models.outputs.openai_direct_counts }} + # The same pair for the direct Anthropic cells (`review-anthropic-direct`). + anthropic_direct: ${{ steps.models.outputs.anthropic_direct }} + anthropic_direct_counts: ${{ steps.models.outputs.anthropic_direct_counts }} + # `anthropic_direct_effort`, validated: always one of the allowed levels. + anthropic_direct_effort: ${{ steps.models.outputs.anthropic_direct_effort }} # The caps DERIVED from `cell_timeout_minutes`, computed here because # Actions expressions have no arithmetic: `${{ x + 15 }}` is a parse # error that fails the workflow at load for every caller. Consumers @@ -2196,6 +2261,11 @@ jobs: # Resolved in the hosted `gate` job: this one follows `runs_on`, so # it must not reference the secret at all. OPENAI_KEY_PRESENT: ${{ needs.gate.outputs.openai_key_present }} + # The same three for the direct Anthropic cells. + ANTHROPIC_DIRECT_MODEL: ${{ inputs.anthropic_direct_model }} + ANTHROPIC_DIRECT_REPLACES: ${{ inputs.anthropic_direct_replaces_cursor_anthropic }} + ANTHROPIC_DIRECT_EFFORT: ${{ inputs.anthropic_direct_effort }} + ANTHROPIC_KEY_PRESENT: ${{ needs.gate.outputs.anthropic_key_present }} run: | cat > /tmp/models.json <<'JSON' [ @@ -2288,6 +2358,48 @@ jobs: fi fi echo "openai_direct_counts=${openai_direct_counts}" >> "$GITHUB_OUTPUT" + # The direct Anthropic cells, by exactly the rules above: one verdict + # first, then the replacement, applied to whatever the OpenAI branch + # left, so the empty-panel check sees both drops. `judge_model` is + # never touched — the judge stays on Cursor. + anthropic_direct=false + if [ -n "$(printf '%s' "${ANTHROPIC_DIRECT_MODEL:-}" | tr -d '[:space:]')" ]; then + if ! printf '%s' "$ANTHROPIC_DIRECT_MODEL" | grep -Eqx '[A-Za-z0-9._-]{1,100}'; then + echo "::warning::anthropic_direct_model is not a plain model id (letters, digits, '.', '_', '-'; at most 100), so the direct-API Anthropic cells are off for this run and the panel is unchanged." + elif [ "${ANTHROPIC_KEY_PRESENT:-false}" != "true" ]; then + echo "::warning::anthropic_direct_model is set but the ANTHROPIC_API_KEY secret did not reach this workflow, so the direct-API Anthropic cells are off for this run and the panel is unchanged. Pass the secret through from the caller to enable it." + else + anthropic_direct=true + fi + fi + echo "anthropic_direct=${anthropic_direct}" >> "$GITHUB_OUTPUT" + case "${ANTHROPIC_DIRECT_EFFORT:-}" in + low|medium|high|xhigh|max) anthropic_direct_effort="$ANTHROPIC_DIRECT_EFFORT" ;; + *) + anthropic_direct_effort=xhigh + if [ "$anthropic_direct" = "true" ]; then + echo "::warning::anthropic_direct_effort is not one of low, medium, high, xhigh, max, so the direct-API Anthropic cells use xhigh." + fi ;; + esac + echo "anthropic_direct_effort=${anthropic_direct_effort}" >> "$GITHUB_OUTPUT" + anthropic_direct_counts=false + if [ "$anthropic_direct" = "true" ] \ + && [ "${ANTHROPIC_DIRECT_REPLACES:-true}" = "true" ]; then + anthropic_direct_counts=true + dropped="$(jq -c '[.[] | select((ascii_downcase | startswith("claude-")))]' /tmp/models.json)" + kept="$(jq -c '[.[] | select((ascii_downcase | startswith("claude-")) | not)]' /tmp/models.json)" + if [ "$(printf '%s' "$dropped" | jq 'length')" -gt 0 ]; then + if [ "$(printf '%s' "$kept" | jq 'length')" -eq 0 ]; then + echo "::error::anthropic_direct_model is set and anthropic_direct_replaces_cursor_anthropic is true, but every id left in the panel list is an Anthropic id, so replacing them would leave no Cursor cell at all. Set anthropic_direct_replaces_cursor_anthropic: false, or widen panel_models." + exit 1 + fi + printf '%s' "$kept" > /tmp/models.json + echo "direct-API Anthropic cells are on, so the Cursor Anthropic cell(s) $dropped were dropped from the panel; remaining: $kept" + else + echo "direct-API Anthropic cells are on, but the panel list carries no Cursor Anthropic id to replace; panel unchanged." + fi + fi + echo "anthropic_direct_counts=${anthropic_direct_counts}" >> "$GITHUB_OUTPUT" echo "models=$(jq -c . /tmp/models.json)" >> "$GITHUB_OUTPUT" - name: Require a pinned workflows_ref @@ -3311,6 +3423,419 @@ jobs: fi echo "Direct-API reviewer cell ${REVIEW_TYPE}/${MODEL} submitted its findings." + review-anthropic-direct: + # The Anthropic twin of `review-openai-direct`: the same OPT-IN pair of + # cells (adversarial + edge-case), reaching their model through the + # Anthropic API directly, via the Claude Code CLI, instead of through + # Cursor. Everything that job's header says — why it is a separate job, how + # the judge picks it up, gating by role (here preflight's + # `anthropic_direct_counts`), scope — holds here; only the differences are + # noted below. + # + # The one deliberate difference: this agent ALSO reads a read-only + # checkout of the PR (`_pr`, its working directory) with Read, Grep and + # Glob — no Bash or other code-running tool — because a diff-only cell + # misses every finding that needs a file outside the diff. + # + # Credential posture: `contents: read` only, no write-scoped token (see + # SECURITY.md); the PR checkout persists no credential. ANTHROPIC_API_KEY + # reaches `claude` in that ONE step's env and is never written to disk, + # and the agent has no tool that can read the environment. Belt and + # braces, groom's finder guard runs on the record before it uploads. + name: '${{ matrix.review_type }} direct (${{ inputs.anthropic_direct_model }})' + needs: [gate, diff-size, preflight, ledger] + if: >- + needs.preflight.outputs.anthropic_direct == 'true' + && needs.gate.outputs.should_run == 'true' + && needs.gate.outputs.already_reviewed != 'true' + && needs.diff-size.outputs.within_cap == 'true' + # GitHub-hosted, never `runs_on`: this job holds ANTHROPIC_API_KEY (see + # `review-openai-direct`). + runs-on: ubuntu-latest + continue-on-error: ${{ needs.preflight.outputs.anthropic_direct_counts != 'true' }} + timeout-minutes: ${{ fromJSON(needs.preflight.outputs.cell_job_timeout) }} + permissions: + contents: read + strategy: + fail-fast: false + matrix: + review_type: [adversarial, edge-case] + steps: + - name: Seed default findings artifact + # Same contract as `review-openai-direct`'s seed. + env: + MODEL: ${{ inputs.anthropic_direct_model }} + REVIEW_TYPE: ${{ matrix.review_type }} + run: | + mkdir -p /tmp/findings-out + python3 - <<'PY' + import json, os + record = { + "model": os.environ["MODEL"], + "review_type": os.environ["REVIEW_TYPE"], + "status": "error", + "error": "Direct-API cell did not reach the submission step.", + "findings": [], + } + with open("/tmp/findings-out/findings.json", "w", encoding="utf-8") as f: + json.dump(record, f, indent=2) + PY + + - name: Require a pinned workflows_ref + # workflows_ref has no default on purpose. GitHub does NOT enforce + # `required: true` for workflow_call inputs, so an omitted input arrives + # as '' and actions/checkout would silently fall back to this repo's + # default branch — running MUTABLE scripts under a pin that claims + # otherwise. Fail fast instead. (BE-5546) + env: + WORKFLOWS_REF: ${{ inputs.workflows_ref }} + WORKFLOW_SHA: ${{ job.workflow_sha }} + run: | + # actions/checkout reads `ref` through core.getInput, which TRIMS, so a + # whitespace-only value is an empty ref to IT while sailing past a bare + # -z test here. Compare the stripped form, and echo only that: dropping + # newlines also stops a multi-line value from smuggling a ::workflow + # command:: into the log, and from satisfying the line-oriented grep + # below on one 40-hex line among many. + REF="$(printf '%s' "$WORKFLOWS_REF" | tr -d '[:space:]')" + if [ -z "$REF" ]; then + echo "::error::workflows_ref is required; pin it to the same commit SHA as the uses: line (see header example)" + exit 1 + fi + if ! printf '%s' "$REF" | grep -Eq '^[0-9a-f]{40}$'; then + echo "::warning::workflows_ref '$REF' is not a full 40-hex commit SHA — branch and tag refs are mutable and can skew between jobs mid-run" + fi + # Equality with the commit the caller's `uses:` line resolved to — the + # same guard public-repo-hygiene.yml runs, and the check that turns + # "workflows_ref must be the same SHA as uses:" from a header comment + # into a fact. `job.workflow_sha` is computed by the RUNNER from the + # `uses:` pin, so a caller cannot set it. Without this, a caller can + # pin `uses:` to one commit and load this workflow's prompts, scripts + # and model list from another. (`job.workflow_sha`, NOT + # `github.job_workflow_sha` — that one is an OIDC token claim, absent + # from the `github` context, and expands to ''.) Both sides are + # lower-cased: hex is case-insensitive, and unlike hygiene's the + # 40-hex test above only WARNS, so an upper-case ref naming the RIGHT + # commit reaches here and must not fail. $REF itself is left alone so + # the messages quote what the caller wrote. (BE-15927) + REF_LC="$(printf '%s' "$REF" | tr 'A-Z' 'a-z')" + SHA="$(printf '%s' "$WORKFLOW_SHA" | tr -d '[:space:]' | tr 'A-Z' 'a-z')" + if [ -z "$SHA" ]; then + # Cannot check, NOT a mismatch. `job.workflow_sha` is the `job`-context + # accessor added in Actions runner v2.334.0, so only an older runner + # gets here. GitHub stops queueing jobs to a self-hosted runner more + # than 30 days behind the latest release, so on any pool — hosted + # `ubuntu-latest`, or the caller's `runs_on` pool the read-only + # model jobs may use — that is a transient window. Warn and continue: a + # property nothing could EVALUATE is not evidence of a violation, and + # failing would take the whole review down on a caller that changed + # nothing. (public-repo-hygiene.yml fails closed in this branch on + # purpose — it is the checker whose own integrity rests on the + # equality. Here the ledger's warn-and-degrade posture is the fit.) + echo "::warning::job.workflow_sha is empty (Actions runner older than v2.334.0?), so workflows_ref could not be cross-checked against the uses: pin. The empty and 40-hex tests above still apply." + elif [ "$REF_LC" != "$SHA" ]; then + echo "::error::workflows_ref ($REF) is not the commit this workflow was called at ($SHA). The uses: pin and workflows_ref MUST name the same commit — otherwise the workflow is pinned but its prompts and scripts are loaded from a different version. Set both to $SHA. (The bump-callers fleet rewrites both together; Dependabot rewrites only uses:, so a caller that lets Dependabot bump this reusable needs a dependabot.yml ignore entry for Comfy-Org/github-workflows.)" + exit 1 + fi + + - name: Load cursor-review assets + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + repository: Comfy-Org/github-workflows + ref: ${{ inputs.workflows_ref }} + path: _cursor_review_assets + persist-credentials: false + + - name: Checkout PR repo + # The read-only tree the agent's file tools read: the default ref, so + # the same commit the Cursor cells review. No credential persisted. + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + path: _pr + fetch-depth: 1 + persist-credentials: false + + - name: Remove symlinks from the PR checkout + # A symlink planted in the PR could lead the file tools outside the + # tree (e.g. to the runner's home), so none survive into the review. + run: find _pr -type l -delete + + - name: Download reviewed diff + # Built once in the diff-size job (generated files already excluded), + # shared here, so this cell needs no PR checkout of its own. + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 + with: + name: pr-diff + path: /tmp + + - name: Download prior-review ledger + # Same artifact, same tolerance as the `review` job: absent on round 1. + continue-on-error: true + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 + with: + name: cursor-review-ledger + path: /tmp/ledger + + - name: Check the API key is present + id: key + # Belt and braces, as in `review-openai-direct`. + env: + ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }} + run: | + if [ -z "${ANTHROPIC_API_KEY:-}" ]; then + echo "::warning::anthropic_direct_model is set but the ANTHROPIC_API_KEY secret is absent, so the direct-API Anthropic cell is skipped. Pass the secret through from the caller to enable it." + python3 -c 'import json; p = "/tmp/findings-out/findings.json"; r = json.load(open(p)); r["error"] = "ANTHROPIC_API_KEY secret was not provided to the reusable workflow."; json.dump(r, open(p, "w"), indent=2)' + echo "present=false" >> "$GITHUB_OUTPUT" + else + echo "present=true" >> "$GITHUB_OUTPUT" + fi + + - name: Install Claude Code CLI + if: steps.key.outputs.present == 'true' + # Pinned in .github/cursor-review/package.json, read exactly as + # `review-openai-direct`'s `Install Codex CLI` reads its pin. + run: | + manifest="$CURSOR_REVIEW_ASSETS/package.json" + if [ ! -f "$manifest" ]; then + echo "::error::$manifest is missing — it is the single source of truth for the direct-API cell's CLI pin." + exit 1 + fi + if ! version=$(jq -r ' + if type != "object" then "" + elif (.dependencies | type) != "object" then "" + else (.dependencies["@anthropic-ai/claude-code"] // "") + end' "$manifest"); then + echo "::error::$manifest is not valid JSON — it is the single source of truth for the direct-API cell's CLI pin." + exit 1 + fi + version="$(printf '%s' "$version" | tr -d '\r\n')" + case "$version" in + ""|*[!0-9.]*) + echo "::error::dependencies[\"@anthropic-ai/claude-code\"] must be an EXACT version (digits and dots only, no ^ or ~); got a value this job will not install." + exit 1 ;; + esac + echo "CLAUDE_CLI_VERSION=$version" >> "$GITHUB_ENV" + npm install -g "@anthropic-ai/claude-code@${version}" + + - name: Verify Claude Code CLI version + if: steps.key.outputs.present == 'true' + # As `Verify Codex CLI version`: `claude --version` prints + # `2.1.286 (Claude Code)`, so the FIRST field is the version. + run: | + installed="$(claude --version 2>/dev/null | head -n 1 | tr -d '\r' | awk '{print $1}')" + if [ "$installed" != "$CLAUDE_CLI_VERSION" ]; then + echo "::error::claude reports '${installed}', expected the pinned '${CLAUDE_CLI_VERSION}'." + exit 1 + fi + echo "claude ${installed}" + + - name: Build prompt + if: steps.key.outputs.present == 'true' + # Byte-identical construction to the Cursor cells' `Build prompt`, so a + # finding difference is attributable to the model or the backend rather + # than to the prompt: the same prompt-.md with the same + # prior-review ledger spliced in, the same marker stripped and + # re-emitted by fence-diff.py, the same new-hunks block from round 2 + # on, and a fresh per-prompt nonce minted into a shell variable (never + # a step `env:`) so a leaked marker from one cell cannot forge a fence + # in another. See that step for the full rationale. + env: + REVIEW_TYPE: ${{ matrix.review_type }} + LEDGER_ROUNDS: ${{ needs.ledger.outputs.rounds }} + run: | + python3 "$CURSOR_REVIEW_ASSETS/build-ledger.py" splice \ + --prompt "$CURSOR_REVIEW_ASSETS/prompt-${REVIEW_TYPE}.md" \ + --marker '=== BEGIN DIFF ===' \ + --insert /tmp/ledger/ledger.md \ + --out /tmp/prompt-head.txt + diff_nonce="$(python3 "$CURSOR_REVIEW_ASSETS/fence-diff.py" mint)" + python3 "$CURSOR_REVIEW_ASSETS/fence-diff.py" strip-marker \ + --file /tmp/prompt-head.txt \ + --marker '=== BEGIN DIFF ===' + { + cat /tmp/prompt-head.txt + python3 "$CURSOR_REVIEW_ASSETS/fence-diff.py" emit \ + --body /tmp/pr-diff.patch --label DIFF --nonce "$diff_nonce" + if [ -s /tmp/pr-diff-new.patch ]; then + echo "" + echo "The block below is the subset of the diff above that changed since" + echo "the last reviewed commit. Prioritize it — but the full diff above" + echo "is still in scope." + python3 "$CURSOR_REVIEW_ASSETS/fence-diff.py" emit \ + --body /tmp/pr-diff-new.patch \ + --label "HUNKS NEW SINCE ROUND ${LEDGER_ROUNDS:-?}" \ + --nonce "$diff_nonce" + fi + } > /tmp/prompt.txt + + - name: Run direct-API review + if: steps.key.outputs.present == 'true' + continue-on-error: true + # Step-capped below the job, as in `review-openai-direct`. + timeout-minutes: ${{ inputs.cell_timeout_minutes }} + working-directory: _pr + env: + MODEL: ${{ inputs.anthropic_direct_model }} + EFFORT: ${{ needs.preflight.outputs.anthropic_direct_effort }} + REVIEW_TYPE: ${{ matrix.review_type }} + # Process env of this ONE step only: never argv, never a file. + ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }} + # No telemetry or error report may carry private-diff context. + CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC: '1' + run: | + mkdir -p /tmp/findings-out + # A fresh, empty config dir under RUNNER_TEMP, as CODEX_HOME is. + CLAUDE_CONFIG_DIR="$(mktemp -d -p "$RUNNER_TEMP")" + export CLAUDE_CONFIG_DIR + python3 "$CURSOR_REVIEW_ASSETS/review-output-mcp.py" \ + --mode reviewer \ + --out /tmp/findings-out/findings.json \ + --model "$MODEL" \ + --review-type "$REVIEW_TYPE" \ + --init + # The SAME MCP submission tool the Cursor and Codex cells use, as the + # only MCP server (`--strict-mcp-config`). The agent's tools are + # Read, Grep and Glob, confined to this checkout by `--restricted`, + # plus the two submission calls. + mcp_config="$(python3 - <<'PY' + import json, os + print(json.dumps({"mcpServers": {"cursor-review-output": { + "command": "python3", + "args": [ + os.path.join(os.environ["CURSOR_REVIEW_ASSETS"], "review-output-mcp.py"), + "--mode", "reviewer", + "--out", "/tmp/findings-out/findings.json", + "--model", os.environ["MODEL"], + "--review-type", os.environ["REVIEW_TYPE"], + ], + }}})) + PY + )" + # The result JSON goes to a file that is never echoed: the run log + # is public and the result is model output steered by PR text. + claude -p --bare --restricted \ + --tools "Read,Grep,Glob" \ + --strict-mcp-config --mcp-config "$mcp_config" \ + --allowedTools "Read,Grep,Glob,mcp__cursor-review-output__cursor_review_record_finding,mcp__cursor-review-output__cursor_review_finish" \ + --permission-prompts none \ + --model "$MODEL" --effort "$EFFORT" \ + --output-format json --no-session-persistence \ + < /tmp/prompt.txt > /tmp/claude-result.json + + - name: Refuse a record carrying the API key + # groom.yml's finder guard: should a prompt-injected agent get the key + # into its findings, error the record rather than publish it (to the + # judge, and from there to the PR). Literal match only. + if: always() + env: + MODEL: ${{ inputs.anthropic_direct_model }} + REVIEW_TYPE: ${{ matrix.review_type }} + ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }} + run: | + set -uo pipefail + out=/tmp/findings-out/findings.json + if [ -n "${ANTHROPIC_API_KEY:-}" ] && [ -f "$out" ] \ + && LC_ALL=C grep -qF -- "$ANTHROPIC_API_KEY" "$out"; then + echo "::error::direct-API Anthropic findings contain ANTHROPIC_API_KEY — refusing to publish (possible prompt-injection exfil)." + python3 - <<'PY' + import json, os + record = { + "model": os.environ["MODEL"], + "review_type": os.environ["REVIEW_TYPE"], + "status": "error", + "error": "findings contained the API key; refusing to publish", + "findings": [], + } + with open("/tmp/findings-out/findings.json", "w", encoding="utf-8") as f: + json.dump(record, f, indent=2) + PY + fi + + - name: Report cell outcome + if: always() + env: + MODEL: ${{ inputs.anthropic_direct_model }} + REVIEW_TYPE: ${{ matrix.review_type }} + COUNTS: ${{ needs.preflight.outputs.anthropic_direct_counts }} + # Status and a count only — NOT the findings verbatim, unlike the Cursor + # cells: the run log is public, and finding bodies are model output + # steered by PR-authored text. The full record still reaches the judge + # through the artifact below. + run: | + read -r status count < <(python3 - <<'PY' + import json, re + try: + with open("/tmp/findings-out/findings.json", encoding="utf-8") as f: + cell = json.load(f) + except (OSError, ValueError): + cell = None + value = cell.get("status") if isinstance(cell, dict) else None + if not isinstance(value, str): + value = "error" + findings = cell.get("findings") if isinstance(cell, dict) else None + print(re.sub(r"[^A-Za-z0-9_.-]", "_", value)[:32] or "error", + len(findings) if isinstance(findings, list) else 0) + PY + ) + echo "Direct-API cell: status=${status}, ${count} finding(s)." + if [ "$status" = "ok" ]; then + echo "Direct-API cell ${REVIEW_TYPE}/${MODEL} submitted its findings." + elif [ "$COUNTS" = "true" ]; then + # Failed AFTER the upload below, by `Fail the leg when the cell did + # not submit`, so the errored record still reaches the judge. + echo "Direct-API cell ${REVIEW_TYPE}/${MODEL} did not submit (status=$status); it replaces the Cursor Anthropic lane, so it counts as a failed reviewer." + else + echo "::warning::Direct-API cell ${REVIEW_TYPE}/${MODEL} did not submit (status=$status). Its record still reaches the judge; this leg stays green because the cell is advisory." + fi + + - name: Upload findings artifact + if: always() + # `findings-*` so `consolidate` globs it; `direct-` so it can never + # collide with a Cursor cell's `findings--` (with + # replacement off the two may share an id) and so `consolidate` can + # tell it apart. The id is charset-checked in preflight. + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 + with: + name: findings-direct-${{ matrix.review_type }}-${{ inputs.anthropic_direct_model }} + path: /tmp/findings-out/findings.json + if-no-files-found: error + retention-days: 7 + + - name: Fail the leg when the cell did not submit + # Only when these cells REPLACE the Cursor Anthropic lane — the same + # rollup-visibility contract as the `review` job's step of this name + # (BE-15554), and like it deliberately AFTER the upload, so the errored + # record still reaches the judge and the consolidated review still + # posts. Side by side the cell is advisory and this step is skipped. + # Same availability-not-attestation scope as that step: the status is + # read from a file this cell's own agent could write. + if: always() && needs.preflight.outputs.anthropic_direct_counts == 'true' + env: + MODEL: ${{ inputs.anthropic_direct_model }} + REVIEW_TYPE: ${{ matrix.review_type }} + run: | + # Read defensively and sanitized for a ::workflow command:: line, as + # in the `review` job (`ValueError` covers invalid UTF-8 too). + status=$(python3 - <<'PY' + import json, re + + try: + with open("/tmp/findings-out/findings.json", encoding="utf-8") as f: + cell = json.load(f) + except (OSError, ValueError): + cell = None + value = cell.get("status") if isinstance(cell, dict) else None + if not isinstance(value, str): + value = "error" + print(re.sub(r"[^A-Za-z0-9_.-]", "_", value)[:32] or "error") + PY + ) + if [ "$status" != "ok" ]; then + echo "::error::Direct-API reviewer cell ${REVIEW_TYPE}/${MODEL} did not submit a review (status=$status). It replaces the Cursor Anthropic lane, so it counts as a failed reviewer; its findings artifact was uploaded and the panel still consolidates." + exit 1 + fi + echo "Direct-API reviewer cell ${REVIEW_TYPE}/${MODEL} submitted its findings." + consolidate: # Adjudicates the panel cell artifacts into one consolidated review. # Calls cursor-agent one more time as the judge, with prompt-judge.md. @@ -3320,13 +3845,13 @@ jobs: # "Credential isolation" note for why the judge must not share a job with # a write-scoped token. name: Consolidate panel - # `review-openai-direct` is here for ORDERING — its artifacts must land - # before `Download panel findings` runs. It never gates this job: when it + # `review-openai-direct` and `review-anthropic-direct` are here for + # ORDERING — their artifacts must land before `Download panel findings` runs. It never gates this job: when it # is skipped (the default) or fails, `!cancelled()` below still runs, and # a failed cell is counted from its artifact (or its absence) instead. Its # result is read once, by `Aggregate panel findings`, as a consistency # check on those artifacts (see aggregate-panel.py). - needs: [gate, diff-size, preflight, review, review-openai-direct, ledger] + needs: [gate, diff-size, preflight, review, review-openai-direct, review-anthropic-direct, ledger] # `!cancelled()`, not `always()`: the judge is the most expensive call in # the workflow, and the only job that does anything with its payload, # `post-review`, is itself `!cancelled()`. On a cancelled run — now an @@ -3553,6 +4078,10 @@ jobs: # post-review failure — either way, panel_inconsistent=true. REVIEW_RESULT: ${{ needs.review.result }} DIRECT_RESULT: ${{ needs.review-openai-direct.result }} + # The same three for the direct Anthropic cells, counted per lab. + ANTHROPIC_DIRECT_MODEL: ${{ needs.preflight.outputs.anthropic_direct == 'true' && inputs.anthropic_direct_model || '' }} + ANTHROPIC_DIRECT_COUNTS: ${{ needs.preflight.outputs.anthropic_direct_counts }} + ANTHROPIC_DIRECT_RESULT: ${{ needs.review-anthropic-direct.result }} run: | # Each artifact directory contains one findings.json. Combine into # a single panel.json array preserving cell metadata, and emit the @@ -3565,6 +4094,9 @@ jobs: --direct-counts "${DIRECT_COUNTS:-false}" \ --review-result "$REVIEW_RESULT" \ --direct-result "${DIRECT_RESULT:-skipped}" \ + --anthropic-direct-model "$ANTHROPIC_DIRECT_MODEL" \ + --anthropic-direct-counts "${ANTHROPIC_DIRECT_COUNTS:-false}" \ + --anthropic-direct-result "${ANTHROPIC_DIRECT_RESULT:-skipped}" \ --out /tmp/panel.json \ --github-output "$GITHUB_OUTPUT" diff --git a/docs/callers/cursor-review.md b/docs/callers/cursor-review.md index ccfacb06..8a8a7b93 100644 --- a/docs/callers/cursor-review.md +++ b/docs/callers/cursor-review.md @@ -53,6 +53,7 @@ Prompts and scripts live in [`.github/cursor-review/`](../../.github/cursor-revi |---|---| | `secrets.CURSOR_API_KEY` | **Required.** Org-level in Comfy-Org. | | `secrets.OPENAI_API_KEY` | Optional. Only read when `openai_direct_model` is set; use a project-scoped service-account key with a spend limit. | +| `secrets.ANTHROPIC_API_KEY` | Optional. Only read when `anthropic_direct_model` is set; use a key scoped to its own Anthropic workspace with a spend limit, never a personal key. | | `secrets.SLACK_BOT_TOKEN` | Optional. Without it the review still posts; only the DMs are skipped. | | `vars.REVIEW_BOT_APP_ID` + `secrets.BOT_APP_PRIVATE_KEY` | Optional. Posts the review as your App instead of `github-actions[bot]`. | | A review label | Default `cursor-review`. Create it in your repo. | @@ -115,6 +116,9 @@ pull-requests: write # posting the consolidated review (and, at the round cap, | `panel_models` | `''` | JSON array of model ids replacing the built-in panel list (each runs both review types; preflight checks each one exists in the live catalog — existence only, not ZDR). Use for per-repo experiments such as a reasoning-tier A/B — and prefer wiring it to a **variable** rather than a literal, see [Running a model experiment](#running-a-model-experiment). | | `openai_direct_model` | `''` | Opt-in. An OpenAI model id reviewed directly through the OpenAI API (pinned Codex CLI) as two cells — **adversarial** and **edge-case**, with the same prompts the Cursor cells use — always on GitHub-hosted `ubuntu-latest`. By default they **replace** the Cursor-hosted OpenAI (`gpt-*`) panel cells — see the next row, which also decides whether they gate. Requires `secrets.OPENAI_API_KEY` and a plain id (`[A-Za-z0-9._-]`); without either, preflight warns, the cells stay off and the panel is unchanged. | | `openai_direct_replaces_cursor_openai` | `true` | Only with `openai_direct_model` (and its key) in effect: drop the Cursor `gpt-*` cell(s) from the panel so the OpenAI lane is reviewed once. The direct cells then **gate** exactly like the Cursor cells they replace: they count toward `Panel integrity` and the auto-approve gate (`approve_max_failed_reviewers`), a failed one turns its leg red, and the review's "did not contribute" line names it, e.g. `gpt-5.6-sol:edge-case (direct, error)`. `false` runs both, for a side-by-side comparison of the two backends; the direct cells are then **advisory** — their findings reach the judge, but they are never counted and their legs never fail the run. Preflight fails rather than leave the Cursor panel empty. | +| `anthropic_direct_model` | `''` | Opt-in. The Anthropic twin of `openai_direct_model`: an Anthropic model id (e.g. `claude-opus-5-5`) reviewed directly through the Anthropic API (pinned Claude Code CLI) as two cells — **adversarial** and **edge-case** — always on GitHub-hosted `ubuntu-latest`, by default **replacing** the Cursor `claude-*` cells. Requires `secrets.ANTHROPIC_API_KEY` and a plain id (`[A-Za-z0-9._-]`); without either, preflight warns, the cells stay off and the panel is unchanged. Unlike the OpenAI cells, the agent also reads a **read-only PR checkout** with Read, Grep and Glob only — no Bash or code-running tool — and `--restricted` confines those file tools to the checkout. **Private repos:** this panel reviews private-repo diffs and the workflow cannot check the Anthropic organization's data-retention terms, so use a key from an organization with zero data retention. | +| `anthropic_direct_effort` | `xhigh` | Reasoning effort for the direct Anthropic cells: `low`, `medium`, `high`, `xhigh` or `max`. Any other value warns and falls back to `xhigh`. | +| `anthropic_direct_replaces_cursor_anthropic` | `true` | Same semantics as `openai_direct_replaces_cursor_openai`, for the Cursor `claude-*` cells: `true` drops them and the direct Anthropic cells **gate** in their place; `false` keeps both, with the direct cells **advisory**. Each lab counts by its own flag, so OpenAI and Anthropic may be set differently. The judge (`judge_model`) is never touched. | | `skip_bot_branch_prefixes` | `ci/bump- chore/refresh- auto/refresh-` | Skip the panel for Bot-authored PRs on these branch prefixes (machine pin bumps / refreshes). `''` to review every bot PR. | | `cell_timeout_minutes` | `15` | Wall-clock cap on ONE reviewer cell's agent step (`Run cursor review`, or `Run direct-API review` for a direct cell). A cell that runs out of clock exits 130 and uploads `status: error` with no findings, which fails its leg and so fails `Panel integrity` — so this is a guillotine, not a safety net. Raise it when cells are dying at the cap rather than hanging: compare how long the cells that DID submit took, and if a meaningful share finish within a few minutes of the cap you are clipping a tail, not catching hangs. It is also the budget a cell's retries share (`resource_exhausted` is retried in-step), so it bounds attempts plus backoff. Three further caps derive from this one and cannot be set separately — each cell's job cap is this + 15 (checkout, CLI install, artifacts), the judge's `Run judge` step cap is this × 2 but never below 30 (it reads every cell's findings at the same tier, and its workload scales with the panel, not this cap), and the judge's job cap is its step + 10. Bounded 10–45 (the floor leaves a late in-step retry room for a full review); preflight fails loud outside that, before any cell spends. | | `diff_size_cap` | `5000` | Skip review above this diff size. An over-cap PR is not silent — see the gotcha below. Under `blocking: true` it is also not green: an unreviewed PR cannot pass the gate. | From f82d7b936f456a1a56edb349ce029dc7bcc7ac27 Mon Sep 17 00:00:00 2001 From: Matt Miller Date: Fri, 9 Oct 2026 19:16:11 +0000 Subject: [PATCH 2/7] fix(cursor-review): harden the direct-Anthropic cells per review - Run every python3 in the PR-cwd step with -I, so a PR-planted json.py cannot load ahead of the stdlib with ANTHROPIC_API_KEY in its env. - Check out the PR head SHA, the reviewed diff's RIGHT side, not the merge ref. - Preflight turns the Anthropic cells off when their id equals openai_direct_model (artifact-name collision, misattributed records). - Refuse any whitespace in either direct model id; grep -x let a multi-line value through on one clean line. Co-Authored-By: Claude Opus 5.5 --- .github/cursor-review/README.md | 2 +- .github/workflows/cursor-review.yml | 40 +++++++++++++++++++++++------ docs/callers/cursor-review.md | 2 +- 3 files changed, 34 insertions(+), 10 deletions(-) diff --git a/.github/cursor-review/README.md b/.github/cursor-review/README.md index 276b4a17..c3267ad6 100644 --- a/.github/cursor-review/README.md +++ b/.github/cursor-review/README.md @@ -392,7 +392,7 @@ All optional except `workflows_ref` (required, no default) — pass them under | `panel_models` | `''` (built-in list) | JSON array of Cursor model ids that **replaces** the panel list; each still runs both review types and is validated against the live catalog by preflight. For per-repo experiments (e.g. `-xhigh` vs `-max` tiers). | | `openai_direct_model` | `''` (off) | OPT-IN. An OpenAI model id reviewed through the OpenAI API directly (pinned Codex CLI, `package.json` here) as two cells (**adversarial** + **edge-case**, the Cursor cells' prompts) on GitHub-hosted runners, by default **replacing** the Cursor `gpt-*` cells (next row). Needs the `OPENAI_API_KEY` secret and a plain id (`[A-Za-z0-9._-]`); without either, preflight warns and leaves the cells off and the panel unchanged. Not part of `panel_models` (not in the Cursor catalog). Whether the cells gate depends on the next row. | | `openai_direct_replaces_cursor_openai` | `true` | When the direct cells are in effect, drop the Cursor `gpt-*` cell(s) so the OpenAI lane is reviewed once — the direct cells then **gate** like the Cursor cells they replace: counted in `Panel integrity` and the auto-approve gate, a failed one red in the rollup and named `(direct, )` in the review's "did not contribute" line. `false` runs both for a side-by-side backend comparison, with the direct cells **advisory** (findings reach the judge; never counted, never red). Preflight fails rather than leave the Cursor panel empty. | -| `anthropic_direct_model` | `''` (off) | OPT-IN. The Anthropic twin of `openai_direct_model`: an Anthropic model id (e.g. `claude-opus-5-5`) reviewed through the Anthropic API directly (pinned Claude Code CLI, `package.json` here) as two cells on GitHub-hosted runners, by default **replacing** the Cursor `claude-*` cells. Needs the `ANTHROPIC_API_KEY` secret and a plain id; without either, preflight warns and leaves the cells off and the panel unchanged. The agent also reads a **read-only PR checkout** (symlinks removed) with Read, Grep and Glob only — no Bash or code-running tool — confined to it by `--restricted`. This panel reviews private-repo diffs and the workflow cannot check the Anthropic organization's data-retention terms: callers on private repos must use a key from an organization with zero data retention. | +| `anthropic_direct_model` | `''` (off) | OPT-IN. The Anthropic twin of `openai_direct_model`: an Anthropic model id (e.g. `claude-opus-5-5`) reviewed through the Anthropic API directly (pinned Claude Code CLI, `package.json` here) as two cells on GitHub-hosted runners, by default **replacing** the Cursor `claude-*` cells. Needs the `ANTHROPIC_API_KEY` secret and a plain id other than `openai_direct_model`'s; without any of those, preflight warns and leaves the cells off and the panel unchanged. The agent also reads a **read-only PR checkout** (symlinks removed) with Read, Grep and Glob only — no Bash or code-running tool — confined to it by `--restricted`. This panel reviews private-repo diffs and the workflow cannot check the Anthropic organization's data-retention terms: callers on private repos must use a key from an organization with zero data retention. | | `anthropic_direct_effort` | `xhigh` | `--effort` for the direct Anthropic cells: `low`, `medium`, `high`, `xhigh` or `max`; anything else warns and falls back to `xhigh`. | | `anthropic_direct_replaces_cursor_anthropic` | `true` | As `openai_direct_replaces_cursor_openai`, for the Cursor `claude-*` cells. Each lab counts by its own flag. Never touches `judge_model`. | | `cell_timeout_minutes` | `15` | Wall-clock cap on one reviewer cell's agent step (Cursor or direct). Over it the cell exits 130, uploads `status: error` with no findings, reddens its leg and fails `Panel integrity`. Shared by the cell's in-step retries, so it bounds attempts + backoff. Derives three more caps that cannot be set alone: cell job = this + 15, `Run judge` step = max(30, this × 2), judge job = that + 10. At the default: 15 / 30 / 30 / 40, today's values exactly. Preflight bounds it 10–45. | diff --git a/.github/workflows/cursor-review.yml b/.github/workflows/cursor-review.yml index 24febcd6..bc6983f1 100644 --- a/.github/workflows/cursor-review.yml +++ b/.github/workflows/cursor-review.yml @@ -322,8 +322,9 @@ on: Cursor model gets. The Anthropic twin of `openai_direct_model`, with the same rules: empty (default) = the job is skipped and nothing changes; requires the `ANTHROPIC_API_KEY` secret and a plain id - (`[A-Za-z0-9._-]`), and set without either, preflight warns, the - cells stay off and the panel is unchanged. Unlike the OpenAI cells, + (`[A-Za-z0-9._-]`) other than `openai_direct_model`'s, and set + without any of those, preflight warns, the cells stay off and the + panel is unchanged. Unlike the OpenAI cells, which see the diff only, the agent also reads a read-only checkout of the PR with Read, Grep and Glob alone — no Bash or other code-running tool — and `--restricted` confines those file tools to @@ -2329,7 +2330,12 @@ jobs: # exactly as it would be without the opt-in. openai_direct=false if [ -n "$(printf '%s' "${OPENAI_DIRECT_MODEL:-}" | tr -d '[:space:]')" ]; then - if ! printf '%s' "$OPENAI_DIRECT_MODEL" | grep -Eqx '[A-Za-z0-9._-]{1,100}'; then + # `grep -x` matches line by line, so a multi-line value would pass + # on any one clean line; refuse every whitespace character (a + # newline included, which a grep could not see) first. + case "$OPENAI_DIRECT_MODEL" in *[[:space:]]*) openai_ws=true ;; *) openai_ws=false ;; esac + if [ "$openai_ws" = "true" ] \ + || ! printf '%s' "$OPENAI_DIRECT_MODEL" | grep -Eqx '[A-Za-z0-9._-]{1,100}'; then echo "::warning::openai_direct_model is not a plain model id (letters, digits, '.', '_', '-'; at most 100), so the direct-API cell is off for this run and the panel is unchanged." elif [ "${OPENAI_KEY_PRESENT:-false}" != "true" ]; then echo "::warning::openai_direct_model is set but the OPENAI_API_KEY secret did not reach this workflow, so the direct-API cell is off for this run and the panel is unchanged. Pass the secret through from the caller to enable it." @@ -2364,10 +2370,22 @@ jobs: # never touched — the judge stays on Cursor. anthropic_direct=false if [ -n "$(printf '%s' "${ANTHROPIC_DIRECT_MODEL:-}" | tr -d '[:space:]')" ]; then - if ! printf '%s' "$ANTHROPIC_DIRECT_MODEL" | grep -Eqx '[A-Za-z0-9._-]{1,100}'; then + # `grep -x` matches line by line, so a multi-line value would pass + # on any one clean line; refuse every whitespace character (a + # newline included, which a grep could not see) first. + case "$ANTHROPIC_DIRECT_MODEL" in *[[:space:]]*) anthropic_ws=true ;; *) anthropic_ws=false ;; esac + if [ "$anthropic_ws" = "true" ] \ + || ! printf '%s' "$ANTHROPIC_DIRECT_MODEL" | grep -Eqx '[A-Za-z0-9._-]{1,100}'; then echo "::warning::anthropic_direct_model is not a plain model id (letters, digits, '.', '_', '-'; at most 100), so the direct-API Anthropic cells are off for this run and the panel is unchanged." elif [ "${ANTHROPIC_KEY_PRESENT:-false}" != "true" ]; then echo "::warning::anthropic_direct_model is set but the ANTHROPIC_API_KEY secret did not reach this workflow, so the direct-API Anthropic cells are off for this run and the panel is unchanged. Pass the secret through from the caller to enable it." + elif [ "$openai_direct" = "true" ] \ + && [ "$(printf '%s' "$ANTHROPIC_DIRECT_MODEL" | tr 'A-Z' 'a-z')" = "$(printf '%s' "$OPENAI_DIRECT_MODEL" | tr 'A-Z' 'a-z')" ]; then + # Both direct jobs name their artifact findings-direct--, + # so one id in both lanes collides (one upload fails) and the + # aggregator, which assigns by model id, credits every record to + # the OpenAI lab. + echo "::warning::anthropic_direct_model is the same id as openai_direct_model, so the direct-API Anthropic cells are off for this run and the panel is unchanged. Give each lane its own lab's model id." else anthropic_direct=true fi @@ -3547,10 +3565,12 @@ jobs: persist-credentials: false - name: Checkout PR repo - # The read-only tree the agent's file tools read: the default ref, so - # the same commit the Cursor cells review. No credential persisted. + # The read-only tree the agent's file tools read: the PR head, the + # RIGHT side of the reviewed diff (the default merge ref would mix in + # base-only changes and shift line numbers). No credential persisted. uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: + ref: ${{ github.event.pull_request.head.sha }} path: _pr fetch-depth: 1 persist-credentials: false @@ -3687,7 +3707,10 @@ jobs: # A fresh, empty config dir under RUNNER_TEMP, as CODEX_HOME is. CLAUDE_CONFIG_DIR="$(mktemp -d -p "$RUNNER_TEMP")" export CLAUDE_CONFIG_DIR - python3 "$CURSOR_REVIEW_ASSETS/review-output-mcp.py" \ + # `-I` on every python3 here: the cwd is the PR checkout, and a stdin + # script puts '' (the cwd) first on sys.path, so a PR-planted json.py + # would run with ANTHROPIC_API_KEY in its env. Isolated mode drops it. + python3 -I "$CURSOR_REVIEW_ASSETS/review-output-mcp.py" \ --mode reviewer \ --out /tmp/findings-out/findings.json \ --model "$MODEL" \ @@ -3697,11 +3720,12 @@ jobs: # only MCP server (`--strict-mcp-config`). The agent's tools are # Read, Grep and Glob, confined to this checkout by `--restricted`, # plus the two submission calls. - mcp_config="$(python3 - <<'PY' + mcp_config="$(python3 -I - <<'PY' import json, os print(json.dumps({"mcpServers": {"cursor-review-output": { "command": "python3", "args": [ + "-I", os.path.join(os.environ["CURSOR_REVIEW_ASSETS"], "review-output-mcp.py"), "--mode", "reviewer", "--out", "/tmp/findings-out/findings.json", diff --git a/docs/callers/cursor-review.md b/docs/callers/cursor-review.md index 8a8a7b93..a5111659 100644 --- a/docs/callers/cursor-review.md +++ b/docs/callers/cursor-review.md @@ -116,7 +116,7 @@ pull-requests: write # posting the consolidated review (and, at the round cap, | `panel_models` | `''` | JSON array of model ids replacing the built-in panel list (each runs both review types; preflight checks each one exists in the live catalog — existence only, not ZDR). Use for per-repo experiments such as a reasoning-tier A/B — and prefer wiring it to a **variable** rather than a literal, see [Running a model experiment](#running-a-model-experiment). | | `openai_direct_model` | `''` | Opt-in. An OpenAI model id reviewed directly through the OpenAI API (pinned Codex CLI) as two cells — **adversarial** and **edge-case**, with the same prompts the Cursor cells use — always on GitHub-hosted `ubuntu-latest`. By default they **replace** the Cursor-hosted OpenAI (`gpt-*`) panel cells — see the next row, which also decides whether they gate. Requires `secrets.OPENAI_API_KEY` and a plain id (`[A-Za-z0-9._-]`); without either, preflight warns, the cells stay off and the panel is unchanged. | | `openai_direct_replaces_cursor_openai` | `true` | Only with `openai_direct_model` (and its key) in effect: drop the Cursor `gpt-*` cell(s) from the panel so the OpenAI lane is reviewed once. The direct cells then **gate** exactly like the Cursor cells they replace: they count toward `Panel integrity` and the auto-approve gate (`approve_max_failed_reviewers`), a failed one turns its leg red, and the review's "did not contribute" line names it, e.g. `gpt-5.6-sol:edge-case (direct, error)`. `false` runs both, for a side-by-side comparison of the two backends; the direct cells are then **advisory** — their findings reach the judge, but they are never counted and their legs never fail the run. Preflight fails rather than leave the Cursor panel empty. | -| `anthropic_direct_model` | `''` | Opt-in. The Anthropic twin of `openai_direct_model`: an Anthropic model id (e.g. `claude-opus-5-5`) reviewed directly through the Anthropic API (pinned Claude Code CLI) as two cells — **adversarial** and **edge-case** — always on GitHub-hosted `ubuntu-latest`, by default **replacing** the Cursor `claude-*` cells. Requires `secrets.ANTHROPIC_API_KEY` and a plain id (`[A-Za-z0-9._-]`); without either, preflight warns, the cells stay off and the panel is unchanged. Unlike the OpenAI cells, the agent also reads a **read-only PR checkout** with Read, Grep and Glob only — no Bash or code-running tool — and `--restricted` confines those file tools to the checkout. **Private repos:** this panel reviews private-repo diffs and the workflow cannot check the Anthropic organization's data-retention terms, so use a key from an organization with zero data retention. | +| `anthropic_direct_model` | `''` | Opt-in. The Anthropic twin of `openai_direct_model`: an Anthropic model id (e.g. `claude-opus-5-5`) reviewed directly through the Anthropic API (pinned Claude Code CLI) as two cells — **adversarial** and **edge-case** — always on GitHub-hosted `ubuntu-latest`, by default **replacing** the Cursor `claude-*` cells. Requires `secrets.ANTHROPIC_API_KEY` and a plain id (`[A-Za-z0-9._-]`) other than `openai_direct_model`'s; without any of those, preflight warns, the cells stay off and the panel is unchanged. Unlike the OpenAI cells, the agent also reads a **read-only PR checkout** with Read, Grep and Glob only — no Bash or code-running tool — and `--restricted` confines those file tools to the checkout. **Private repos:** this panel reviews private-repo diffs and the workflow cannot check the Anthropic organization's data-retention terms, so use a key from an organization with zero data retention. | | `anthropic_direct_effort` | `xhigh` | Reasoning effort for the direct Anthropic cells: `low`, `medium`, `high`, `xhigh` or `max`. Any other value warns and falls back to `xhigh`. | | `anthropic_direct_replaces_cursor_anthropic` | `true` | Same semantics as `openai_direct_replaces_cursor_openai`, for the Cursor `claude-*` cells: `true` drops them and the direct Anthropic cells **gate** in their place; `false` keeps both, with the direct cells **advisory**. Each lab counts by its own flag, so OpenAI and Anthropic may be set differently. The judge (`judge_model`) is never touched. | | `skip_bot_branch_prefixes` | `ci/bump- chore/refresh- auto/refresh-` | Skip the panel for Bot-authored PRs on these branch prefixes (machine pin bumps / refreshes). `''` to review every bot PR. | From 49bec9595c3ce343800a2b17c688327a8292af2e Mon Sep 17 00:00:00 2001 From: Matt Miller Date: Fri, 9 Oct 2026 19:21:21 +0000 Subject: [PATCH 3/7] fix(cursor-review): load no PR-authored Claude config in the direct-Anthropic cells Pass --setting-sources "" to claude (as groom.yml does) and remove the PR checkout's .claude/, .mcp.json and CLAUDE.local.md before the agent runs, so PR-planted settings cannot re-point ANTHROPIC_BASE_URL, add hooks or widen file access. Pinned by a job-isolation test. Co-Authored-By: Claude Opus 5.5 --- .../tests/test_workflow_job_isolation.py | 15 +++++++++++++++ .github/workflows/cursor-review.yml | 11 ++++++++++- 2 files changed, 25 insertions(+), 1 deletion(-) diff --git a/.github/cursor-review/tests/test_workflow_job_isolation.py b/.github/cursor-review/tests/test_workflow_job_isolation.py index a22dfcd4..63714eec 100644 --- a/.github/cursor-review/tests/test_workflow_job_isolation.py +++ b/.github/cursor-review/tests/test_workflow_job_isolation.py @@ -315,6 +315,21 @@ def test_the_anthropic_key_reaches_only_the_gate_and_its_direct_cells(self): runs_on = [l.strip() for l in self.jobs["review-anthropic-direct"] if l.startswith(" runs-on:")] self.assertEqual(runs_on, ["runs-on: ubuntu-latest"]) + def test_the_anthropic_cells_load_no_pr_authored_claude_config(self): + # `claude` runs with the PR checkout as cwd, so PR-authored + # `.claude/settings*.json` or `.mcp.json` could re-point the API key at + # another host. Pin both halves: no setting source is loaded, and the + # files are removed from the checkout before the agent runs. + body = code_lines(self.jobs["review-anthropic-direct"]) + invocations = [l for l in body if "claude -p" in l] + self.assertEqual(len(invocations), 1, invocations) + for flag in ("--bare", "--restricted", '--setting-sources ""'): + self.assertIn(flag, invocations[0]) + self.assertTrue( + any("rm -rf" in l and "_pr/.claude" in l and "_pr/.mcp.json" in l for l in body), + "review-anthropic-direct no longer removes the PR's .claude/ and .mcp.json", + ) + def test_only_credential_free_jobs_follow_the_runs_on_input(self): # `runs_on` hands jobs to a caller-chosen pool where a prompt-injected # model cell may have run. Anything holding the bot key, a write scope, diff --git a/.github/workflows/cursor-review.yml b/.github/workflows/cursor-review.yml index bc6983f1..0443856d 100644 --- a/.github/workflows/cursor-review.yml +++ b/.github/workflows/cursor-review.yml @@ -3580,6 +3580,15 @@ jobs: # tree (e.g. to the runner's home), so none survive into the review. run: find _pr -type l -delete + - name: Remove PR-authored Claude config + # The Cursor cells' `rm -rf .cursor`, for this CLI: `_pr` is claude's + # cwd, so a PR-planted `.claude/settings*.json` (env such as + # ANTHROPIC_BASE_URL, hooks, extra directories) or `.mcp.json` must + # never be there to load. `--restricted` and `--setting-sources ""` + # already ignore them; this removes them outright. CLAUDE.md stays — + # `--bare` skips its auto-discovery, and it is ordinary review context. + run: rm -rf _pr/.claude _pr/.mcp.json _pr/CLAUDE.local.md + - name: Download reviewed diff # Built once in the diff-size job (generated files already excluded), # shared here, so this cell needs no PR checkout of its own. @@ -3737,7 +3746,7 @@ jobs: )" # The result JSON goes to a file that is never echoed: the run log # is public and the result is model output steered by PR text. - claude -p --bare --restricted \ + claude -p --bare --restricted --setting-sources "" \ --tools "Read,Grep,Glob" \ --strict-mcp-config --mcp-config "$mcp_config" \ --allowedTools "Read,Grep,Glob,mcp__cursor-review-output__cursor_review_record_finding,mcp__cursor-review-output__cursor_review_finish" \ From e2a926cd2596e832455e9ecc1516fa404c7f5520 Mon Sep 17 00:00:00 2001 From: Matt Miller Date: Fri, 9 Oct 2026 20:40:30 +0000 Subject: [PATCH 4/7] fix(cursor-review): deny /proc and /sys to the direct-Anthropic agent independently of --restricted ANTHROPIC_API_KEY is in the env of the process serving Read/Grep/Glob, so /proc/self/environ is the key. --restricted alone confined the file tools; add explicit deny rules as a second layer, and fail the cell by name when the pinned CLI stops advertising any flag the confinement relies on. Co-Authored-By: Claude Opus 5.5 --- .github/cursor-review/README.md | 2 +- .../tests/test_workflow_job_isolation.py | 22 ++++++++++++++++ .github/workflows/cursor-review.yml | 25 ++++++++++++++++--- docs/callers/cursor-review.md | 2 +- 4 files changed, 45 insertions(+), 6 deletions(-) diff --git a/.github/cursor-review/README.md b/.github/cursor-review/README.md index c3267ad6..bb412330 100644 --- a/.github/cursor-review/README.md +++ b/.github/cursor-review/README.md @@ -392,7 +392,7 @@ All optional except `workflows_ref` (required, no default) — pass them under | `panel_models` | `''` (built-in list) | JSON array of Cursor model ids that **replaces** the panel list; each still runs both review types and is validated against the live catalog by preflight. For per-repo experiments (e.g. `-xhigh` vs `-max` tiers). | | `openai_direct_model` | `''` (off) | OPT-IN. An OpenAI model id reviewed through the OpenAI API directly (pinned Codex CLI, `package.json` here) as two cells (**adversarial** + **edge-case**, the Cursor cells' prompts) on GitHub-hosted runners, by default **replacing** the Cursor `gpt-*` cells (next row). Needs the `OPENAI_API_KEY` secret and a plain id (`[A-Za-z0-9._-]`); without either, preflight warns and leaves the cells off and the panel unchanged. Not part of `panel_models` (not in the Cursor catalog). Whether the cells gate depends on the next row. | | `openai_direct_replaces_cursor_openai` | `true` | When the direct cells are in effect, drop the Cursor `gpt-*` cell(s) so the OpenAI lane is reviewed once — the direct cells then **gate** like the Cursor cells they replace: counted in `Panel integrity` and the auto-approve gate, a failed one red in the rollup and named `(direct, )` in the review's "did not contribute" line. `false` runs both for a side-by-side backend comparison, with the direct cells **advisory** (findings reach the judge; never counted, never red). Preflight fails rather than leave the Cursor panel empty. | -| `anthropic_direct_model` | `''` (off) | OPT-IN. The Anthropic twin of `openai_direct_model`: an Anthropic model id (e.g. `claude-opus-5-5`) reviewed through the Anthropic API directly (pinned Claude Code CLI, `package.json` here) as two cells on GitHub-hosted runners, by default **replacing** the Cursor `claude-*` cells. Needs the `ANTHROPIC_API_KEY` secret and a plain id other than `openai_direct_model`'s; without any of those, preflight warns and leaves the cells off and the panel unchanged. The agent also reads a **read-only PR checkout** (symlinks removed) with Read, Grep and Glob only — no Bash or code-running tool — confined to it by `--restricted`. This panel reviews private-repo diffs and the workflow cannot check the Anthropic organization's data-retention terms: callers on private repos must use a key from an organization with zero data retention. | +| `anthropic_direct_model` | `''` (off) | OPT-IN. The Anthropic twin of `openai_direct_model`: an Anthropic model id (e.g. `claude-opus-5-5`) reviewed through the Anthropic API directly (pinned Claude Code CLI, `package.json` here) as two cells on GitHub-hosted runners, by default **replacing** the Cursor `claude-*` cells. Needs the `ANTHROPIC_API_KEY` secret and a plain id other than `openai_direct_model`'s; without any of those, preflight warns and leaves the cells off and the panel unchanged. The agent also reads a **read-only PR checkout** (symlinks removed) with Read, Grep and Glob only — no Bash or code-running tool — confined to it by `--restricted`, with `/proc` and `/sys` also denied by explicit deny rules. This panel reviews private-repo diffs and the workflow cannot check the Anthropic organization's data-retention terms: callers on private repos must use a key from an organization with zero data retention. | | `anthropic_direct_effort` | `xhigh` | `--effort` for the direct Anthropic cells: `low`, `medium`, `high`, `xhigh` or `max`; anything else warns and falls back to `xhigh`. | | `anthropic_direct_replaces_cursor_anthropic` | `true` | As `openai_direct_replaces_cursor_openai`, for the Cursor `claude-*` cells. Each lab counts by its own flag. Never touches `judge_model`. | | `cell_timeout_minutes` | `15` | Wall-clock cap on one reviewer cell's agent step (Cursor or direct). Over it the cell exits 130, uploads `status: error` with no findings, reddens its leg and fails `Panel integrity`. Shared by the cell's in-step retries, so it bounds attempts + backoff. Derives three more caps that cannot be set alone: cell job = this + 15, `Run judge` step = max(30, this × 2), judge job = that + 10. At the default: 15 / 30 / 30 / 40, today's values exactly. Preflight bounds it 10–45. | diff --git a/.github/cursor-review/tests/test_workflow_job_isolation.py b/.github/cursor-review/tests/test_workflow_job_isolation.py index 63714eec..a54b43f2 100644 --- a/.github/cursor-review/tests/test_workflow_job_isolation.py +++ b/.github/cursor-review/tests/test_workflow_job_isolation.py @@ -330,6 +330,28 @@ def test_the_anthropic_cells_load_no_pr_authored_claude_config(self): "review-anthropic-direct no longer removes the PR's .claude/ and .mcp.json", ) + def test_the_anthropic_cells_deny_proc_independently_of_restricted(self): + # ANTHROPIC_API_KEY is in the env of the process serving Read/Grep/Glob, + # so `/proc/self/environ` is the key. `--restricted` confines the file + # tools to the checkout; the deny rules are a second layer that holds + # even if a CLI bump loosens that flag, and the verify step fails the + # cell by name if the pinned CLI stops advertising any flag relied on. + body = code_lines(self.jobs["review-anthropic-direct"]) + invocation = [l for l in body if "claude -p" in l] + self.assertEqual(len(invocation), 1, invocation) + start = body.index(invocation[0]) + argv = [] + for line in body[start:]: + argv.append(line) + if not line.rstrip().endswith("\\"): + break + argv = " ".join(argv) + self.assertIn('--disallowedTools "Read(//proc/**),Read(//sys/**)"', argv) + self.assertIn("--permission-prompts none", argv) + help_check = " ".join(body) + for flag in ("--restricted", "--disallowedTools", "--permission-prompts", "--setting-sources"): + self.assertRegex(help_check, rf"for flag in [^;]*{flag}\b", flag) + def test_only_credential_free_jobs_follow_the_runs_on_input(self): # `runs_on` hands jobs to a caller-chosen pool where a prompt-injected # model cell may have run. Anything holding the bot key, a write scope, diff --git a/.github/workflows/cursor-review.yml b/.github/workflows/cursor-review.yml index 0443856d..d9b8f15e 100644 --- a/.github/workflows/cursor-review.yml +++ b/.github/workflows/cursor-review.yml @@ -3457,9 +3457,11 @@ jobs: # # Credential posture: `contents: read` only, no write-scoped token (see # SECURITY.md); the PR checkout persists no credential. ANTHROPIC_API_KEY - # reaches `claude` in that ONE step's env and is never written to disk, - # and the agent has no tool that can read the environment. Belt and - # braces, groom's finder guard runs on the record before it uploads. + # reaches `claude` in that ONE step's env and is never written to disk. + # The agent cannot run code, and its file tools cannot reach + # `/proc/self/environ`: `--restricted` confines them to the checkout, and + # deny rules on /proc and /sys hold independently of that flag. The + # literal-match guard on the record before upload is a last tripwire. name: '${{ matrix.review_type }} direct (${{ inputs.anthropic_direct_model }})' needs: [gate, diff-size, preflight, ledger] if: >- @@ -3657,6 +3659,17 @@ jobs: exit 1 fi echo "claude ${installed}" + # The confinement rests on these flags, so a bump whose CLI dropped + # or renamed one fails HERE, by name, rather than at argv parsing + # mid-review — or, worse, by silently running unconfined. + help="$(claude --help 2>&1)" + for flag in --bare --restricted --setting-sources --strict-mcp-config \ + --disallowedTools --permission-prompts --no-session-persistence; do + if ! printf '%s\n' "$help" | grep -qE -- "(^|[[:space:],])${flag}([[:space:],]|$)"; then + echo "::error::claude ${installed} does not advertise ${flag}, which the direct-API cell's confinement relies on." + exit 1 + fi + done - name: Build prompt if: steps.key.outputs.present == 'true' @@ -3728,7 +3741,10 @@ jobs: # The SAME MCP submission tool the Cursor and Codex cells use, as the # only MCP server (`--strict-mcp-config`). The agent's tools are # Read, Grep and Glob, confined to this checkout by `--restricted`, - # plus the two submission calls. + # plus the two submission calls. The `--disallowedTools` deny rules + # (a Read rule covers Grep and Glob too) are a second, independent + # layer over /proc and /sys — `/proc/self/environ` holds the key — so + # a CLI bump that loosened `--restricted` alone still cannot reach it. mcp_config="$(python3 -I - <<'PY' import json, os print(json.dumps({"mcpServers": {"cursor-review-output": { @@ -3750,6 +3766,7 @@ jobs: --tools "Read,Grep,Glob" \ --strict-mcp-config --mcp-config "$mcp_config" \ --allowedTools "Read,Grep,Glob,mcp__cursor-review-output__cursor_review_record_finding,mcp__cursor-review-output__cursor_review_finish" \ + --disallowedTools "Read(//proc/**),Read(//sys/**)" \ --permission-prompts none \ --model "$MODEL" --effort "$EFFORT" \ --output-format json --no-session-persistence \ diff --git a/docs/callers/cursor-review.md b/docs/callers/cursor-review.md index a5111659..2b6cc60d 100644 --- a/docs/callers/cursor-review.md +++ b/docs/callers/cursor-review.md @@ -116,7 +116,7 @@ pull-requests: write # posting the consolidated review (and, at the round cap, | `panel_models` | `''` | JSON array of model ids replacing the built-in panel list (each runs both review types; preflight checks each one exists in the live catalog — existence only, not ZDR). Use for per-repo experiments such as a reasoning-tier A/B — and prefer wiring it to a **variable** rather than a literal, see [Running a model experiment](#running-a-model-experiment). | | `openai_direct_model` | `''` | Opt-in. An OpenAI model id reviewed directly through the OpenAI API (pinned Codex CLI) as two cells — **adversarial** and **edge-case**, with the same prompts the Cursor cells use — always on GitHub-hosted `ubuntu-latest`. By default they **replace** the Cursor-hosted OpenAI (`gpt-*`) panel cells — see the next row, which also decides whether they gate. Requires `secrets.OPENAI_API_KEY` and a plain id (`[A-Za-z0-9._-]`); without either, preflight warns, the cells stay off and the panel is unchanged. | | `openai_direct_replaces_cursor_openai` | `true` | Only with `openai_direct_model` (and its key) in effect: drop the Cursor `gpt-*` cell(s) from the panel so the OpenAI lane is reviewed once. The direct cells then **gate** exactly like the Cursor cells they replace: they count toward `Panel integrity` and the auto-approve gate (`approve_max_failed_reviewers`), a failed one turns its leg red, and the review's "did not contribute" line names it, e.g. `gpt-5.6-sol:edge-case (direct, error)`. `false` runs both, for a side-by-side comparison of the two backends; the direct cells are then **advisory** — their findings reach the judge, but they are never counted and their legs never fail the run. Preflight fails rather than leave the Cursor panel empty. | -| `anthropic_direct_model` | `''` | Opt-in. The Anthropic twin of `openai_direct_model`: an Anthropic model id (e.g. `claude-opus-5-5`) reviewed directly through the Anthropic API (pinned Claude Code CLI) as two cells — **adversarial** and **edge-case** — always on GitHub-hosted `ubuntu-latest`, by default **replacing** the Cursor `claude-*` cells. Requires `secrets.ANTHROPIC_API_KEY` and a plain id (`[A-Za-z0-9._-]`) other than `openai_direct_model`'s; without any of those, preflight warns, the cells stay off and the panel is unchanged. Unlike the OpenAI cells, the agent also reads a **read-only PR checkout** with Read, Grep and Glob only — no Bash or code-running tool — and `--restricted` confines those file tools to the checkout. **Private repos:** this panel reviews private-repo diffs and the workflow cannot check the Anthropic organization's data-retention terms, so use a key from an organization with zero data retention. | +| `anthropic_direct_model` | `''` | Opt-in. The Anthropic twin of `openai_direct_model`: an Anthropic model id (e.g. `claude-opus-5-5`) reviewed directly through the Anthropic API (pinned Claude Code CLI) as two cells — **adversarial** and **edge-case** — always on GitHub-hosted `ubuntu-latest`, by default **replacing** the Cursor `claude-*` cells. Requires `secrets.ANTHROPIC_API_KEY` and a plain id (`[A-Za-z0-9._-]`) other than `openai_direct_model`'s; without any of those, preflight warns, the cells stay off and the panel is unchanged. Unlike the OpenAI cells, the agent also reads a **read-only PR checkout** with Read, Grep and Glob only — no Bash or code-running tool — and `--restricted` confines those file tools to the checkout (`/proc` and `/sys` are also denied by explicit deny rules). **Private repos:** this panel reviews private-repo diffs and the workflow cannot check the Anthropic organization's data-retention terms, so use a key from an organization with zero data retention. | | `anthropic_direct_effort` | `xhigh` | Reasoning effort for the direct Anthropic cells: `low`, `medium`, `high`, `xhigh` or `max`. Any other value warns and falls back to `xhigh`. | | `anthropic_direct_replaces_cursor_anthropic` | `true` | Same semantics as `openai_direct_replaces_cursor_openai`, for the Cursor `claude-*` cells: `true` drops them and the direct Anthropic cells **gate** in their place; `false` keeps both, with the direct cells **advisory**. Each lab counts by its own flag, so OpenAI and Anthropic may be set differently. The judge (`judge_model`) is never touched. | | `skip_bot_branch_prefixes` | `ci/bump- chore/refresh- auto/refresh-` | Skip the panel for Bot-authored PRs on these branch prefixes (machine pin bumps / refreshes). `''` to review every bot PR. | From 1290970b1c41e2c561869d8f5278e5f33b429c78 Mon Sep 17 00:00:00 2001 From: Matt Miller Date: Fri, 9 Oct 2026 15:15:20 -0700 Subject: [PATCH 5/7] fix(cursor-review): keep Grep and Glob in the direct-Anthropic cells by dropping --bare (BE-19913) On the pinned Claude Code CLI, --bare silently drops Grep and Glob from --tools "Read,Grep,Glob", so the agent had Read alone and could not search the PR checkout it was given to read. Everything --bare would skip is already off: --setting-sources "" and a fresh CLAUDE_CONFIG_DIR load no settings, hooks, plugins or CLAUDE.md instructions; --strict-mcp-config admits only the submission server; CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC stops telemetry; auth is the env key alone. Checked on Linux with the pinned 2.1.286 binary: with --bare only Read is offered, without it Glob, Grep and Read; Read, Grep and Glob on /proc are refused by --restricted alone and by the deny rules alone (and succeed with neither, so the probe is real); a PR-planted CLAUDE.md telling the model to answer one word was ignored. A regression test pins the absence of --bare. Also fixes the consolidate needs comment's grammar now that it names two jobs. --- .../tests/test_workflow_job_isolation.py | 19 +++++++++++- .github/workflows/cursor-review.yml | 29 +++++++++++++------ 2 files changed, 38 insertions(+), 10 deletions(-) diff --git a/.github/cursor-review/tests/test_workflow_job_isolation.py b/.github/cursor-review/tests/test_workflow_job_isolation.py index a54b43f2..9ae6ffd8 100644 --- a/.github/cursor-review/tests/test_workflow_job_isolation.py +++ b/.github/cursor-review/tests/test_workflow_job_isolation.py @@ -323,13 +323,30 @@ def test_the_anthropic_cells_load_no_pr_authored_claude_config(self): body = code_lines(self.jobs["review-anthropic-direct"]) invocations = [l for l in body if "claude -p" in l] self.assertEqual(len(invocations), 1, invocations) - for flag in ("--bare", "--restricted", '--setting-sources ""'): + for flag in ("--restricted", '--setting-sources ""'): self.assertIn(flag, invocations[0]) self.assertTrue( any("rm -rf" in l and "_pr/.claude" in l and "_pr/.mcp.json" in l for l in body), "review-anthropic-direct no longer removes the PR's .claude/ and .mcp.json", ) + def test_the_anthropic_cells_keep_their_search_tools(self): + # On the pinned CLI `--bare` silently drops Grep and Glob from + # `--tools "Read,Grep,Glob"`, leaving Read alone: the agent could open + # files it already knows of but never search the checkout, which is + # the reason it reads the checkout at all. `--setting-sources ""` and + # the fresh CLAUDE_CONFIG_DIR already cover what `--bare` would skip. + body = code_lines(self.jobs["review-anthropic-direct"]) + start = next(i for i, l in enumerate(body) if "claude -p" in l) + argv = [] + for line in body[start:]: + argv.append(line) + if not line.rstrip().endswith("\\"): + break + argv = " ".join(argv) + self.assertNotIn("--bare", argv) + self.assertIn('--tools "Read,Grep,Glob"', argv) + def test_the_anthropic_cells_deny_proc_independently_of_restricted(self): # ANTHROPIC_API_KEY is in the env of the process serving Read/Grep/Glob, # so `/proc/self/environ` is the key. `--restricted` confines the file diff --git a/.github/workflows/cursor-review.yml b/.github/workflows/cursor-review.yml index d9b8f15e..6ad30b5e 100644 --- a/.github/workflows/cursor-review.yml +++ b/.github/workflows/cursor-review.yml @@ -3587,8 +3587,10 @@ jobs: # cwd, so a PR-planted `.claude/settings*.json` (env such as # ANTHROPIC_BASE_URL, hooks, extra directories) or `.mcp.json` must # never be there to load. `--restricted` and `--setting-sources ""` - # already ignore them; this removes them outright. CLAUDE.md stays — - # `--bare` skips its auto-discovery, and it is ordinary review context. + # already ignore them; this removes them outright. CLAUDE.md stays: + # `--setting-sources ""` keeps it from loading as instructions (on the + # pinned CLI a planted "reply with one word" CLAUDE.md was ignored), + # and the agent can still Read it as ordinary review context. run: rm -rf _pr/.claude _pr/.mcp.json _pr/CLAUDE.local.md - name: Download reviewed diff @@ -3663,7 +3665,7 @@ jobs: # or renamed one fails HERE, by name, rather than at argv parsing # mid-review — or, worse, by silently running unconfined. help="$(claude --help 2>&1)" - for flag in --bare --restricted --setting-sources --strict-mcp-config \ + for flag in --restricted --setting-sources --strict-mcp-config \ --disallowedTools --permission-prompts --no-session-persistence; do if ! printf '%s\n' "$help" | grep -qE -- "(^|[[:space:],])${flag}([[:space:],]|$)"; then echo "::error::claude ${installed} does not advertise ${flag}, which the direct-API cell's confinement relies on." @@ -3762,7 +3764,15 @@ jobs: )" # The result JSON goes to a file that is never echoed: the run log # is public and the result is model output steered by PR text. - claude -p --bare --restricted --setting-sources "" \ + # Deliberately NOT `--bare`: on the pinned CLI it drops Grep and Glob + # from `--tools`, leaving the agent Read alone, which cannot search + # the checkout. What `--bare` would skip is already off here: no + # settings, hooks or plugins (`--setting-sources ""` and a fresh + # CLAUDE_CONFIG_DIR, which also keep CLAUDE.md from loading as + # instructions), no other MCP server (`--strict-mcp-config`), no + # telemetry (CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC), and auth is + # the env key alone. + claude -p --restricted --setting-sources "" \ --tools "Read,Grep,Glob" \ --strict-mcp-config --mcp-config "$mcp_config" \ --allowedTools "Read,Grep,Glob,mcp__cursor-review-output__cursor_review_record_finding,mcp__cursor-review-output__cursor_review_finish" \ @@ -3896,11 +3906,12 @@ jobs: # a write-scoped token. name: Consolidate panel # `review-openai-direct` and `review-anthropic-direct` are here for - # ORDERING — their artifacts must land before `Download panel findings` runs. It never gates this job: when it - # is skipped (the default) or fails, `!cancelled()` below still runs, and - # a failed cell is counted from its artifact (or its absence) instead. Its - # result is read once, by `Aggregate panel findings`, as a consistency - # check on those artifacts (see aggregate-panel.py). + # ORDERING — their artifacts must land before `Download panel findings` + # runs. Neither gates this job: when one is skipped (the default) or + # fails, `!cancelled()` below still runs, and a failed cell is counted + # from its artifact (or its absence) instead. Each one's result is read + # once, by `Aggregate panel findings`, as a consistency check on those + # artifacts (see aggregate-panel.py). needs: [gate, diff-size, preflight, review, review-openai-direct, review-anthropic-direct, ledger] # `!cancelled()`, not `always()`: the judge is the most expensive call in # the workflow, and the only job that does anything with its payload, From e2ebf0fd5ed86303900e7119654bae1ea58e3d0b Mon Sep 17 00:00:00 2001 From: Matt Miller Date: Fri, 9 Oct 2026 15:15:20 -0700 Subject: [PATCH 6/7] feat(cursor-review): record each direct-Anthropic cell's token usage as an artifact (BE-19913) The API returns exact token counts per call, and Claude Code totals them in its result JSON. A new step copies those counts (and model ids, run attempt, turns, duration) into usage-direct--, kept 30 days, so a caller can measure what the cells cost without an Admin API key. It copies integers only: no prices, and nothing model-written, since the result is model output steered by PR text. A cell killed by its timeout leaves no result and is recorded as measured=false. The artifact is not findings-*, so consolidate never reads it as a cell, and its upload is best-effort. Executed tests cover the record's fields, the exclusion of model text and prices, unusable results, non-integer counts, and the artifact name. --- .github/cursor-review/README.md | 2 +- .../tests/test_workflow_panel_integrity.py | 93 ++++++++++++++++++- .github/workflows/cursor-review.yml | 77 +++++++++++++++ docs/callers/cursor-review.md | 2 +- 4 files changed, 171 insertions(+), 3 deletions(-) diff --git a/.github/cursor-review/README.md b/.github/cursor-review/README.md index bb412330..0f675f89 100644 --- a/.github/cursor-review/README.md +++ b/.github/cursor-review/README.md @@ -392,7 +392,7 @@ All optional except `workflows_ref` (required, no default) — pass them under | `panel_models` | `''` (built-in list) | JSON array of Cursor model ids that **replaces** the panel list; each still runs both review types and is validated against the live catalog by preflight. For per-repo experiments (e.g. `-xhigh` vs `-max` tiers). | | `openai_direct_model` | `''` (off) | OPT-IN. An OpenAI model id reviewed through the OpenAI API directly (pinned Codex CLI, `package.json` here) as two cells (**adversarial** + **edge-case**, the Cursor cells' prompts) on GitHub-hosted runners, by default **replacing** the Cursor `gpt-*` cells (next row). Needs the `OPENAI_API_KEY` secret and a plain id (`[A-Za-z0-9._-]`); without either, preflight warns and leaves the cells off and the panel unchanged. Not part of `panel_models` (not in the Cursor catalog). Whether the cells gate depends on the next row. | | `openai_direct_replaces_cursor_openai` | `true` | When the direct cells are in effect, drop the Cursor `gpt-*` cell(s) so the OpenAI lane is reviewed once — the direct cells then **gate** like the Cursor cells they replace: counted in `Panel integrity` and the auto-approve gate, a failed one red in the rollup and named `(direct, )` in the review's "did not contribute" line. `false` runs both for a side-by-side backend comparison, with the direct cells **advisory** (findings reach the judge; never counted, never red). Preflight fails rather than leave the Cursor panel empty. | -| `anthropic_direct_model` | `''` (off) | OPT-IN. The Anthropic twin of `openai_direct_model`: an Anthropic model id (e.g. `claude-opus-5-5`) reviewed through the Anthropic API directly (pinned Claude Code CLI, `package.json` here) as two cells on GitHub-hosted runners, by default **replacing** the Cursor `claude-*` cells. Needs the `ANTHROPIC_API_KEY` secret and a plain id other than `openai_direct_model`'s; without any of those, preflight warns and leaves the cells off and the panel unchanged. The agent also reads a **read-only PR checkout** (symlinks removed) with Read, Grep and Glob only — no Bash or code-running tool — confined to it by `--restricted`, with `/proc` and `/sys` also denied by explicit deny rules. This panel reviews private-repo diffs and the workflow cannot check the Anthropic organization's data-retention terms: callers on private repos must use a key from an organization with zero data retention. | +| `anthropic_direct_model` | `''` (off) | OPT-IN. The Anthropic twin of `openai_direct_model`: an Anthropic model id (e.g. `claude-opus-5-5`) reviewed through the Anthropic API directly (pinned Claude Code CLI, `package.json` here) as two cells on GitHub-hosted runners, by default **replacing** the Cursor `claude-*` cells. Needs the `ANTHROPIC_API_KEY` secret and a plain id other than `openai_direct_model`'s; without any of those, preflight warns and leaves the cells off and the panel unchanged. The agent also reads a **read-only PR checkout** (symlinks removed) with Read, Grep and Glob only — no Bash or code-running tool — confined to it by `--restricted`, with `/proc` and `/sys` also denied by explicit deny rules. This panel reviews private-repo diffs and the workflow cannot check the Anthropic organization's data-retention terms: callers on private repos must use a key from an organization with zero data retention. Each cell also uploads a `usage-direct--` artifact (kept 30 days) with the API's token counts for that cell, so cost can be measured without an Admin API key. | | `anthropic_direct_effort` | `xhigh` | `--effort` for the direct Anthropic cells: `low`, `medium`, `high`, `xhigh` or `max`; anything else warns and falls back to `xhigh`. | | `anthropic_direct_replaces_cursor_anthropic` | `true` | As `openai_direct_replaces_cursor_openai`, for the Cursor `claude-*` cells. Each lab counts by its own flag. Never touches `judge_model`. | | `cell_timeout_minutes` | `15` | Wall-clock cap on one reviewer cell's agent step (Cursor or direct). Over it the cell exits 130, uploads `status: error` with no findings, reddens its leg and fails `Panel integrity`. Shared by the cell's in-step retries, so it bounds attempts + backoff. Derives three more caps that cannot be set alone: cell job = this + 15, `Run judge` step = max(30, this × 2), judge job = that + 10. At the default: 15 / 30 / 30 / 40, today's values exactly. Preflight bounds it 10–45. | diff --git a/.github/cursor-review/tests/test_workflow_panel_integrity.py b/.github/cursor-review/tests/test_workflow_panel_integrity.py index a34069cb..55d15c66 100644 --- a/.github/cursor-review/tests/test_workflow_panel_integrity.py +++ b/.github/cursor-review/tests/test_workflow_panel_integrity.py @@ -681,7 +681,7 @@ def test_the_deliberate_skips_are_still_skips(self): self.assertIn(gate, self.condition, f"`{PANEL_JOB}` lost gate `{gate}`") -CONTEXT_REF = re.compile(r"\b(?:needs|inputs|steps|vars|matrix)(?:\.[A-Za-z0-9_-]+)+") +CONTEXT_REF = re.compile(r"\b(?:needs|inputs|steps|vars|matrix|github)(?:\.[A-Za-z0-9_-]+)+") def evaluate(expression, context): @@ -1252,5 +1252,96 @@ def test_the_judge_model_is_untouched(self): "the Anthropic replacement reaches the judge model", ) + +class RecordTokenUsageTest(unittest.TestCase): + """`Record token usage` in `review-anthropic-direct`, executed: the record + it uploads carries the API's token counts and nothing model-written (the + result JSON is model output steered by PR text) and no price, and a cell + that left no result is marked unmeasured instead of failing the step.""" + + CONTEXT = { + "inputs.anthropic_direct_model": "claude-opus-5-5", + "matrix.review_type": "edge-case", + "github.run_attempt": "2", + } + + def setUp(self): + job = split_jobs(read_workflow())["review-anthropic-direct"] + self.step = step_named(job, "Record token usage") + self.upload = step_named(job, "Upload usage artifact") + self.assertIsNotNone(self.step, "review-anthropic-direct lost its `Record token usage` step") + self.assertIsNotNone(self.upload, "review-anthropic-direct lost its `Upload usage artifact` step") + + def record(self, result): + """Run the step over `result` (None: no result file at all) and return + (the usage record it wrote, the step's stdout).""" + workdir = tempfile.mkdtemp() + self.addCleanup(shutil.rmtree, workdir) + if result is not None: + with open(os.path.join(workdir, "claude-result.json"), "w", encoding="utf-8") as f: + f.write(result if isinstance(result, str) else json.dumps(result)) + done, _ = run_step(self.step, self.CONTEXT, workdir) + self.assertEqual(done.returncode, 0, done.stderr) + with open(os.path.join(workdir, "usage-out", "usage.json"), encoding="utf-8") as f: + return json.load(f), done.stdout + + def test_copies_the_token_counts_and_nothing_else(self): + record, stdout = self.record({ + "result": "MODEL-WRITTEN TEXT", + "session_id": "session-1", + "total_cost_usd": 1.2345, + "usage": { + "input_tokens": 26, "output_tokens": 5312, "cache_read_input_tokens": 397506, + "cache_creation_input_tokens": 43925, + "cache_creation": {"ephemeral_1h_input_tokens": 43925, "ephemeral_5m_input_tokens": 0}, + }, + "modelUsage": { + "claude-opus-5-5": {"inputTokens": 26, "outputTokens": 5312, "cacheReadInputTokens": 397506, + "cacheCreationInputTokens": 43925, "costUSD": 0.5372}, + "not a model id; $(x)": {"inputTokens": 1}, + }, + "num_turns": 12, + "duration_ms": 65000, + }) + self.assertEqual( + (record["model"], record["review_type"], record["run_attempt"], record["measured"]), + ("claude-opus-5-5", "edge-case", 2, True), + ) + self.assertEqual(record["usage"], { + "input_tokens": 26, "output_tokens": 5312, "cache_read_input_tokens": 397506, + "cache_creation_input_tokens": 43925, "ephemeral_5m_input_tokens": 0, + "ephemeral_1h_input_tokens": 43925, + }) + self.assertEqual(record["models"], {"claude-opus-5-5": { + "input_tokens": 26, "output_tokens": 5312, "cache_read_input_tokens": 397506, + "cache_creation_input_tokens": 43925, + }}) + self.assertEqual((record["num_turns"], record["duration_ms"]), (12, 65000)) + text = json.dumps(record) + for leaked in ("MODEL-WRITTEN TEXT", "session-1", "1.2345", "0.5372", "$(x)"): + self.assertNotIn(leaked, text) + self.assertNotIn("5312", stdout, "token counts belong in the artifact, not the public log") + + def test_a_cell_with_no_usable_result_is_unmeasured(self): + for result in (None, "", "{truncated", "[]", {"usage": "n/a"}): + with self.subTest(result=result): + record, _ = self.record(result) + self.assertFalse(record["measured"]) + self.assertNotIn("usage", record) + + def test_a_count_that_is_not_a_plain_integer_reads_as_zero(self): + record, _ = self.record({"usage": { + "input_tokens": "9", "output_tokens": -5, "cache_read_input_tokens": True, + "cache_creation_input_tokens": 1.5, "cache_creation": ["not", "a", "mapping"], + }}) + self.assertTrue(record["measured"]) + self.assertEqual(set(record["usage"].values()), {0}) + + def test_the_upload_cannot_reach_the_judge_or_fail_the_cell(self): + name = block_mapping(self.upload, " with:", 10)["name"] + self.assertTrue(name.startswith("usage-direct-"), name) + self.assertFalse(name.startswith("findings-"), "consolidate downloads `findings-*` as panel cells") + self.assertEqual(step_scalar(self.upload, "continue-on-error"), "true") + if __name__ == "__main__": unittest.main(verbosity=2) diff --git a/.github/workflows/cursor-review.yml b/.github/workflows/cursor-review.yml index 6ad30b5e..345be3ef 100644 --- a/.github/workflows/cursor-review.yml +++ b/.github/workflows/cursor-review.yml @@ -3861,6 +3861,83 @@ jobs: if-no-files-found: error retention-days: 7 + - name: Record token usage + # The API's own token counts for this cell, as Claude Code totals them + # across its calls, so a caller can measure what the cells cost without + # an Admin API key. Counts only — no prices (they vary by contract and + # over time) and nothing model-written: the result JSON is model output + # steered by PR text, so only its integers and model ids are copied. + # A cell killed by its timeout leaves no result, so `measured` is false. + if: always() && steps.key.outputs.present == 'true' + env: + MODEL: ${{ inputs.anthropic_direct_model }} + REVIEW_TYPE: ${{ matrix.review_type }} + RUN_ATTEMPT: ${{ github.run_attempt }} + run: | + mkdir -p /tmp/usage-out + python3 -I - <<'PY' + import json, os, re + + def count(value): + return value if type(value) is int and value >= 0 else 0 + + try: + with open("/tmp/claude-result.json", encoding="utf-8") as f: + result = json.load(f) + except (OSError, ValueError): + result = None + if not isinstance(result, dict): + result = {} + usage = result.get("usage") + record = { + "model": os.environ["MODEL"], + "review_type": os.environ["REVIEW_TYPE"], + "run_attempt": int(os.environ.get("RUN_ATTEMPT") or 0), + "measured": isinstance(usage, dict), + } + if isinstance(usage, dict): + written = usage.get("cache_creation") + written = written if isinstance(written, dict) else {} + record["usage"] = { + "input_tokens": count(usage.get("input_tokens")), + "output_tokens": count(usage.get("output_tokens")), + "cache_read_input_tokens": count(usage.get("cache_read_input_tokens")), + "cache_creation_input_tokens": count(usage.get("cache_creation_input_tokens")), + "ephemeral_5m_input_tokens": count(written.get("ephemeral_5m_input_tokens")), + "ephemeral_1h_input_tokens": count(written.get("ephemeral_1h_input_tokens")), + } + # Per model, in case the CLI ever spends on a second one. + per_model = result.get("modelUsage") + record["models"] = { + name: { + "input_tokens": count(m.get("inputTokens")), + "output_tokens": count(m.get("outputTokens")), + "cache_read_input_tokens": count(m.get("cacheReadInputTokens")), + "cache_creation_input_tokens": count(m.get("cacheCreationInputTokens")), + } + for name, m in (per_model.items() if isinstance(per_model, dict) else []) + if re.fullmatch(r"[A-Za-z0-9._-]{1,100}", name) and isinstance(m, dict) + } + record["num_turns"] = count(result.get("num_turns")) + record["duration_ms"] = count(result.get("duration_ms")) + with open("/tmp/usage-out/usage.json", "w", encoding="utf-8") as f: + json.dump(record, f, indent=2) + print(f"Token usage recorded: measured={str(record['measured']).lower()}.") + PY + + - name: Upload usage artifact + # Not `findings-*`, so `consolidate` never reads it as a cell. Kept + # longer than the findings so a weekly cost report can still read it. + # Best effort: accounting must never fail a review cell. + if: always() && steps.key.outputs.present == 'true' + continue-on-error: true + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 + with: + name: usage-direct-${{ matrix.review_type }}-${{ inputs.anthropic_direct_model }} + path: /tmp/usage-out/usage.json + if-no-files-found: ignore + retention-days: 30 + - name: Fail the leg when the cell did not submit # Only when these cells REPLACE the Cursor Anthropic lane — the same # rollup-visibility contract as the `review` job's step of this name diff --git a/docs/callers/cursor-review.md b/docs/callers/cursor-review.md index 2b6cc60d..9de6b535 100644 --- a/docs/callers/cursor-review.md +++ b/docs/callers/cursor-review.md @@ -116,7 +116,7 @@ pull-requests: write # posting the consolidated review (and, at the round cap, | `panel_models` | `''` | JSON array of model ids replacing the built-in panel list (each runs both review types; preflight checks each one exists in the live catalog — existence only, not ZDR). Use for per-repo experiments such as a reasoning-tier A/B — and prefer wiring it to a **variable** rather than a literal, see [Running a model experiment](#running-a-model-experiment). | | `openai_direct_model` | `''` | Opt-in. An OpenAI model id reviewed directly through the OpenAI API (pinned Codex CLI) as two cells — **adversarial** and **edge-case**, with the same prompts the Cursor cells use — always on GitHub-hosted `ubuntu-latest`. By default they **replace** the Cursor-hosted OpenAI (`gpt-*`) panel cells — see the next row, which also decides whether they gate. Requires `secrets.OPENAI_API_KEY` and a plain id (`[A-Za-z0-9._-]`); without either, preflight warns, the cells stay off and the panel is unchanged. | | `openai_direct_replaces_cursor_openai` | `true` | Only with `openai_direct_model` (and its key) in effect: drop the Cursor `gpt-*` cell(s) from the panel so the OpenAI lane is reviewed once. The direct cells then **gate** exactly like the Cursor cells they replace: they count toward `Panel integrity` and the auto-approve gate (`approve_max_failed_reviewers`), a failed one turns its leg red, and the review's "did not contribute" line names it, e.g. `gpt-5.6-sol:edge-case (direct, error)`. `false` runs both, for a side-by-side comparison of the two backends; the direct cells are then **advisory** — their findings reach the judge, but they are never counted and their legs never fail the run. Preflight fails rather than leave the Cursor panel empty. | -| `anthropic_direct_model` | `''` | Opt-in. The Anthropic twin of `openai_direct_model`: an Anthropic model id (e.g. `claude-opus-5-5`) reviewed directly through the Anthropic API (pinned Claude Code CLI) as two cells — **adversarial** and **edge-case** — always on GitHub-hosted `ubuntu-latest`, by default **replacing** the Cursor `claude-*` cells. Requires `secrets.ANTHROPIC_API_KEY` and a plain id (`[A-Za-z0-9._-]`) other than `openai_direct_model`'s; without any of those, preflight warns, the cells stay off and the panel is unchanged. Unlike the OpenAI cells, the agent also reads a **read-only PR checkout** with Read, Grep and Glob only — no Bash or code-running tool — and `--restricted` confines those file tools to the checkout (`/proc` and `/sys` are also denied by explicit deny rules). **Private repos:** this panel reviews private-repo diffs and the workflow cannot check the Anthropic organization's data-retention terms, so use a key from an organization with zero data retention. | +| `anthropic_direct_model` | `''` | Opt-in. The Anthropic twin of `openai_direct_model`: an Anthropic model id (e.g. `claude-opus-5-5`) reviewed directly through the Anthropic API (pinned Claude Code CLI) as two cells — **adversarial** and **edge-case** — always on GitHub-hosted `ubuntu-latest`, by default **replacing** the Cursor `claude-*` cells. Requires `secrets.ANTHROPIC_API_KEY` and a plain id (`[A-Za-z0-9._-]`) other than `openai_direct_model`'s; without any of those, preflight warns, the cells stay off and the panel is unchanged. Unlike the OpenAI cells, the agent also reads a **read-only PR checkout** with Read, Grep and Glob only — no Bash or code-running tool — and `--restricted` confines those file tools to the checkout (`/proc` and `/sys` are also denied by explicit deny rules). **Private repos:** this panel reviews private-repo diffs and the workflow cannot check the Anthropic organization's data-retention terms, so use a key from an organization with zero data retention. Each cell also uploads a `usage-direct--` artifact (kept 30 days) with the API's token counts for that cell, so cost can be measured without an Admin API key. | | `anthropic_direct_effort` | `xhigh` | Reasoning effort for the direct Anthropic cells: `low`, `medium`, `high`, `xhigh` or `max`. Any other value warns and falls back to `xhigh`. | | `anthropic_direct_replaces_cursor_anthropic` | `true` | Same semantics as `openai_direct_replaces_cursor_openai`, for the Cursor `claude-*` cells: `true` drops them and the direct Anthropic cells **gate** in their place; `false` keeps both, with the direct cells **advisory**. Each lab counts by its own flag, so OpenAI and Anthropic may be set differently. The judge (`judge_model`) is never touched. | | `skip_bot_branch_prefixes` | `ci/bump- chore/refresh- auto/refresh-` | Skip the panel for Bot-authored PRs on these branch prefixes (machine pin bumps / refreshes). `''` to review every bot PR. | From 7b0f72b2cca783f6cc22d7d072268bd740dda190 Mon Sep 17 00:00:00 2001 From: Matt Miller Date: Fri, 9 Oct 2026 15:22:20 -0700 Subject: [PATCH 7/7] fix(cursor-review): make usage recording best-effort and refuse malformed counts (BE-19913) Record token usage now has continue-on-error, like its upload: a failure there would turn a cell that submitted ok red, and the aggregator would then flag the panel as inconsistent. A present but malformed count (a string, negative, boolean or float), a non-mapping cache split, or a malformed model id now makes the record measured=false instead of writing a zero that a cost report would read as a measured, free cell; an absent count still reads as 0. The log names malformed fields only, never values. --- .../tests/test_workflow_panel_integrity.py | 43 +++++++++-- .github/workflows/cursor-review.yml | 74 ++++++++++++------- 2 files changed, 83 insertions(+), 34 deletions(-) diff --git a/.github/cursor-review/tests/test_workflow_panel_integrity.py b/.github/cursor-review/tests/test_workflow_panel_integrity.py index 55d15c66..9323e8ac 100644 --- a/.github/cursor-review/tests/test_workflow_panel_integrity.py +++ b/.github/cursor-review/tests/test_workflow_panel_integrity.py @@ -1298,7 +1298,6 @@ def test_copies_the_token_counts_and_nothing_else(self): "modelUsage": { "claude-opus-5-5": {"inputTokens": 26, "outputTokens": 5312, "cacheReadInputTokens": 397506, "cacheCreationInputTokens": 43925, "costUSD": 0.5372}, - "not a model id; $(x)": {"inputTokens": 1}, }, "num_turns": 12, "duration_ms": 65000, @@ -1318,7 +1317,7 @@ def test_copies_the_token_counts_and_nothing_else(self): }}) self.assertEqual((record["num_turns"], record["duration_ms"]), (12, 65000)) text = json.dumps(record) - for leaked in ("MODEL-WRITTEN TEXT", "session-1", "1.2345", "0.5372", "$(x)"): + for leaked in ("MODEL-WRITTEN TEXT", "session-1", "1.2345", "0.5372"): self.assertNotIn(leaked, text) self.assertNotIn("5312", stdout, "token counts belong in the artifact, not the public log") @@ -1329,19 +1328,47 @@ def test_a_cell_with_no_usable_result_is_unmeasured(self): self.assertFalse(record["measured"]) self.assertNotIn("usage", record) - def test_a_count_that_is_not_a_plain_integer_reads_as_zero(self): - record, _ = self.record({"usage": { - "input_tokens": "9", "output_tokens": -5, "cache_read_input_tokens": True, - "cache_creation_input_tokens": 1.5, "cache_creation": ["not", "a", "mapping"], - }}) + def test_a_malformed_count_makes_the_record_unmeasured(self): + # A guessed zero would read as a measured, free cell in a cost report. + cases = { + "a string": {"input_tokens": "900"}, + "a negative": {"output_tokens": -5}, + "a boolean": {"cache_read_input_tokens": True}, + "a float": {"cache_creation_input_tokens": 1.5}, + "a non-mapping cache split": {"cache_creation": ["not", "a", "mapping"]}, + } + for label, usage in cases.items(): + with self.subTest(label): + record, stdout = self.record({"usage": usage}) + self.assertFalse(record["measured"]) + self.assertNotIn("usage", record) + self.assertIn("malformed:", stdout) + self.assertNotIn("900", stdout, "values never reach the public log") + + def test_a_malformed_model_id_makes_the_record_unmeasured_and_is_not_echoed(self): + record, stdout = self.record({ + "usage": {"input_tokens": 1, "output_tokens": 1}, + "modelUsage": {"not a model id; $(x)": {"inputTokens": 1}}, + }) + self.assertFalse(record["measured"]) + self.assertNotIn("$(x)", json.dumps(record) + stdout) + + def test_an_absent_count_reads_as_zero_and_stays_measured(self): + # Older CLIs report no cache-write split; that is a measured zero. + record, _ = self.record({"usage": {"input_tokens": 7, "output_tokens": 3}}) self.assertTrue(record["measured"]) - self.assertEqual(set(record["usage"].values()), {0}) + self.assertEqual(record["usage"]["input_tokens"], 7) + self.assertEqual(record["usage"]["ephemeral_1h_input_tokens"], 0) + self.assertEqual(record["models"], {}) def test_the_upload_cannot_reach_the_judge_or_fail_the_cell(self): name = block_mapping(self.upload, " with:", 10)["name"] self.assertTrue(name.startswith("usage-direct-"), name) self.assertFalse(name.startswith("findings-"), "consolidate downloads `findings-*` as panel cells") self.assertEqual(step_scalar(self.upload, "continue-on-error"), "true") + # A failed record step would turn a cell that submitted `ok` red, and + # the aggregator would then flag the whole panel as inconsistent. + self.assertEqual(step_scalar(self.step, "continue-on-error"), "true") if __name__ == "__main__": unittest.main(verbosity=2) diff --git a/.github/workflows/cursor-review.yml b/.github/workflows/cursor-review.yml index 345be3ef..b83eea4a 100644 --- a/.github/workflows/cursor-review.yml +++ b/.github/workflows/cursor-review.yml @@ -3867,8 +3867,11 @@ jobs: # an Admin API key. Counts only — no prices (they vary by contract and # over time) and nothing model-written: the result JSON is model output # steered by PR text, so only its integers and model ids are copied. - # A cell killed by its timeout leaves no result, so `measured` is false. + # `measured` is false when the cell left no result (a timeout kills + # it first) or any count in it is malformed, never a guessed zero. + # Best effort, like the upload: accounting must never fail a cell. if: always() && steps.key.outputs.present == 'true' + continue-on-error: true env: MODEL: ${{ inputs.anthropic_direct_model }} REVIEW_TYPE: ${{ matrix.review_type }} @@ -3878,8 +3881,23 @@ jobs: python3 -I - <<'PY' import json, os, re - def count(value): - return value if type(value) is int and value >= 0 else 0 + malformed = [] + + def count(value, field): + # A token count is a plain non-negative integer; absent reads as + # 0. Anything else makes the whole record unmeasured. + if value is None: + return 0 + if type(value) is int and value >= 0: + return value + malformed.append(field) + return 0 + + def mapping(value, field): + if value is None or isinstance(value, dict): + return value or {} + malformed.append(field) + return {} try: with open("/tmp/claude-result.json", encoding="utf-8") as f: @@ -3893,36 +3911,40 @@ jobs: "model": os.environ["MODEL"], "review_type": os.environ["REVIEW_TYPE"], "run_attempt": int(os.environ.get("RUN_ATTEMPT") or 0), - "measured": isinstance(usage, dict), + "measured": False, } if isinstance(usage, dict): - written = usage.get("cache_creation") - written = written if isinstance(written, dict) else {} - record["usage"] = { - "input_tokens": count(usage.get("input_tokens")), - "output_tokens": count(usage.get("output_tokens")), - "cache_read_input_tokens": count(usage.get("cache_read_input_tokens")), - "cache_creation_input_tokens": count(usage.get("cache_creation_input_tokens")), - "ephemeral_5m_input_tokens": count(written.get("ephemeral_5m_input_tokens")), - "ephemeral_1h_input_tokens": count(written.get("ephemeral_1h_input_tokens")), + written = mapping(usage.get("cache_creation"), "cache_creation") + totals = { + "input_tokens": count(usage.get("input_tokens"), "input_tokens"), + "output_tokens": count(usage.get("output_tokens"), "output_tokens"), + "cache_read_input_tokens": count(usage.get("cache_read_input_tokens"), "cache_read_input_tokens"), + "cache_creation_input_tokens": count(usage.get("cache_creation_input_tokens"), "cache_creation_input_tokens"), + "ephemeral_5m_input_tokens": count(written.get("ephemeral_5m_input_tokens"), "ephemeral_5m_input_tokens"), + "ephemeral_1h_input_tokens": count(written.get("ephemeral_1h_input_tokens"), "ephemeral_1h_input_tokens"), } # Per model, in case the CLI ever spends on a second one. - per_model = result.get("modelUsage") - record["models"] = { - name: { - "input_tokens": count(m.get("inputTokens")), - "output_tokens": count(m.get("outputTokens")), - "cache_read_input_tokens": count(m.get("cacheReadInputTokens")), - "cache_creation_input_tokens": count(m.get("cacheCreationInputTokens")), + models = {} + for name, m in mapping(result.get("modelUsage"), "modelUsage").items(): + if not re.fullmatch(r"[A-Za-z0-9._-]{1,100}", name) or not isinstance(m, dict): + malformed.append("modelUsage") + continue + models[name] = { + "input_tokens": count(m.get("inputTokens"), "inputTokens"), + "output_tokens": count(m.get("outputTokens"), "outputTokens"), + "cache_read_input_tokens": count(m.get("cacheReadInputTokens"), "cacheReadInputTokens"), + "cache_creation_input_tokens": count(m.get("cacheCreationInputTokens"), "cacheCreationInputTokens"), } - for name, m in (per_model.items() if isinstance(per_model, dict) else []) - if re.fullmatch(r"[A-Za-z0-9._-]{1,100}", name) and isinstance(m, dict) - } - record["num_turns"] = count(result.get("num_turns")) - record["duration_ms"] = count(result.get("duration_ms")) + if not malformed: + record.update(measured=True, usage=totals, models=models) + for field in ("num_turns", "duration_ms"): + value = result.get(field) + record[field] = value if type(value) is int and value >= 0 else 0 with open("/tmp/usage-out/usage.json", "w", encoding="utf-8") as f: json.dump(record, f, indent=2) - print(f"Token usage recorded: measured={str(record['measured']).lower()}.") + # Field names only, never values: the run log is public. + print(f"Token usage recorded: measured={str(record['measured']).lower()}" + + (f" (malformed: {', '.join(sorted(set(malformed)))})." if malformed else ".")) PY - name: Upload usage artifact