From 998545deffe0f5028310ca7b1a4fc8958bf81c6f Mon Sep 17 00:00:00 2001 From: Matt Miller Date: Fri, 9 Oct 2026 22:56:38 +0000 Subject: [PATCH 1/2] feat(cursor-review): optional direct-Anthropic judge via the pinned Claude Code CLI (BE-19914) The judge's twin of the direct-Anthropic cells, reusing their CLI pin, install/verify steps and ANTHROPIC_API_KEY secret: - inputs judge_direct_model / judge_direct_effort; preflight resolves judge_direct=true only for a plain id with the key present, else warns and the Cursor judge runs - consolidate: `Run judge (direct)` runs claude -p in the PR checkout (symlinks and PR Claude config removed first) with Read/Grep/Glob, the same review-output-mcp.py --mode judge tool and file, the same step cap and absorption; `Run judge` (Cursor) is unchanged and runs otherwise - exfil guard replaces a final review carrying the key with the --init seed, so the degraded panel-union path posts; the log shows status and finding count only - the judge record is labelled ` (direct)` - consolidate runs on ubuntu-latest whenever ANTHROPIC_API_KEY is passed, not only when the direct judge runs: its steps reference the secret, so the job is sent it either way. gate now resolves key presence on every event for that switch - confinement follows the merged cells, not the ticket's flag list: no --bare (drops Grep/Glob on the pinned CLI), plus --setting-sources "" and the /proc and /sys deny rules - catalog-drift's last-checked window 40 -> 50 lines, as the new preflight env lines pushed the audit comment past it Co-Authored-By: Claude Opus 5.5 --- .github/cursor-review/README.md | 6 +- .github/cursor-review/catalog-drift.py | 2 +- .github/cursor-review/package.json | 2 +- .../tests/test_workflow_job_isolation.py | 21 +- .../tests/test_workflow_judge_direct.py | 350 ++++++++++++++++++ .../tests/test_workflow_panel_integrity.py | 7 +- .github/workflows/cursor-review.yml | 287 +++++++++++++- docs/callers/cursor-review.md | 6 +- 8 files changed, 655 insertions(+), 26 deletions(-) create mode 100644 .github/cursor-review/tests/test_workflow_judge_direct.py diff --git a/.github/cursor-review/README.md b/.github/cursor-review/README.md index 0f675f89..0b6a9023 100644 --- a/.github/cursor-review/README.md +++ b/.github/cursor-review/README.md @@ -327,7 +327,7 @@ jobs: |---|---|---|---| | Secret | `CURSOR_API_KEY` | **yes** | Bills the panel + judge `cursor-agent` calls. | | Secret | `OPENAI_API_KEY` | no | Only with `openai_direct_model`: bills the optional direct-OpenAI cells. Use a project-scoped service-account key with a spend limit. | -| Secret | `ANTHROPIC_API_KEY` | no | Only with `anthropic_direct_model`: bills the optional direct-Anthropic cells. Use a key scoped to its own Anthropic workspace with a spend limit, never a personal key. | +| Secret | `ANTHROPIC_API_KEY` | no | Only with `anthropic_direct_model` or `judge_direct_model`: bills the optional direct-Anthropic cells and direct judge. Passing it moves `consolidate` to GitHub-hosted `ubuntu-latest` whatever `runs_on` says. Use a key scoped to its own Anthropic workspace with a spend limit, never a personal key. | | Secret | `SLACK_BOT_TOKEN` | no | Enables start/complete DMs to the triggerer. | | Variable | `CURSOR_REVIEW_DM_EMAIL_MAP` | no | Maps GitHub logins → emails for Slack DM lookup. | @@ -395,6 +395,8 @@ All optional except `workflows_ref` (required, no default) — pass them under | `anthropic_direct_model` | `''` (off) | OPT-IN. The Anthropic twin of `openai_direct_model`: an Anthropic model id (e.g. `claude-opus-5-5`) reviewed through the Anthropic API directly (pinned Claude Code CLI, `package.json` here) as two cells on GitHub-hosted runners, by default **replacing** the Cursor `claude-*` cells. Needs the `ANTHROPIC_API_KEY` secret and a plain id other than `openai_direct_model`'s; without any of those, preflight warns and leaves the cells off and the panel unchanged. The agent also reads a **read-only PR checkout** (symlinks removed) with Read, Grep and Glob only — no Bash or code-running tool — confined to it by `--restricted`, with `/proc` and `/sys` also denied by explicit deny rules. This panel reviews private-repo diffs and the workflow cannot check the Anthropic organization's data-retention terms: callers on private repos must use a key from an organization with zero data retention. Each cell also uploads a `usage-direct--` artifact (kept 30 days) with the API's token counts for that cell, so cost can be measured without an Admin API key. | | `anthropic_direct_effort` | `xhigh` | `--effort` for the direct Anthropic cells: `low`, `medium`, `high`, `xhigh` or `max`; anything else warns and falls back to `xhigh`. | | `anthropic_direct_replaces_cursor_anthropic` | `true` | As `openai_direct_replaces_cursor_openai`, for the Cursor `claude-*` cells. Each lab counts by its own flag. Never touches `judge_model`. | +| `judge_direct_model` | `''` (off) | OPT-IN. An Anthropic model id (e.g. `claude-opus-5-5`) that runs the **judge** through the Anthropic API directly instead of `judge_model` through Cursor, with the same pinned Claude Code CLI and `ANTHROPIC_API_KEY` as `anthropic_direct_model`. Needs that secret and a plain id (`[A-Za-z0-9._-]`, at most 100); without either, preflight warns and the Cursor judge runs. Same prompt, structured-output tool, step timeout and degraded panel-union fallback as the Cursor judge; the agent reads the PR checkout (symlinks removed) with Read, Grep and Glob only — no Bash or code-running tool — confined to it by `--restricted`, with `/proc` and `/sys` also denied by explicit deny rules. A final review containing the key is replaced by the error seed, so the panel union posts instead; the log shows the judge's status and finding count only. `consolidate` runs on GitHub-hosted `ubuntu-latest` whenever `ANTHROPIC_API_KEY` is passed, whatever `runs_on` says. This judge reads private-repo diffs and the workflow cannot check the Anthropic organization's data-retention terms: callers on private repos must use a key from an organization with zero data retention. | +| `judge_direct_effort` | `xhigh` | `--effort` for the direct judge: `low`, `medium`, `high`, `xhigh` or `max`; anything else warns and falls back to `xhigh`. | | `cell_timeout_minutes` | `15` | Wall-clock cap on one reviewer cell's agent step (Cursor or direct). Over it the cell exits 130, uploads `status: error` with no findings, reddens its leg and fails `Panel integrity`. Shared by the cell's in-step retries, so it bounds attempts + backoff. Derives three more caps that cannot be set alone: cell job = this + 15, `Run judge` step = max(30, this × 2), judge job = that + 10. At the default: 15 / 30 / 30 / 40, today's values exactly. Preflight bounds it 10–45. | | `skip_bot_branch_prefixes` | `ci/bump- chore/refresh- auto/refresh-` | Skip the panel when the PR author is a Bot **and** the head branch starts with one of these prefixes (machine pin bumps / catalog refreshes). `''` reviews every bot PR. | | `diff_size_cap` | `5000` | Max counted changed lines (after generated-file exclusion and comment discounting); larger PRs are skipped. | @@ -414,7 +416,7 @@ All optional except `workflows_ref` (required, no default) — pass them under | `defer_approval` | `false` | For a caller that runs cursor-approve after this workflow: an APPROVE outcome still reports `approve_gate` = `pass` but posts no approval and resolves no thread, and withdraws the approver's own earlier auto-approvals and requests for changes, so cursor-approve's decide is the only approver. Pass the workflow's `approve_scope_effective` output (the scope `decide` actually gated under, `full` on any fallback) to cursor-approve's `approve_scope`, so its resolution honours non-gating marks only after a `delta` round. | | `approve_max_failed_reviewers` | `0` | Errored panel reviewers (status `error`) auto-approve tolerates before withholding its decision. `0` → any reviewer that did not complete means no decision. Never tolerates another status, more than N, or a review type with no completed reviewer; tolerated cells are named in the decision. | | `max_rounds` | `5` | Per-PR review-round cap; `0` → none. Counts the posting identity's consolidated reviews since `needs-human-review` was last removed; at or over the cap the panel is skipped, the PR is labelled `needs-human-review`, one comment lists the latest round's open findings above the threshold, and the `approve_gate` output is `capped`. Fails open on a read error. See [the setup guide](../../docs/callers/cursor-review.md#round-cap-and-the-approve_gate-output). | -| `runs_on` | `'"ubuntu-latest"'` | JSON-encoded `runs-on` (same shape as `cursor-review-auto-label.yml`'s) for the four credential-free heavy jobs only: `diff-size`, `preflight`, the `review` panel and `consolidate`. The gate, ledger, Slack notices and every job holding the App token or `pull-requests: write` stay on GitHub-hosted `ubuntu-latest`. Those four run models with unsandboxed shell over PR code, so use a self-hosted pool only if each job gets a fresh pod (fresh filesystem and `$HOME`), holds no cloud identity or token, and can't reach private networks or the metadata server. The image must be linux/x64 with bash, git, curl, jq, python3, gh, tar and GNU coreutils. | +| `runs_on` | `'"ubuntu-latest"'` | JSON-encoded `runs-on` (same shape as `cursor-review-auto-label.yml`'s) for the four credential-free heavy jobs only: `diff-size`, `preflight`, the `review` panel and `consolidate` (which stays on `ubuntu-latest` whenever `ANTHROPIC_API_KEY` is passed, for the direct judge). The gate, ledger, Slack notices and every job holding the App token or `pull-requests: write` stay on GitHub-hosted `ubuntu-latest`. Those four run models with unsandboxed shell over PR code, so use a self-hosted pool only if each job gets a fresh pod (fresh filesystem and `$HOME`), holds no cloud identity or token, and can't reach private networks or the metadata server. The image must be linux/x64 with bash, git, curl, jq, python3, gh, tar and GNU coreutils. | ### `workflows_ref` must equal the `uses:` pin diff --git a/.github/cursor-review/catalog-drift.py b/.github/cursor-review/catalog-drift.py index ac4c3faa..cff2fa32 100644 --- a/.github/cursor-review/catalog-drift.py +++ b/.github/cursor-review/catalog-drift.py @@ -133,7 +133,7 @@ # whole file) stops an unrelated `last checked` elsewhere in cursor-review.yml # from shadowing the real pin-adjacent date; the real-workflow test guards the # window from being too tight. -LAST_CHECKED_WINDOW = 40 +LAST_CHECKED_WINDOW = 50 # The reasoning/speed suffixes Cursor appends to a model family, and the ONE # place they are enumerated. Cursor ships a family at every reasoning tier and diff --git a/.github/cursor-review/package.json b/.github/cursor-review/package.json index c2718213..8bebfbc2 100644 --- a/.github/cursor-review/package.json +++ b/.github/cursor-review/package.json @@ -1,7 +1,7 @@ { "name": "comfy-cursor-review-direct-cli", "private": true, - "description": "Manifest whose ONLY job is to be the single, Dependabot-visible source of truth for the CLI versions cursor-review.yml's optional direct-API review cells install: @openai/codex (the direct OpenAI cells) and @anthropic-ai/claude-code (the direct Anthropic cells). Nothing is ever `npm install`ed from this directory: the job reads the pin out of this file and runs `npm install -g @`. It is a real npm manifest rather than a workflow `env:` constant purely so Dependabot's npm ecosystem can see the pin and open a bump PR — an inline `npm install -g pkg@1.2.3` inside a `run:` step is invisible to every ecosystem, which is how the groom CLI pin silently rotted (see .github/groom/package.json, the precedent this mirrors). Keep the version EXACT (no ^ or ~): the agent CLI is executable supply chain for a step that reads untrusted repo content, and the job rejects a non-exact pin at run time.", + "description": "Manifest whose ONLY job is to be the single, Dependabot-visible source of truth for the CLI versions cursor-review.yml's optional direct-API review cells install: @openai/codex (the direct OpenAI cells) and @anthropic-ai/claude-code (the direct Anthropic cells and the direct judge). Nothing is ever `npm install`ed from this directory: the job reads the pin out of this file and runs `npm install -g @`. It is a real npm manifest rather than a workflow `env:` constant purely so Dependabot's npm ecosystem can see the pin and open a bump PR — an inline `npm install -g pkg@1.2.3` inside a `run:` step is invisible to every ecosystem, which is how the groom CLI pin silently rotted (see .github/groom/package.json, the precedent this mirrors). Keep the version EXACT (no ^ or ~): the agent CLI is executable supply chain for a step that reads untrusted repo content, and the job rejects a non-exact pin at run time.", "dependencies": { "@openai/codex": "0.162.0", "@anthropic-ai/claude-code": "2.1.286" diff --git a/.github/cursor-review/tests/test_workflow_job_isolation.py b/.github/cursor-review/tests/test_workflow_job_isolation.py index 9ae6ffd8..a92d0674 100644 --- a/.github/cursor-review/tests/test_workflow_job_isolation.py +++ b/.github/cursor-review/tests/test_workflow_job_isolation.py @@ -304,14 +304,15 @@ def test_the_bot_key_is_confined_to_the_checkout_free_posting_jobs(self): "auto-retry"}) def test_the_anthropic_key_reaches_only_the_gate_and_its_direct_cells(self): - # `gate` only resolves a presence boolean; the cells are the one - # consumer, and they hold a model key beside a PR checkout, so they - # must stay GitHub-hosted. + # `gate` only resolves a presence boolean; the cells and the direct + # judge (`consolidate`) are the consumers, and they hold a model key + # beside a PR checkout, so they must be GitHub-hosted whenever they can + # hold it — pinned for `consolidate` in test_workflow_judge_direct.py. holders = { name for name, body in self.jobs.items() if any("secrets.ANTHROPIC_API_KEY" in l for l in code_lines(body)) } - self.assertEqual(holders, {"gate", "review-anthropic-direct"}) + self.assertEqual(holders, {"gate", "review-anthropic-direct", "consolidate"}) runs_on = [l.strip() for l in self.jobs["review-anthropic-direct"] if l.startswith(" runs-on:")] self.assertEqual(runs_on, ["runs-on: ubuntu-latest"]) @@ -387,9 +388,17 @@ def test_only_credential_free_jobs_follow_the_runs_on_input(self): for name in routed: body = self.jobs[name] self.assertFalse(references_bot_key(body), name) + # `consolidate` may also read ANTHROPIC_API_KEY, for the direct + # judge, ONLY because its runs-on leaves the pool whenever that key + # is passed (test_workflow_judge_direct.py executes that switch). + allowed = ("CURSOR_API_KEY", "ANTHROPIC_API_KEY") if name == "consolidate" else ("CURSOR_API_KEY",) + if name == "consolidate": + runs_on = next(l for l in body if l.startswith(" runs-on:")) + self.assertIn("needs.gate.outputs.anthropic_key_present == 'true'", runs_on) + self.assertIn("&& 'ubuntu-latest' ||", runs_on) self.assertFalse( - any("secrets." in l and "CURSOR_API_KEY" not in l for l in code_lines(body)), - f"job `{name}` follows runs_on but reads a secret other than CURSOR_API_KEY", + any("secrets." in l and not any(k in l for k in allowed) for l in code_lines(body)), + f"job `{name}` follows runs_on but reads a secret other than {', '.join(allowed)}", ) for scope, value in sorted((job_permissions(body) or {"": "missing"}).items()): self.assertIn(value, READ_ONLY_VALUES, f"`{name}` grants `{scope}: {value}`") diff --git a/.github/cursor-review/tests/test_workflow_judge_direct.py b/.github/cursor-review/tests/test_workflow_judge_direct.py new file mode 100644 index 00000000..356e6110 --- /dev/null +++ b/.github/cursor-review/tests/test_workflow_judge_direct.py @@ -0,0 +1,350 @@ +"""The optional direct-Anthropic judge in cursor-review.yml (`judge_direct_model`). + +Three properties, each one a single line in the workflow that nothing else +would notice breaking: + + * preflight's resolution — `judge_direct=true` only for a plain id with the + key present, else a warning and the Cursor judge. EXECUTED: the real + `Define panel models` script, run by test_workflow_panel_integrity's helper. + * job isolation — `consolidate` follows `runs_on`, so ANTHROPIC_API_KEY must + appear only in steps gated on the direct judge, and the job's `runs-on:` + must resolve to `ubuntu-latest` whenever it can hold the key (the key is + sent for every step that references it, skipped ones included). + * the exfil guard — a final review carrying the key is replaced by the + `--init` error seed, so `Build consolidated findings file` takes the + degraded panel-union path. EXECUTED, both steps. + +Run: python3 -m unittest discover -s .github/cursor-review/tests -p 'test_*.py' +""" + +import json +import os +import re +import shutil +import subprocess +import tempfile +import unittest + +from test_workflow_panel_integrity import ( + ASSETS, + AGGREGATE_STEP, + PANEL_MODELS_STEP, + block_mapping, + code_lines, + evaluate, + read_workflow, + render, + run_block, + run_step, + split_jobs, + split_steps, + step_named, + step_scalar, +) + +JUDGE_MODEL = "claude-opus-5-5" +CURSOR_JUDGE = "claude-opus-4-8-thinking-max" +KEY = "sk-ant-test-0123456789abcdef" +RUNS_ON_INPUT = "fromJSON(inputs.runs_on || '\"ubuntu-latest\"')" + +CONFIGURE_STEP = "Configure structured final review" +CURSOR_JUDGE_STEP = "Run judge" +DIRECT_JUDGE_STEP = "Run judge (direct)" +GUARD_STEP = "Refuse a judge review carrying the API key" +BUILD_STEP = "Build consolidated findings file" + + +def jobs(): + return split_jobs(read_workflow()) + + +def step(job_lines, name): + body = step_named(job_lines, name) + if body is None: + raise AssertionError(f"step `{name}` is gone") + return body + + +def run_with(step_lines, context, workdir, extra_env): + """`run_step`, plus env the runner supplies from outside the step's own + `env:` (the workflow-level JUDGE_MODEL).""" + script = run_block(step_lines) + env = {"PATH": os.environ.get("PATH", ""), "CURSOR_REVIEW_ASSETS": ASSETS, + "GITHUB_OUTPUT": os.path.join(workdir, "github-output"), **extra_env} + open(env["GITHUB_OUTPUT"], "w", encoding="utf-8").close() + for name, value in block_mapping(step_lines, " env:", 10).items(): + env[name] = render(value, context) + return subprocess.run( + [shutil.which("bash"), "-e", "-c", script.replace("/tmp/", workdir + "/")], + env=env, cwd=workdir, capture_output=True, text=True, timeout=60, + ) + + +class PreflightResolvesTheDirectJudgeTest(unittest.TestCase): + @classmethod + def setUpClass(cls): + for tool in ("bash", "jq", "python3"): + if shutil.which(tool) is None: + raise AssertionError(f"`{tool}` is required to execute the workflow steps") + cls.preflight = jobs()["preflight"] + + def resolve(self, model, key_present="true", effort="xhigh"): + """(needs.preflight.outputs.*, stdout) after `Define panel models`.""" + with tempfile.TemporaryDirectory() as workdir: + done, step_outputs = run_step( + step(self.preflight, PANEL_MODELS_STEP), + { + "inputs.judge_direct_model": model, + "inputs.judge_direct_effort": effort, + "needs.gate.outputs.anthropic_key_present": key_present, + }, + workdir, + ) + self.assertEqual(done.returncode, 0, done.stdout + done.stderr) + steps = {"steps.models.outputs.%s" % k: v for k, v in step_outputs.items()} + outputs = { + name: render(value, steps) + for name, value in block_mapping(self.preflight, " outputs:", 6).items() + } + return outputs, done.stdout + + def test_set_with_the_key_present_runs_direct(self): + outputs, log = self.resolve(JUDGE_MODEL, effort="max") + self.assertEqual(outputs["judge_direct"], "true") + self.assertEqual(outputs["judge_direct_effort"], "max") + self.assertNotIn("::warning::judge_direct", log) + + def test_unset_changes_nothing(self): + outputs, log = self.resolve("", key_present="") + baseline, _ = self.resolve("", key_present="", effort="") + self.assertEqual(outputs["judge_direct"], "false") + self.assertNotIn("judge_direct", log) + # The panel list is the judge-free baseline's: the judge never edits it. + self.assertEqual(outputs["models"], baseline["models"]) + self.assertEqual(outputs["models"], self.resolve(JUDGE_MODEL)[0]["models"]) + + def test_set_without_the_key_warns_and_stays_on_cursor(self): + for key_present in ("false", ""): + with self.subTest(key_present=key_present): + outputs, log = self.resolve(JUDGE_MODEL, key_present=key_present) + self.assertEqual(outputs["judge_direct"], "false") + self.assertIn("::warning::judge_direct_model is set but the ANTHROPIC_API_KEY", log) + + def test_a_non_plain_id_warns_and_stays_on_cursor(self): + for model in ("claude opus", "claude/opus", "a" * 101, "claude-opus-5-5\nx", "$(id)"): + with self.subTest(model=model): + outputs, log = self.resolve(model) + self.assertEqual(outputs["judge_direct"], "false") + self.assertIn("::warning::judge_direct_model is not a plain model id", log) + + def test_an_unknown_effort_falls_back_to_xhigh(self): + outputs, log = self.resolve(JUDGE_MODEL, effort="extreme") + self.assertEqual(outputs["judge_direct"], "true") + self.assertEqual(outputs["judge_direct_effort"], "xhigh") + self.assertIn("::warning::judge_direct_effort", log) + + +class ConsolidateJobIsolationTest(unittest.TestCase): + @classmethod + def setUpClass(cls): + cls.consolidate = jobs()["consolidate"] + cls.steps = split_steps(cls.consolidate) + + def runs_on(self, judge_direct, key_present): + line = next(l for l in code_lines(self.consolidate) if l.startswith(" runs-on:")) + expression = line.split(":", 1)[1].strip() + self.assertIn(RUNS_ON_INPUT, expression, "the runs_on fallback is no longer today's") + # `fromJSON` is outside `evaluate`'s grammar; a sentinel string stands + # in for whatever pool the caller's `runs_on` names. + expression = expression.replace(RUNS_ON_INPUT, "'POOL'") + return evaluate(expression, { + "needs.preflight.outputs.judge_direct": judge_direct, + "needs.gate.outputs.anthropic_key_present": key_present, + }) + + def test_it_is_hosted_whenever_it_can_hold_the_key(self): + self.assertEqual(self.runs_on("true", "true"), "ubuntu-latest") + self.assertEqual(self.runs_on("true", ""), "ubuntu-latest") + # Key passed for the cells only: the direct judge's steps still + # reference it, so it is still sent to this job. + self.assertEqual(self.runs_on("false", "true"), "ubuntu-latest") + + def test_without_the_key_it_follows_runs_on_as_today(self): + self.assertEqual(self.runs_on("false", "false"), "POOL") + self.assertEqual(self.runs_on("", ""), "POOL") + + def test_the_key_is_only_in_steps_gated_on_the_direct_judge(self): + holders = [] + for name, body in self.steps: + if any("secrets.ANTHROPIC_API_KEY" in l for l in code_lines(body)): + holders.append(name) + self.assertIn("needs.preflight.outputs.judge_direct == 'true'", step_scalar(body, "if") or "", name) + self.assertEqual(sorted(holders), sorted([DIRECT_JUDGE_STEP, GUARD_STEP])) + + def test_exactly_one_judge_step_runs(self): + context = {"steps.aggregate.outputs.ok_count": "3", "steps.claude_verify.outcome": "success"} + for judge_direct, expected in (("true", DIRECT_JUDGE_STEP), ("false", CURSOR_JUDGE_STEP), ("", CURSOR_JUDGE_STEP)): + ran = [ + name for name in (CURSOR_JUDGE_STEP, DIRECT_JUDGE_STEP) + if evaluate(step_scalar(step(self.consolidate, name), "if"), + {**context, "needs.preflight.outputs.judge_direct": judge_direct}) + ] + self.assertEqual(ran, [expected], judge_direct) + # A CLI that did not install or verify clean never runs: the seed then + # takes the degraded path, like a failed Cursor judge. + self.assertFalse(evaluate(step_scalar(step(self.consolidate, DIRECT_JUDGE_STEP), "if"), { + **context, "needs.preflight.outputs.judge_direct": "true", "steps.claude_verify.outcome": "failure"})) + + def test_the_cursor_judge_step_is_unchanged_otherwise(self): + body = code_lines(step(self.consolidate, CURSOR_JUDGE_STEP)) + self.assertFalse(any("ANTHROPIC" in l or "claude" in l for l in body)) + self.assertTrue(any(l.strip().startswith("cursor-agent") for l in body)) + self.assertEqual(step_scalar(step(self.consolidate, CURSOR_JUDGE_STEP), "continue-on-error"), "true") + + def test_the_direct_judge_matches_the_cursor_judges_cap_and_absorption(self): + cursor, direct = step(self.consolidate, CURSOR_JUDGE_STEP), step(self.consolidate, DIRECT_JUDGE_STEP) + for key in ("timeout-minutes", "continue-on-error"): + self.assertEqual(step_scalar(direct, key), step_scalar(cursor, key), key) + + def test_the_direct_judge_is_confined(self): + body = code_lines(step(self.consolidate, DIRECT_JUDGE_STEP)) + start = next(i for i, l in enumerate(body) if l.strip().startswith("claude -p")) + argv = [] + for line in body[start:]: + argv.append(line.strip()) + if not line.rstrip().endswith("\\"): + break + argv = " ".join(argv) + for flag in ( + "--restricted", '--setting-sources ""', '--tools "Read,Grep,Glob"', "--strict-mcp-config", + '--allowedTools "Read,Grep,Glob,mcp__cursor-review-output__cursor_review_submit_final"', + '--disallowedTools "Read(//proc/**),Read(//sys/**)"', "--permission-prompts none", + '--model "$JUDGE_DIRECT_MODEL"', '--effort "$JUDGE_DIRECT_EFFORT"', + "--output-format json", "--no-session-persistence", "< /tmp/judge-prompt.txt", + ): + self.assertIn(flag, argv) + # Drops Grep and Glob on the pinned CLI (see the cells' twin test). + self.assertNotIn("--bare", argv) + script = "\n".join(body) + self.assertLess(script.index("find . -type l -delete"), script.index("claude -p")) + self.assertIn('--mode", "judge"', script) + self.assertIn('"--out", "/tmp/judge-findings.json"', script) + self.assertIn('mktemp -d -p "$RUNNER_TEMP"', script) + env = block_mapping(step(self.consolidate, DIRECT_JUDGE_STEP), " env:", 10) + self.assertEqual(env.get("CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC"), "'1'") + # The result is model output steered by PR text: never echoed. + self.assertFalse(any(re.search(r"\b(cat|head|tail)\b.*judge-claude-result", l) for l in body)) + + def test_the_cli_is_installed_from_the_manifest_only_when_direct(self): + for name in ("Install Claude Code CLI", "Verify Claude Code CLI version"): + body = step(self.consolidate, name) + self.assertIn("needs.preflight.outputs.judge_direct == 'true'", step_scalar(body, "if")) + install = "\n".join(code_lines(step(self.consolidate, "Install Claude Code CLI"))) + self.assertIn('manifest="$CURSOR_REVIEW_ASSETS/package.json"', install) + self.assertNotRegex(install, r"claude-code@\d") + # The Cursor CLI install is untouched and unconditional. + cursor = step(self.consolidate, "Install Cursor agent CLI") + self.assertIsNone(step_scalar(cursor, "if")) + + +class ExfilGuardTest(unittest.TestCase): + @classmethod + def setUpClass(cls): + cls.consolidate = jobs()["consolidate"] + + def setUp(self): + tmp = tempfile.TemporaryDirectory() + self.addCleanup(tmp.cleanup) + self.workdir = tmp.name + self.context = { + "needs.preflight.outputs.judge_direct": "true", + "inputs.judge_direct_model": JUDGE_MODEL, + "secrets.ANTHROPIC_API_KEY": KEY, + } + + def configure(self, judge_direct): + done = run_with(step(self.consolidate, CONFIGURE_STEP), + {**self.context, "needs.preflight.outputs.judge_direct": judge_direct}, + self.workdir, {"JUDGE_MODEL": CURSOR_JUDGE}) + self.assertEqual(done.returncode, 0, done.stdout + done.stderr) + return self.read() + + def read(self): + with open(os.path.join(self.workdir, "judge-findings.json"), encoding="utf-8") as f: + return json.load(f) + + def submit(self, body): + record = {"model": f"{JUDGE_MODEL} (direct)", "review_type": "judge", "status": "ok", + "findings": [{"file": "a.py", "line": 1, "side": "RIGHT", "severity": "high", "body": body}]} + with open(os.path.join(self.workdir, "judge-findings.json"), "w", encoding="utf-8") as f: + json.dump(record, f) + return record + + def guard(self): + body = step(self.consolidate, GUARD_STEP) + self.assertTrue(evaluate(step_scalar(body, "if"), self.context)) + return run_step(body, self.context, self.workdir)[0] + + def build(self): + with open(os.path.join(self.workdir, "panel.json"), "w", encoding="utf-8") as f: + json.dump([{"model": "kimi-k3-max", "review_type": "adversarial", "status": "ok", + "findings": [{"file": "b.py", "line": 2, "body": "panel finding"}]}], f) + done, outputs = run_step(step(self.consolidate, BUILD_STEP), + {"steps.aggregate.outputs.ok_count": "1", + "steps.aggregate.outputs.panel_inconsistent": "false"}, + self.workdir) + self.assertEqual(done.returncode, 0, done.stdout + done.stderr) + with open(os.path.join(self.workdir, "consolidated.json"), encoding="utf-8") as f: + return outputs, json.load(f) + + def test_the_seed_names_the_judge_that_will_run(self): + self.assertEqual(self.configure("true")["model"], f"{JUDGE_MODEL} (direct)") + self.assertEqual(self.configure("false")["model"], CURSOR_JUDGE) + + def test_a_review_carrying_the_key_is_replaced_by_the_seed(self): + self.configure("true") + self.submit(f"leaked {KEY} here") + done = self.guard() + self.assertEqual(done.returncode, 0, done.stdout + done.stderr) + self.assertIn("::error::", done.stdout) + self.assertNotIn(KEY, done.stdout + done.stderr) + record = self.read() + self.assertEqual(record["status"], "error") + self.assertEqual(record["findings"], []) + self.assertEqual(record["model"], f"{JUDGE_MODEL} (direct)") + self.assertNotIn(KEY, json.dumps(record)) + outputs, consolidated = self.build() + self.assertEqual(outputs["degraded"], "true") + self.assertEqual([f["body"] for f in consolidated["findings"]], ["panel finding"]) + + def test_a_clean_review_flows_through_unchanged(self): + self.configure("true") + submitted = self.submit("a real finding") + done = self.guard() + self.assertEqual(done.returncode, 0, done.stdout + done.stderr) + self.assertEqual(self.read(), submitted) + # Status and count only: the run log is public. + self.assertIn("Direct judge: status=ok, 1 finding(s).", done.stdout) + self.assertNotIn("a real finding", done.stdout) + outputs, consolidated = self.build() + self.assertEqual(outputs["degraded"], "false") + self.assertEqual(consolidated["findings"], submitted["findings"]) + + def test_a_judge_that_never_submitted_takes_the_degraded_path(self): + self.configure("true") + done = self.guard() + self.assertEqual(done.returncode, 0, done.stdout + done.stderr) + self.assertIn("Direct judge: status=error, 0 finding(s).", done.stdout) + outputs, _ = self.build() + self.assertEqual(outputs["degraded"], "true") + + def test_an_untrusted_status_is_not_echoed(self): + with open(os.path.join(self.workdir, "judge-findings.json"), "w", encoding="utf-8") as f: + json.dump({"status": "::error::forged", "findings": "x"}, f) + done = self.guard() + self.assertIn("Direct judge: status=?, 0 finding(s).", done.stdout) + self.assertNotIn("forged", done.stdout) + + +if __name__ == "__main__": + unittest.main() diff --git a/.github/cursor-review/tests/test_workflow_panel_integrity.py b/.github/cursor-review/tests/test_workflow_panel_integrity.py index 9323e8ac..02f0a089 100644 --- a/.github/cursor-review/tests/test_workflow_panel_integrity.py +++ b/.github/cursor-review/tests/test_workflow_panel_integrity.py @@ -681,7 +681,7 @@ def test_the_deliberate_skips_are_still_skips(self): self.assertIn(gate, self.condition, f"`{PANEL_JOB}` lost gate `{gate}`") -CONTEXT_REF = re.compile(r"\b(?:needs|inputs|steps|vars|matrix|github)(?:\.[A-Za-z0-9_-]+)+") +CONTEXT_REF = re.compile(r"\b(?:needs|inputs|steps|vars|matrix|github|secrets)(?:\.[A-Za-z0-9_-]+)+") def evaluate(expression, context): @@ -1248,7 +1248,10 @@ class AnthropicDirectCellsGateWhenTheyReplaceTest(DirectCellsGateWhenTheyReplace def test_the_judge_model_is_untouched(self): step = self.step(self.preflight, PANEL_MODELS_STEP) self.assertFalse( - any("judge" in line.lower() for line in code_lines(step) if "anthropic" in line.lower()), + # `anthropic_direct`, not `anthropic`: the direct JUDGE's own + # resolution in this step names ANTHROPIC_API_KEY beside + # `judge_direct_model`, and is not the cells' replacement. + any("judge" in line.lower() for line in code_lines(step) if "anthropic_direct" in line.lower()), "the Anthropic replacement reaches the judge model", ) diff --git a/.github/workflows/cursor-review.yml b/.github/workflows/cursor-review.yml index b83eea4a..9ef763f0 100644 --- a/.github/workflows/cursor-review.yml +++ b/.github/workflows/cursor-review.yml @@ -357,6 +357,36 @@ on: type: boolean required: false default: true + judge_direct_model: + description: >- + OPT-IN. An Anthropic model id (e.g. `claude-opus-5-5`) that runs the + JUDGE through the Anthropic API directly, via the same pinned Claude + Code CLI and `ANTHROPIC_API_KEY` secret as `anthropic_direct_model`, + instead of `judge_model` through Cursor. Empty (default) = the judge + stays on Cursor and nothing changes. Requires a plain id + (`[A-Za-z0-9._-]`, at most 100) and the key; set without either, + preflight warns and the Cursor judge runs. Same prompt, same + structured-output tool, same timeout and same degraded panel-union + fallback as the Cursor judge. The agent reads the PR checkout with + Read, Grep and Glob alone — no Bash or other code-running tool — + confined to it by `--restricted`. `consolidate` then runs on + GitHub-hosted `ubuntu-latest` whatever `runs_on` says, because it + holds the key. This judge reads private-repo diffs and the workflow + cannot check the Anthropic organization's data-retention terms, so + a caller on a private repo must use a key from an organization with + zero data retention. + type: string + required: false + default: '' + judge_direct_effort: + description: >- + Reasoning effort for the direct judge (`--effort`): one of `low`, + `medium`, `high`, `xhigh`, `max`. Any other value warns and falls + back to `xhigh`, the default. Ignored unless `judge_direct_model` is + in effect. + type: string + required: false + default: xhigh skip_bot_branch_prefixes: description: >- Space-separated head-branch prefixes that skip the review when the PR @@ -552,7 +582,9 @@ on: '["self-hosted", "linux", "x64"]'. Every other job — the gate, the ledger, the Slack notices, and every job that mints the App token or holds `pull-requests: write` — stays on GitHub-hosted `ubuntu-latest` - regardless, so a pool never sees a write credential. Review and + regardless, so a pool never sees a write credential — and so does + `consolidate` whenever `ANTHROPIC_API_KEY` is passed, for the direct + judge (`judge_direct_model`). Review and consolidate run models with unsandboxed shell over the PR's code, so pick a pool that gives each job a fresh pod (fresh filesystem and $HOME), carries no cloud identity or service-account token, and @@ -578,11 +610,15 @@ on: required: false ANTHROPIC_API_KEY: description: >- - Anthropic API key for the optional direct-API Anthropic review cells. - Required only when the `anthropic_direct_model` input is set; ignored - otherwise. Use a key scoped to its own Anthropic workspace with a - spend limit, never a personal key — the cells bill per call against - it directly rather than against a pooled seat. + Anthropic API key for the optional direct-API Anthropic review cells + and direct judge. Required only when the `anthropic_direct_model` or + `judge_direct_model` input is set; otherwise unused — but whenever it + is passed, `consolidate` runs on GitHub-hosted `ubuntu-latest` rather + than `runs_on`, since the direct judge's steps reference it and a job + is sent every secret its steps reference, skipped steps included. + Use a key scoped to its own Anthropic workspace with a spend limit, + never a personal key — the cells and judge bill per call against it + directly rather than against a pooled seat. required: false SLACK_BOT_TOKEN: description: Slack bot token for the start/complete DM notifications. Optional — DMs are skipped if absent. @@ -750,9 +786,11 @@ jobs: - name: Resolve ANTHROPIC_API_KEY presence # Same as `Resolve OPENAI_API_KEY presence` above, for the direct - # Anthropic cells. + # Anthropic cells and the direct judge. Unlike that one it runs on + # every event: `consolidate` reads it to pick its runner, and the + # direct judge's steps reference the secret, so a passed key must keep + # that job GitHub-hosted even with neither direct input set. id: anthropic_key - if: inputs.anthropic_direct_model != '' env: ANTHROPIC_KEY_PRESENT: ${{ secrets.ANTHROPIC_API_KEY != '' }} run: echo "present=${ANTHROPIC_KEY_PRESENT}" >> "$GITHUB_OUTPUT" @@ -2097,6 +2135,12 @@ jobs: anthropic_direct_counts: ${{ steps.models.outputs.anthropic_direct_counts }} # `anthropic_direct_effort`, validated: always one of the allowed levels. anthropic_direct_effort: ${{ steps.models.outputs.anthropic_direct_effort }} + # 'true' only when the judge can actually run direct: `judge_direct_model` + # is a plain id and ANTHROPIC_API_KEY reached this workflow. `consolidate` + # switches its runner and its judge steps on THIS, never the raw input. + judge_direct: ${{ steps.models.outputs.judge_direct }} + # `judge_direct_effort`, validated: always one of the allowed levels. + judge_direct_effort: ${{ steps.models.outputs.judge_direct_effort }} # The caps DERIVED from `cell_timeout_minutes`, computed here because # Actions expressions have no arithmetic: `${{ x + 15 }}` is a parse # error that fails the workflow at load for every caller. Consumers @@ -2267,6 +2311,9 @@ jobs: ANTHROPIC_DIRECT_REPLACES: ${{ inputs.anthropic_direct_replaces_cursor_anthropic }} ANTHROPIC_DIRECT_EFFORT: ${{ inputs.anthropic_direct_effort }} ANTHROPIC_KEY_PRESENT: ${{ needs.gate.outputs.anthropic_key_present }} + # The direct judge (touches no panel list). + JUDGE_DIRECT_MODEL: ${{ inputs.judge_direct_model }} + JUDGE_DIRECT_EFFORT: ${{ inputs.judge_direct_effort }} run: | cat > /tmp/models.json <<'JSON' [ @@ -2418,6 +2465,32 @@ jobs: fi fi echo "anthropic_direct_counts=${anthropic_direct_counts}" >> "$GITHUB_OUTPUT" + # The direct judge, by the cells' rules: a plain id (no whitespace at + # all) and the key present, or it warns and the Cursor judge runs. + # No collision check — the judge uploads no per-model artifact. + judge_direct=false + if [ -n "$(printf '%s' "${JUDGE_DIRECT_MODEL:-}" | tr -d '[:space:]')" ]; then + case "$JUDGE_DIRECT_MODEL" in *[[:space:]]*) judge_ws=true ;; *) judge_ws=false ;; esac + if [ "$judge_ws" = "true" ] \ + || ! printf '%s' "$JUDGE_DIRECT_MODEL" | grep -Eqx '[A-Za-z0-9._-]{1,100}'; then + echo "::warning::judge_direct_model is not a plain model id (letters, digits, '.', '_', '-'; at most 100), so the judge runs on Cursor (judge_model) for this run." + elif [ "${ANTHROPIC_KEY_PRESENT:-false}" != "true" ]; then + echo "::warning::judge_direct_model is set but the ANTHROPIC_API_KEY secret did not reach this workflow, so the judge runs on Cursor (judge_model) for this run. Pass the secret through from the caller to enable it." + else + judge_direct=true + echo "The judge runs direct through the Anthropic API ($JUDGE_DIRECT_MODEL)." + fi + fi + echo "judge_direct=${judge_direct}" >> "$GITHUB_OUTPUT" + case "${JUDGE_DIRECT_EFFORT:-}" in + low|medium|high|xhigh|max) judge_direct_effort="$JUDGE_DIRECT_EFFORT" ;; + *) + judge_direct_effort=xhigh + if [ "$judge_direct" = "true" ]; then + echo "::warning::judge_direct_effort is not one of low, medium, high, xhigh, max, so the direct judge uses xhigh." + fi ;; + esac + echo "judge_direct_effort=${judge_direct_effort}" >> "$GITHUB_OUTPUT" echo "models=$(jq -c . /tmp/models.json)" >> "$GITHUB_OUTPUT" - name: Require a pinned workflows_ref @@ -3997,7 +4070,10 @@ jobs: consolidate: # Adjudicates the panel cell artifacts into one consolidated review. - # Calls cursor-agent one more time as the judge, with prompt-judge.md. + # Calls cursor-agent one more time as the judge, with prompt-judge.md — + # or, with `judge_direct_model` in effect (preflight's `judge_direct`), + # `claude -p` through the Anthropic API instead: same prompt, same + # structured-output tool, same file, same degraded fallback. # # Holds no write credential: the payload is handed to the `post-review` job # below as an artifact, and that job does the mint + post. See the header's @@ -4022,7 +4098,14 @@ jobs: # was cancelled, so a red or cancelled matrix cell still reaches the judge, # which is what `always()` was here for. if: ${{ !cancelled() && needs.gate.outputs.should_run == 'true' && needs.gate.outputs.already_reviewed != 'true' && needs.diff-size.outputs.within_cap == 'true' && needs.review.result != 'skipped' }} - runs-on: ${{ fromJSON(inputs.runs_on || '"ubuntu-latest"') }} + # GitHub-hosted whenever this job can hold ANTHROPIC_API_KEY, `runs_on` + # otherwise. The direct judge's steps reference the secret, and a job is + # sent every secret its steps reference — skipped steps included — so the + # switch keys on the key being PASSED (`anthropic_key_present`), not only + # on the direct judge running. `judge_direct` implies it; both are named + # so neither half can be dropped alone. With no key passed, today's + # `runs_on` routing is unchanged. + runs-on: ${{ (needs.preflight.outputs.judge_direct == 'true' || needs.gate.outputs.anthropic_key_present == 'true') && 'ubuntu-latest' || fromJSON(inputs.runs_on || '"ubuntu-latest"') }} # ABOVE the `review` cap, never under it. The judge reads every cell's # artifact with the same top reasoning tier one cell uses, so it is the # strictly harder call — yet at 15m it was capped at HALF the panel it @@ -4162,6 +4245,61 @@ jobs: - name: Install Cursor agent CLI run: bash "$CURSOR_REVIEW_ASSETS/install-cursor-cli.sh" + - name: Install Claude Code CLI + id: claude_install + if: needs.preflight.outputs.judge_direct == 'true' + # `review-anthropic-direct`'s install, verbatim: the pin is read out of + # .github/cursor-review/package.json, its one home. Absorbed, unlike + # there: a judge that cannot start must cost the adjudication only, so + # `Run judge (direct)` is skipped and the seed takes the degraded + # panel-union path, as a failed Cursor judge does. + continue-on-error: true + run: | + manifest="$CURSOR_REVIEW_ASSETS/package.json" + if [ ! -f "$manifest" ]; then + echo "::error::$manifest is missing — it is the single source of truth for the direct judge's CLI pin." + exit 1 + fi + if ! version=$(jq -r ' + if type != "object" then "" + elif (.dependencies | type) != "object" then "" + else (.dependencies["@anthropic-ai/claude-code"] // "") + end' "$manifest"); then + echo "::error::$manifest is not valid JSON — it is the single source of truth for the direct judge's CLI pin." + exit 1 + fi + version="$(printf '%s' "$version" | tr -d '\r\n')" + case "$version" in + ""|*[!0-9.]*) + echo "::error::dependencies[\"@anthropic-ai/claude-code\"] must be an EXACT version (digits and dots only, no ^ or ~); got a value this job will not install." + exit 1 ;; + esac + echo "CLAUDE_CLI_VERSION=$version" >> "$GITHUB_ENV" + npm install -g "@anthropic-ai/claude-code@${version}" + + - name: Verify Claude Code CLI version + id: claude_verify + if: needs.preflight.outputs.judge_direct == 'true' && steps.claude_install.outcome == 'success' + # `review-anthropic-direct`'s check, verbatim, and absorbed for the + # same reason as the install: a CLI that is off-pin, or no longer + # advertises a flag the confinement relies on, never runs. + continue-on-error: true + run: | + installed="$(claude --version 2>/dev/null | head -n 1 | tr -d '\r' | awk '{print $1}')" + if [ "$installed" != "$CLAUDE_CLI_VERSION" ]; then + echo "::error::claude reports '${installed}', expected the pinned '${CLAUDE_CLI_VERSION}'." + exit 1 + fi + echo "claude ${installed}" + help="$(claude --help 2>&1)" + for flag in --restricted --setting-sources --strict-mcp-config \ + --disallowedTools --permission-prompts --no-session-persistence; do + if ! printf '%s\n' "$help" | grep -qE -- "(^|[[:space:],])${flag}([[:space:],]|$)"; then + echo "::error::claude ${installed} does not advertise ${flag}, which the direct judge's confinement relies on." + exit 1 + fi + done + - name: Download prior-review ledger id: ledger_download continue-on-error: true @@ -4177,11 +4315,21 @@ jobs: pattern: findings-* - name: Configure structured final review + env: + # The seed names the judge that will run: ` (direct)` for the + # direct judge, so a judge that never submits is not attributed to + # the Cursor model. + JUDGE_DIRECT: ${{ needs.preflight.outputs.judge_direct }} + JUDGE_DIRECT_MODEL: ${{ inputs.judge_direct_model }} run: | + judge_label="$JUDGE_MODEL" + if [ "$JUDGE_DIRECT" = "true" ]; then + judge_label="$JUDGE_DIRECT_MODEL (direct)" + fi python3 "$CURSOR_REVIEW_ASSETS/review-output-mcp.py" \ --mode judge \ --out /tmp/judge-findings.json \ - --model "$JUDGE_MODEL" \ + --model "$judge_label" \ --review-type judge \ --init @@ -4313,7 +4461,7 @@ jobs: echo "Judge prompt size: $(wc -c < /tmp/judge-prompt.txt) bytes" - name: Run judge - if: steps.aggregate.outputs.ok_count != '0' + if: steps.aggregate.outputs.ok_count != '0' && needs.preflight.outputs.judge_direct != 'true' # Time-boxed BELOW the job cap and absorbed, so a hung judge loses only # the adjudication: the steps below then read the `--init` seed left by # `Configure structured final review` (status=error), take the degraded @@ -4356,6 +4504,121 @@ jobs: cat /tmp/judge-findings.json echo "" + - name: Run judge (direct) + # `Run judge` above, through the Anthropic API instead of Cursor: the + # same /tmp/judge-prompt.txt, the same `review-output-mcp.py --mode + # judge` writing the same /tmp/judge-findings.json, the same step cap + # and the same absorption — a judge that fails, times out or never + # submits leaves the seed, and `Build consolidated findings file` takes + # the degraded panel-union path exactly as it does for Cursor. Skipped + # (seed left in place, same path) when the CLI did not install clean. + # + # Confinement, as in `review-anthropic-direct`: ANTHROPIC_API_KEY is + # in this ONE step's env (never argv, never a file), and the agent has + # Read, Grep and Glob — no Bash or other code-running tool — confined + # to the PR checkout (its cwd) by `--restricted`, with /proc and /sys + # denied independently of that flag, since /proc/self/environ is the + # key. The result JSON is never echoed: the run log is public and the + # result is model output steered by PR text. + if: steps.aggregate.outputs.ok_count != '0' && needs.preflight.outputs.judge_direct == 'true' && steps.claude_verify.outcome == 'success' + continue-on-error: true + timeout-minutes: ${{ fromJSON(needs.preflight.outputs.judge_step_timeout) }} + env: + JUDGE_DIRECT_MODEL: ${{ inputs.judge_direct_model }} + JUDGE_DIRECT_EFFORT: ${{ needs.preflight.outputs.judge_direct_effort }} + ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }} + # No telemetry or error report may carry private-diff context. + CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC: '1' + run: | + if [ -z "${ANTHROPIC_API_KEY:-}" ]; then + # Belt and braces: preflight already required the key. + echo "::warning::judge_direct_model is in effect but ANTHROPIC_API_KEY is empty here, so the judge did not run." + exit 0 + fi + # A planted symlink could lead the file tools outside the tree, and + # a planted .claude/settings*.json or .mcp.json must never be there + # to load (`--restricted` and `--setting-sources ""` ignore them + # already; this removes them outright). + find . -type l -delete + rm -rf ./.claude ./.mcp.json ./CLAUDE.local.md + CLAUDE_CONFIG_DIR="$(mktemp -d -p "$RUNNER_TEMP")" + export CLAUDE_CONFIG_DIR + # `-I`: the cwd is the PR checkout, and a stdin script puts '' first + # on sys.path, so a PR-planted json.py would run beside the key. + mcp_config="$(python3 -I - <<'PY' + import json, os + print(json.dumps({"mcpServers": {"cursor-review-output": { + "command": "python3", + "args": [ + "-I", + os.path.join(os.environ["CURSOR_REVIEW_ASSETS"], "review-output-mcp.py"), + "--mode", "judge", + "--out", "/tmp/judge-findings.json", + "--model", os.environ["JUDGE_DIRECT_MODEL"] + " (direct)", + "--review-type", "judge", + ], + }}})) + PY + )" + # Deliberately NOT `--bare`: on the pinned CLI it drops Grep and + # Glob from `--tools` (see `review-anthropic-direct`). What it would + # skip is already off: `--setting-sources ""` and a fresh + # CLAUDE_CONFIG_DIR, `--strict-mcp-config`, and + # CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC. + claude -p --restricted --setting-sources "" \ + --tools "Read,Grep,Glob" \ + --strict-mcp-config --mcp-config "$mcp_config" \ + --allowedTools "Read,Grep,Glob,mcp__cursor-review-output__cursor_review_submit_final" \ + --disallowedTools "Read(//proc/**),Read(//sys/**)" \ + --permission-prompts none \ + --model "$JUDGE_DIRECT_MODEL" --effort "$JUDGE_DIRECT_EFFORT" \ + --output-format json --no-session-persistence \ + < /tmp/judge-prompt.txt > /tmp/judge-claude-result.json \ + && JUDGE_EXIT=0 || JUDGE_EXIT=$? + echo "claude (judge) exit code: $JUDGE_EXIT" + + - name: Refuse a judge review carrying the API key + # The direct cells' exfil guard, for the judge: should a + # prompt-injected judge get the key into its final review, replace the + # review with the `--init` error seed, so the degraded path posts the + # panel union instead. Literal match only. `always()`, so a judge + # killed by its step cap AFTER submitting is still checked. Then the + # status and finding count — never the review itself (see above). + if: always() && needs.preflight.outputs.judge_direct == 'true' + env: + JUDGE_DIRECT_MODEL: ${{ inputs.judge_direct_model }} + ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }} + run: | + set -uo pipefail + out=/tmp/judge-findings.json + if [ -n "${ANTHROPIC_API_KEY:-}" ] && [ -f "$out" ] \ + && LC_ALL=C grep -qF -- "$ANTHROPIC_API_KEY" "$out"; then + echo "::error::the direct judge's final review contains ANTHROPIC_API_KEY — refusing to publish it (possible prompt-injection exfil); the panel union is posted instead." + python3 "$CURSOR_REVIEW_ASSETS/review-output-mcp.py" \ + --mode judge \ + --out "$out" \ + --model "$JUDGE_DIRECT_MODEL (direct)" \ + --review-type judge \ + --init + fi + python3 - <<'PY' + import json + try: + with open("/tmp/judge-findings.json", encoding="utf-8") as f: + record = json.load(f) + except (OSError, ValueError): + record = {} + if not isinstance(record, dict): + record = {} + status = record.get("status") + findings = record.get("findings") + # Fixed vocabulary only: `status` is written by a tool the agent + # drives, so anything else is reported as `?`, never echoed. + status = status if status in ("ok", "error") else "?" + count = len(findings) if isinstance(findings, list) else 0 + print(f"Direct judge: status={status}, {count} finding(s).") + PY + - name: Build consolidated findings file id: consolidated env: diff --git a/docs/callers/cursor-review.md b/docs/callers/cursor-review.md index 9de6b535..d91ea21d 100644 --- a/docs/callers/cursor-review.md +++ b/docs/callers/cursor-review.md @@ -53,7 +53,7 @@ Prompts and scripts live in [`.github/cursor-review/`](../../.github/cursor-revi |---|---| | `secrets.CURSOR_API_KEY` | **Required.** Org-level in Comfy-Org. | | `secrets.OPENAI_API_KEY` | Optional. Only read when `openai_direct_model` is set; use a project-scoped service-account key with a spend limit. | -| `secrets.ANTHROPIC_API_KEY` | Optional. Only read when `anthropic_direct_model` is set; use a key scoped to its own Anthropic workspace with a spend limit, never a personal key. | +| `secrets.ANTHROPIC_API_KEY` | Optional. Only used when `anthropic_direct_model` or `judge_direct_model` is set (and, when passed, `consolidate` runs on GitHub-hosted `ubuntu-latest` whatever `runs_on` says); use a key scoped to its own Anthropic workspace with a spend limit, never a personal key. | | `secrets.SLACK_BOT_TOKEN` | Optional. Without it the review still posts; only the DMs are skipped. | | `vars.REVIEW_BOT_APP_ID` + `secrets.BOT_APP_PRIVATE_KEY` | Optional. Posts the review as your App instead of `github-actions[bot]`. | | A review label | Default `cursor-review`. Create it in your repo. | @@ -119,6 +119,8 @@ pull-requests: write # posting the consolidated review (and, at the round cap, | `anthropic_direct_model` | `''` | Opt-in. The Anthropic twin of `openai_direct_model`: an Anthropic model id (e.g. `claude-opus-5-5`) reviewed directly through the Anthropic API (pinned Claude Code CLI) as two cells — **adversarial** and **edge-case** — always on GitHub-hosted `ubuntu-latest`, by default **replacing** the Cursor `claude-*` cells. Requires `secrets.ANTHROPIC_API_KEY` and a plain id (`[A-Za-z0-9._-]`) other than `openai_direct_model`'s; without any of those, preflight warns, the cells stay off and the panel is unchanged. Unlike the OpenAI cells, the agent also reads a **read-only PR checkout** with Read, Grep and Glob only — no Bash or code-running tool — and `--restricted` confines those file tools to the checkout (`/proc` and `/sys` are also denied by explicit deny rules). **Private repos:** this panel reviews private-repo diffs and the workflow cannot check the Anthropic organization's data-retention terms, so use a key from an organization with zero data retention. Each cell also uploads a `usage-direct--` artifact (kept 30 days) with the API's token counts for that cell, so cost can be measured without an Admin API key. | | `anthropic_direct_effort` | `xhigh` | Reasoning effort for the direct Anthropic cells: `low`, `medium`, `high`, `xhigh` or `max`. Any other value warns and falls back to `xhigh`. | | `anthropic_direct_replaces_cursor_anthropic` | `true` | Same semantics as `openai_direct_replaces_cursor_openai`, for the Cursor `claude-*` cells: `true` drops them and the direct Anthropic cells **gate** in their place; `false` keeps both, with the direct cells **advisory**. Each lab counts by its own flag, so OpenAI and Anthropic may be set differently. The judge (`judge_model`) is never touched. | +| `judge_direct_model` | `''` | Opt-in. An Anthropic model id (e.g. `claude-opus-5-5`) that runs the **judge** directly through the Anthropic API (the same pinned Claude Code CLI and `secrets.ANTHROPIC_API_KEY` as `anthropic_direct_model`) instead of `judge_model` through Cursor. Requires the secret and a plain id (`[A-Za-z0-9._-]`); without either, preflight warns and the Cursor judge runs. Same prompt, output tool, timeout and degraded panel-union fallback as the Cursor judge; the agent reads the PR checkout with Read, Grep and Glob only — no Bash or code-running tool — confined to it by `--restricted` (`/proc` and `/sys` are also denied by explicit deny rules). `consolidate` runs on GitHub-hosted `ubuntu-latest` regardless of `runs_on` whenever the secret is passed. **Private repos:** this judge reads private-repo diffs and the workflow cannot check the Anthropic organization's data-retention terms, so use a key from an organization with zero data retention. | +| `judge_direct_effort` | `xhigh` | Reasoning effort for the direct judge: `low`, `medium`, `high`, `xhigh` or `max`. Any other value warns and falls back to `xhigh`. | | `skip_bot_branch_prefixes` | `ci/bump- chore/refresh- auto/refresh-` | Skip the panel for Bot-authored PRs on these branch prefixes (machine pin bumps / refreshes). `''` to review every bot PR. | | `cell_timeout_minutes` | `15` | Wall-clock cap on ONE reviewer cell's agent step (`Run cursor review`, or `Run direct-API review` for a direct cell). A cell that runs out of clock exits 130 and uploads `status: error` with no findings, which fails its leg and so fails `Panel integrity` — so this is a guillotine, not a safety net. Raise it when cells are dying at the cap rather than hanging: compare how long the cells that DID submit took, and if a meaningful share finish within a few minutes of the cap you are clipping a tail, not catching hangs. It is also the budget a cell's retries share (`resource_exhausted` is retried in-step), so it bounds attempts plus backoff. Three further caps derive from this one and cannot be set separately — each cell's job cap is this + 15 (checkout, CLI install, artifacts), the judge's `Run judge` step cap is this × 2 but never below 30 (it reads every cell's findings at the same tier, and its workload scales with the panel, not this cap), and the judge's job cap is its step + 10. Bounded 10–45 (the floor leaves a late in-step retry room for a full review); preflight fails loud outside that, before any cell spends. | | `diff_size_cap` | `5000` | Skip review above this diff size. An over-cap PR is not silent — see the gotcha below. Under `blocking: true` it is also not green: an unreviewed PR cannot pass the gate. | @@ -138,7 +140,7 @@ pull-requests: write # posting the consolidated review (and, at the round cap, | `defer_approval` | `false` | Only for a caller that runs [cursor-approve](cursor-approve.md) after this workflow: a round that would approve still reports `approve_gate` = `pass`, but posts **no** approval and resolves no thread, and withdraws the approver's own earlier auto-approvals and requests for changes instead — so cursor-approve's decide is the only approver. Requests changes as usual. No effect without `approve_max_severity`. | | `approve_max_failed_reviewers` | `0` | How many panel reviewers may **error** (the cell ran but failed, timed out, or never uploaded) before auto-approve withholds its decision. `0` keeps the strict rule: any reviewer that did not complete means no decision. `N` tolerates up to N errored reviewers and names them in the decision (e.g. "approved with 1/6 reviewers errored: `:edge-case`"). See [auto-approve](#auto-approve). No effect without `approve_max_severity`. | | `max_rounds` | `5` | Cap on review rounds per PR (`0` → no cap). At or over it, no panel runs: the PR is labelled `needs-human-review`, one comment lists the latest round's open findings above the threshold, and the `approve_gate` output is `capped`. Removing the label resets the count. See [round cap](#round-cap-and-the-approve_gate-output). | -| `runs_on` | `'"ubuntu-latest"'` | JSON-encoded `runs-on` for `diff-size`, `preflight`, the `review` panel and `consolidate` only — the jobs that hold no write credential. Every other job stays on GitHub-hosted `ubuntu-latest`. A self-hosted pool is fine only if it is one-job-per-fresh-pod, identity-free and private-network-isolated (those jobs run models with shell over PR code), on linux/x64 with bash, git, curl, jq, python3, gh, tar and GNU coreutils. Empty falls back to the default, so `${{ vars.CURSOR_REVIEW_RUNS_ON }}` is safe while the variable is unset; e.g. `'["self-hosted", "linux", "x64"]'`. | +| `runs_on` | `'"ubuntu-latest"'` | JSON-encoded `runs-on` for `diff-size`, `preflight`, the `review` panel and `consolidate` only — the jobs that hold no write credential (`consolidate` stays on `ubuntu-latest` whenever `ANTHROPIC_API_KEY` is passed). Every other job stays on GitHub-hosted `ubuntu-latest`. A self-hosted pool is fine only if it is one-job-per-fresh-pod, identity-free and private-network-isolated (those jobs run models with shell over PR code), on linux/x64 with bash, git, curl, jq, python3, gh, tar and GNU coreutils. Empty falls back to the default, so `${{ vars.CURSOR_REVIEW_RUNS_ON }}` is safe while the variable is unset; e.g. `'["self-hosted", "linux", "x64"]'`. | ## Running a model experiment From 3880687bc9fb71e3547a65af4a3c33e16a12b232 Mon Sep 17 00:00:00 2001 From: Matt Miller Date: Fri, 9 Oct 2026 23:18:29 +0000 Subject: [PATCH 2/2] fix(cursor-review): run the direct judge in its own job so ANTHROPIC_API_KEY never reaches consolidate MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review round on the direct judge (BE-19914): - High: a job is sent every secret its steps reference, skipped steps included, so putting the direct judge's steps in `consolidate` sent the key to the job running the Cursor judge (an unsandboxed shell agent over PR text) for any caller that passes it — including callers passing it only for the direct cells. The judge now runs in a new `judge-direct` job (GitHub-hosted, contents: read, no shell agent) that rebuilds the judge input with steps byte-identical to consolidate's and hands back /tmp/judge-findings.json as an artifact. `consolidate` adopts it over its seed only from a job that succeeded and only when it is a JSON object, so a forged same-name upload (which 409s the real one) degrades to the panel union. `consolidate`'s runs-on is back to today's routing, and the gate's key-presence step is again conditional (now on either direct input). - Critical: every stdin Python in the job holding the key is `python3 -I`, including the exfil guard and its --init call. - Medium: the Claude CLI install and its check are time-boxed (5m / 2m), so a hung registry fetch trips the absorbed step, not the job cap. - Low: the Cursor CLI install is skipped when the judge runs direct. Co-Authored-By: Claude Opus 5.5 --- .github/cursor-review/README.md | 6 +- .../tests/test_workflow_job_isolation.py | 25 +- .../tests/test_workflow_judge_direct.py | 194 +++-- .../tests/test_workflow_ref_guard_parity.py | 2 +- .../tests/test_check_workflow_pins.py | 8 +- .github/workflows/cursor-review.yml | 705 ++++++++++++------ docs/callers/cursor-review.md | 6 +- 7 files changed, 644 insertions(+), 302 deletions(-) diff --git a/.github/cursor-review/README.md b/.github/cursor-review/README.md index 0b6a9023..fbcbc45c 100644 --- a/.github/cursor-review/README.md +++ b/.github/cursor-review/README.md @@ -327,7 +327,7 @@ jobs: |---|---|---|---| | Secret | `CURSOR_API_KEY` | **yes** | Bills the panel + judge `cursor-agent` calls. | | Secret | `OPENAI_API_KEY` | no | Only with `openai_direct_model`: bills the optional direct-OpenAI cells. Use a project-scoped service-account key with a spend limit. | -| Secret | `ANTHROPIC_API_KEY` | no | Only with `anthropic_direct_model` or `judge_direct_model`: bills the optional direct-Anthropic cells and direct judge. Passing it moves `consolidate` to GitHub-hosted `ubuntu-latest` whatever `runs_on` says. Use a key scoped to its own Anthropic workspace with a spend limit, never a personal key. | +| Secret | `ANTHROPIC_API_KEY` | no | Only with `anthropic_direct_model` or `judge_direct_model`: bills the optional direct-Anthropic cells and direct judge. Use a key scoped to its own Anthropic workspace with a spend limit, never a personal key. | | Secret | `SLACK_BOT_TOKEN` | no | Enables start/complete DMs to the triggerer. | | Variable | `CURSOR_REVIEW_DM_EMAIL_MAP` | no | Maps GitHub logins → emails for Slack DM lookup. | @@ -395,7 +395,7 @@ All optional except `workflows_ref` (required, no default) — pass them under | `anthropic_direct_model` | `''` (off) | OPT-IN. The Anthropic twin of `openai_direct_model`: an Anthropic model id (e.g. `claude-opus-5-5`) reviewed through the Anthropic API directly (pinned Claude Code CLI, `package.json` here) as two cells on GitHub-hosted runners, by default **replacing** the Cursor `claude-*` cells. Needs the `ANTHROPIC_API_KEY` secret and a plain id other than `openai_direct_model`'s; without any of those, preflight warns and leaves the cells off and the panel unchanged. The agent also reads a **read-only PR checkout** (symlinks removed) with Read, Grep and Glob only — no Bash or code-running tool — confined to it by `--restricted`, with `/proc` and `/sys` also denied by explicit deny rules. This panel reviews private-repo diffs and the workflow cannot check the Anthropic organization's data-retention terms: callers on private repos must use a key from an organization with zero data retention. Each cell also uploads a `usage-direct--` artifact (kept 30 days) with the API's token counts for that cell, so cost can be measured without an Admin API key. | | `anthropic_direct_effort` | `xhigh` | `--effort` for the direct Anthropic cells: `low`, `medium`, `high`, `xhigh` or `max`; anything else warns and falls back to `xhigh`. | | `anthropic_direct_replaces_cursor_anthropic` | `true` | As `openai_direct_replaces_cursor_openai`, for the Cursor `claude-*` cells. Each lab counts by its own flag. Never touches `judge_model`. | -| `judge_direct_model` | `''` (off) | OPT-IN. An Anthropic model id (e.g. `claude-opus-5-5`) that runs the **judge** through the Anthropic API directly instead of `judge_model` through Cursor, with the same pinned Claude Code CLI and `ANTHROPIC_API_KEY` as `anthropic_direct_model`. Needs that secret and a plain id (`[A-Za-z0-9._-]`, at most 100); without either, preflight warns and the Cursor judge runs. Same prompt, structured-output tool, step timeout and degraded panel-union fallback as the Cursor judge; the agent reads the PR checkout (symlinks removed) with Read, Grep and Glob only — no Bash or code-running tool — confined to it by `--restricted`, with `/proc` and `/sys` also denied by explicit deny rules. A final review containing the key is replaced by the error seed, so the panel union posts instead; the log shows the judge's status and finding count only. `consolidate` runs on GitHub-hosted `ubuntu-latest` whenever `ANTHROPIC_API_KEY` is passed, whatever `runs_on` says. This judge reads private-repo diffs and the workflow cannot check the Anthropic organization's data-retention terms: callers on private repos must use a key from an organization with zero data retention. | +| `judge_direct_model` | `''` (off) | OPT-IN. An Anthropic model id (e.g. `claude-opus-5-5`) that runs the **judge** through the Anthropic API directly instead of `judge_model` through Cursor, with the same pinned Claude Code CLI and `ANTHROPIC_API_KEY` as `anthropic_direct_model`. Needs that secret and a plain id (`[A-Za-z0-9._-]`, at most 100); without either, preflight warns and the Cursor judge runs. Same prompt, structured-output tool, step timeout and degraded panel-union fallback as the Cursor judge; the agent reads the PR checkout (symlinks removed) with Read, Grep and Glob only — no Bash or code-running tool — confined to it by `--restricted`, with `/proc` and `/sys` also denied by explicit deny rules. A final review containing the key is replaced by the error seed, so the panel union posts instead; the log shows the judge's status and finding count only. The judge runs in its own `judge-direct` job on GitHub-hosted `ubuntu-latest`, whatever `runs_on` says, so the key never reaches `consolidate`. This judge reads private-repo diffs and the workflow cannot check the Anthropic organization's data-retention terms: callers on private repos must use a key from an organization with zero data retention. | | `judge_direct_effort` | `xhigh` | `--effort` for the direct judge: `low`, `medium`, `high`, `xhigh` or `max`; anything else warns and falls back to `xhigh`. | | `cell_timeout_minutes` | `15` | Wall-clock cap on one reviewer cell's agent step (Cursor or direct). Over it the cell exits 130, uploads `status: error` with no findings, reddens its leg and fails `Panel integrity`. Shared by the cell's in-step retries, so it bounds attempts + backoff. Derives three more caps that cannot be set alone: cell job = this + 15, `Run judge` step = max(30, this × 2), judge job = that + 10. At the default: 15 / 30 / 30 / 40, today's values exactly. Preflight bounds it 10–45. | | `skip_bot_branch_prefixes` | `ci/bump- chore/refresh- auto/refresh-` | Skip the panel when the PR author is a Bot **and** the head branch starts with one of these prefixes (machine pin bumps / catalog refreshes). `''` reviews every bot PR. | @@ -416,7 +416,7 @@ All optional except `workflows_ref` (required, no default) — pass them under | `defer_approval` | `false` | For a caller that runs cursor-approve after this workflow: an APPROVE outcome still reports `approve_gate` = `pass` but posts no approval and resolves no thread, and withdraws the approver's own earlier auto-approvals and requests for changes, so cursor-approve's decide is the only approver. Pass the workflow's `approve_scope_effective` output (the scope `decide` actually gated under, `full` on any fallback) to cursor-approve's `approve_scope`, so its resolution honours non-gating marks only after a `delta` round. | | `approve_max_failed_reviewers` | `0` | Errored panel reviewers (status `error`) auto-approve tolerates before withholding its decision. `0` → any reviewer that did not complete means no decision. Never tolerates another status, more than N, or a review type with no completed reviewer; tolerated cells are named in the decision. | | `max_rounds` | `5` | Per-PR review-round cap; `0` → none. Counts the posting identity's consolidated reviews since `needs-human-review` was last removed; at or over the cap the panel is skipped, the PR is labelled `needs-human-review`, one comment lists the latest round's open findings above the threshold, and the `approve_gate` output is `capped`. Fails open on a read error. See [the setup guide](../../docs/callers/cursor-review.md#round-cap-and-the-approve_gate-output). | -| `runs_on` | `'"ubuntu-latest"'` | JSON-encoded `runs-on` (same shape as `cursor-review-auto-label.yml`'s) for the four credential-free heavy jobs only: `diff-size`, `preflight`, the `review` panel and `consolidate` (which stays on `ubuntu-latest` whenever `ANTHROPIC_API_KEY` is passed, for the direct judge). The gate, ledger, Slack notices and every job holding the App token or `pull-requests: write` stay on GitHub-hosted `ubuntu-latest`. Those four run models with unsandboxed shell over PR code, so use a self-hosted pool only if each job gets a fresh pod (fresh filesystem and `$HOME`), holds no cloud identity or token, and can't reach private networks or the metadata server. The image must be linux/x64 with bash, git, curl, jq, python3, gh, tar and GNU coreutils. | +| `runs_on` | `'"ubuntu-latest"'` | JSON-encoded `runs-on` (same shape as `cursor-review-auto-label.yml`'s) for the four credential-free heavy jobs only: `diff-size`, `preflight`, the `review` panel and `consolidate`. The gate, ledger, Slack notices and every job holding the App token or `pull-requests: write` stay on GitHub-hosted `ubuntu-latest`. Those four run models with unsandboxed shell over PR code, so use a self-hosted pool only if each job gets a fresh pod (fresh filesystem and `$HOME`), holds no cloud identity or token, and can't reach private networks or the metadata server. The image must be linux/x64 with bash, git, curl, jq, python3, gh, tar and GNU coreutils. | ### `workflows_ref` must equal the `uses:` pin diff --git a/.github/cursor-review/tests/test_workflow_job_isolation.py b/.github/cursor-review/tests/test_workflow_job_isolation.py index a92d0674..16a4d5b9 100644 --- a/.github/cursor-review/tests/test_workflow_job_isolation.py +++ b/.github/cursor-review/tests/test_workflow_job_isolation.py @@ -305,16 +305,17 @@ def test_the_bot_key_is_confined_to_the_checkout_free_posting_jobs(self): def test_the_anthropic_key_reaches_only_the_gate_and_its_direct_cells(self): # `gate` only resolves a presence boolean; the cells and the direct - # judge (`consolidate`) are the consumers, and they hold a model key - # beside a PR checkout, so they must be GitHub-hosted whenever they can - # hold it — pinned for `consolidate` in test_workflow_judge_direct.py. + # judge (`judge-direct`, never `consolidate` and its Cursor shell + # agent) are the consumers, and they hold a model key beside a PR + # checkout, so they must stay GitHub-hosted. holders = { name for name, body in self.jobs.items() if any("secrets.ANTHROPIC_API_KEY" in l for l in code_lines(body)) } - self.assertEqual(holders, {"gate", "review-anthropic-direct", "consolidate"}) - runs_on = [l.strip() for l in self.jobs["review-anthropic-direct"] if l.startswith(" runs-on:")] - self.assertEqual(runs_on, ["runs-on: ubuntu-latest"]) + self.assertEqual(holders, {"gate", "review-anthropic-direct", "judge-direct"}) + for name in ("review-anthropic-direct", "judge-direct"): + runs_on = [l.strip() for l in self.jobs[name] if l.startswith(" runs-on:")] + self.assertEqual(runs_on, ["runs-on: ubuntu-latest"], name) def test_the_anthropic_cells_load_no_pr_authored_claude_config(self): # `claude` runs with the PR checkout as cwd, so PR-authored @@ -388,17 +389,9 @@ def test_only_credential_free_jobs_follow_the_runs_on_input(self): for name in routed: body = self.jobs[name] self.assertFalse(references_bot_key(body), name) - # `consolidate` may also read ANTHROPIC_API_KEY, for the direct - # judge, ONLY because its runs-on leaves the pool whenever that key - # is passed (test_workflow_judge_direct.py executes that switch). - allowed = ("CURSOR_API_KEY", "ANTHROPIC_API_KEY") if name == "consolidate" else ("CURSOR_API_KEY",) - if name == "consolidate": - runs_on = next(l for l in body if l.startswith(" runs-on:")) - self.assertIn("needs.gate.outputs.anthropic_key_present == 'true'", runs_on) - self.assertIn("&& 'ubuntu-latest' ||", runs_on) self.assertFalse( - any("secrets." in l and not any(k in l for k in allowed) for l in code_lines(body)), - f"job `{name}` follows runs_on but reads a secret other than {', '.join(allowed)}", + any("secrets." in l and "CURSOR_API_KEY" not in l for l in code_lines(body)), + f"job `{name}` follows runs_on but reads a secret other than CURSOR_API_KEY", ) for scope, value in sorted((job_permissions(body) or {"": "missing"}).items()): self.assertIn(value, READ_ONLY_VALUES, f"`{name}` grants `{scope}: {value}`") diff --git a/.github/cursor-review/tests/test_workflow_judge_direct.py b/.github/cursor-review/tests/test_workflow_judge_direct.py index 356e6110..ad0f2b4e 100644 --- a/.github/cursor-review/tests/test_workflow_judge_direct.py +++ b/.github/cursor-review/tests/test_workflow_judge_direct.py @@ -1,15 +1,17 @@ """The optional direct-Anthropic judge in cursor-review.yml (`judge_direct_model`). -Three properties, each one a single line in the workflow that nothing else +Four properties, each one a single line in the workflow that nothing else would notice breaking: * preflight's resolution — `judge_direct=true` only for a plain id with the key present, else a warning and the Cursor judge. EXECUTED: the real `Define panel models` script, run by test_workflow_panel_integrity's helper. - * job isolation — `consolidate` follows `runs_on`, so ANTHROPIC_API_KEY must - appear only in steps gated on the direct judge, and the job's `runs-on:` - must resolve to `ubuntu-latest` whenever it can hold the key (the key is - sent for every step that references it, skipped ones included). + * job isolation — a job is sent every secret its steps reference, skipped + ones included, so ANTHROPIC_API_KEY lives in its own `judge-direct` job + (hosted, no shell agent, every stdin Python `-I`) and never in + `consolidate`, which follows `runs_on` and runs the Cursor shell agent. + * the hand-back — `consolidate` adopts the `judge-direct` review over its + seed only from a job that succeeded, and only a JSON object. EXECUTED. * the exfil guard — a final review carrying the key is replaced by the `--init` error seed, so `Build consolidated findings file` takes the degraded panel-union path. EXECUTED, both steps. @@ -45,13 +47,21 @@ JUDGE_MODEL = "claude-opus-5-5" CURSOR_JUDGE = "claude-opus-4-8-thinking-max" KEY = "sk-ant-test-0123456789abcdef" -RUNS_ON_INPUT = "fromJSON(inputs.runs_on || '\"ubuntu-latest\"')" CONFIGURE_STEP = "Configure structured final review" CURSOR_JUDGE_STEP = "Run judge" DIRECT_JUDGE_STEP = "Run judge (direct)" GUARD_STEP = "Refuse a judge review carrying the API key" BUILD_STEP = "Build consolidated findings file" +ADOPT_STEP = "Adopt direct judge review" +DOWNLOAD_STEP = "Download direct judge review" +UPLOAD_STEP = "Upload direct judge review" +ARTIFACT = "cursor-review-judge-direct" +# Rebuilt in `judge-direct` exactly as in `consolidate`, so both judges read +# the same prompt over the same panel. +SHARED_STEPS = ("Checkout PR repo", "Load cursor-review assets", "Download reviewed diff", + "Download prior-review ledger", "Download panel findings", + AGGREGATE_STEP, "Build judge prompt") def jobs(): @@ -144,56 +154,73 @@ def test_an_unknown_effort_falls_back_to_xhigh(self): self.assertIn("::warning::judge_direct_effort", log) -class ConsolidateJobIsolationTest(unittest.TestCase): +class JudgeDirectJobIsolationTest(unittest.TestCase): @classmethod def setUpClass(cls): - cls.consolidate = jobs()["consolidate"] - cls.steps = split_steps(cls.consolidate) - - def runs_on(self, judge_direct, key_present): - line = next(l for l in code_lines(self.consolidate) if l.startswith(" runs-on:")) - expression = line.split(":", 1)[1].strip() - self.assertIn(RUNS_ON_INPUT, expression, "the runs_on fallback is no longer today's") - # `fromJSON` is outside `evaluate`'s grammar; a sentinel string stands - # in for whatever pool the caller's `runs_on` names. - expression = expression.replace(RUNS_ON_INPUT, "'POOL'") - return evaluate(expression, { - "needs.preflight.outputs.judge_direct": judge_direct, - "needs.gate.outputs.anthropic_key_present": key_present, - }) - - def test_it_is_hosted_whenever_it_can_hold_the_key(self): - self.assertEqual(self.runs_on("true", "true"), "ubuntu-latest") - self.assertEqual(self.runs_on("true", ""), "ubuntu-latest") - # Key passed for the cells only: the direct judge's steps still - # reference it, so it is still sent to this job. - self.assertEqual(self.runs_on("false", "true"), "ubuntu-latest") - - def test_without_the_key_it_follows_runs_on_as_today(self): - self.assertEqual(self.runs_on("false", "false"), "POOL") - self.assertEqual(self.runs_on("", ""), "POOL") - - def test_the_key_is_only_in_steps_gated_on_the_direct_judge(self): - holders = [] - for name, body in self.steps: - if any("secrets.ANTHROPIC_API_KEY" in l for l in code_lines(body)): - holders.append(name) - self.assertIn("needs.preflight.outputs.judge_direct == 'true'", step_scalar(body, "if") or "", name) - self.assertEqual(sorted(holders), sorted([DIRECT_JUDGE_STEP, GUARD_STEP])) - - def test_exactly_one_judge_step_runs(self): - context = {"steps.aggregate.outputs.ok_count": "3", "steps.claude_verify.outcome": "success"} - for judge_direct, expected in (("true", DIRECT_JUDGE_STEP), ("false", CURSOR_JUDGE_STEP), ("", CURSOR_JUDGE_STEP)): - ran = [ - name for name in (CURSOR_JUDGE_STEP, DIRECT_JUDGE_STEP) - if evaluate(step_scalar(step(self.consolidate, name), "if"), - {**context, "needs.preflight.outputs.judge_direct": judge_direct}) - ] - self.assertEqual(ran, [expected], judge_direct) + all_jobs = jobs() + cls.consolidate = all_jobs["consolidate"] + cls.direct = all_jobs["judge-direct"] + + def job_scalar(self, job_lines, key): + line = next(l for l in code_lines(job_lines) if l.startswith(f" {key}:")) + return line.split(":", 1)[1].strip() + + def test_the_key_never_reaches_consolidate(self): + self.assertFalse(any("ANTHROPIC_API_KEY" in l for l in code_lines(self.consolidate))) + self.assertIn("runs-on: ${{ fromJSON(inputs.runs_on", "\n".join(code_lines(self.consolidate))) + + def test_the_direct_job_is_hosted_and_read_only(self): + self.assertEqual(self.job_scalar(self.direct, "runs-on"), "ubuntu-latest") + self.assertEqual(block_mapping(self.direct, " permissions:", 6), {"contents": "read"}) + self.assertEqual(self.job_scalar(self.direct, "timeout-minutes"), + self.job_scalar(self.consolidate, "timeout-minutes")) + + def test_the_direct_job_runs_no_shell_agent_and_no_bare_stdin_python(self): + body = code_lines(self.direct) + self.assertFalse(any("cursor-agent" in l or "install-cursor-cli" in l for l in body)) + # A stdin script puts the cwd — the PR checkout — first on sys.path. + self.assertFalse([l for l in body if re.search(r"python3 -(\s|$)", l)]) + self.assertEqual([l.strip().split()[0] for l in body if re.search(r"python3 -I -", l)], + ["mcp_config=\"$(python3", "python3"]) + + def test_it_runs_only_when_consolidate_would_and_the_judge_is_direct(self): + direct_if = self.job_scalar(self.direct, "if") + consolidate_if = self.job_scalar(self.consolidate, "if") + self.assertEqual(direct_if, consolidate_if.replace( + " }}", " && needs.preflight.outputs.judge_direct == 'true' }}")) + self.assertEqual(self.job_scalar(self.direct, "needs"), + self.job_scalar(self.consolidate, "needs").replace(", judge-direct]", "]")) + + def test_the_rebuilt_judge_input_is_consolidates(self): + for name in SHARED_STEPS: + with self.subTest(step=name): + self.assertEqual(step(self.direct, name), step(self.consolidate, name)) + + def test_exactly_one_judge_runs(self): + context = {"steps.aggregate.outputs.ok_count": "3"} + for judge_direct, cursor_runs in (("true", False), ("false", True), ("", True)): + ctx = {**context, "needs.preflight.outputs.judge_direct": judge_direct} + self.assertEqual(bool(evaluate(step_scalar(step(self.consolidate, CURSOR_JUDGE_STEP), "if"), ctx)), + cursor_runs, judge_direct) + self.assertEqual(bool(evaluate(step_scalar(step(self.consolidate, ADOPT_STEP), "if"), ctx)), + not cursor_runs, judge_direct) # A CLI that did not install or verify clean never runs: the seed then # takes the degraded path, like a failed Cursor judge. - self.assertFalse(evaluate(step_scalar(step(self.consolidate, DIRECT_JUDGE_STEP), "if"), { - **context, "needs.preflight.outputs.judge_direct": "true", "steps.claude_verify.outcome": "failure"})) + run_if = step_scalar(step(self.direct, DIRECT_JUDGE_STEP), "if") + self.assertTrue(evaluate(run_if, {**context, "steps.claude_verify.outcome": "success"})) + self.assertFalse(evaluate(run_if, {**context, "steps.claude_verify.outcome": "failure"})) + + def test_the_review_is_adopted_only_from_a_job_that_succeeded(self): + download = step(self.consolidate, DOWNLOAD_STEP) + base = {"steps.aggregate.outputs.ok_count": "3", "needs.preflight.outputs.judge_direct": "true"} + self.assertTrue(evaluate(step_scalar(download, "if"), {**base, "needs.judge-direct.result": "success"})) + for result in ("failure", "cancelled", "skipped"): + self.assertFalse(evaluate(step_scalar(download, "if"), {**base, "needs.judge-direct.result": result})) + self.assertIn(f"name: {ARTIFACT}", "\n".join(code_lines(download))) + upload = step(self.direct, UPLOAD_STEP) + self.assertIn(f"name: {ARTIFACT}", "\n".join(code_lines(upload))) + # A 409 on the name must fail the job, so it is never absorbed. + self.assertIsNone(step_scalar(upload, "continue-on-error")) def test_the_cursor_judge_step_is_unchanged_otherwise(self): body = code_lines(step(self.consolidate, CURSOR_JUDGE_STEP)) @@ -202,12 +229,12 @@ def test_the_cursor_judge_step_is_unchanged_otherwise(self): self.assertEqual(step_scalar(step(self.consolidate, CURSOR_JUDGE_STEP), "continue-on-error"), "true") def test_the_direct_judge_matches_the_cursor_judges_cap_and_absorption(self): - cursor, direct = step(self.consolidate, CURSOR_JUDGE_STEP), step(self.consolidate, DIRECT_JUDGE_STEP) + cursor, direct = step(self.consolidate, CURSOR_JUDGE_STEP), step(self.direct, DIRECT_JUDGE_STEP) for key in ("timeout-minutes", "continue-on-error"): self.assertEqual(step_scalar(direct, key), step_scalar(cursor, key), key) def test_the_direct_judge_is_confined(self): - body = code_lines(step(self.consolidate, DIRECT_JUDGE_STEP)) + body = code_lines(step(self.direct, DIRECT_JUDGE_STEP)) start = next(i for i, l in enumerate(body) if l.strip().startswith("claude -p")) argv = [] for line in body[start:]: @@ -230,27 +257,30 @@ def test_the_direct_judge_is_confined(self): self.assertIn('--mode", "judge"', script) self.assertIn('"--out", "/tmp/judge-findings.json"', script) self.assertIn('mktemp -d -p "$RUNNER_TEMP"', script) - env = block_mapping(step(self.consolidate, DIRECT_JUDGE_STEP), " env:", 10) + env = block_mapping(step(self.direct, DIRECT_JUDGE_STEP), " env:", 10) self.assertEqual(env.get("CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC"), "'1'") # The result is model output steered by PR text: never echoed. self.assertFalse(any(re.search(r"\b(cat|head|tail)\b.*judge-claude-result", l) for l in body)) - def test_the_cli_is_installed_from_the_manifest_only_when_direct(self): - for name in ("Install Claude Code CLI", "Verify Claude Code CLI version"): - body = step(self.consolidate, name) - self.assertIn("needs.preflight.outputs.judge_direct == 'true'", step_scalar(body, "if")) - install = "\n".join(code_lines(step(self.consolidate, "Install Claude Code CLI"))) + def test_the_cli_is_installed_from_the_manifest_and_time_boxed(self): + for name, cap in (("Install Claude Code CLI", "5"), ("Verify Claude Code CLI version", "2")): + body = step(self.direct, name) + self.assertEqual(step_scalar(body, "continue-on-error"), "true", name) + self.assertEqual(step_scalar(body, "timeout-minutes"), cap, name) + install = "\n".join(code_lines(step(self.direct, "Install Claude Code CLI"))) self.assertIn('manifest="$CURSOR_REVIEW_ASSETS/package.json"', install) self.assertNotRegex(install, r"claude-code@\d") - # The Cursor CLI install is untouched and unconditional. + # The Cursor CLI is installed only for the Cursor judge. cursor = step(self.consolidate, "Install Cursor agent CLI") - self.assertIsNone(step_scalar(cursor, "if")) + self.assertEqual(step_scalar(cursor, "if"), "needs.preflight.outputs.judge_direct != 'true'") class ExfilGuardTest(unittest.TestCase): @classmethod def setUpClass(cls): - cls.consolidate = jobs()["consolidate"] + all_jobs = jobs() + cls.consolidate = all_jobs["consolidate"] + cls.direct = all_jobs["judge-direct"] def setUp(self): tmp = tempfile.TemporaryDirectory() @@ -281,10 +311,26 @@ def submit(self, body): return record def guard(self): - body = step(self.consolidate, GUARD_STEP) - self.assertTrue(evaluate(step_scalar(body, "if"), self.context)) + body = step(self.direct, GUARD_STEP) + self.assertEqual(step_scalar(body, "if"), "always()") return run_step(body, self.context, self.workdir)[0] + def hand_back(self, record=None): + """`judge-direct`'s file into consolidate's seed via `Adopt direct + judge review`. `record` replaces the uploaded file when given.""" + src = os.path.join(self.workdir, "judge-direct") + os.makedirs(src, exist_ok=True) + if record is not None: + with open(os.path.join(src, "judge-findings.json"), "w", encoding="utf-8") as f: + f.write(record) + else: + shutil.copyfile(os.path.join(self.workdir, "judge-findings.json"), + os.path.join(src, "judge-findings.json")) + self.configure("true") + done = run_step(step(self.consolidate, ADOPT_STEP), self.context, self.workdir)[0] + self.assertEqual(done.returncode, 0, done.stdout + done.stderr) + return done + def build(self): with open(os.path.join(self.workdir, "panel.json"), "w", encoding="utf-8") as f: json.dump([{"model": "kimi-k3-max", "review_type": "adversarial", "status": "ok", @@ -313,6 +359,7 @@ def test_a_review_carrying_the_key_is_replaced_by_the_seed(self): self.assertEqual(record["findings"], []) self.assertEqual(record["model"], f"{JUDGE_MODEL} (direct)") self.assertNotIn(KEY, json.dumps(record)) + self.hand_back() outputs, consolidated = self.build() self.assertEqual(outputs["degraded"], "true") self.assertEqual([f["body"] for f in consolidated["findings"]], ["panel finding"]) @@ -326,6 +373,8 @@ def test_a_clean_review_flows_through_unchanged(self): # Status and count only: the run log is public. self.assertIn("Direct judge: status=ok, 1 finding(s).", done.stdout) self.assertNotIn("a real finding", done.stdout) + self.hand_back() + self.assertEqual(self.read(), submitted) outputs, consolidated = self.build() self.assertEqual(outputs["degraded"], "false") self.assertEqual(consolidated["findings"], submitted["findings"]) @@ -335,6 +384,7 @@ def test_a_judge_that_never_submitted_takes_the_degraded_path(self): done = self.guard() self.assertEqual(done.returncode, 0, done.stdout + done.stderr) self.assertIn("Direct judge: status=error, 0 finding(s).", done.stdout) + self.hand_back() outputs, _ = self.build() self.assertEqual(outputs["degraded"], "true") @@ -345,6 +395,20 @@ def test_an_untrusted_status_is_not_echoed(self): self.assertIn("Direct judge: status=?, 0 finding(s).", done.stdout) self.assertNotIn("forged", done.stdout) + def test_a_non_object_or_missing_hand_back_leaves_the_seed(self): + for record in ("[1, 2]", "not json", '"ok"'): + with self.subTest(record=record): + done = self.hand_back(record) + self.assertIn("seed stays", done.stdout) + self.assertEqual(self.read()["status"], "error") + outputs, _ = self.build() + self.assertEqual(outputs["degraded"], "true") + os.remove(os.path.join(self.workdir, "judge-direct", "judge-findings.json")) + self.configure("true") + done = run_step(step(self.consolidate, ADOPT_STEP), self.context, self.workdir)[0] + self.assertEqual(done.returncode, 0, done.stdout + done.stderr) + self.assertEqual(self.read()["status"], "error") + if __name__ == "__main__": unittest.main() diff --git a/.github/cursor-review/tests/test_workflow_ref_guard_parity.py b/.github/cursor-review/tests/test_workflow_ref_guard_parity.py index 9cd749f9..c6c754a4 100644 --- a/.github/cursor-review/tests/test_workflow_ref_guard_parity.py +++ b/.github/cursor-review/tests/test_workflow_ref_guard_parity.py @@ -42,7 +42,7 @@ # The guard steps. `.strip()`-compared so the assertion is about the step name, # not about the indentation of whichever job happens to hold it. GUARD_STEP_NAME = "- name: Require a pinned workflows_ref" -EXPECTED_GUARDS = 13 +EXPECTED_GUARDS = 14 # The ledger's exempt resolver, addressed by its step id rather than its name so # a comment rewrite above it cannot move the anchor. diff --git a/.github/workflow-pins/tests/test_check_workflow_pins.py b/.github/workflow-pins/tests/test_check_workflow_pins.py index cb030c5c..4a9c6eaa 100644 --- a/.github/workflow-pins/tests/test_check_workflow_pins.py +++ b/.github/workflow-pins/tests/test_check_workflow_pins.py @@ -3504,7 +3504,7 @@ def test_this_repos_own_workflows_guard_every_ref_checkout(self): self.assertEqual(cwp.find_unguarded_ref_checkouts(lines), [], name) self.assertEqual( seen, - 24, + 25, "expected the 12 guarded sites BE-5546 fixed + pr-size.yml's (BE-5858) " "+ cursor-review.yml's preflight (hard guard) site picked up merging " "main + cursor-review.yml's diff-size job's check-pr-size-tool " @@ -3540,7 +3540,11 @@ def test_this_repos_own_workflows_guard_every_ref_checkout(self): "the direct-OpenAI cell loads its prompt, fence-diff.py and " "review-output-mcp.py from `workflows_ref` in its own job and " "carries its own copy of the guard. The 24th is its Anthropic twin, " - "`review-anthropic-direct` (BE-19913), which carries its own copy too.", + "`review-anthropic-direct` (BE-19913), which carries its own copy too. " + "The 25th is cursor-review.yml's opt-in `judge-direct` job (BE-19914): " + "the direct-Anthropic judge runs in its own job so ANTHROPIC_API_KEY " + "never reaches `consolidate`, loads its prompt and scripts from " + "`workflows_ref` and carries its own copy of the guard.", ) WORKFLOWS_DIR = os.path.normpath( diff --git a/.github/workflows/cursor-review.yml b/.github/workflows/cursor-review.yml index 9ef763f0..6161ddaf 100644 --- a/.github/workflows/cursor-review.yml +++ b/.github/workflows/cursor-review.yml @@ -369,12 +369,12 @@ on: structured-output tool, same timeout and same degraded panel-union fallback as the Cursor judge. The agent reads the PR checkout with Read, Grep and Glob alone — no Bash or other code-running tool — - confined to it by `--restricted`. `consolidate` then runs on - GitHub-hosted `ubuntu-latest` whatever `runs_on` says, because it - holds the key. This judge reads private-repo diffs and the workflow - cannot check the Anthropic organization's data-retention terms, so - a caller on a private repo must use a key from an organization with - zero data retention. + confined to it by `--restricted`. It runs in its own `judge-direct` + job on GitHub-hosted `ubuntu-latest`, whatever `runs_on` says, so the + key never reaches `consolidate`. This judge reads private-repo diffs + and the workflow cannot check the Anthropic organization's + data-retention terms, so a caller on a private repo must use a key + from an organization with zero data retention. type: string required: false default: '' @@ -582,11 +582,10 @@ on: '["self-hosted", "linux", "x64"]'. Every other job — the gate, the ledger, the Slack notices, and every job that mints the App token or holds `pull-requests: write` — stays on GitHub-hosted `ubuntu-latest` - regardless, so a pool never sees a write credential — and so does - `consolidate` whenever `ANTHROPIC_API_KEY` is passed, for the direct - judge (`judge_direct_model`). Review and - consolidate run models with unsandboxed shell over the PR's code, so - pick a pool that gives each job a fresh pod (fresh filesystem and + regardless, so a pool never sees a write credential, as do the + direct-API cells and the direct judge, which hold a model key. + Review and consolidate run models with unsandboxed shell over the + PR's code, so pick a pool that gives each job a fresh pod (fresh filesystem and $HOME), carries no cloud identity or service-account token, and cannot reach private networks or the metadata server — anything a prompt-injected PR runs there can reach whatever the runner can. The @@ -612,13 +611,12 @@ on: description: >- Anthropic API key for the optional direct-API Anthropic review cells and direct judge. Required only when the `anthropic_direct_model` or - `judge_direct_model` input is set; otherwise unused — but whenever it - is passed, `consolidate` runs on GitHub-hosted `ubuntu-latest` rather - than `runs_on`, since the direct judge's steps reference it and a job - is sent every secret its steps reference, skipped steps included. - Use a key scoped to its own Anthropic workspace with a spend limit, - never a personal key — the cells and judge bill per call against it - directly rather than against a pooled seat. + `judge_direct_model` input is set; ignored otherwise. Only the gate + (a presence check), the direct cells and the `judge-direct` job read + it, all on GitHub-hosted `ubuntu-latest`. Use a key scoped to its + own Anthropic workspace with a spend limit, never a personal key — + the cells and judge bill per call against it directly rather than + against a pooled seat. required: false SLACK_BOT_TOKEN: description: Slack bot token for the start/complete DM notifications. Optional — DMs are skipped if absent. @@ -786,11 +784,9 @@ jobs: - name: Resolve ANTHROPIC_API_KEY presence # Same as `Resolve OPENAI_API_KEY presence` above, for the direct - # Anthropic cells and the direct judge. Unlike that one it runs on - # every event: `consolidate` reads it to pick its runner, and the - # direct judge's steps reference the secret, so a passed key must keep - # that job GitHub-hosted even with neither direct input set. + # Anthropic cells and the direct judge. id: anthropic_key + if: inputs.anthropic_direct_model != '' || inputs.judge_direct_model != '' env: ANTHROPIC_KEY_PRESENT: ${{ secrets.ANTHROPIC_API_KEY != '' }} run: echo "present=${ANTHROPIC_KEY_PRESENT}" >> "$GITHUB_OUTPUT" @@ -4068,98 +4064,35 @@ jobs: fi echo "Direct-API reviewer cell ${REVIEW_TYPE}/${MODEL} submitted its findings." - consolidate: - # Adjudicates the panel cell artifacts into one consolidated review. - # Calls cursor-agent one more time as the judge, with prompt-judge.md — - # or, with `judge_direct_model` in effect (preflight's `judge_direct`), - # `claude -p` through the Anthropic API instead: same prompt, same - # structured-output tool, same file, same degraded fallback. + judge-direct: + # The direct-Anthropic judge (`judge_direct_model`, preflight's + # `judge_direct`): `consolidate`'s Cursor judge, through `claude -p` and + # the Anthropic API. Its own job so ANTHROPIC_API_KEY — sent to a job for + # every step that references it, skipped steps included — reaches ONLY a + # job whose one agent has Read, Grep and Glob and no shell, on + # GitHub-hosted `ubuntu-latest`, never `consolidate` (a Cursor shell agent + # over PR text, on the caller's `runs_on` pool). # - # Holds no write credential: the payload is handed to the `post-review` job - # below as an artifact, and that job does the mint + post. See the header's - # "Credential isolation" note for why the judge must not share a job with - # a write-scoped token. - name: Consolidate panel - # `review-openai-direct` and `review-anthropic-direct` are here for - # ORDERING — their artifacts must land before `Download panel findings` - # runs. Neither gates this job: when one is skipped (the default) or - # fails, `!cancelled()` below still runs, and a failed cell is counted - # from its artifact (or its absence) instead. Each one's result is read - # once, by `Aggregate panel findings`, as a consistency check on those - # artifacts (see aggregate-panel.py). + # It rebuilds the judge's input exactly as `consolidate` does — the + # `Aggregate panel findings` and `Build judge prompt` steps are + # byte-identical to consolidate's (pinned by test_workflow_judge_direct.py) + # — then hands back ONE file, /tmp/judge-findings.json, as the + # `cursor-review-judge-direct` artifact. `consolidate` adopts it over its + # own seed, so everything downstream (the degraded fallback, the + # consolidated file, the post) is the Cursor judge's path unchanged. + name: Direct judge needs: [gate, diff-size, preflight, review, review-openai-direct, review-anthropic-direct, ledger] - # `!cancelled()`, not `always()`: the judge is the most expensive call in - # the workflow, and the only job that does anything with its payload, - # `post-review`, is itself `!cancelled()`. On a cancelled run — now an - # advertised escape hatch, since applying `skip-cursor-review` mid-panel - # cancels the run — `always()` spent up to 30 minutes adjudicating partial - # cell output that nothing could ever post. Individual cell FAILURES are a - # different thing and are still judged: `cancelled()` asks whether the RUN - # was cancelled, so a red or cancelled matrix cell still reaches the judge, - # which is what `always()` was here for. - if: ${{ !cancelled() && needs.gate.outputs.should_run == 'true' && needs.gate.outputs.already_reviewed != 'true' && needs.diff-size.outputs.within_cap == 'true' && needs.review.result != 'skipped' }} - # GitHub-hosted whenever this job can hold ANTHROPIC_API_KEY, `runs_on` - # otherwise. The direct judge's steps reference the secret, and a job is - # sent every secret its steps reference — skipped steps included — so the - # switch keys on the key being PASSED (`anthropic_key_present`), not only - # on the direct judge running. `judge_direct` implies it; both are named - # so neither half can be dropped alone. With no key passed, today's - # `runs_on` routing is unchanged. - runs-on: ${{ (needs.preflight.outputs.judge_direct == 'true' || needs.gate.outputs.anthropic_key_present == 'true') && 'ubuntu-latest' || fromJSON(inputs.runs_on || '"ubuntu-latest"') }} - # ABOVE the `review` cap, never under it. The judge reads every cell's - # artifact with the same top reasoning tier one cell uses, so it is the - # strictly harder call — yet at 15m it was capped at HALF the panel it - # adjudicates. A diff slow enough to stretch the panel stretches the judge - # with it, so the failure mode was the expensive one: every cell runs green - # for ~25m, then the judge is killed at the last step and every cell's work - # is discarded with nothing posted to the PR. GitHub reports a - # timeout-minutes kill as `cancelled`, not `failed`, so it reads on the run - # page like a human stopped it — which is how this went unnoticed. Observed - # twice in one day on a consumer repo, both at 15m0Xs (~3% of runs). - # - # Deliberately longer than the `Run judge` step cap below, same idiom as - # the `ledger` job: raising the JOB cap alone would only move the cliff, - # since a job-level kill takes `Build consolidated findings file` and the - # payload upload with it and is the one failure mode that can never reach - # the degraded panel-union fallback. The step trips first and is absorbed, - # so a hung judge costs the adjudication, not the panel. The slack over - # the step (10m) covers this job's own overhead — a `fetch-depth: 0` - # checkout, the CLI install, three artifact downloads and the upload — - # which a panel cell does not all pay, so the judge's effective MODEL - # budget is the step cap, not this one. - # - # DERIVED from `cell_timeout_minutes` (step = max(30, x 2), job = step - # + 10 slack), so raising a - # cell's cap carries the judge with it instead of leaving it behind. + # `consolidate`'s condition, plus the direct judge being in effect. + if: ${{ !cancelled() && needs.gate.outputs.should_run == 'true' && needs.gate.outputs.already_reviewed != 'true' && needs.diff-size.outputs.within_cap == 'true' && needs.review.result != 'skipped' && needs.preflight.outputs.judge_direct == 'true' }} + # Holds ANTHROPIC_API_KEY: never the caller's `runs_on` pool. + runs-on: ubuntu-latest + # `consolidate`'s cap, for the same reason: the judge step below trips + # first and is absorbed, so the upload always runs. The 10m slack covers + # this job's overhead — checkout, three downloads, the CLI install + # (itself capped at 5m) and its check (2m), and the upload. timeout-minutes: ${{ fromJSON(needs.preflight.outputs.judge_job_timeout) }} - # contents: read ONLY. This job checks out PR-authored code and runs the - # judge agent over it, so it must carry nothing write-scoped — `post-review` - # below holds pull-requests: write and the bot credentials instead. permissions: contents: read - # Panel-integrity facts, for the `Panel integrity` job at the bottom of this - # file. They are the same numbers the steps below already print; a job output - # is what makes them readable from OUTSIDE this job's log, which is the whole - # point — `Aggregate panel findings` reporting "2/6 cells contributed" in a - # log nobody opens is not a signal a merge gate can act on. - outputs: - ok_count: ${{ steps.aggregate.outputs.ok_count }} - total: ${{ steps.aggregate.outputs.total }} - # 'true' when a counted cell artifact may not be its own cell's: a - # reviewer matrix did not succeed although every counted cell of it - # reads `ok`, or a record is `mismatch`. Written by aggregate-panel.py - # BEFORE the judge runs in this job; `Auto-approve decision` and `Panel - # integrity` read it from here, never from consolidated.json. - panel_inconsistent: ${{ steps.aggregate.outputs.panel_inconsistent }} - degraded: ${{ steps.consolidated.outputs.degraded }} - # Untrusted: this is whatever string the judge agent's own tool wrote as - # `status` in /tmp/judge-findings.json, in a job running `cursor-agent - # --trust` over PR-authored text. It is neither validated against an enum - # nor escaped on the way out. Anything that consumes it must flatten AND - # percent-escape it before it reaches a ::workflow command:: line — - # stripping newlines alone is not enough, because the runner unescapes - # `%0A` back into one. `Panel integrity` does both. - judge_status: ${{ steps.consolidated.outputs.judge_status }} steps: - name: Checkout PR repo uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 @@ -4242,18 +4175,17 @@ jobs: name: pr-diff path: /tmp - - name: Install Cursor agent CLI - run: bash "$CURSOR_REVIEW_ASSETS/install-cursor-cli.sh" - - name: Install Claude Code CLI id: claude_install - if: needs.preflight.outputs.judge_direct == 'true' # `review-anthropic-direct`'s install, verbatim: the pin is read out of # .github/cursor-review/package.json, its one home. Absorbed, unlike # there: a judge that cannot start must cost the adjudication only, so # `Run judge (direct)` is skipped and the seed takes the degraded - # panel-union path, as a failed Cursor judge does. + # panel-union path, as a failed Cursor judge does. Time-boxed for the + # same reason: a hung registry fetch must trip this step, not the job + # cap, whose kill never reaches the upload below. continue-on-error: true + timeout-minutes: 5 run: | manifest="$CURSOR_REVIEW_ASSETS/package.json" if [ ! -f "$manifest" ]; then @@ -4279,11 +4211,12 @@ jobs: - name: Verify Claude Code CLI version id: claude_verify - if: needs.preflight.outputs.judge_direct == 'true' && steps.claude_install.outcome == 'success' + if: steps.claude_install.outcome == 'success' # `review-anthropic-direct`'s check, verbatim, and absorbed for the # same reason as the install: a CLI that is off-pin, or no longer # advertises a flag the confinement relies on, never runs. continue-on-error: true + timeout-minutes: 2 run: | installed="$(claude --version 2>/dev/null | head -n 1 | tr -d '\r' | awk '{print $1}')" if [ "$installed" != "$CLAUDE_CLI_VERSION" ]; then @@ -4315,62 +4248,18 @@ jobs: pattern: findings-* - name: Configure structured final review + # The `--init` seed, as in `consolidate`: a judge that never submits + # leaves status=error, and `consolidate` takes the degraded path. env: - # The seed names the judge that will run: ` (direct)` for the - # direct judge, so a judge that never submits is not attributed to - # the Cursor model. - JUDGE_DIRECT: ${{ needs.preflight.outputs.judge_direct }} JUDGE_DIRECT_MODEL: ${{ inputs.judge_direct_model }} run: | - judge_label="$JUDGE_MODEL" - if [ "$JUDGE_DIRECT" = "true" ]; then - judge_label="$JUDGE_DIRECT_MODEL (direct)" - fi - python3 "$CURSOR_REVIEW_ASSETS/review-output-mcp.py" \ + python3 -I "$CURSOR_REVIEW_ASSETS/review-output-mcp.py" \ --mode judge \ --out /tmp/judge-findings.json \ - --model "$judge_label" \ + --model "$JUDGE_DIRECT_MODEL (direct)" \ --review-type judge \ --init - rm -rf -- .cursor - mkdir -m 700 .cursor - python3 - <<'PY' - import json, os, sys, tempfile - - if os.path.islink(".cursor"): - raise SystemExit("Refusing to write Cursor config through a symlinked .cursor directory") - - config = { - "mcpServers": { - "cursor-review-output": { - "type": "stdio", - "command": sys.executable, - "args": [ - os.path.join(os.environ["CURSOR_REVIEW_ASSETS"], "review-output-mcp.py"), - "--mode", "judge", - "--out", "/tmp/judge-findings.json", - "--model", os.environ["JUDGE_MODEL"], - "--review-type", "judge", - ], - } - } - } - cli_config = { - "permissions": { - "allow": [ - "Mcp(cursor-review-output:cursor_review_submit_final)", - ], - "deny": [], - } - } - for filename, value in (("mcp.json", config), ("cli.json", cli_config)): - fd, temporary = tempfile.mkstemp(dir=".cursor", prefix=f".{filename}-") - with os.fdopen(fd, "w", encoding="utf-8") as output: - json.dump(value, output) - os.replace(temporary, os.path.join(".cursor", filename)) - PY - - name: Aggregate panel findings id: aggregate env: @@ -4460,52 +4349,8 @@ jobs: echo "Judge prompt size: $(wc -c < /tmp/judge-prompt.txt) bytes" - - name: Run judge - if: steps.aggregate.outputs.ok_count != '0' && needs.preflight.outputs.judge_direct != 'true' - # Time-boxed BELOW the job cap and absorbed, so a hung judge loses only - # the adjudication: the steps below then read the `--init` seed left by - # `Configure structured final review` (status=error), take the degraded - # panel-union path and still post what the cells found. At least the - # `review` job cap at the default (30) and always at least twice the - # cell step cap, so the judge is never given less model time than one of - # the cells it reads. `continue-on-error` is what makes the - # timeout absorbable — without it the step failure fails the job and - # lands back on the all-cells-discarded outcome this is fixing. It - # hides no real signal: the cursor-agent exit code is already captured - # in-script, and a judge that did not submit is reported to the PR as - # `judge_status` by `Post review`. - continue-on-error: true - # DERIVED: twice `cell_timeout_minutes`, never below 30. The judge - # adjudicates EVERY cell's findings at the same reasoning tier, so a - # budget equal to one cell's is the floor, not the target — and its - # workload scales with the panel, so a tightened cell cap must not - # shrink it below today's 30. - timeout-minutes: ${{ fromJSON(needs.preflight.outputs.judge_step_timeout) }} - env: - CURSOR_API_KEY: ${{ secrets.CURSOR_API_KEY }} - run: | - cursor-agent \ - --print \ - --trust \ - --approve-mcps \ - --model "$JUDGE_MODEL" \ - < /tmp/judge-prompt.txt \ - > /tmp/judge-raw.txt 2>/tmp/judge-stderr.txt \ - && JUDGE_EXIT=0 || JUDGE_EXIT=$? - echo "cursor-agent (judge) exit code: $JUDGE_EXIT" - - echo "=== Judge raw output (first 1000 chars) ===" - head -c 1000 /tmp/judge-raw.txt - echo "" - echo "=== Judge stderr ===" - cat /tmp/judge-stderr.txt - - echo "=== Structured final review ===" - cat /tmp/judge-findings.json - echo "" - - name: Run judge (direct) - # `Run judge` above, through the Anthropic API instead of Cursor: the + # `consolidate`'s `Run judge`, through the Anthropic API instead of Cursor: the # same /tmp/judge-prompt.txt, the same `review-output-mcp.py --mode # judge` writing the same /tmp/judge-findings.json, the same step cap # and the same absorption — a judge that fails, times out or never @@ -4520,7 +4365,7 @@ jobs: # denied independently of that flag, since /proc/self/environ is the # key. The result JSON is never echoed: the run log is public and the # result is model output steered by PR text. - if: steps.aggregate.outputs.ok_count != '0' && needs.preflight.outputs.judge_direct == 'true' && steps.claude_verify.outcome == 'success' + if: steps.aggregate.outputs.ok_count != '0' && steps.claude_verify.outcome == 'success' continue-on-error: true timeout-minutes: ${{ fromJSON(needs.preflight.outputs.judge_step_timeout) }} env: @@ -4584,7 +4429,7 @@ jobs: # panel union instead. Literal match only. `always()`, so a judge # killed by its step cap AFTER submitting is still checked. Then the # status and finding count — never the review itself (see above). - if: always() && needs.preflight.outputs.judge_direct == 'true' + if: always() env: JUDGE_DIRECT_MODEL: ${{ inputs.judge_direct_model }} ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }} @@ -4594,14 +4439,17 @@ jobs: if [ -n "${ANTHROPIC_API_KEY:-}" ] && [ -f "$out" ] \ && LC_ALL=C grep -qF -- "$ANTHROPIC_API_KEY" "$out"; then echo "::error::the direct judge's final review contains ANTHROPIC_API_KEY — refusing to publish it (possible prompt-injection exfil); the panel union is posted instead." - python3 "$CURSOR_REVIEW_ASSETS/review-output-mcp.py" \ + python3 -I "$CURSOR_REVIEW_ASSETS/review-output-mcp.py" \ --mode judge \ --out "$out" \ --model "$JUDGE_DIRECT_MODEL (direct)" \ --review-type judge \ --init fi - python3 - <<'PY' + # `-I`, as everywhere in this job: the cwd is the PR checkout, and a + # stdin script puts '' first on sys.path, so a PR-planted json.py + # would run beside the key. + python3 -I - <<'PY' import json try: with open("/tmp/judge-findings.json", encoding="utf-8") as f: @@ -4619,6 +4467,439 @@ jobs: print(f"Direct judge: status={status}, {count} finding(s).") PY + - name: Upload direct judge review + # Always, so `consolidate` sees the seed rather than nothing on a + # judge that did not submit. NOT absorbed: a failed upload (a 409 + # because something else took the name first included) fails this + # job, and `consolidate` adopts only from a job that succeeded. + if: always() + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 + with: + name: cursor-review-judge-direct + path: /tmp/judge-findings.json + if-no-files-found: error + retention-days: 7 + + consolidate: + # Adjudicates the panel cell artifacts into one consolidated review. + # Calls cursor-agent one more time as the judge, with prompt-judge.md — + # or, with `judge_direct_model` in effect (preflight's `judge_direct`), + # adopts the review the `judge-direct` job above wrote instead: same + # prompt, same structured-output tool, same file, same degraded fallback. + # That judge runs in its OWN job so ANTHROPIC_API_KEY never shares one + # with this job's Cursor judge (an unsandboxed shell agent over PR text) + # or its `runs_on` pool. + # + # Holds no write credential: the payload is handed to the `post-review` job + # below as an artifact, and that job does the mint + post. See the header's + # "Credential isolation" note for why the judge must not share a job with + # a write-scoped token. + name: Consolidate panel + # `review-openai-direct` and `review-anthropic-direct` are here for + # ORDERING — their artifacts must land before `Download panel findings` + # runs (and `judge-direct`'s before `Download direct judge review`). None + # gates this job: when one is skipped (the default) or + # fails, `!cancelled()` below still runs, and a failed cell is counted + # from its artifact (or its absence) instead. Each one's result is read + # once, by `Aggregate panel findings`, as a consistency check on those + # artifacts (see aggregate-panel.py). + needs: [gate, diff-size, preflight, review, review-openai-direct, review-anthropic-direct, ledger, judge-direct] + # `!cancelled()`, not `always()`: the judge is the most expensive call in + # the workflow, and the only job that does anything with its payload, + # `post-review`, is itself `!cancelled()`. On a cancelled run — now an + # advertised escape hatch, since applying `skip-cursor-review` mid-panel + # cancels the run — `always()` spent up to 30 minutes adjudicating partial + # cell output that nothing could ever post. Individual cell FAILURES are a + # different thing and are still judged: `cancelled()` asks whether the RUN + # was cancelled, so a red or cancelled matrix cell still reaches the judge, + # which is what `always()` was here for. + if: ${{ !cancelled() && needs.gate.outputs.should_run == 'true' && needs.gate.outputs.already_reviewed != 'true' && needs.diff-size.outputs.within_cap == 'true' && needs.review.result != 'skipped' }} + runs-on: ${{ fromJSON(inputs.runs_on || '"ubuntu-latest"') }} + # ABOVE the `review` cap, never under it. The judge reads every cell's + # artifact with the same top reasoning tier one cell uses, so it is the + # strictly harder call — yet at 15m it was capped at HALF the panel it + # adjudicates. A diff slow enough to stretch the panel stretches the judge + # with it, so the failure mode was the expensive one: every cell runs green + # for ~25m, then the judge is killed at the last step and every cell's work + # is discarded with nothing posted to the PR. GitHub reports a + # timeout-minutes kill as `cancelled`, not `failed`, so it reads on the run + # page like a human stopped it — which is how this went unnoticed. Observed + # twice in one day on a consumer repo, both at 15m0Xs (~3% of runs). + # + # Deliberately longer than the `Run judge` step cap below, same idiom as + # the `ledger` job: raising the JOB cap alone would only move the cliff, + # since a job-level kill takes `Build consolidated findings file` and the + # payload upload with it and is the one failure mode that can never reach + # the degraded panel-union fallback. The step trips first and is absorbed, + # so a hung judge costs the adjudication, not the panel. The slack over + # the step (10m) covers this job's own overhead — a `fetch-depth: 0` + # checkout, the CLI install, three artifact downloads and the upload — + # which a panel cell does not all pay, so the judge's effective MODEL + # budget is the step cap, not this one. + # + # DERIVED from `cell_timeout_minutes` (step = max(30, x 2), job = step + # + 10 slack), so raising a + # cell's cap carries the judge with it instead of leaving it behind. + timeout-minutes: ${{ fromJSON(needs.preflight.outputs.judge_job_timeout) }} + # contents: read ONLY. This job checks out PR-authored code and runs the + # judge agent over it, so it must carry nothing write-scoped — `post-review` + # below holds pull-requests: write and the bot credentials instead. + permissions: + contents: read + # Panel-integrity facts, for the `Panel integrity` job at the bottom of this + # file. They are the same numbers the steps below already print; a job output + # is what makes them readable from OUTSIDE this job's log, which is the whole + # point — `Aggregate panel findings` reporting "2/6 cells contributed" in a + # log nobody opens is not a signal a merge gate can act on. + outputs: + ok_count: ${{ steps.aggregate.outputs.ok_count }} + total: ${{ steps.aggregate.outputs.total }} + # 'true' when a counted cell artifact may not be its own cell's: a + # reviewer matrix did not succeed although every counted cell of it + # reads `ok`, or a record is `mismatch`. Written by aggregate-panel.py + # BEFORE the judge runs in this job; `Auto-approve decision` and `Panel + # integrity` read it from here, never from consolidated.json. + panel_inconsistent: ${{ steps.aggregate.outputs.panel_inconsistent }} + degraded: ${{ steps.consolidated.outputs.degraded }} + # Untrusted: this is whatever string the judge agent's own tool wrote as + # `status` in /tmp/judge-findings.json, in a job running `cursor-agent + # --trust` over PR-authored text. It is neither validated against an enum + # nor escaped on the way out. Anything that consumes it must flatten AND + # percent-escape it before it reaches a ::workflow command:: line — + # stripping newlines alone is not enough, because the runner unescapes + # `%0A` back into one. `Panel integrity` does both. + judge_status: ${{ steps.consolidated.outputs.judge_status }} + steps: + - name: Checkout PR repo + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + fetch-depth: 0 + persist-credentials: false + + - name: Require a pinned workflows_ref + # workflows_ref has no default on purpose. GitHub does NOT enforce + # `required: true` for workflow_call inputs, so an omitted input arrives + # as '' and actions/checkout would silently fall back to this repo's + # default branch — running MUTABLE scripts under a pin that claims + # otherwise. Fail fast instead. (BE-5546) + env: + WORKFLOWS_REF: ${{ inputs.workflows_ref }} + WORKFLOW_SHA: ${{ job.workflow_sha }} + run: | + # actions/checkout reads `ref` through core.getInput, which TRIMS, so a + # whitespace-only value is an empty ref to IT while sailing past a bare + # -z test here. Compare the stripped form, and echo only that: dropping + # newlines also stops a multi-line value from smuggling a ::workflow + # command:: into the log, and from satisfying the line-oriented grep + # below on one 40-hex line among many. + REF="$(printf '%s' "$WORKFLOWS_REF" | tr -d '[:space:]')" + if [ -z "$REF" ]; then + echo "::error::workflows_ref is required; pin it to the same commit SHA as the uses: line (see header example)" + exit 1 + fi + if ! printf '%s' "$REF" | grep -Eq '^[0-9a-f]{40}$'; then + echo "::warning::workflows_ref '$REF' is not a full 40-hex commit SHA — branch and tag refs are mutable and can skew between jobs mid-run" + fi + # Equality with the commit the caller's `uses:` line resolved to — the + # same guard public-repo-hygiene.yml runs, and the check that turns + # "workflows_ref must be the same SHA as uses:" from a header comment + # into a fact. `job.workflow_sha` is computed by the RUNNER from the + # `uses:` pin, so a caller cannot set it. Without this, a caller can + # pin `uses:` to one commit and load this workflow's prompts, scripts + # and model list from another. (`job.workflow_sha`, NOT + # `github.job_workflow_sha` — that one is an OIDC token claim, absent + # from the `github` context, and expands to ''.) Both sides are + # lower-cased: hex is case-insensitive, and unlike hygiene's the + # 40-hex test above only WARNS, so an upper-case ref naming the RIGHT + # commit reaches here and must not fail. $REF itself is left alone so + # the messages quote what the caller wrote. (BE-15927) + REF_LC="$(printf '%s' "$REF" | tr 'A-Z' 'a-z')" + SHA="$(printf '%s' "$WORKFLOW_SHA" | tr -d '[:space:]' | tr 'A-Z' 'a-z')" + if [ -z "$SHA" ]; then + # Cannot check, NOT a mismatch. `job.workflow_sha` is the `job`-context + # accessor added in Actions runner v2.334.0, so only an older runner + # gets here. GitHub stops queueing jobs to a self-hosted runner more + # than 30 days behind the latest release, so on any pool — hosted + # `ubuntu-latest`, or the caller's `runs_on` pool the read-only + # model jobs may use — that is a transient window. Warn and continue: a + # property nothing could EVALUATE is not evidence of a violation, and + # failing would take the whole review down on a caller that changed + # nothing. (public-repo-hygiene.yml fails closed in this branch on + # purpose — it is the checker whose own integrity rests on the + # equality. Here the ledger's warn-and-degrade posture is the fit.) + echo "::warning::job.workflow_sha is empty (Actions runner older than v2.334.0?), so workflows_ref could not be cross-checked against the uses: pin. The empty and 40-hex tests above still apply." + elif [ "$REF_LC" != "$SHA" ]; then + echo "::error::workflows_ref ($REF) is not the commit this workflow was called at ($SHA). The uses: pin and workflows_ref MUST name the same commit — otherwise the workflow is pinned but its prompts and scripts are loaded from a different version. Set both to $SHA. (The bump-callers fleet rewrites both together; Dependabot rewrites only uses:, so a caller that lets Dependabot bump this reusable needs a dependabot.yml ignore entry for Comfy-Org/github-workflows.)" + exit 1 + fi + + - name: Load cursor-review assets + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + repository: Comfy-Org/github-workflows + ref: ${{ inputs.workflows_ref }} + path: _cursor_review_assets + persist-credentials: false + + - name: Download reviewed diff + # Same generated-file-excluded diff (plus the incremental "hunks new + # since the last reviewed round" diff) the panel reviewed, built once + # in diff-size — so the judge adjudicates against exactly what the + # panel saw, and this job needs no PR checkout. + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 + with: + name: pr-diff + path: /tmp + + - name: Install Cursor agent CLI + # Not needed when the judge runs direct, and a transient Cursor + # download failure here would fail this job before the degraded + # panel-union path, discarding a completed panel. + if: needs.preflight.outputs.judge_direct != 'true' + run: bash "$CURSOR_REVIEW_ASSETS/install-cursor-cli.sh" + + - name: Download prior-review ledger + id: ledger_download + continue-on-error: true + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 + with: + name: cursor-review-ledger + path: /tmp/ledger + + - name: Download panel findings + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 + with: + path: /tmp/panel + pattern: findings-* + + - name: Configure structured final review + env: + # The seed names the judge that will run: ` (direct)` for the + # direct judge, so a judge that never submits is not attributed to + # the Cursor model. + JUDGE_DIRECT: ${{ needs.preflight.outputs.judge_direct }} + JUDGE_DIRECT_MODEL: ${{ inputs.judge_direct_model }} + run: | + judge_label="$JUDGE_MODEL" + if [ "$JUDGE_DIRECT" = "true" ]; then + judge_label="$JUDGE_DIRECT_MODEL (direct)" + fi + python3 "$CURSOR_REVIEW_ASSETS/review-output-mcp.py" \ + --mode judge \ + --out /tmp/judge-findings.json \ + --model "$judge_label" \ + --review-type judge \ + --init + + rm -rf -- .cursor + mkdir -m 700 .cursor + python3 - <<'PY' + import json, os, sys, tempfile + + if os.path.islink(".cursor"): + raise SystemExit("Refusing to write Cursor config through a symlinked .cursor directory") + + config = { + "mcpServers": { + "cursor-review-output": { + "type": "stdio", + "command": sys.executable, + "args": [ + os.path.join(os.environ["CURSOR_REVIEW_ASSETS"], "review-output-mcp.py"), + "--mode", "judge", + "--out", "/tmp/judge-findings.json", + "--model", os.environ["JUDGE_MODEL"], + "--review-type", "judge", + ], + } + } + } + cli_config = { + "permissions": { + "allow": [ + "Mcp(cursor-review-output:cursor_review_submit_final)", + ], + "deny": [], + } + } + for filename, value in (("mcp.json", config), ("cli.json", cli_config)): + fd, temporary = tempfile.mkstemp(dir=".cursor", prefix=f".{filename}-") + with os.fdopen(fd, "w", encoding="utf-8") as output: + json.dump(value, output) + os.replace(temporary, os.path.join(".cursor", filename)) + PY + + - name: Aggregate panel findings + id: aggregate + env: + PANEL_MODELS: ${{ needs.preflight.outputs.models }} + # The direct-API cells COUNT only when they replace the Cursor OpenAI + # lane; side by side they stay advisory. See aggregate-panel.py. + DIRECT_MODEL: ${{ needs.preflight.outputs.openai_direct == 'true' && inputs.openai_direct_model || '' }} + DIRECT_COUNTS: ${{ needs.preflight.outputs.openai_direct_counts }} + # The matrices' RESULTS, not their artifacts: a cell can upload under + # another cell's name, but it cannot turn the honest leg's 409 green. + # `Fail the leg when the cell did not submit` reds every non-ok cell, + # so a red matrix whose artifacts all read ok is a forgery or a + # post-review failure — either way, panel_inconsistent=true. + REVIEW_RESULT: ${{ needs.review.result }} + DIRECT_RESULT: ${{ needs.review-openai-direct.result }} + # The same three for the direct Anthropic cells, counted per lab. + ANTHROPIC_DIRECT_MODEL: ${{ needs.preflight.outputs.anthropic_direct == 'true' && inputs.anthropic_direct_model || '' }} + ANTHROPIC_DIRECT_COUNTS: ${{ needs.preflight.outputs.anthropic_direct_counts }} + ANTHROPIC_DIRECT_RESULT: ${{ needs.review-anthropic-direct.result }} + run: | + # Each artifact directory contains one findings.json. Combine into + # a single panel.json array preserving cell metadata, and emit the + # ok_count/total/panel_inconsistent `Panel integrity` and auto-approve + # gate on. + python3 "$CURSOR_REVIEW_ASSETS/aggregate-panel.py" \ + --panel-dir /tmp/panel \ + --models "$PANEL_MODELS" \ + --direct-model "$DIRECT_MODEL" \ + --direct-counts "${DIRECT_COUNTS:-false}" \ + --review-result "$REVIEW_RESULT" \ + --direct-result "${DIRECT_RESULT:-skipped}" \ + --anthropic-direct-model "$ANTHROPIC_DIRECT_MODEL" \ + --anthropic-direct-counts "${ANTHROPIC_DIRECT_COUNTS:-false}" \ + --anthropic-direct-result "${ANTHROPIC_DIRECT_RESULT:-skipped}" \ + --out /tmp/panel.json \ + --github-output "$GITHUB_OUTPUT" + + - name: Build judge prompt + if: steps.aggregate.outputs.ok_count != '0' + env: + LEDGER_ROUNDS: ${{ needs.ledger.outputs.rounds }} + run: | + # The judge is the repeat-policy enforcement point, so it gets the + # judge-flavored ledger block (repeat_of requirement + the per-review + # repeat cap). Byte-identical to the old prompt when there's no ledger. + python3 "$CURSOR_REVIEW_ASSETS/build-ledger.py" splice \ + --prompt "$CURSOR_REVIEW_ASSETS/prompt-judge.md" \ + --marker '=== BEGIN PANEL FINDINGS ===' \ + --insert /tmp/ledger/ledger-judge.md \ + --out /tmp/judge-prompt-head.txt + + # Nonce'd fences as in the panel (BE-7645) — the judge reads the same + # attacker-authored diff, so it gets the same delimiting. Its nonce is + # minted HERE and is its OWN: the panel jobs never see it. That is + # what stops a laundering path — panel.json is model output produced + # with the panel's nonce in context, and it is spliced ABOVE the diff + # here, so a shared value could be echoed by a steered panel model into + # a region the judge has been told is authoritative and used to forge a + # close/open fence pair around a fake diff. + # + # panel.json is fenced with the same nonce for the same reason: it is + # model output derived from the untrusted diff, so "only a marker + # carrying this nonce ends a region" has to hold for it too, not just + # for the diff. Today only json.dump's newline escaping keeps a forged + # marker off a line start — that is not a control to rely on. + judge_nonce="$(python3 "$CURSOR_REVIEW_ASSETS/fence-diff.py" mint)" + + python3 "$CURSOR_REVIEW_ASSETS/fence-diff.py" strip-marker \ + --file /tmp/judge-prompt-head.txt \ + --marker '=== BEGIN PANEL FINDINGS ===' + + { + cat /tmp/judge-prompt-head.txt + python3 "$CURSOR_REVIEW_ASSETS/fence-diff.py" emit \ + --body /tmp/panel.json --label "PANEL FINDINGS" --nonce "$judge_nonce" + echo "" + python3 "$CURSOR_REVIEW_ASSETS/fence-diff.py" emit \ + --body /tmp/pr-diff.patch --label DIFF --nonce "$judge_nonce" + if [ -s /tmp/pr-diff-new.patch ]; then + echo "" + python3 "$CURSOR_REVIEW_ASSETS/fence-diff.py" emit \ + --body /tmp/pr-diff-new.patch \ + --label "HUNKS NEW SINCE ROUND ${LEDGER_ROUNDS:-?}" \ + --nonce "$judge_nonce" + fi + } > /tmp/judge-prompt.txt + + echo "Judge prompt size: $(wc -c < /tmp/judge-prompt.txt) bytes" + + - name: Run judge + if: steps.aggregate.outputs.ok_count != '0' && needs.preflight.outputs.judge_direct != 'true' + # Time-boxed BELOW the job cap and absorbed, so a hung judge loses only + # the adjudication: the steps below then read the `--init` seed left by + # `Configure structured final review` (status=error), take the degraded + # panel-union path and still post what the cells found. At least the + # `review` job cap at the default (30) and always at least twice the + # cell step cap, so the judge is never given less model time than one of + # the cells it reads. `continue-on-error` is what makes the + # timeout absorbable — without it the step failure fails the job and + # lands back on the all-cells-discarded outcome this is fixing. It + # hides no real signal: the cursor-agent exit code is already captured + # in-script, and a judge that did not submit is reported to the PR as + # `judge_status` by `Post review`. + continue-on-error: true + # DERIVED: twice `cell_timeout_minutes`, never below 30. The judge + # adjudicates EVERY cell's findings at the same reasoning tier, so a + # budget equal to one cell's is the floor, not the target — and its + # workload scales with the panel, so a tightened cell cap must not + # shrink it below today's 30. + timeout-minutes: ${{ fromJSON(needs.preflight.outputs.judge_step_timeout) }} + env: + CURSOR_API_KEY: ${{ secrets.CURSOR_API_KEY }} + run: | + cursor-agent \ + --print \ + --trust \ + --approve-mcps \ + --model "$JUDGE_MODEL" \ + < /tmp/judge-prompt.txt \ + > /tmp/judge-raw.txt 2>/tmp/judge-stderr.txt \ + && JUDGE_EXIT=0 || JUDGE_EXIT=$? + echo "cursor-agent (judge) exit code: $JUDGE_EXIT" + + echo "=== Judge raw output (first 1000 chars) ===" + head -c 1000 /tmp/judge-raw.txt + echo "" + echo "=== Judge stderr ===" + cat /tmp/judge-stderr.txt + + echo "=== Structured final review ===" + cat /tmp/judge-findings.json + echo "" + + - name: Download direct judge review + # Only from a `judge-direct` job that SUCCEEDED. A panel cell runs an + # agent over PR code before that job starts and could upload an + # artifact under this name first; the job's own upload then 409s, the + # job fails, and nothing is adopted — the seed below stays and the + # degraded panel-union path posts instead. + if: steps.aggregate.outputs.ok_count != '0' && needs.preflight.outputs.judge_direct == 'true' && needs.judge-direct.result == 'success' + continue-on-error: true + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 + with: + name: cursor-review-judge-direct + path: /tmp/judge-direct + + - name: Adopt direct judge review + # Over the `--init` seed `Configure structured final review` wrote, + # and only when it is a JSON object: anything else leaves the seed, + # so `Build consolidated findings file` takes the degraded path + # exactly as for a Cursor judge that never submitted. + if: steps.aggregate.outputs.ok_count != '0' && needs.preflight.outputs.judge_direct == 'true' + run: | + python3 -I - <<'PY' + import json, os, shutil + + src = "/tmp/judge-direct/judge-findings.json" + try: + if os.path.islink(src): + raise ValueError("symlink") + with open(src, encoding="utf-8") as f: + record = json.load(f) + except (OSError, ValueError): + record = None + if isinstance(record, dict): + shutil.copyfile(src, "/tmp/judge-findings.json") + print("Adopted the direct judge's review.") + else: + print("No usable direct judge review — the seed stays (degraded panel-union path).") + PY + - name: Build consolidated findings file id: consolidated env: diff --git a/docs/callers/cursor-review.md b/docs/callers/cursor-review.md index d91ea21d..86173dfa 100644 --- a/docs/callers/cursor-review.md +++ b/docs/callers/cursor-review.md @@ -53,7 +53,7 @@ Prompts and scripts live in [`.github/cursor-review/`](../../.github/cursor-revi |---|---| | `secrets.CURSOR_API_KEY` | **Required.** Org-level in Comfy-Org. | | `secrets.OPENAI_API_KEY` | Optional. Only read when `openai_direct_model` is set; use a project-scoped service-account key with a spend limit. | -| `secrets.ANTHROPIC_API_KEY` | Optional. Only used when `anthropic_direct_model` or `judge_direct_model` is set (and, when passed, `consolidate` runs on GitHub-hosted `ubuntu-latest` whatever `runs_on` says); use a key scoped to its own Anthropic workspace with a spend limit, never a personal key. | +| `secrets.ANTHROPIC_API_KEY` | Optional. Only used when `anthropic_direct_model` or `judge_direct_model` is set; use a key scoped to its own Anthropic workspace with a spend limit, never a personal key. | | `secrets.SLACK_BOT_TOKEN` | Optional. Without it the review still posts; only the DMs are skipped. | | `vars.REVIEW_BOT_APP_ID` + `secrets.BOT_APP_PRIVATE_KEY` | Optional. Posts the review as your App instead of `github-actions[bot]`. | | A review label | Default `cursor-review`. Create it in your repo. | @@ -119,7 +119,7 @@ pull-requests: write # posting the consolidated review (and, at the round cap, | `anthropic_direct_model` | `''` | Opt-in. The Anthropic twin of `openai_direct_model`: an Anthropic model id (e.g. `claude-opus-5-5`) reviewed directly through the Anthropic API (pinned Claude Code CLI) as two cells — **adversarial** and **edge-case** — always on GitHub-hosted `ubuntu-latest`, by default **replacing** the Cursor `claude-*` cells. Requires `secrets.ANTHROPIC_API_KEY` and a plain id (`[A-Za-z0-9._-]`) other than `openai_direct_model`'s; without any of those, preflight warns, the cells stay off and the panel is unchanged. Unlike the OpenAI cells, the agent also reads a **read-only PR checkout** with Read, Grep and Glob only — no Bash or code-running tool — and `--restricted` confines those file tools to the checkout (`/proc` and `/sys` are also denied by explicit deny rules). **Private repos:** this panel reviews private-repo diffs and the workflow cannot check the Anthropic organization's data-retention terms, so use a key from an organization with zero data retention. Each cell also uploads a `usage-direct--` artifact (kept 30 days) with the API's token counts for that cell, so cost can be measured without an Admin API key. | | `anthropic_direct_effort` | `xhigh` | Reasoning effort for the direct Anthropic cells: `low`, `medium`, `high`, `xhigh` or `max`. Any other value warns and falls back to `xhigh`. | | `anthropic_direct_replaces_cursor_anthropic` | `true` | Same semantics as `openai_direct_replaces_cursor_openai`, for the Cursor `claude-*` cells: `true` drops them and the direct Anthropic cells **gate** in their place; `false` keeps both, with the direct cells **advisory**. Each lab counts by its own flag, so OpenAI and Anthropic may be set differently. The judge (`judge_model`) is never touched. | -| `judge_direct_model` | `''` | Opt-in. An Anthropic model id (e.g. `claude-opus-5-5`) that runs the **judge** directly through the Anthropic API (the same pinned Claude Code CLI and `secrets.ANTHROPIC_API_KEY` as `anthropic_direct_model`) instead of `judge_model` through Cursor. Requires the secret and a plain id (`[A-Za-z0-9._-]`); without either, preflight warns and the Cursor judge runs. Same prompt, output tool, timeout and degraded panel-union fallback as the Cursor judge; the agent reads the PR checkout with Read, Grep and Glob only — no Bash or code-running tool — confined to it by `--restricted` (`/proc` and `/sys` are also denied by explicit deny rules). `consolidate` runs on GitHub-hosted `ubuntu-latest` regardless of `runs_on` whenever the secret is passed. **Private repos:** this judge reads private-repo diffs and the workflow cannot check the Anthropic organization's data-retention terms, so use a key from an organization with zero data retention. | +| `judge_direct_model` | `''` | Opt-in. An Anthropic model id (e.g. `claude-opus-5-5`) that runs the **judge** directly through the Anthropic API (the same pinned Claude Code CLI and `secrets.ANTHROPIC_API_KEY` as `anthropic_direct_model`) instead of `judge_model` through Cursor. Requires the secret and a plain id (`[A-Za-z0-9._-]`); without either, preflight warns and the Cursor judge runs. Same prompt, output tool, timeout and degraded panel-union fallback as the Cursor judge; the agent reads the PR checkout with Read, Grep and Glob only — no Bash or code-running tool — confined to it by `--restricted` (`/proc` and `/sys` are also denied by explicit deny rules). It runs in its own `judge-direct` job on GitHub-hosted `ubuntu-latest` regardless of `runs_on`, so the key never reaches `consolidate`. **Private repos:** this judge reads private-repo diffs and the workflow cannot check the Anthropic organization's data-retention terms, so use a key from an organization with zero data retention. | | `judge_direct_effort` | `xhigh` | Reasoning effort for the direct judge: `low`, `medium`, `high`, `xhigh` or `max`. Any other value warns and falls back to `xhigh`. | | `skip_bot_branch_prefixes` | `ci/bump- chore/refresh- auto/refresh-` | Skip the panel for Bot-authored PRs on these branch prefixes (machine pin bumps / refreshes). `''` to review every bot PR. | | `cell_timeout_minutes` | `15` | Wall-clock cap on ONE reviewer cell's agent step (`Run cursor review`, or `Run direct-API review` for a direct cell). A cell that runs out of clock exits 130 and uploads `status: error` with no findings, which fails its leg and so fails `Panel integrity` — so this is a guillotine, not a safety net. Raise it when cells are dying at the cap rather than hanging: compare how long the cells that DID submit took, and if a meaningful share finish within a few minutes of the cap you are clipping a tail, not catching hangs. It is also the budget a cell's retries share (`resource_exhausted` is retried in-step), so it bounds attempts plus backoff. Three further caps derive from this one and cannot be set separately — each cell's job cap is this + 15 (checkout, CLI install, artifacts), the judge's `Run judge` step cap is this × 2 but never below 30 (it reads every cell's findings at the same tier, and its workload scales with the panel, not this cap), and the judge's job cap is its step + 10. Bounded 10–45 (the floor leaves a late in-step retry room for a full review); preflight fails loud outside that, before any cell spends. | @@ -140,7 +140,7 @@ pull-requests: write # posting the consolidated review (and, at the round cap, | `defer_approval` | `false` | Only for a caller that runs [cursor-approve](cursor-approve.md) after this workflow: a round that would approve still reports `approve_gate` = `pass`, but posts **no** approval and resolves no thread, and withdraws the approver's own earlier auto-approvals and requests for changes instead — so cursor-approve's decide is the only approver. Requests changes as usual. No effect without `approve_max_severity`. | | `approve_max_failed_reviewers` | `0` | How many panel reviewers may **error** (the cell ran but failed, timed out, or never uploaded) before auto-approve withholds its decision. `0` keeps the strict rule: any reviewer that did not complete means no decision. `N` tolerates up to N errored reviewers and names them in the decision (e.g. "approved with 1/6 reviewers errored: `:edge-case`"). See [auto-approve](#auto-approve). No effect without `approve_max_severity`. | | `max_rounds` | `5` | Cap on review rounds per PR (`0` → no cap). At or over it, no panel runs: the PR is labelled `needs-human-review`, one comment lists the latest round's open findings above the threshold, and the `approve_gate` output is `capped`. Removing the label resets the count. See [round cap](#round-cap-and-the-approve_gate-output). | -| `runs_on` | `'"ubuntu-latest"'` | JSON-encoded `runs-on` for `diff-size`, `preflight`, the `review` panel and `consolidate` only — the jobs that hold no write credential (`consolidate` stays on `ubuntu-latest` whenever `ANTHROPIC_API_KEY` is passed). Every other job stays on GitHub-hosted `ubuntu-latest`. A self-hosted pool is fine only if it is one-job-per-fresh-pod, identity-free and private-network-isolated (those jobs run models with shell over PR code), on linux/x64 with bash, git, curl, jq, python3, gh, tar and GNU coreutils. Empty falls back to the default, so `${{ vars.CURSOR_REVIEW_RUNS_ON }}` is safe while the variable is unset; e.g. `'["self-hosted", "linux", "x64"]'`. | +| `runs_on` | `'"ubuntu-latest"'` | JSON-encoded `runs-on` for `diff-size`, `preflight`, the `review` panel and `consolidate` only — the jobs that hold no write credential. Every other job stays on GitHub-hosted `ubuntu-latest`. A self-hosted pool is fine only if it is one-job-per-fresh-pod, identity-free and private-network-isolated (those jobs run models with shell over PR code), on linux/x64 with bash, git, curl, jq, python3, gh, tar and GNU coreutils. Empty falls back to the default, so `${{ vars.CURSOR_REVIEW_RUNS_ON }}` is safe while the variable is unset; e.g. `'["self-hosted", "linux", "x64"]'`. | ## Running a model experiment