Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/cursor-review/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -395,7 +395,7 @@ All optional except `workflows_ref` (required, no default) — pass them under
| `anthropic_direct_model` | `''` (off) | OPT-IN. The Anthropic twin of `openai_direct_model`: an Anthropic model id (e.g. `claude-opus-5-5`) reviewed through the Anthropic API directly (pinned Claude Code CLI, `package.json` here) as two cells on GitHub-hosted runners, by default **replacing** the Cursor `claude-*` cells. Needs the `ANTHROPIC_API_KEY` secret and a plain id other than `openai_direct_model`'s; without any of those, preflight warns and leaves the cells off and the panel unchanged. The agent also reads a **read-only PR checkout** (symlinks removed) with Read, Grep and Glob only — no Bash or code-running tool — confined to it by `--restricted`, with `/proc` and `/sys` also denied by explicit deny rules. This panel reviews private-repo diffs and the workflow cannot check the Anthropic organization's data-retention terms: callers on private repos must use a key from an organization with zero data retention. Each cell also uploads a `usage-direct-<review_type>-<model>` artifact (kept 30 days) with the API's token counts for that cell, so cost can be measured without an Admin API key. |
| `anthropic_direct_effort` | `xhigh` | `--effort` for the direct Anthropic cells: `low`, `medium`, `high`, `xhigh` or `max`; anything else warns and falls back to `xhigh`. |
| `anthropic_direct_replaces_cursor_anthropic` | `true` | As `openai_direct_replaces_cursor_openai`, for the Cursor `claude-*` cells. Each lab counts by its own flag. Never touches `judge_model`. |
| `judge_direct_model` | `''` (off) | OPT-IN. An Anthropic model id (e.g. `claude-opus-5-5`) that runs the **judge** through the Anthropic API directly instead of `judge_model` through Cursor, with the same pinned Claude Code CLI and `ANTHROPIC_API_KEY` as `anthropic_direct_model`. Needs that secret and a plain id (`[A-Za-z0-9._-]`, at most 100); without either, preflight warns and the Cursor judge runs. Same prompt, structured-output tool, step timeout and degraded panel-union fallback as the Cursor judge; the agent reads the PR checkout (symlinks removed) with Read, Grep and Glob only — no Bash or code-running tool — confined to it by `--restricted`, with `/proc` and `/sys` also denied by explicit deny rules. A final review containing the key is replaced by the error seed, so the panel union posts instead; the log shows the judge's status and finding count only. The judge runs in its own `judge-direct` job on GitHub-hosted `ubuntu-latest`, whatever `runs_on` says, so the key never reaches `consolidate`. This judge reads private-repo diffs and the workflow cannot check the Anthropic organization's data-retention terms: callers on private repos must use a key from an organization with zero data retention. |
| `judge_direct_model` | `''` (off) | OPT-IN. An Anthropic model id (e.g. `claude-opus-5-5`) that runs the **judge** through the Anthropic API directly instead of `judge_model` through Cursor, with the same pinned Claude Code CLI and `ANTHROPIC_API_KEY` as `anthropic_direct_model`. Needs that secret and a plain id (`[A-Za-z0-9._-]`, at most 100); without either, preflight warns and the Cursor judge runs. Same prompt, structured-output tool, step timeout and degraded panel-union fallback as the Cursor judge; the agent reads the PR checkout (symlinks removed) with Read, Grep and Glob only — no Bash or code-running tool — confined to it by `--restricted`, with `/proc` and `/sys` also denied by explicit deny rules. A final review containing the key is replaced by the error seed, so the panel union posts instead; the log shows the judge's status and finding count only. The judge runs in its own `judge-direct` job on GitHub-hosted `ubuntu-latest`, whatever `runs_on` says, so the key never reaches `consolidate`. This judge reads private-repo diffs and the workflow cannot check the Anthropic organization's data-retention terms: callers on private repos must use a key from an organization with zero data retention. The judge also uploads a `usage-direct-judge-<model>` artifact (kept 30 days) with the API's token counts for the judge, the same record the direct cells upload. |
| `judge_direct_effort` | `xhigh` | `--effort` for the direct judge: `low`, `medium`, `high`, `xhigh` or `max`; anything else warns and falls back to `xhigh`. |
| `cell_timeout_minutes` | `15` | Wall-clock cap on one reviewer cell's agent step (Cursor or direct). Over it the cell exits 130, uploads `status: error` with no findings, reddens its leg and fails `Panel integrity`. Shared by the cell's in-step retries, so it bounds attempts + backoff. Derives three more caps that cannot be set alone: cell job = this + 15, `Run judge` step = max(30, this × 2), judge job = that + 10. At the default: 15 / 30 / 30 / 40, today's values exactly. Preflight bounds it 10–45. |
| `skip_bot_branch_prefixes` | `ci/bump- chore/refresh- auto/refresh-` | Skip the panel when the PR author is a Bot **and** the head branch starts with one of these prefixes (machine pin bumps / catalog refreshes). `''` reviews every bot PR. |
Expand Down
107 changes: 105 additions & 2 deletions .github/cursor-review/tests/test_workflow_judge_direct.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
"""The optional direct-Anthropic judge in cursor-review.yml (`judge_direct_model`).

Four properties, each one a single line in the workflow that nothing else
Five properties, each one a single line in the workflow that nothing else
would notice breaking:

* preflight's resolution — `judge_direct=true` only for a plain id with the
Expand All @@ -15,6 +15,10 @@
* the exfil guard — a final review carrying the key is replaced by the
`--init` error seed, so `Build consolidated findings file` takes the
degraded panel-union path. EXECUTED, both steps.
* token usage — the judge records the API's token counts as
`usage-direct-judge-<model>` with the cells' script, byte for byte but
for the result path, whenever the judge step ran, and never fails the
job. EXECUTED.

Run: python3 -m unittest discover -s .github/cursor-review/tests -p 'test_*.py'
"""
Expand Down Expand Up @@ -56,6 +60,9 @@
ADOPT_STEP = "Adopt direct judge review"
DOWNLOAD_STEP = "Download direct judge review"
UPLOAD_STEP = "Upload direct judge review"
RECORD_STEP = "Record token usage"
USAGE_UPLOAD_STEP = "Upload usage artifact"
USAGE_WARN_STEP = "Warn on unrecorded usage"
ARTIFACT = "cursor-review-judge-direct"
# Rebuilt in `judge-direct` exactly as in `consolidate`, so both judges read
# the same prompt over the same panel.
Expand Down Expand Up @@ -180,8 +187,9 @@ def test_the_direct_job_runs_no_shell_agent_and_no_bare_stdin_python(self):
self.assertFalse(any("cursor-agent" in l or "install-cursor-cli" in l for l in body))
# A stdin script puts the cwd — the PR checkout — first on sys.path.
self.assertFalse([l for l in body if re.search(r"python3 -(\s|$)", l)])
# The judge's MCP config, the exfil guard's count and `Record token usage`.
self.assertEqual([l.strip().split()[0] for l in body if re.search(r"python3 -I -", l)],
["mcp_config=\"$(python3", "python3"])
["mcp_config=\"$(python3", "python3", "python3"])

def test_it_runs_only_when_consolidate_would_and_the_judge_is_direct(self):
direct_if = self.job_scalar(self.direct, "if")
Expand Down Expand Up @@ -410,5 +418,100 @@ def test_a_non_object_or_missing_hand_back_leaves_the_seed(self):
self.assertEqual(self.read()["status"], "error")


class JudgeTokenUsageTest(unittest.TestCase):
"""`Record token usage` and `Upload usage artifact` in `judge-direct`: the
direct cells' accounting, for the judge. RecordTokenUsageTest covers the
script itself; this pins that the judge runs the same script over its own
result, when, and that accounting can never fail or reach the hand-off."""

CONTEXT = {"inputs.judge_direct_model": JUDGE_MODEL, "github.run_attempt": "1"}

@classmethod
def setUpClass(cls):
all_jobs = jobs()
cls.cell = all_jobs["review-anthropic-direct"]
cls.direct = all_jobs["judge-direct"]
cls.record_step = step(cls.direct, RECORD_STEP)
cls.upload = step(cls.direct, USAGE_UPLOAD_STEP)
cls.warn = step(cls.direct, USAGE_WARN_STEP)

def record(self, result):
workdir = tempfile.mkdtemp()
self.addCleanup(shutil.rmtree, workdir)
if result is not None:
with open(os.path.join(workdir, "judge-claude-result.json"), "w", encoding="utf-8") as f:
json.dump(result, f)
done, _ = run_step(self.record_step, self.CONTEXT, workdir)
self.assertEqual(done.returncode, 0, done.stderr)
with open(os.path.join(workdir, "usage-out", "usage.json"), encoding="utf-8") as f:
return json.load(f)

def test_the_script_is_the_cells_but_for_the_result_path(self):
cell = run_block(step(self.cell, RECORD_STEP))
self.assertIn("/tmp/claude-result.json", cell)
self.assertEqual(run_block(self.record_step),
cell.replace("/tmp/claude-result.json", "/tmp/judge-claude-result.json"))
# ...which is where `Run judge (direct)` writes it.
self.assertIn("> /tmp/judge-claude-result.json", "\n".join(code_lines(step(self.direct, DIRECT_JUDGE_STEP))))

def test_it_records_the_judges_counts_and_nothing_model_written(self):
record = self.record({
"result": "MODEL-WRITTEN TEXT",
"total_cost_usd": 0.4321,
"usage": {"input_tokens": 12, "output_tokens": 2100, "cache_read_input_tokens": 90000,
"cache_creation_input_tokens": 30000,
"cache_creation": {"ephemeral_5m_input_tokens": 30000, "ephemeral_1h_input_tokens": 0}},
"modelUsage": {JUDGE_MODEL: {"inputTokens": 12, "outputTokens": 2100,
"cacheReadInputTokens": 90000, "cacheCreationInputTokens": 30000}},
"num_turns": 4,
"duration_ms": 41000,
})
self.assertEqual((record["model"], record["review_type"], record["run_attempt"], record["measured"]),
(JUDGE_MODEL, "judge", 1, True))
self.assertEqual((record["usage"]["output_tokens"], record["usage"]["ephemeral_5m_input_tokens"]),
(2100, 30000))
for leaked in ("MODEL-WRITTEN TEXT", "0.4321"):
self.assertNotIn(leaked, json.dumps(record))

def test_a_judge_that_left_no_result_is_unmeasured(self):
# Its cap killed it, or it never got a word out: tokens may still have
# been spent, so the record says unknown, never a measured zero.
record = self.record(None)
self.assertFalse(record["measured"])
self.assertNotIn("usage", record)

def test_it_runs_whenever_the_judge_step_did(self):
self.assertEqual(step_scalar(step(self.direct, DIRECT_JUDGE_STEP), "id"), "judge_direct")
for body in (self.record_step, self.upload):
condition = step_scalar(body, "if")
for outcome, runs in (("success", True), ("failure", True), ("cancelled", True), ("skipped", False)):
with self.subTest(outcome=outcome):
self.assertEqual(bool(evaluate(condition, {"steps.judge_direct.outcome": outcome})), runs)

def test_accounting_never_fails_the_job_or_reaches_the_hand_off(self):
for body in (self.record_step, self.upload, self.warn):
self.assertEqual(step_scalar(body, "continue-on-error"), "true")
# `continue-on-error` does not absorb the job cap: a stalled
# accounting step must not cancel a job whose review is uploaded.
self.assertIn(step_scalar(body, "timeout-minutes"), ("1", "2"))
self.assertFalse(any("ANTHROPIC_API_KEY" in l for l in code_lines(body)))
name = block_mapping(self.upload, " with:", 10)["name"]
self.assertEqual(name, "usage-direct-judge-${{ inputs.judge_direct_model }}")
self.assertFalse(name.startswith("findings-"), "consolidate downloads `findings-*` as panel cells")
self.assertNotEqual(name, ARTIFACT)
# Last, after the review's own upload, so it can never delay the hand-off.
names = [l.strip()[len("- name: "):] for l in self.direct if l.startswith(" - name: ")]
self.assertEqual(names[-4:], [UPLOAD_STEP, RECORD_STEP, USAGE_UPLOAD_STEP, USAGE_WARN_STEP])

def test_a_failed_usage_upload_is_flagged(self):
# A squatted name 409s under `continue-on-error`; the run must say so.
self.assertEqual(step_scalar(self.upload, "id"), "usage_upload")
condition = step_scalar(self.warn, "if")
for outcome, runs in (("success", False), ("failure", True), ("skipped", False)):
with self.subTest(outcome=outcome):
self.assertEqual(bool(evaluate(condition, {"steps.usage_upload.outcome": outcome})), runs)
self.assertTrue(any("::warning::" in l for l in code_lines(self.warn)))


if __name__ == "__main__":
unittest.main()
Loading
Loading