diff --git a/.github/cursor-review/README.md b/.github/cursor-review/README.md index fbcbc45c..5031f8f3 100644 --- a/.github/cursor-review/README.md +++ b/.github/cursor-review/README.md @@ -395,7 +395,7 @@ All optional except `workflows_ref` (required, no default) — pass them under | `anthropic_direct_model` | `''` (off) | OPT-IN. The Anthropic twin of `openai_direct_model`: an Anthropic model id (e.g. `claude-opus-5-5`) reviewed through the Anthropic API directly (pinned Claude Code CLI, `package.json` here) as two cells on GitHub-hosted runners, by default **replacing** the Cursor `claude-*` cells. Needs the `ANTHROPIC_API_KEY` secret and a plain id other than `openai_direct_model`'s; without any of those, preflight warns and leaves the cells off and the panel unchanged. The agent also reads a **read-only PR checkout** (symlinks removed) with Read, Grep and Glob only — no Bash or code-running tool — confined to it by `--restricted`, with `/proc` and `/sys` also denied by explicit deny rules. This panel reviews private-repo diffs and the workflow cannot check the Anthropic organization's data-retention terms: callers on private repos must use a key from an organization with zero data retention. Each cell also uploads a `usage-direct--` artifact (kept 30 days) with the API's token counts for that cell, so cost can be measured without an Admin API key. | | `anthropic_direct_effort` | `xhigh` | `--effort` for the direct Anthropic cells: `low`, `medium`, `high`, `xhigh` or `max`; anything else warns and falls back to `xhigh`. | | `anthropic_direct_replaces_cursor_anthropic` | `true` | As `openai_direct_replaces_cursor_openai`, for the Cursor `claude-*` cells. Each lab counts by its own flag. Never touches `judge_model`. | -| `judge_direct_model` | `''` (off) | OPT-IN. An Anthropic model id (e.g. `claude-opus-5-5`) that runs the **judge** through the Anthropic API directly instead of `judge_model` through Cursor, with the same pinned Claude Code CLI and `ANTHROPIC_API_KEY` as `anthropic_direct_model`. Needs that secret and a plain id (`[A-Za-z0-9._-]`, at most 100); without either, preflight warns and the Cursor judge runs. Same prompt, structured-output tool, step timeout and degraded panel-union fallback as the Cursor judge; the agent reads the PR checkout (symlinks removed) with Read, Grep and Glob only — no Bash or code-running tool — confined to it by `--restricted`, with `/proc` and `/sys` also denied by explicit deny rules. A final review containing the key is replaced by the error seed, so the panel union posts instead; the log shows the judge's status and finding count only. The judge runs in its own `judge-direct` job on GitHub-hosted `ubuntu-latest`, whatever `runs_on` says, so the key never reaches `consolidate`. This judge reads private-repo diffs and the workflow cannot check the Anthropic organization's data-retention terms: callers on private repos must use a key from an organization with zero data retention. | +| `judge_direct_model` | `''` (off) | OPT-IN. An Anthropic model id (e.g. `claude-opus-5-5`) that runs the **judge** through the Anthropic API directly instead of `judge_model` through Cursor, with the same pinned Claude Code CLI and `ANTHROPIC_API_KEY` as `anthropic_direct_model`. Needs that secret and a plain id (`[A-Za-z0-9._-]`, at most 100); without either, preflight warns and the Cursor judge runs. Same prompt, structured-output tool, step timeout and degraded panel-union fallback as the Cursor judge; the agent reads the PR checkout (symlinks removed) with Read, Grep and Glob only — no Bash or code-running tool — confined to it by `--restricted`, with `/proc` and `/sys` also denied by explicit deny rules. A final review containing the key is replaced by the error seed, so the panel union posts instead; the log shows the judge's status and finding count only. The judge runs in its own `judge-direct` job on GitHub-hosted `ubuntu-latest`, whatever `runs_on` says, so the key never reaches `consolidate`. This judge reads private-repo diffs and the workflow cannot check the Anthropic organization's data-retention terms: callers on private repos must use a key from an organization with zero data retention. The judge also uploads a `usage-direct-judge-` artifact (kept 30 days) with the API's token counts for the judge, the same record the direct cells upload. | | `judge_direct_effort` | `xhigh` | `--effort` for the direct judge: `low`, `medium`, `high`, `xhigh` or `max`; anything else warns and falls back to `xhigh`. | | `cell_timeout_minutes` | `15` | Wall-clock cap on one reviewer cell's agent step (Cursor or direct). Over it the cell exits 130, uploads `status: error` with no findings, reddens its leg and fails `Panel integrity`. Shared by the cell's in-step retries, so it bounds attempts + backoff. Derives three more caps that cannot be set alone: cell job = this + 15, `Run judge` step = max(30, this × 2), judge job = that + 10. At the default: 15 / 30 / 30 / 40, today's values exactly. Preflight bounds it 10–45. | | `skip_bot_branch_prefixes` | `ci/bump- chore/refresh- auto/refresh-` | Skip the panel when the PR author is a Bot **and** the head branch starts with one of these prefixes (machine pin bumps / catalog refreshes). `''` reviews every bot PR. | diff --git a/.github/cursor-review/tests/test_workflow_judge_direct.py b/.github/cursor-review/tests/test_workflow_judge_direct.py index ad0f2b4e..8c90b824 100644 --- a/.github/cursor-review/tests/test_workflow_judge_direct.py +++ b/.github/cursor-review/tests/test_workflow_judge_direct.py @@ -1,6 +1,6 @@ """The optional direct-Anthropic judge in cursor-review.yml (`judge_direct_model`). -Four properties, each one a single line in the workflow that nothing else +Five properties, each one a single line in the workflow that nothing else would notice breaking: * preflight's resolution — `judge_direct=true` only for a plain id with the @@ -15,6 +15,10 @@ * the exfil guard — a final review carrying the key is replaced by the `--init` error seed, so `Build consolidated findings file` takes the degraded panel-union path. EXECUTED, both steps. + * token usage — the judge records the API's token counts as + `usage-direct-judge-` with the cells' script, byte for byte but + for the result path, whenever the judge step ran, and never fails the + job. EXECUTED. Run: python3 -m unittest discover -s .github/cursor-review/tests -p 'test_*.py' """ @@ -56,6 +60,9 @@ ADOPT_STEP = "Adopt direct judge review" DOWNLOAD_STEP = "Download direct judge review" UPLOAD_STEP = "Upload direct judge review" +RECORD_STEP = "Record token usage" +USAGE_UPLOAD_STEP = "Upload usage artifact" +USAGE_WARN_STEP = "Warn on unrecorded usage" ARTIFACT = "cursor-review-judge-direct" # Rebuilt in `judge-direct` exactly as in `consolidate`, so both judges read # the same prompt over the same panel. @@ -180,8 +187,9 @@ def test_the_direct_job_runs_no_shell_agent_and_no_bare_stdin_python(self): self.assertFalse(any("cursor-agent" in l or "install-cursor-cli" in l for l in body)) # A stdin script puts the cwd — the PR checkout — first on sys.path. self.assertFalse([l for l in body if re.search(r"python3 -(\s|$)", l)]) + # The judge's MCP config, the exfil guard's count and `Record token usage`. self.assertEqual([l.strip().split()[0] for l in body if re.search(r"python3 -I -", l)], - ["mcp_config=\"$(python3", "python3"]) + ["mcp_config=\"$(python3", "python3", "python3"]) def test_it_runs_only_when_consolidate_would_and_the_judge_is_direct(self): direct_if = self.job_scalar(self.direct, "if") @@ -410,5 +418,100 @@ def test_a_non_object_or_missing_hand_back_leaves_the_seed(self): self.assertEqual(self.read()["status"], "error") +class JudgeTokenUsageTest(unittest.TestCase): + """`Record token usage` and `Upload usage artifact` in `judge-direct`: the + direct cells' accounting, for the judge. RecordTokenUsageTest covers the + script itself; this pins that the judge runs the same script over its own + result, when, and that accounting can never fail or reach the hand-off.""" + + CONTEXT = {"inputs.judge_direct_model": JUDGE_MODEL, "github.run_attempt": "1"} + + @classmethod + def setUpClass(cls): + all_jobs = jobs() + cls.cell = all_jobs["review-anthropic-direct"] + cls.direct = all_jobs["judge-direct"] + cls.record_step = step(cls.direct, RECORD_STEP) + cls.upload = step(cls.direct, USAGE_UPLOAD_STEP) + cls.warn = step(cls.direct, USAGE_WARN_STEP) + + def record(self, result): + workdir = tempfile.mkdtemp() + self.addCleanup(shutil.rmtree, workdir) + if result is not None: + with open(os.path.join(workdir, "judge-claude-result.json"), "w", encoding="utf-8") as f: + json.dump(result, f) + done, _ = run_step(self.record_step, self.CONTEXT, workdir) + self.assertEqual(done.returncode, 0, done.stderr) + with open(os.path.join(workdir, "usage-out", "usage.json"), encoding="utf-8") as f: + return json.load(f) + + def test_the_script_is_the_cells_but_for_the_result_path(self): + cell = run_block(step(self.cell, RECORD_STEP)) + self.assertIn("/tmp/claude-result.json", cell) + self.assertEqual(run_block(self.record_step), + cell.replace("/tmp/claude-result.json", "/tmp/judge-claude-result.json")) + # ...which is where `Run judge (direct)` writes it. + self.assertIn("> /tmp/judge-claude-result.json", "\n".join(code_lines(step(self.direct, DIRECT_JUDGE_STEP)))) + + def test_it_records_the_judges_counts_and_nothing_model_written(self): + record = self.record({ + "result": "MODEL-WRITTEN TEXT", + "total_cost_usd": 0.4321, + "usage": {"input_tokens": 12, "output_tokens": 2100, "cache_read_input_tokens": 90000, + "cache_creation_input_tokens": 30000, + "cache_creation": {"ephemeral_5m_input_tokens": 30000, "ephemeral_1h_input_tokens": 0}}, + "modelUsage": {JUDGE_MODEL: {"inputTokens": 12, "outputTokens": 2100, + "cacheReadInputTokens": 90000, "cacheCreationInputTokens": 30000}}, + "num_turns": 4, + "duration_ms": 41000, + }) + self.assertEqual((record["model"], record["review_type"], record["run_attempt"], record["measured"]), + (JUDGE_MODEL, "judge", 1, True)) + self.assertEqual((record["usage"]["output_tokens"], record["usage"]["ephemeral_5m_input_tokens"]), + (2100, 30000)) + for leaked in ("MODEL-WRITTEN TEXT", "0.4321"): + self.assertNotIn(leaked, json.dumps(record)) + + def test_a_judge_that_left_no_result_is_unmeasured(self): + # Its cap killed it, or it never got a word out: tokens may still have + # been spent, so the record says unknown, never a measured zero. + record = self.record(None) + self.assertFalse(record["measured"]) + self.assertNotIn("usage", record) + + def test_it_runs_whenever_the_judge_step_did(self): + self.assertEqual(step_scalar(step(self.direct, DIRECT_JUDGE_STEP), "id"), "judge_direct") + for body in (self.record_step, self.upload): + condition = step_scalar(body, "if") + for outcome, runs in (("success", True), ("failure", True), ("cancelled", True), ("skipped", False)): + with self.subTest(outcome=outcome): + self.assertEqual(bool(evaluate(condition, {"steps.judge_direct.outcome": outcome})), runs) + + def test_accounting_never_fails_the_job_or_reaches_the_hand_off(self): + for body in (self.record_step, self.upload, self.warn): + self.assertEqual(step_scalar(body, "continue-on-error"), "true") + # `continue-on-error` does not absorb the job cap: a stalled + # accounting step must not cancel a job whose review is uploaded. + self.assertIn(step_scalar(body, "timeout-minutes"), ("1", "2")) + self.assertFalse(any("ANTHROPIC_API_KEY" in l for l in code_lines(body))) + name = block_mapping(self.upload, " with:", 10)["name"] + self.assertEqual(name, "usage-direct-judge-${{ inputs.judge_direct_model }}") + self.assertFalse(name.startswith("findings-"), "consolidate downloads `findings-*` as panel cells") + self.assertNotEqual(name, ARTIFACT) + # Last, after the review's own upload, so it can never delay the hand-off. + names = [l.strip()[len("- name: "):] for l in self.direct if l.startswith(" - name: ")] + self.assertEqual(names[-4:], [UPLOAD_STEP, RECORD_STEP, USAGE_UPLOAD_STEP, USAGE_WARN_STEP]) + + def test_a_failed_usage_upload_is_flagged(self): + # A squatted name 409s under `continue-on-error`; the run must say so. + self.assertEqual(step_scalar(self.upload, "id"), "usage_upload") + condition = step_scalar(self.warn, "if") + for outcome, runs in (("success", False), ("failure", True), ("skipped", False)): + with self.subTest(outcome=outcome): + self.assertEqual(bool(evaluate(condition, {"steps.usage_upload.outcome": outcome})), runs) + self.assertTrue(any("::warning::" in l for l in code_lines(self.warn))) + + if __name__ == "__main__": unittest.main() diff --git a/.github/workflows/cursor-review.yml b/.github/workflows/cursor-review.yml index 6161ddaf..dd48c49a 100644 --- a/.github/workflows/cursor-review.yml +++ b/.github/workflows/cursor-review.yml @@ -4080,6 +4080,8 @@ jobs: # `cursor-review-judge-direct` artifact. `consolidate` adopts it over its # own seed, so everything downstream (the degraded fallback, the # consolidated file, the post) is the Cursor judge's path unchanged. + # It also uploads the API's token counts for the judge as + # `usage-direct-judge-`, which nothing in this workflow reads. name: Direct judge needs: [gate, diff-size, preflight, review, review-openai-direct, review-anthropic-direct, ledger] # `consolidate`'s condition, plus the direct judge being in effect. @@ -4365,6 +4367,7 @@ jobs: # denied independently of that flag, since /proc/self/environ is the # key. The result JSON is never echoed: the run log is public and the # result is model output steered by PR text. + id: judge_direct if: steps.aggregate.outputs.ok_count != '0' && steps.claude_verify.outcome == 'success' continue-on-error: true timeout-minutes: ${{ fromJSON(needs.preflight.outputs.judge_step_timeout) }} @@ -4480,6 +4483,123 @@ jobs: if-no-files-found: error retention-days: 7 + - name: Record token usage + # `review-anthropic-direct`'s step, over the judge's result: the API's + # own token counts for the judge, so a caller can measure what the + # direct judge costs as it does the direct cells. Counts only — nothing + # model-written, no prices — and `measured` is false when the judge + # left no usable result. The script is the cells', byte for byte, but + # for the result path (test_workflow_judge_direct.py pins that). It + # runs whenever the judge step did, so a judge its cap killed is still + # recorded, as unmeasured. Best effort, and after the review's upload: + # accounting must never fail or delay the judge's hand-off. Time-boxed + # too: `continue-on-error` does not absorb the JOB cap, and a job that + # ends cancelled loses its already-uploaded review to `consolidate`. + if: always() && steps.judge_direct.outcome != 'skipped' + continue-on-error: true + timeout-minutes: 1 + env: + MODEL: ${{ inputs.judge_direct_model }} + REVIEW_TYPE: judge + RUN_ATTEMPT: ${{ github.run_attempt }} + run: | + mkdir -p /tmp/usage-out + python3 -I - <<'PY' + import json, os, re + + malformed = [] + + def count(value, field): + # A token count is a plain non-negative integer; absent reads as + # 0. Anything else makes the whole record unmeasured. + if value is None: + return 0 + if type(value) is int and value >= 0: + return value + malformed.append(field) + return 0 + + def mapping(value, field): + if value is None or isinstance(value, dict): + return value or {} + malformed.append(field) + return {} + + try: + with open("/tmp/judge-claude-result.json", encoding="utf-8") as f: + result = json.load(f) + except (OSError, ValueError): + result = None + if not isinstance(result, dict): + result = {} + usage = result.get("usage") + record = { + "model": os.environ["MODEL"], + "review_type": os.environ["REVIEW_TYPE"], + "run_attempt": int(os.environ.get("RUN_ATTEMPT") or 0), + "measured": False, + } + if isinstance(usage, dict): + written = mapping(usage.get("cache_creation"), "cache_creation") + totals = { + "input_tokens": count(usage.get("input_tokens"), "input_tokens"), + "output_tokens": count(usage.get("output_tokens"), "output_tokens"), + "cache_read_input_tokens": count(usage.get("cache_read_input_tokens"), "cache_read_input_tokens"), + "cache_creation_input_tokens": count(usage.get("cache_creation_input_tokens"), "cache_creation_input_tokens"), + "ephemeral_5m_input_tokens": count(written.get("ephemeral_5m_input_tokens"), "ephemeral_5m_input_tokens"), + "ephemeral_1h_input_tokens": count(written.get("ephemeral_1h_input_tokens"), "ephemeral_1h_input_tokens"), + } + # Per model, in case the CLI ever spends on a second one. + models = {} + for name, m in mapping(result.get("modelUsage"), "modelUsage").items(): + if not re.fullmatch(r"[A-Za-z0-9._-]{1,100}", name) or not isinstance(m, dict): + malformed.append("modelUsage") + continue + models[name] = { + "input_tokens": count(m.get("inputTokens"), "inputTokens"), + "output_tokens": count(m.get("outputTokens"), "outputTokens"), + "cache_read_input_tokens": count(m.get("cacheReadInputTokens"), "cacheReadInputTokens"), + "cache_creation_input_tokens": count(m.get("cacheCreationInputTokens"), "cacheCreationInputTokens"), + } + if not malformed: + record.update(measured=True, usage=totals, models=models) + for field in ("num_turns", "duration_ms"): + value = result.get(field) + record[field] = value if type(value) is int and value >= 0 else 0 + with open("/tmp/usage-out/usage.json", "w", encoding="utf-8") as f: + json.dump(record, f, indent=2) + # Field names only, never values: the run log is public. + print(f"Token usage recorded: measured={str(record['measured']).lower()}" + + (f" (malformed: {', '.join(sorted(set(malformed)))})." if malformed else ".")) + PY + + - name: Upload usage artifact + # The cells' `usage-direct--` family, review type + # `judge`. Neither `findings-*` nor `cursor-review-judge-direct`, so + # neither `consolidate` nor the hand-back ever reads it. Best effort, + # and time-boxed for the same reason as `Record token usage`. + id: usage_upload + if: always() && steps.judge_direct.outcome != 'skipped' + continue-on-error: true + timeout-minutes: 2 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 + with: + name: usage-direct-judge-${{ inputs.judge_direct_model }} + path: /tmp/usage-out/usage.json + if-no-files-found: ignore + retention-days: 30 + + - name: Warn on unrecorded usage + # The name is fixed and known before the job runs, so a panel cell (an + # agent over PR code that runs first) can upload under it and make this + # upload 409. `continue-on-error` would hide that, leaving the squatter's + # counts as the judge's record; say so on the run. `usage-direct-*` + # records are unauthenticated either way: treat them as indicative. + if: always() && steps.usage_upload.outcome == 'failure' + continue-on-error: true + timeout-minutes: 1 + run: echo "::warning::The direct judge's usage artifact did not upload; any usage-direct-judge-* record on this run is not the judge's." + consolidate: # Adjudicates the panel cell artifacts into one consolidated review. # Calls cursor-agent one more time as the judge, with prompt-judge.md — diff --git a/docs/callers/cursor-review.md b/docs/callers/cursor-review.md index 86173dfa..ecdfd89d 100644 --- a/docs/callers/cursor-review.md +++ b/docs/callers/cursor-review.md @@ -119,7 +119,7 @@ pull-requests: write # posting the consolidated review (and, at the round cap, | `anthropic_direct_model` | `''` | Opt-in. The Anthropic twin of `openai_direct_model`: an Anthropic model id (e.g. `claude-opus-5-5`) reviewed directly through the Anthropic API (pinned Claude Code CLI) as two cells — **adversarial** and **edge-case** — always on GitHub-hosted `ubuntu-latest`, by default **replacing** the Cursor `claude-*` cells. Requires `secrets.ANTHROPIC_API_KEY` and a plain id (`[A-Za-z0-9._-]`) other than `openai_direct_model`'s; without any of those, preflight warns, the cells stay off and the panel is unchanged. Unlike the OpenAI cells, the agent also reads a **read-only PR checkout** with Read, Grep and Glob only — no Bash or code-running tool — and `--restricted` confines those file tools to the checkout (`/proc` and `/sys` are also denied by explicit deny rules). **Private repos:** this panel reviews private-repo diffs and the workflow cannot check the Anthropic organization's data-retention terms, so use a key from an organization with zero data retention. Each cell also uploads a `usage-direct--` artifact (kept 30 days) with the API's token counts for that cell, so cost can be measured without an Admin API key. | | `anthropic_direct_effort` | `xhigh` | Reasoning effort for the direct Anthropic cells: `low`, `medium`, `high`, `xhigh` or `max`. Any other value warns and falls back to `xhigh`. | | `anthropic_direct_replaces_cursor_anthropic` | `true` | Same semantics as `openai_direct_replaces_cursor_openai`, for the Cursor `claude-*` cells: `true` drops them and the direct Anthropic cells **gate** in their place; `false` keeps both, with the direct cells **advisory**. Each lab counts by its own flag, so OpenAI and Anthropic may be set differently. The judge (`judge_model`) is never touched. | -| `judge_direct_model` | `''` | Opt-in. An Anthropic model id (e.g. `claude-opus-5-5`) that runs the **judge** directly through the Anthropic API (the same pinned Claude Code CLI and `secrets.ANTHROPIC_API_KEY` as `anthropic_direct_model`) instead of `judge_model` through Cursor. Requires the secret and a plain id (`[A-Za-z0-9._-]`); without either, preflight warns and the Cursor judge runs. Same prompt, output tool, timeout and degraded panel-union fallback as the Cursor judge; the agent reads the PR checkout with Read, Grep and Glob only — no Bash or code-running tool — confined to it by `--restricted` (`/proc` and `/sys` are also denied by explicit deny rules). It runs in its own `judge-direct` job on GitHub-hosted `ubuntu-latest` regardless of `runs_on`, so the key never reaches `consolidate`. **Private repos:** this judge reads private-repo diffs and the workflow cannot check the Anthropic organization's data-retention terms, so use a key from an organization with zero data retention. | +| `judge_direct_model` | `''` | Opt-in. An Anthropic model id (e.g. `claude-opus-5-5`) that runs the **judge** directly through the Anthropic API (the same pinned Claude Code CLI and `secrets.ANTHROPIC_API_KEY` as `anthropic_direct_model`) instead of `judge_model` through Cursor. Requires the secret and a plain id (`[A-Za-z0-9._-]`); without either, preflight warns and the Cursor judge runs. Same prompt, output tool, timeout and degraded panel-union fallback as the Cursor judge; the agent reads the PR checkout with Read, Grep and Glob only — no Bash or code-running tool — confined to it by `--restricted` (`/proc` and `/sys` are also denied by explicit deny rules). It runs in its own `judge-direct` job on GitHub-hosted `ubuntu-latest` regardless of `runs_on`, so the key never reaches `consolidate`. **Private repos:** this judge reads private-repo diffs and the workflow cannot check the Anthropic organization's data-retention terms, so use a key from an organization with zero data retention. The judge also uploads a `usage-direct-judge-` artifact (kept 30 days) with the API's token counts for the judge, the same record the direct cells upload. | | `judge_direct_effort` | `xhigh` | Reasoning effort for the direct judge: `low`, `medium`, `high`, `xhigh` or `max`. Any other value warns and falls back to `xhigh`. | | `skip_bot_branch_prefixes` | `ci/bump- chore/refresh- auto/refresh-` | Skip the panel for Bot-authored PRs on these branch prefixes (machine pin bumps / refreshes). `''` to review every bot PR. | | `cell_timeout_minutes` | `15` | Wall-clock cap on ONE reviewer cell's agent step (`Run cursor review`, or `Run direct-API review` for a direct cell). A cell that runs out of clock exits 130 and uploads `status: error` with no findings, which fails its leg and so fails `Panel integrity` — so this is a guillotine, not a safety net. Raise it when cells are dying at the cap rather than hanging: compare how long the cells that DID submit took, and if a meaningful share finish within a few minutes of the cap you are clipping a tail, not catching hangs. It is also the budget a cell's retries share (`resource_exhausted` is retried in-step), so it bounds attempts plus backoff. Three further caps derive from this one and cannot be set separately — each cell's job cap is this + 15 (checkout, CLI install, artifacts), the judge's `Run judge` step cap is this × 2 but never below 30 (it reads every cell's findings at the same tier, and its workload scales with the panel, not this cap), and the judge's job cap is its step + 10. Bounded 10–45 (the floor leaves a late in-step retry room for a full review); preflight fails loud outside that, before any cell spends. |