Skip to content

feat(cursor-review): record the direct judge's token usage - #401

Merged
mattmillerai merged 2 commits into
mainfrom
matt/judge-direct-usage
Oct 9, 2026
Merged

mattmillerai merged 2 commits into
mainfrom
matt/judge-direct-usage

Conversation

@mattmillerai

Copy link
Copy Markdown
Member

ELI-5

Each direct-Anthropic review cell already saves how many tokens it used, so you can work out what it cost. The direct judge from #399 doesn't save anything. This PR makes the judge save the same token counts, so its cost can be measured too.

Summary

judge-direct gains the cells' two accounting steps, Record token usage and Upload usage artifact.

  • Same script as the cells. The record script matches the cells' byte for byte except for the result path. Run judge (direct) already writes Claude Code's result JSON to /tmp/judge-claude-result.json, and the script reads it from there. A test pins the match, so the two scripts can't drift apart.
    • The script copies only integer token counts and model ids. Nothing model-written goes in, and no prices.
    • If the judge left no usable result, the record says measured: false instead of a guessed zero.
  • Artifact: usage-direct-judge-<model>, kept for 30 days.
    • It belongs to the cells' usage-direct-<review_type>-<model> family, with review type judge.
    • The name doesn't match findings-* and isn't cursor-review-judge-direct, so neither consolidate nor the hand-back reads it.
  • When it runs: whenever the judge step ran. The condition is always() && steps.judge_direct.outcome != 'skipped', and Run judge (direct) gains id: judge_direct. A judge killed by its time cap is still recorded, as unmeasured.
  • Can't affect the review:
    • Both steps are continue-on-error.
    • They run after the review's own upload, so accounting can neither fail nor delay the hand-off.
    • Neither step references ANTHROPIC_API_KEY.
  • Docs: the judge_direct_model rows in .github/cursor-review/README.md and docs/callers/cursor-review.md now mention the artifact.

Tests

  • New JudgeTokenUsageTest in test_workflow_judge_direct.py. It checks:
    • Identity: the judge's script equals the cells' script with only the result path changed, and that path is where Run judge (direct) writes.
    • Executed over a judge result:
      • the counts are copied, with review_type: judge;
      • the model text and the CLI's cost figure are dropped;
      • with no result file at all, the record is unmeasured.
    • The if:, evaluated for success, failure, cancelled and skipped.
    • Safety: both steps are best effort and key-free, the artifact name is right, and they come last in the job.
  • test_the_direct_job_runs_no_shell_agent_and_no_bare_stdin_python now expects three python3 -I - scripts, all with -I.
  • Mutation check: pointing the judge's record step back at the cells' result path fails two of the new tests.
  • python3 -m unittest discover -s .github/cursor-review/tests -p 'test_*.py': 1334 tests, OK.
  • python3 -m unittest discover -s .github/workflow-pins/tests -p 'test_*.py': 551 tests, OK (43 skipped).
  • actionlint: 80 findings on main and 80 on this branch, so none are new. They are the existing job.workflow_sha and SC2018/SC2019 notes.
  • Not run: a live round with the direct judge. No caller has enabled it yet.

🤖 Generated with Claude Code

The direct-Anthropic cells upload the API's token counts as
`usage-direct-<review_type>-<model>`; the direct judge recorded nothing,
so its cost could not be measured. `judge-direct` now runs the cells'
`Record token usage` script over its own result (byte for byte but the
result path, pinned by a test) and uploads `usage-direct-judge-<model>`.
Both steps run whenever the judge step did, are best effort, and come
after the review's upload, so accounting never fails or delays the
hand-off.
@mattmillerai mattmillerai added cursor-review Multi-model cursor review agent-coded Authored by the agent-work loop labels Oct 9, 2026
@coderabbitai

coderabbitai Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

  • Run on-demand review

This review includes 4 billable files and costs up to $1.00.

  • Ask an admin to make reviews automatic

Open in CodeRabbit

Reviews can continue after your included limit without a manual trigger. An admin must approve usage-based billing.

Or wait 22 minutes for your next included review.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available. Your 132 included PR review attempts over the past 7 days set your current allowance at 1 review per hour.

Learn how review limits work.

Review configuration:

⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: ASSERTIVE
  • Plan: Team
  • Run ID: d4b76247-d615-41fa-b3bc-0246a04c9bd5

📥 Commits

Reviewing files that changed from the base of the PR and between ede6008 and a2fb19e.


📒 Files selected for processing (4)
  • .github/cursor-review/README.md
  • .github/cursor-review/tests/test_workflow_judge_direct.py
  • .github/workflows/cursor-review.yml
  • docs/callers/cursor-review.md


  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 Cursor Review — Consolidated panel

Triggered by @mattmillerai.

Found 2 finding(s).

Severity Count
🟡 Medium 1
🟢 Low 1

Panel: 6/6 reviewers contributed findings.

Comment thread .github/workflows/cursor-review.yml Resolved
Comment thread .github/workflows/cursor-review.yml Resolved
…e upload

- Record token usage / Upload usage artifact get timeout-minutes (1/2):
  continue-on-error does not absorb the job cap, and a cancelled
  judge-direct job loses its already-uploaded review to consolidate.
- New 'Warn on unrecorded usage' step emits ::warning:: when the usage
  upload fails (e.g. a 409 on a squatted artifact name), so a forged
  record is not silently left as the judge's.
- Tests pin both.
@mattmillerai
mattmillerai merged commit cc22b08 into main Oct 9, 2026
8 checks passed
@mattmillerai
mattmillerai deleted the matt/judge-direct-usage branch October 9, 2026 23:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent-coded Authored by the agent-work loop cursor-review Multi-model cursor review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants