Repository navigation
feat(cursor-review): record the direct judge's token usage - #401
Merged
Merged
Conversation
The direct-Anthropic cells upload the API's token counts as `usage-direct-<review_type>-<model>`; the direct judge recorded nothing, so its cost could not be measured. `judge-direct` now runs the cells' `Record token usage` script over its own result (byte for byte but the result path, pinned by a test) and uploads `usage-direct-judge-<model>`. Both steps run whenever the judge step did, are best effort, and come after the review's upload, so accounting never fails or delays the hand-off.
Contributor
|
Warning Review limit reached
This review includes 4 billable files and costs up to $1.00.
Reviews can continue after your included limit without a manual trigger. An admin must approve usage-based billing. Or wait 22 minutes for your next included review. View limit detailsLimit details: You’ve used the included review currently available. Your 132 included PR review attempts over the past 7 days set your current allowance at 1 review per hour. Review configuration: ⚙️ Run configuration
📒 Files selected for processing (4)
Comment |
There was a problem hiding this comment.
🔍 Cursor Review — Consolidated panel
Triggered by @mattmillerai.
Found 2 finding(s).
| Severity | Count |
|---|---|
| 🟡 Medium | 1 |
| 🟢 Low | 1 |
Panel: 6/6 reviewers contributed findings.
huntcsg
approved these changes
Oct 9, 2026
…e upload - Record token usage / Upload usage artifact get timeout-minutes (1/2): continue-on-error does not absorb the job cap, and a cancelled judge-direct job loses its already-uploaded review to consolidate. - New 'Warn on unrecorded usage' step emits ::warning:: when the usage upload fails (e.g. a 409 on a squatted artifact name), so a forged record is not silently left as the judge's. - Tests pin both.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ELI-5
Each direct-Anthropic review cell already saves how many tokens it used, so you can work out what it cost. The direct judge from #399 doesn't save anything. This PR makes the judge save the same token counts, so its cost can be measured too.
Summary
judge-directgains the cells' two accounting steps,Record token usageandUpload usage artifact.Run judge (direct)already writes Claude Code's result JSON to/tmp/judge-claude-result.json, and the script reads it from there. A test pins the match, so the two scripts can't drift apart.measured: falseinstead of a guessed zero.usage-direct-judge-<model>, kept for 30 days.usage-direct-<review_type>-<model>family, with review typejudge.findings-*and isn'tcursor-review-judge-direct, so neitherconsolidatenor the hand-back reads it.always() && steps.judge_direct.outcome != 'skipped', andRun judge (direct)gainsid: judge_direct. A judge killed by its time cap is still recorded, as unmeasured.continue-on-error.ANTHROPIC_API_KEY.judge_direct_modelrows in.github/cursor-review/README.mdanddocs/callers/cursor-review.mdnow mention the artifact.Tests
JudgeTokenUsageTestintest_workflow_judge_direct.py. It checks:Run judge (direct)writes.review_type: judge;if:, evaluated forsuccess,failure,cancelledandskipped.test_the_direct_job_runs_no_shell_agent_and_no_bare_stdin_pythonnow expects threepython3 -I -scripts, all with-I.python3 -m unittest discover -s .github/cursor-review/tests -p 'test_*.py': 1334 tests, OK.python3 -m unittest discover -s .github/workflow-pins/tests -p 'test_*.py': 551 tests, OK (43 skipped).mainand 80 on this branch, so none are new. They are the existingjob.workflow_shaand SC2018/SC2019 notes.🤖 Generated with Claude Code