feat(predict/evaluate): claude pair → claude-opus-5-5 (proc-v9) — HELD until after the long-conference release - #1964
Draft
modelmirror wants to merge 2 commits into
Draft
modelmirror wants to merge 2 commits into
modelmirror wants to merge 2 commits into
Conversation
… to 2.1.280 Opus 5.5 needs Claude Code 2.1.280 or newer; the pin moves in step across run-analytics, run-backtest and integration-test. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…-v9, held) DEFAULT_MODELS moves, MODEL_RATES prices Opus 5.5 with its own cache-read rate, the claude-code-action pin moves to one bundling Claude Code 2.1.280, and the claude pair is re-blessed as proc-v9 with a draft freeze-record entry. Held until the long-conference release is minted. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1963
What changed and why
This moves the claude predictor and evaluator from
claude-fable-5-1toclaude-opus-5-5.pricing.DEFAULT_MODELS["claude-code"]→claude-opus-5-5. No actor pins a model, so this is what the fleet runs.MODEL_RATES["claude-opus-5-5"]is $4 / $20 per MTok.ModelRategains an optionalcache_read_per_mtok, and Opus 5.5 carries its published $0.20. The uniform 0.1× multiplier would bill its cache reads at twice that.usage.jsonis never re-priced, so every Opus cell would carry a ~20% overstatement permanently. Every other model keeps the multiplier. A test pins the override.claude-code-actionpin moves to9ca9355in all five places. That commit is v1.0.231 plus a single version-only commit bundling Claude Code 2.1.280, the first CLI that accepts the model id. No tagged release bundles 2.1.280 yet. Rebase step: re-pin to the first tagged release that does.proc-v9: the claude predictor and judge digests are new, and the four codex/gemini digests carry forward with proc-v8's bless moment. Bless moments are step 2's forecast.FROZEN_SINCEis guessed late (2026-10-21).‹›placeholders mark what the rebase fills in: the landing date, the declaration it cites, and the action release.docs/budget.md: per event goes from $15–17 to about $10. The claude pair is priced by re-costing its 122 proc-v8 predictions and 27 gradings at Opus rates: ~$1.35 a prediction, $1.30–1.40 a grading. That assumes Opus spends the same tokens, and the doc says so.config/predictors.yamlcomment example is updated (not hashed). The prompts'MODEL_IDexample is left alone on purpose, because it is hashed and editing it would move all six digests.Artifact served: every claude prediction and evaluation from proc-v9 on, plus the ledger's cost figures.
Before this can merge (in order)
mainbefore 2026-09-28 (the long conference). The third shape de-counts; it is pre-registered only if a dated entry closes the proc-v8 counted window while its outcomes are unknown. Today 0 of the 122 cohort events have resolved. This is a separate, small PR; see claude pair → Claude Opus 5.5 (proc-v9): predict/evaluate model, pricing and budget #1963.results/ot2026-longconf). After this lands, the live frozen scope no longer reproduces the release's figures.fedcourts process-digest --all, and the step-0 grep onorigin/main;prereg/proc-v9.Effect check (after the carrying promotion)
gh workflow run integration-test.yml --repo ModelMirrorAI/fedcourtsai --ref main -f scenario=engine-actions-smoke -f engine=claude-code -f deploy-environment=prodis green, and the resolve step namesclaude-opus-5-5.process_version.label: proc-v9andusage.json"model": "claude-opus-5-5".uv run fedcourts usage-summaryover those cells replaces the priced budget figure.Review
stats-reviewer
model:pin on claude-baseline, or an unblessed Opus shakedown actor.code-reviewer
predictors.yamlexample, and the comment nits.Gate
Not run: docs-reviewer and workflow-reviewer on this delta. #1962's workflow review covers the CLI pin. Run both at the rebase, since the text will change anyway.
This PR touches
.github/workflows/andconfig/, so it is maintainer-merged. It is also held; I will not merge it.🤖 Generated with Claude Code