Skip to content

feat(predict/evaluate): claude pair → claude-opus-5-5 (proc-v9) — HELD until after the long-conference release - #1964

Draft
modelmirror wants to merge 2 commits into
stagingfrom
feat/opus-5-5-predict-evaluate
Draft

modelmirror wants to merge 2 commits into
stagingfrom
feat/opus-5-5-predict-evaluate

Conversation

@modelmirror

Copy link
Copy Markdown
Collaborator

⛔ Do not merge. Held until results/ot2026-longconf is minted (~2026-10-20). Draft; it is stacked on #1962, whose commit appears here until that PR merges.

Closes #1963

What changed and why

This moves the claude predictor and evaluator from claude-fable-5-1 to claude-opus-5-5.

  • pricing.DEFAULT_MODELS["claude-code"]claude-opus-5-5. No actor pins a model, so this is what the fleet runs.
  • MODEL_RATES["claude-opus-5-5"] is $4 / $20 per MTok. ModelRate gains an optional cache_read_per_mtok, and Opus 5.5 carries its published $0.20. The uniform 0.1× multiplier would bill its cache reads at twice that. usage.json is never re-priced, so every Opus cell would carry a ~20% overstatement permanently. Every other model keeps the multiplier. A test pins the override.
  • The claude-code-action pin moves to 9ca9355 in all five places. That commit is v1.0.231 plus a single version-only commit bundling Claude Code 2.1.280, the first CLI that accepts the model id. No tagged release bundles 2.1.280 yet. Rebase step: re-pin to the first tagged release that does.
  • Re-bless → proc-v9: the claude predictor and judge digests are new, and the four codex/gemini digests carry forward with proc-v8's bless moment. Bless moments are step 2's forecast. FROZEN_SINCE is guessed late (2026-10-21).
  • Draft freeze-record entry for the third supersession shape. ‹› placeholders mark what the rebase fills in: the landing date, the declaration it cites, and the action release.
  • docs/budget.md: per event goes from $15–17 to about $10. The claude pair is priced by re-costing its 122 proc-v8 predictions and 27 gradings at Opus rates: ~$1.35 a prediction, $1.30–1.40 a grading. That assumes Opus spends the same tokens, and the doc says so.
  • Tests that pin the default are updated. The config/predictors.yaml comment example is updated (not hashed). The prompts' MODEL_ID example is left alone on purpose, because it is hashed and editing it would move all six digests.

Artifact served: every claude prediction and evaluation from proc-v9 on, plus the ledger's cost figures.

Before this can merge (in order)

  1. A declaration entry on main before 2026-09-28 (the long conference). The third shape de-counts; it is pre-registered only if a dated entry closes the proc-v8 counted window while its outcomes are unknown. Today 0 of the 122 cohort events have resolved. This is a separate, small PR; see claude pair → Claude Opus 5.5 (proc-v9): predict/evaluate model, pricing and budget #1963.
  2. The long-conference release is minted (results/ot2026-longconf). After this lands, the live frozen scope no longer reproduces the release's figures.
  3. Rebase and re-read:
  4. After the carrying promotion, run step 4: correct the bless moments and instant, then tag prereg/proc-v9.

Effect check (after the carrying promotion)

  • gh workflow run integration-test.yml --repo ModelMirrorAI/fedcourtsai --ref main -f scenario=engine-actions-smoke -f engine=claude-code -f deploy-environment=prod is green, and the resolve step names claude-opus-5-5.
  • The first claude predict/evaluate cells after it carry process_version.label: proc-v9 and usage.json "model": "claude-opus-5-5".
  • uv run fedcourts usage-summary over those cells replaces the priced budget figure.

Review

stats-reviewer

  • One blocker: the declaration was folded into this held entry, which would land after the outcomes. Resolved by splitting it out (step 1 above); the entry now cites it.
  • Recommendations adopted:
    • "closed counted window" wording instead of "shakedown", which would contradict the release;
    • the release-first gate;
    • the evaluation count across all three judges;
    • the per-model cache-read rate;
    • budget figures labelled as priced;
    • the step-4 wording for both directions.
  • Its alternatives are presented in claude pair → Claude Opus 5.5 (proc-v9): predict/evaluate model, pricing and budget #1963, not chosen here: evaluator-only via a model: pin on claude-baseline, or an unblessed Opus shakedown actor.

code-reviewer

Gate

  • Full test suite: 4,985 pass. One secret-scan timing test failed under parallel load and passes alone.
  • ruff and mypy are clean, schemas have no drift, and zizmor is clean.

Not run: docs-reviewer and workflow-reviewer on this delta. #1962's workflow review covers the CLI pin. Run both at the rebase, since the text will change anyway.

This PR touches .github/workflows/ and config/, so it is maintainer-merged. It is also held; I will not merge it.

🤖 Generated with Claude Code

modelmirror and others added 2 commits September 22, 2026 17:12
… to 2.1.280

Opus 5.5 needs Claude Code 2.1.280 or newer; the pin moves in step across
run-analytics, run-backtest and integration-test.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…-v9, held)

DEFAULT_MODELS moves, MODEL_RATES prices Opus 5.5 with its own cache-read
rate, the claude-code-action pin moves to one bundling Claude Code 2.1.280,
and the claude pair is re-blessed as proc-v9 with a draft freeze-record
entry. Held until the long-conference release is minted.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant