Skip to content

Promotion 2026-09-23: staging → main - #1977

Merged
modelmirror merged 24 commits into
mainfrom
staging
Sep 23, 2026
Merged

modelmirror merged 24 commits into
mainfrom
staging

Conversation

@modelmirror

Copy link
Copy Markdown
Collaborator

Promotion batch — 2026-09-23

Staging → main, ahead of the 2026-09-28 long conference. No process digest moves: no prompt, predictor/evaluator config, DEFAULT_MODELS, ENGINE_RETRIEVAL or MCP manifest entry is in the batch, and tests/test_process_version.py -k blessed is green at the staging head. No forward cell's execution path changes: live run-predict / run-evaluate call their engines through the actions and CLIs directly. The shared pipeline/runner.py change (#1959) reaches only local-cascade (engine-smoke) and the back-test.

PR What Class
#1958 Weekly digest restructured around three production windows ops reporting; run-ops.yml comments
#1959 Back-test: a failed engine call is one lost cell; a spent quota stops that engine, never the campaign back-test; shared runner retry classification
#1962 qp-topic labeler offers claude-opus-5-5; Claude CLI pin 2.1.259 → 2.1.280 (analytics, integration-test, back-test) workflow
#1970 boto3/botocore, pypdf 6.19.0, ruff patch bumps deps (lock only)
#1972 Back-test per-cell leakage disclosure record (recorded, not excluded) back-test report schema; metrics/README reading rule
#1973 Plain-language case summaries lane (summarize.yml, Claude Sonnet 5, environment's Anthropic key) new workflow; ci.yml main-base admits summaries/refresh; SECURITY.md
#1974 Back-test runs each engine's replay cells in its own lane back-test; run-backtest.yml comments

Held out on purpose (drafts): #1964 (Opus 5.5 predict/evaluate, proc-v9 — after the long-conference release), #1971 (actions group incl. claude-code-action → v1.0.231, which bundles CLI 2.1.278 < 2.1.280), #1844 (engine-smoke provisioning — conflicts with this batch; rebase after).

Engine evidence: a full green scenario=all run at 98c4607c1 (run 35901466753; all six engine-smoke / engine-actions-smoke legs green — this matters because #1962 moves the Claude CLI the engine-smoke leg runs). The head then moved by a data-only re-sync (seed commit 81a0153aa, corpus/corpus.db.ref only), which cannot reach a cell, so the head is carried by an all-offline run plus the promote:skip-engine-smoke label, per When the head moves after a green all in docs/pipeline.md. The engine evidence for this batch's code is the 98c4607c1 run above.

Effect checks (report each as it lands)

  1. Summaries lane registered, held behind the rehearsal. summarize.yml appears in Actions once this merges:
    gh workflow list --all | grep -i summar
    
    Then the staging rehearsal — the lane's first real model call:
    gh workflow run summarize.yml --ref staging -f limit=10
    
    Expected: plan reports ~10 planned with a cost range; the run waits on review; after release generate reports ~10 written with cost from response usage (≈$1–2 total), files in the case-summaries artifact, no PR. The summaries: plain-language case summaries per snapshot #1968 spot check (~10 summaries against their filings, factual errors and slant) is done on that artifact.
    Until the spot check passes, reject the daily main run's review hold (03:43 UTC) — it will plan the full ~195-case backfill. Once released after the spot check, it opens a summaries/refresh PR with ~195 files:
    gh run list --workflow summarize.yml --branch main --limit 3
    gh pr list --head summaries/refresh --state open --json number,title,url
    
  2. Weekly digest (feat(ops): restructure the weekly digest around three production windows #1958). The Monday 2026-09-28 digest opens with ## Produced this week, then ## Produced this month and ## Produced this term (OT2025, …) before ## Backtest results, with no ## Health questions or ## Analytics state.
  3. Back-test lanes (perf(backtest): run each engine's replay cells in its own lane #1974) and disclosure record (feat(backtest): per-cell leakage disclosure record #1972, fix(backtest): a failed engine call is one lost cell, and a spent quota stops that engine — never the campaign #1959). On the 2026-10-03 run-backtest cron (spread, once released):
    gh run list --workflow run-backtest.yml --limit 5
    gh run view <run-id> --json jobs --jq '.jobs[] | select(.name == "backtest") | ((.completedAt | fromdateiso8601) - (.startedAt | fromdateiso8601)) / 60'
    gh run view <run-id> --json jobs --jq '.jobs[] | select(.name == "backtest") | .databaseId'
    gh run view <run-id> --log --job <backtest-job-id> | grep -oE '\[(claude-code|codex|gemini)\] ' | sort | uniq -c
    
    Expected: all three engine prefixes interleaved in the log; the backtest job well under the 330-minute ceiling (~40–60 min of cells at 10 petitions); no ::error::cert-backtest annotation and no harness-error in provenance.lost_cells; the report carries provenance.disclosures and provenance.disclosure_tally. On a day gemini's allowance is spent, gemini's cells read quota-exhausted in lost_cells and the job log prints the quota drop line, with the campaign completing.
  4. Opus labeler option (ci(qp-topic-label): offer claude-opus-5-5; claude CLI pin → 2.1.280 #1962). On a prod qp-topic-label dispatch with label_model=claude-opus-5-5, the labels artifact's batch records labeler: claude-code/claude-opus-5-5. (Rehearse on staging first if wanted.)
  5. Deps (chore(deps): Bump the python group with 3 updates #1970). No run-time observable beyond a green gate; pypdf's extraction change is limited to Arabic-Indic digits.

🤖 Generated with Claude Code

modelmirror and others added 24 commits September 21, 2026 19:44
The weekly performance digest drops its Health questions and Analytics
state blocks — the boards are read on the website, and the health bullets
restated the production section — and reports production over three
windows instead: the week, the trailing month, and the October Term to
date. Each block is one shape (cells, events, measured spend, role x stage
table) so the three are read the same way.

The month block's window is the ex-post spend backstop's own, taken from
its config, and the backstop's verdict closes that block: the verdict and
the census printed beside it can then never describe different periods.
The Term block closes with the forward cells scored under the process in
force — a forward cell is minted once at its event and never again, so the
Term is the period that count is worth reading over — stated as a figure,
with the frozen-shakedown state named rather than shown as a bare zero.

`cell_census` and `spend_over` take a `since` cutoff so the Term window can
be pinned to the instant its Term opened: an October Term opens at midnight
on 1 October and the digest renders mid-morning, so a day count would cut
the Term's own first morning out of its own census.

Artifact: the `weekly-digest` issue run-ops opens on the Monday tick.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The weekly performance digest now reads: produced this week, produced this
month (the spend backstop's own trailing window, closing with its verdict),
produced this term (from the Term's 1 October, pinned to that instant rather
than counted back in days, closing with the forward cells scored under the
process in force, stated as a ledger-to-date figure), then the back-test
results unchanged. The health questions and the analytics-state block are
gone: the boards are read on the site, and the health bullets restated the
production block. The ops report, the daily digest and every process-digest
input are untouched.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ed-windows

feat(ops): restructure the weekly digest around three production windows
…aign

A cert back-test campaign ended with no report when one engine's daily
quota ran out: the CLI answered 429 on every retry, the runner classified
it transient, exhausted its retry budget and raised `EngineFailed`, and
`_replay_item_cells` — which absorbed only `EngineUnavailable` — let it
escape, discarding every finished cell with the work root.

Three changes, so a campaign always lands `metrics/cert-backtest.json`:

- `_replay_item_cells` absorbs `EngineFailed` as one lost cell, recorded
  under `provenance.lost_cells` with the new `engine-failed` reason and
  printed as it happens, exactly as an unreadable artifact is.
- The runner classifies a terminal quota marker (the `gemini` CLI's
  `TerminalQuotaError` and the prose it prints with it) as non-transient,
  so the cell fails after one attempt rather than after the backoff
  budget, and raises `EngineQuotaExhausted` naming the cause. An ordinary
  bare-429 throttle keeps its retries.
- Once an engine reports its quota spent, the campaign stops attempting
  that engine's cells and records each under the `quota-exhausted`
  reason, so the report's counts stay honest while no further backoff is
  spent on a certain failure. Tracked apart from `unavailable`: an engine
  whose binary was missing and one whose allowance ran out are different
  facts, and the report says which.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…d-is-a-lost-cell

fix(backtest): a failed engine call is one lost cell, and a spent quota stops that engine — never the campaign
chore: sync main into staging
… to 2.1.280

Opus 5.5 needs Claude Code 2.1.280 or newer; the pin moves in step across
run-analytics, run-backtest and integration-test.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ci(qp-topic-label): offer claude-opus-5-5; claude CLI pin → 2.1.280
chore: sync main into staging
Read each replayed cell's flags.json back beside its prediction and record
it on the report as provenance.disclosures (category, severity and a text
rule's exposure-candidate reading per note) with a per-predictor tally. The
note text goes to the run log, credential-redacted, never into metrics/.
Nothing is excluded: the back-test runs no evaluator, so the reading rules
state the direction and the digest line and review PR name the candidates.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Bumps the python group with 3 updates: [boto3](https://github.com/boto/boto3), [pypdf](https://github.com/py-pdf/pypdf) and [ruff](https://github.com/astral-sh/ruff).


Updates `boto3` from 1.43.93 to 1.43.98
- [Release notes](https://github.com/boto/boto3/releases)
- [Commits](boto/boto3@1.43.93...1.43.98)

Updates `pypdf` from 6.18.1 to 6.19.0
- [Release notes](https://github.com/py-pdf/pypdf/releases)
- [Changelog](https://github.com/py-pdf/pypdf/blob/main/CHANGELOG.md)
- [Commits](py-pdf/pypdf@6.18.1...6.19.0)

Updates `ruff` from 0.16.7 to 0.16.8
- [Release notes](https://github.com/astral-sh/ruff/releases)
- [Changelog](https://github.com/astral-sh/ruff/blob/main/CHANGELOG.md)
- [Commits](astral-sh/ruff@0.16.7...0.16.8)

---
updated-dependencies:
- dependency-name: boto3
  dependency-version: 1.43.98
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: python
- dependency-name: pypdf
  dependency-version: 6.19.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: python
- dependency-name: ruff
  dependency-version: 0.16.8
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: python
...

Signed-off-by: dependabot[bot] <support@github.com>
… candidates

Resolve review findings: a lost cell's exposure candidate is named on the
review PR in its own clause, the headline carries direction and unreadable
notes, scored is settled against the final board, the rule's vocabulary and
its comma-split miss are stated and pinned, and the digest says an unmarked
note is not a cleared one.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A content-keyed summary lane: summarize-plan derives the predicted cases whose
newest record differs from the one their newest committed summary was written
from; summarize writes one tool-less Messages API call's validated body per
case under harness front matter; summarize.yml runs it behind the review hold
with the corpus role and a dedicated API key in separate jobs, publishing a
reviewed summaries/refresh PR on main.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Reject markup in summary bodies, summarize only supremecourt.gov-shaped
records, add a time budget and retry-after handling, skip an unreadable
response per case, pin the lane in the workflow invariant tests, and bring
the security, data-sources and pipeline docs in step.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Provision every petition's case tree first, then run one worker per engine,
each walking all petitions' cells for that engine serially, the lanes at once.
Same-engine cells never overlap (codex's per-process auth home), quota and
missing-binary state is lane-local, results merge deterministically, and an
unexpected fault becomes harness-error losses for that lane's remaining cells.
Lane output is engine-prefixed; --workers 1 runs the lanes serially. The
run-backtest ceiling comment states the lane wall clock; the cap is unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…hon-50ed3bdc63

chore(deps): Bump the python group with 3 updates
…he measured basis

Resolve review findings: pump threads are daemons joined with a bound and
keep draining on a failed write; a harness-error loss prints an ::error::
annotation after the report is written; the concurrency test drives several
petitions per lane; bounded-width, mid-petition and absorption-fault paths
are tested; docs and the workflow comment give the claude rate, the hung-lane
limit, and soften the log-prefix claim.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The lane shares the per-environment Anthropic key the other Claude lanes
use (prod's on main, staging's on a staging run) rather than a dedicated
secret; the key still reaches only the generate job's two steps.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Resolve the stats review: lanes finish petitions at different speeds, so in
one shared tree which peer forecasts sat beside a cell depended on timing.
Each lane now runs under <work-dir>/<engine>/, provisioned with identical
inputs, which also narrows gemini's admitted directory to gemini's cells.
State the wall-clock basis with its n and range, say the harness-fault
annotation does not fail the step, and note the bounded stderr drain.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
feat(summaries): plain-language case summaries per snapshot record
…closure

feat(backtest): per-cell leakage disclosure record
perf(backtest): run each engine's replay cells in its own lane
chore: sync main into staging
chore: sync main into staging
@modelmirror modelmirror added the promote:skip-engine-smoke Promotion PR: waive the engine smokes. Batch merges with no real-engine evidence. label Sep 23, 2026
@modelmirror
modelmirror merged commit 32e0804 into main Sep 23, 2026
19 checks passed

Copy link
Copy Markdown
Collaborator Author

Tagged promotion/2026-09-23 (annotated) on merge commit 32e0804. Effect check 1, first half: gh workflow list --all | grep -i summar → summarize active 365433562 — the lane is registered. The staging rehearsal and the other checks are pending; each is reported here as it lands.

This branch was successfully deployed

2 active deployments
staging — c9bce058 Deployed Sep 23, 2026 by modelmirror via generate #2
review — c9bce058 Deployed Sep 23, 2026 by modelmirror via approval #2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

promote:skip-engine-smoke Promotion PR: waive the engine smokes. Batch merges with no real-engine evidence.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant