diff --git a/.github/workflows/ci-cursor-review.yml b/.github/workflows/ci-cursor-review.yml index 02c8170a..823dd4d9 100644 --- a/.github/workflows/ci-cursor-review.yml +++ b/.github/workflows/ci-cursor-review.yml @@ -37,50 +37,12 @@ jobs: with: # Keep the assets ref in lock-step with the `uses:` ref above. workflows_ref: eef55d3565eef918d29cdfc076bbfb51d12a8e32 - # Panel model override, as a VARIABLE so a reasoning-tier A/B needs no PR - # and no fleet bump — set it, watch a few rounds, unset it to revert. The - # reusable's `panel_models` treats empty as "use the built-in list", so an - # unset variable is exactly today's behaviour; a repo variable overrides an - # org one of the same name, so this can be set org-wide later without - # touching this file. - # - # The trade-off this buys, stated plainly: panel composition becomes - # mutable state OUTSIDE the SHA-pinned tree the header above argues for. - # While the variable is set, nothing in review, CODEOWNERS or git records - # which models judged a PR — the run log is the only trace — and anyone - # with variable write can narrow the panel to one cheap model. At ORG - # level it retargets every enrolled caller at once. That write is - # admin-only, not open to anyone who can push, which is why a variable is - # acceptable here at all; keep it repo-level while experimenting, and - # delete it rather than leaving a stale list in force. - # - # The value is a JSON array of CURSOR CATALOG ids — tier suffix and all - # (`claude-opus-5-thinking-xhigh`, not `claude-opus-5`). It is deliberately - # NOT the org's `CLAUDE_MODEL_*` variables: those hold Anthropic API slugs, - # which the preflight's live-catalog check would reject. Keep the two - # namespaces apart. - # - # Preflight checks that every id EXISTS in the live catalog and fails loud - # before any cell spends, so a typo costs nothing. It does NOT check ZDR - # eligibility — an id on a catalog line reading `NO ZDR` matches like any - # other, and `catalog-drift.py` (which does detect NO-ZDR) parses the - # built-in heredoc and never sees an overridden list. Confirm ZDR yourself - # before setting this. - # - # The documented first experiment is a SINGLE-model tier drop: the - # Anthropic cell from `-max` to `-xhigh` (the judge already made that move - # in #243 on spend grounds), with the other two cells held fixed so the - # result is attributable: - # gh variable set CURSOR_PANEL_MODELS \ - # --repo Comfy-Org/github-workflows \ - # --body '["gpt-5.6-sol-max","claude-opus-5-thinking-xhigh","kimi-k3-high"]' - # - # `judge_model` is deliberately NOT wired the same way. Its input carries a - # real default that catalog-drift.py parses out of cursor-review.yml, and a - # caller passing an explicitly-empty value gets '' rather than that default - # — so the override would have to restate the default here and drift from - # it. Changing that belongs in the reusable, not in a caller. - panel_models: ${{ vars.CURSOR_PANEL_MODELS }} + # No `panel_models:` here on purpose. The reusable now reads the + # CURSOR_PANEL_MODELS variable itself, resolved against this repo and the + # org, so wiring the same variable through an input would be redundant. + # Set the variable to run a panel experiment; see + # docs/callers/cursor-review.md "Running a model experiment" for the + # catalog-id and ZDR caveats. # No vendored/generated content in this repo; defensive excludes only. # These live in extra_generated_globs, not diff_excludes: since BE-7633 # diff_excludes leaves the reviewed diff but is still COUNTED against diff --git a/.github/workflows/cursor-review.yml b/.github/workflows/cursor-review.yml index 6216414e..576f4e42 100644 --- a/.github/workflows/cursor-review.yml +++ b/.github/workflows/cursor-review.yml @@ -1953,7 +1953,19 @@ jobs: # `-xhigh` come from the same spend review. The `panel_models` input # overrides this list per caller for A/B experiments. env: - PANEL_MODELS_OVERRIDE: ${{ inputs.panel_models }} + # Falls back to a VARIABLE when the caller passes no input, so the + # panel list is settable in ONE place for the whole fleet with no + # caller edit. `vars` inside a reusable resolves against the CALLER's + # repo and org — same as CURSOR_REVIEW_DM_EMAIL_MAP further down — so + # an org-level CURSOR_PANEL_MODELS reaches every enrolled repo, and a + # repo-level one of the same name overrides it for that repo alone. + # Unset at both levels is '' → the built-in list below. An explicit + # `panel_models` input still wins, so a caller pinning its own list is + # unaffected. + PANEL_MODELS_OVERRIDE: ${{ inputs.panel_models || vars.CURSOR_PANEL_MODELS }} + # Which of the two won, for the log line below. With two sources the + # effective list alone no longer says where to go change it. + PANEL_MODELS_INPUT: ${{ inputs.panel_models }} run: | cat > /tmp/models.json <<'JSON' [ @@ -1989,7 +2001,12 @@ jobs: exit 1 fi printf '%s' "$PANEL_MODELS_OVERRIDE" | jq -c -s '.[0] | unique' > /tmp/models.json - echo "panel_models override in effect: $(cat /tmp/models.json)" + if [ -n "$(printf '%s' "$PANEL_MODELS_INPUT" | tr -d '[:space:]')" ]; then + SRC="panel_models input" + else + SRC="CURSOR_PANEL_MODELS variable" + fi + echo "panel_models override in effect (from the $SRC): $(cat /tmp/models.json)" else # Logged in BOTH branches so the run says which list produced the # review. A caller driving panel_models from a variable makes the @@ -1997,7 +2014,7 @@ jobs: # line a mid-experiment deletion is indistinguishable in the log # from one that was never set — the panel quietly reverts and the # rounds either side of it look the same. - echo "panel_models not set — using the built-in list: $(jq -c . /tmp/models.json)" + echo "panel_models not set (no input, no CURSOR_PANEL_MODELS variable) — using the built-in list: $(jq -c . /tmp/models.json)" fi echo "models=$(jq -c . /tmp/models.json)" >> "$GITHUB_OUTPUT" diff --git a/docs/callers/cursor-review.md b/docs/callers/cursor-review.md index 440916d8..2c007db4 100644 --- a/docs/callers/cursor-review.md +++ b/docs/callers/cursor-review.md @@ -134,20 +134,29 @@ pull-requests: write # posting the consolidated review (and, at the round cap, ## Running a model experiment -`panel_models` replaces the built-in panel list. Wire it to a **variable** -rather than a literal, so a reasoning-tier or model-version A/B needs no PR and -no SHA-bump across the fleet: +**No caller change is needed.** The reusable reads the variable +`CURSOR_PANEL_MODELS` itself, so setting it is the whole operation: -```yaml - with: - panel_models: ${{ vars.CURSOR_PANEL_MODELS }} +```bash +# one repo +gh variable set CURSOR_PANEL_MODELS --repo / \ + --body '["gpt-5.6-sol-max","claude-opus-5-thinking-xhigh","kimi-k3-high"]' + +# or the whole fleet at once (visibility must include private repos) +gh variable set CURSOR_PANEL_MODELS --org --visibility all \ + --body '["gpt-5.6-sol-max","claude-opus-5-thinking-xhigh","kimi-k3-high"]' ``` -An unset variable is exactly today's behaviour — the reusable reads empty as -"use the built-in list" — so the wiring is inert until you set it, and deleting -the variable (or every variable) falls back rather than failing. The run log -names which list is in effect either way, so a deletion mid-experiment is -visible instead of looking like a round that was never overridden. +`vars` inside a reusable workflow resolves against the **caller's** repo and +org, so an org-level value reaches every enrolled caller, and a repo-level value +of the same name wins for that repo alone. Unset at both levels, the panel runs +the built-in list — so the mechanism is inert until you use it, and `gh variable +delete` is the revert. The run log names which source is in effect on every +round, so a deletion mid-experiment is visible rather than looking like a round +that was never overridden. + +The `panel_models` **input** still exists and takes precedence, for a caller +that wants to pin its own list regardless of what any variable says. **Any override forfeits drift coverage, variable included.** `catalog-drift.py` parses the panel pins out of the heredoc in @@ -156,25 +165,16 @@ auditing the built-in list while the panel runs yours. A variable is not in the tracked tree at all — no review, no grep, no diff. Watch the catalog yourself for the duration of an experiment. -What the variable buys over a literal is therefore not auditing — it is the -revert, and where the fallback lands. Unset, the panel runs the built-in list, -which *is* the drift-audited one, so an experiment ends with a `gh variable -delete` and reverts back into coverage. A caller literal **is** the fallback: -there is nothing to delete, it needs a PR plus a fleet SHA bump to change, and -it holds its snapshot unaudited while the shared default moves on without it. -So prefer the variable, and treat a set variable as temporary by default. - -A **repo** variable overrides an **org** variable of the same name, so one repo -can trial a list the rest of the org is not on, and the org value can be set -later without touching any caller. Note what an org-level value means: it -retargets every enrolled caller at once, and panel composition then lives in -mutable state outside the SHA-pinned tree every caller otherwise pins, with the -run log as the only record of which models judged a given PR. +So treat a set variable as temporary, and once a list has proven itself, move it +into the built-in heredoc and delete the variable — that puts it back under +drift coverage for everyone. -```bash -gh variable set CURSOR_PANEL_MODELS --repo / \ - --body '["gpt-5.6-sol-max","claude-opus-5-thinking-xhigh","kimi-k3-high"]' -``` +Two further consequences of the variable being caller-side, worth knowing rather +than discovering: an **org** value retargets every enrolled caller at once, and +any caller repo's admin can set a **repo** value to narrow the panel judging +their own PRs — with no PR, no CODEOWNERS review and the run log as the only +record. That write is repo-admin, who can already swap the pin or delete the +caller outright, so it is not a new trust boundary; it is a quieter one. Two things to get right: