From 610e05b7a4a69190ea106762e7b1fec2ef99b3c9 Mon Sep 17 00:00:00 2001 From: Matt Miller Date: Wed, 7 Oct 2026 15:46:53 -0700 Subject: [PATCH] feat(cursor-review): read the panel list from CURSOR_PANEL_MODELS, so one org variable drives the fleet MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `panel_models` was an input only, so overriding the panel meant editing every caller to wire a variable through — #367 did that for this repo's caller and nothing else, which is not what "a global thing you can reference and override" means. The reusable now falls back to the CURSOR_PANEL_MODELS variable when the caller passes no input. `vars` inside a reusable resolves against the CALLER's repo and org (verified: CURSOR_REVIEW_DM_EMAIL_MAP is set on a caller, not here, and works), so: - set it at the org -> every enrolled caller picks it up, no edits - set it on a repo -> that repo wins, org value untouched - unset at both -> the built-in list, i.e. today's behaviour - delete to revert -> no PR, no fleet bump The `panel_models` input still exists and takes precedence, for a caller that wants its own list regardless of any variable. Two follow-ons: - This repo's own caller no longer wires the variable through an input, which would now be redundant: -44 lines of comment, replaced by a pointer to the docs. Its panel is driven by the same variable as everyone else's. - The round log names WHICH source won, not just the effective list. With two sources, the list alone no longer says where to change it. Unchanged: an override still forfeits catalog-drift coverage, since extract_panel_models() reads the heredoc. The docs say so, and say to fold a proven list back into the heredoc and delete the variable. Co-Authored-By: Claude Opus 5 --- .github/workflows/ci-cursor-review.yml | 50 +++------------------- .github/workflows/cursor-review.yml | 23 ++++++++-- docs/callers/cursor-review.md | 58 +++++++++++++------------- 3 files changed, 55 insertions(+), 76 deletions(-) diff --git a/.github/workflows/ci-cursor-review.yml b/.github/workflows/ci-cursor-review.yml index 02c8170a..823dd4d9 100644 --- a/.github/workflows/ci-cursor-review.yml +++ b/.github/workflows/ci-cursor-review.yml @@ -37,50 +37,12 @@ jobs: with: # Keep the assets ref in lock-step with the `uses:` ref above. workflows_ref: eef55d3565eef918d29cdfc076bbfb51d12a8e32 - # Panel model override, as a VARIABLE so a reasoning-tier A/B needs no PR - # and no fleet bump — set it, watch a few rounds, unset it to revert. The - # reusable's `panel_models` treats empty as "use the built-in list", so an - # unset variable is exactly today's behaviour; a repo variable overrides an - # org one of the same name, so this can be set org-wide later without - # touching this file. - # - # The trade-off this buys, stated plainly: panel composition becomes - # mutable state OUTSIDE the SHA-pinned tree the header above argues for. - # While the variable is set, nothing in review, CODEOWNERS or git records - # which models judged a PR — the run log is the only trace — and anyone - # with variable write can narrow the panel to one cheap model. At ORG - # level it retargets every enrolled caller at once. That write is - # admin-only, not open to anyone who can push, which is why a variable is - # acceptable here at all; keep it repo-level while experimenting, and - # delete it rather than leaving a stale list in force. - # - # The value is a JSON array of CURSOR CATALOG ids — tier suffix and all - # (`claude-opus-5-thinking-xhigh`, not `claude-opus-5`). It is deliberately - # NOT the org's `CLAUDE_MODEL_*` variables: those hold Anthropic API slugs, - # which the preflight's live-catalog check would reject. Keep the two - # namespaces apart. - # - # Preflight checks that every id EXISTS in the live catalog and fails loud - # before any cell spends, so a typo costs nothing. It does NOT check ZDR - # eligibility — an id on a catalog line reading `NO ZDR` matches like any - # other, and `catalog-drift.py` (which does detect NO-ZDR) parses the - # built-in heredoc and never sees an overridden list. Confirm ZDR yourself - # before setting this. - # - # The documented first experiment is a SINGLE-model tier drop: the - # Anthropic cell from `-max` to `-xhigh` (the judge already made that move - # in #243 on spend grounds), with the other two cells held fixed so the - # result is attributable: - # gh variable set CURSOR_PANEL_MODELS \ - # --repo Comfy-Org/github-workflows \ - # --body '["gpt-5.6-sol-max","claude-opus-5-thinking-xhigh","kimi-k3-high"]' - # - # `judge_model` is deliberately NOT wired the same way. Its input carries a - # real default that catalog-drift.py parses out of cursor-review.yml, and a - # caller passing an explicitly-empty value gets '' rather than that default - # — so the override would have to restate the default here and drift from - # it. Changing that belongs in the reusable, not in a caller. - panel_models: ${{ vars.CURSOR_PANEL_MODELS }} + # No `panel_models:` here on purpose. The reusable now reads the + # CURSOR_PANEL_MODELS variable itself, resolved against this repo and the + # org, so wiring the same variable through an input would be redundant. + # Set the variable to run a panel experiment; see + # docs/callers/cursor-review.md "Running a model experiment" for the + # catalog-id and ZDR caveats. # No vendored/generated content in this repo; defensive excludes only. # These live in extra_generated_globs, not diff_excludes: since BE-7633 # diff_excludes leaves the reviewed diff but is still COUNTED against diff --git a/.github/workflows/cursor-review.yml b/.github/workflows/cursor-review.yml index 6216414e..576f4e42 100644 --- a/.github/workflows/cursor-review.yml +++ b/.github/workflows/cursor-review.yml @@ -1953,7 +1953,19 @@ jobs: # `-xhigh` come from the same spend review. The `panel_models` input # overrides this list per caller for A/B experiments. env: - PANEL_MODELS_OVERRIDE: ${{ inputs.panel_models }} + # Falls back to a VARIABLE when the caller passes no input, so the + # panel list is settable in ONE place for the whole fleet with no + # caller edit. `vars` inside a reusable resolves against the CALLER's + # repo and org — same as CURSOR_REVIEW_DM_EMAIL_MAP further down — so + # an org-level CURSOR_PANEL_MODELS reaches every enrolled repo, and a + # repo-level one of the same name overrides it for that repo alone. + # Unset at both levels is '' → the built-in list below. An explicit + # `panel_models` input still wins, so a caller pinning its own list is + # unaffected. + PANEL_MODELS_OVERRIDE: ${{ inputs.panel_models || vars.CURSOR_PANEL_MODELS }} + # Which of the two won, for the log line below. With two sources the + # effective list alone no longer says where to go change it. + PANEL_MODELS_INPUT: ${{ inputs.panel_models }} run: | cat > /tmp/models.json <<'JSON' [ @@ -1989,7 +2001,12 @@ jobs: exit 1 fi printf '%s' "$PANEL_MODELS_OVERRIDE" | jq -c -s '.[0] | unique' > /tmp/models.json - echo "panel_models override in effect: $(cat /tmp/models.json)" + if [ -n "$(printf '%s' "$PANEL_MODELS_INPUT" | tr -d '[:space:]')" ]; then + SRC="panel_models input" + else + SRC="CURSOR_PANEL_MODELS variable" + fi + echo "panel_models override in effect (from the $SRC): $(cat /tmp/models.json)" else # Logged in BOTH branches so the run says which list produced the # review. A caller driving panel_models from a variable makes the @@ -1997,7 +2014,7 @@ jobs: # line a mid-experiment deletion is indistinguishable in the log # from one that was never set — the panel quietly reverts and the # rounds either side of it look the same. - echo "panel_models not set — using the built-in list: $(jq -c . /tmp/models.json)" + echo "panel_models not set (no input, no CURSOR_PANEL_MODELS variable) — using the built-in list: $(jq -c . /tmp/models.json)" fi echo "models=$(jq -c . /tmp/models.json)" >> "$GITHUB_OUTPUT" diff --git a/docs/callers/cursor-review.md b/docs/callers/cursor-review.md index 440916d8..2c007db4 100644 --- a/docs/callers/cursor-review.md +++ b/docs/callers/cursor-review.md @@ -134,20 +134,29 @@ pull-requests: write # posting the consolidated review (and, at the round cap, ## Running a model experiment -`panel_models` replaces the built-in panel list. Wire it to a **variable** -rather than a literal, so a reasoning-tier or model-version A/B needs no PR and -no SHA-bump across the fleet: +**No caller change is needed.** The reusable reads the variable +`CURSOR_PANEL_MODELS` itself, so setting it is the whole operation: -```yaml - with: - panel_models: ${{ vars.CURSOR_PANEL_MODELS }} +```bash +# one repo +gh variable set CURSOR_PANEL_MODELS --repo / \ + --body '["gpt-5.6-sol-max","claude-opus-5-thinking-xhigh","kimi-k3-high"]' + +# or the whole fleet at once (visibility must include private repos) +gh variable set CURSOR_PANEL_MODELS --org --visibility all \ + --body '["gpt-5.6-sol-max","claude-opus-5-thinking-xhigh","kimi-k3-high"]' ``` -An unset variable is exactly today's behaviour — the reusable reads empty as -"use the built-in list" — so the wiring is inert until you set it, and deleting -the variable (or every variable) falls back rather than failing. The run log -names which list is in effect either way, so a deletion mid-experiment is -visible instead of looking like a round that was never overridden. +`vars` inside a reusable workflow resolves against the **caller's** repo and +org, so an org-level value reaches every enrolled caller, and a repo-level value +of the same name wins for that repo alone. Unset at both levels, the panel runs +the built-in list — so the mechanism is inert until you use it, and `gh variable +delete` is the revert. The run log names which source is in effect on every +round, so a deletion mid-experiment is visible rather than looking like a round +that was never overridden. + +The `panel_models` **input** still exists and takes precedence, for a caller +that wants to pin its own list regardless of what any variable says. **Any override forfeits drift coverage, variable included.** `catalog-drift.py` parses the panel pins out of the heredoc in @@ -156,25 +165,16 @@ auditing the built-in list while the panel runs yours. A variable is not in the tracked tree at all — no review, no grep, no diff. Watch the catalog yourself for the duration of an experiment. -What the variable buys over a literal is therefore not auditing — it is the -revert, and where the fallback lands. Unset, the panel runs the built-in list, -which *is* the drift-audited one, so an experiment ends with a `gh variable -delete` and reverts back into coverage. A caller literal **is** the fallback: -there is nothing to delete, it needs a PR plus a fleet SHA bump to change, and -it holds its snapshot unaudited while the shared default moves on without it. -So prefer the variable, and treat a set variable as temporary by default. - -A **repo** variable overrides an **org** variable of the same name, so one repo -can trial a list the rest of the org is not on, and the org value can be set -later without touching any caller. Note what an org-level value means: it -retargets every enrolled caller at once, and panel composition then lives in -mutable state outside the SHA-pinned tree every caller otherwise pins, with the -run log as the only record of which models judged a given PR. +So treat a set variable as temporary, and once a list has proven itself, move it +into the built-in heredoc and delete the variable — that puts it back under +drift coverage for everyone. -```bash -gh variable set CURSOR_PANEL_MODELS --repo / \ - --body '["gpt-5.6-sol-max","claude-opus-5-thinking-xhigh","kimi-k3-high"]' -``` +Two further consequences of the variable being caller-side, worth knowing rather +than discovering: an **org** value retargets every enrolled caller at once, and +any caller repo's admin can set a **repo** value to narrow the panel judging +their own PRs — with no PR, no CODEOWNERS review and the run log as the only +record. That write is repo-admin, who can already swap the pin or delete the +caller outright, so it is not a new trust boundary; it is a quieter one. Two things to get right: