Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 6 additions & 44 deletions .github/workflows/ci-cursor-review.yml
Original file line number Diff line number Diff line change
Expand Up @@ -37,50 +37,12 @@ jobs:
with:
# Keep the assets ref in lock-step with the `uses:` ref above.
workflows_ref: eef55d3565eef918d29cdfc076bbfb51d12a8e32
# Panel model override, as a VARIABLE so a reasoning-tier A/B needs no PR
# and no fleet bump — set it, watch a few rounds, unset it to revert. The
# reusable's `panel_models` treats empty as "use the built-in list", so an
# unset variable is exactly today's behaviour; a repo variable overrides an
# org one of the same name, so this can be set org-wide later without
# touching this file.
#
# The trade-off this buys, stated plainly: panel composition becomes
# mutable state OUTSIDE the SHA-pinned tree the header above argues for.
# While the variable is set, nothing in review, CODEOWNERS or git records
# which models judged a PR — the run log is the only trace — and anyone
# with variable write can narrow the panel to one cheap model. At ORG
# level it retargets every enrolled caller at once. That write is
# admin-only, not open to anyone who can push, which is why a variable is
# acceptable here at all; keep it repo-level while experimenting, and
# delete it rather than leaving a stale list in force.
#
# The value is a JSON array of CURSOR CATALOG ids — tier suffix and all
# (`claude-opus-5-thinking-xhigh`, not `claude-opus-5`). It is deliberately
# NOT the org's `CLAUDE_MODEL_*` variables: those hold Anthropic API slugs,
# which the preflight's live-catalog check would reject. Keep the two
# namespaces apart.
#
# Preflight checks that every id EXISTS in the live catalog and fails loud
# before any cell spends, so a typo costs nothing. It does NOT check ZDR
# eligibility — an id on a catalog line reading `NO ZDR` matches like any
# other, and `catalog-drift.py` (which does detect NO-ZDR) parses the
# built-in heredoc and never sees an overridden list. Confirm ZDR yourself
# before setting this.
#
# The documented first experiment is a SINGLE-model tier drop: the
# Anthropic cell from `-max` to `-xhigh` (the judge already made that move
# in #243 on spend grounds), with the other two cells held fixed so the
# result is attributable:
# gh variable set CURSOR_PANEL_MODELS \
# --repo Comfy-Org/github-workflows \
# --body '["gpt-5.6-sol-max","claude-opus-5-thinking-xhigh","kimi-k3-high"]'
#
# `judge_model` is deliberately NOT wired the same way. Its input carries a
# real default that catalog-drift.py parses out of cursor-review.yml, and a
# caller passing an explicitly-empty value gets '' rather than that default
# — so the override would have to restate the default here and drift from
# it. Changing that belongs in the reusable, not in a caller.
panel_models: ${{ vars.CURSOR_PANEL_MODELS }}
# No `panel_models:` here on purpose. The reusable now reads the
# CURSOR_PANEL_MODELS variable itself, resolved against this repo and the
# org, so wiring the same variable through an input would be redundant.
# Set the variable to run a panel experiment; see
# docs/callers/cursor-review.md "Running a model experiment" for the
# catalog-id and ZDR caveats.
# No vendored/generated content in this repo; defensive excludes only.
# These live in extra_generated_globs, not diff_excludes: since BE-7633
# diff_excludes leaves the reviewed diff but is still COUNTED against
Expand Down
23 changes: 20 additions & 3 deletions .github/workflows/cursor-review.yml
Original file line number Diff line number Diff line change
Expand Up @@ -1953,7 +1953,19 @@ jobs:
# `-xhigh` come from the same spend review. The `panel_models` input
# overrides this list per caller for A/B experiments.
env:
PANEL_MODELS_OVERRIDE: ${{ inputs.panel_models }}
# Falls back to a VARIABLE when the caller passes no input, so the
# panel list is settable in ONE place for the whole fleet with no
# caller edit. `vars` inside a reusable resolves against the CALLER's
# repo and org — same as CURSOR_REVIEW_DM_EMAIL_MAP further down — so
# an org-level CURSOR_PANEL_MODELS reaches every enrolled repo, and a
# repo-level one of the same name overrides it for that repo alone.
# Unset at both levels is '' → the built-in list below. An explicit
# `panel_models` input still wins, so a caller pinning its own list is
# unaffected.
PANEL_MODELS_OVERRIDE: ${{ inputs.panel_models || vars.CURSOR_PANEL_MODELS }}
# Which of the two won, for the log line below. With two sources the
# effective list alone no longer says where to go change it.
PANEL_MODELS_INPUT: ${{ inputs.panel_models }}
run: |
cat > /tmp/models.json <<'JSON'
[
Expand Down Expand Up @@ -1989,15 +2001,20 @@ jobs:
exit 1
fi
printf '%s' "$PANEL_MODELS_OVERRIDE" | jq -c -s '.[0] | unique' > /tmp/models.json
echo "panel_models override in effect: $(cat /tmp/models.json)"
if [ -n "$(printf '%s' "$PANEL_MODELS_INPUT" | tr -d '[:space:]')" ]; then
SRC="panel_models input"
else
SRC="CURSOR_PANEL_MODELS variable"
fi
echo "panel_models override in effect (from the $SRC): $(cat /tmp/models.json)"
else
# Logged in BOTH branches so the run says which list produced the
# review. A caller driving panel_models from a variable makes the
# empty case reachable by DELETING that variable, and without this
# line a mid-experiment deletion is indistinguishable in the log
# from one that was never set — the panel quietly reverts and the
# rounds either side of it look the same.
echo "panel_models not set — using the built-in list: $(jq -c . /tmp/models.json)"
echo "panel_models not set (no input, no CURSOR_PANEL_MODELS variable) — using the built-in list: $(jq -c . /tmp/models.json)"
fi
echo "models=$(jq -c . /tmp/models.json)" >> "$GITHUB_OUTPUT"

Expand Down
58 changes: 29 additions & 29 deletions docs/callers/cursor-review.md
Original file line number Diff line number Diff line change
Expand Up @@ -134,20 +134,29 @@ pull-requests: write # posting the consolidated review (and, at the round cap,

## Running a model experiment

`panel_models` replaces the built-in panel list. Wire it to a **variable**
rather than a literal, so a reasoning-tier or model-version A/B needs no PR and
no SHA-bump across the fleet:
**No caller change is needed.** The reusable reads the variable
`CURSOR_PANEL_MODELS` itself, so setting it is the whole operation:

```yaml
with:
panel_models: ${{ vars.CURSOR_PANEL_MODELS }}
```bash
# one repo
gh variable set CURSOR_PANEL_MODELS --repo <owner>/<repo> \
--body '["gpt-5.6-sol-max","claude-opus-5-thinking-xhigh","kimi-k3-high"]'

# or the whole fleet at once (visibility must include private repos)
gh variable set CURSOR_PANEL_MODELS --org <org> --visibility all \
--body '["gpt-5.6-sol-max","claude-opus-5-thinking-xhigh","kimi-k3-high"]'
```

An unset variable is exactly today's behaviour — the reusable reads empty as
"use the built-in list" — so the wiring is inert until you set it, and deleting
the variable (or every variable) falls back rather than failing. The run log
names which list is in effect either way, so a deletion mid-experiment is
visible instead of looking like a round that was never overridden.
`vars` inside a reusable workflow resolves against the **caller's** repo and
org, so an org-level value reaches every enrolled caller, and a repo-level value
of the same name wins for that repo alone. Unset at both levels, the panel runs
the built-in list — so the mechanism is inert until you use it, and `gh variable
delete` is the revert. The run log names which source is in effect on every
round, so a deletion mid-experiment is visible rather than looking like a round
that was never overridden.

The `panel_models` **input** still exists and takes precedence, for a caller
that wants to pin its own list regardless of what any variable says.

**Any override forfeits drift coverage, variable included.**
`catalog-drift.py` parses the panel pins out of the heredoc in
Expand All @@ -156,25 +165,16 @@ auditing the built-in list while the panel runs yours. A variable is not in the
tracked tree at all — no review, no grep, no diff. Watch the catalog yourself
for the duration of an experiment.

What the variable buys over a literal is therefore not auditing — it is the
revert, and where the fallback lands. Unset, the panel runs the built-in list,
which *is* the drift-audited one, so an experiment ends with a `gh variable
delete` and reverts back into coverage. A caller literal **is** the fallback:
there is nothing to delete, it needs a PR plus a fleet SHA bump to change, and
it holds its snapshot unaudited while the shared default moves on without it.
So prefer the variable, and treat a set variable as temporary by default.

A **repo** variable overrides an **org** variable of the same name, so one repo
can trial a list the rest of the org is not on, and the org value can be set
later without touching any caller. Note what an org-level value means: it
retargets every enrolled caller at once, and panel composition then lives in
mutable state outside the SHA-pinned tree every caller otherwise pins, with the
run log as the only record of which models judged a given PR.
So treat a set variable as temporary, and once a list has proven itself, move it
into the built-in heredoc and delete the variable — that puts it back under
drift coverage for everyone.

```bash
gh variable set CURSOR_PANEL_MODELS --repo <owner>/<repo> \
--body '["gpt-5.6-sol-max","claude-opus-5-thinking-xhigh","kimi-k3-high"]'
```
Two further consequences of the variable being caller-side, worth knowing rather
than discovering: an **org** value retargets every enrolled caller at once, and
any caller repo's admin can set a **repo** value to narrow the panel judging
their own PRs — with no PR, no CODEOWNERS review and the run log as the only
record. That write is repo-admin, who can already swap the pin or delete the
caller outright, so it is not a new trust boundary; it is a quieter one.

Two things to get right:

Expand Down
Loading