diff --git a/.github/workflows/ci-cursor-review.yml b/.github/workflows/ci-cursor-review.yml index 2207b6f9..e7765619 100644 --- a/.github/workflows/ci-cursor-review.yml +++ b/.github/workflows/ci-cursor-review.yml @@ -37,6 +37,50 @@ jobs: with: # Keep the assets ref in lock-step with the `uses:` ref above. workflows_ref: 0eac47eef787b9cf9c9ee7c0e8249d470e70e8af + # Panel model override, as a VARIABLE so a reasoning-tier A/B needs no PR + # and no fleet bump — set it, watch a few rounds, unset it to revert. The + # reusable's `panel_models` treats empty as "use the built-in list", so an + # unset variable is exactly today's behaviour; a repo variable overrides an + # org one of the same name, so this can be set org-wide later without + # touching this file. + # + # The trade-off this buys, stated plainly: panel composition becomes + # mutable state OUTSIDE the SHA-pinned tree the header above argues for. + # While the variable is set, nothing in review, CODEOWNERS or git records + # which models judged a PR — the run log is the only trace — and anyone + # with variable write can narrow the panel to one cheap model. At ORG + # level it retargets every enrolled caller at once. That write is + # admin-only, not open to anyone who can push, which is why a variable is + # acceptable here at all; keep it repo-level while experimenting, and + # delete it rather than leaving a stale list in force. + # + # The value is a JSON array of CURSOR CATALOG ids — tier suffix and all + # (`claude-opus-5-thinking-xhigh`, not `claude-opus-5`). It is deliberately + # NOT the org's `CLAUDE_MODEL_*` variables: those hold Anthropic API slugs, + # which the preflight's live-catalog check would reject. Keep the two + # namespaces apart. + # + # Preflight checks that every id EXISTS in the live catalog and fails loud + # before any cell spends, so a typo costs nothing. It does NOT check ZDR + # eligibility — an id on a catalog line reading `NO ZDR` matches like any + # other, and `catalog-drift.py` (which does detect NO-ZDR) parses the + # built-in heredoc and never sees an overridden list. Confirm ZDR yourself + # before setting this. + # + # The documented first experiment is a SINGLE-model tier drop: the + # Anthropic cell from `-max` to `-xhigh` (the judge already made that move + # in #243 on spend grounds), with the other two cells held fixed so the + # result is attributable: + # gh variable set CURSOR_PANEL_MODELS \ + # --repo Comfy-Org/github-workflows \ + # --body '["gpt-5.6-sol-max","claude-opus-5-thinking-xhigh","kimi-k3-high"]' + # + # `judge_model` is deliberately NOT wired the same way. Its input carries a + # real default that catalog-drift.py parses out of cursor-review.yml, and a + # caller passing an explicitly-empty value gets '' rather than that default + # — so the override would have to restate the default here and drift from + # it. Changing that belongs in the reusable, not in a caller. + panel_models: ${{ vars.CURSOR_PANEL_MODELS }} # No vendored/generated content in this repo; defensive excludes only. # These live in extra_generated_globs, not diff_excludes: since BE-7633 # diff_excludes leaves the reviewed diff but is still COUNTED against diff --git a/.github/workflows/cursor-review.yml b/.github/workflows/cursor-review.yml index 70457962..6216414e 100644 --- a/.github/workflows/cursor-review.yml +++ b/.github/workflows/cursor-review.yml @@ -1966,13 +1966,38 @@ jobs: # Caller override. Must be a non-empty JSON array of catalog-shaped # ids; the shape check here and the live-catalog check below both # fail loud, so a bad override costs zero cells. - if ! printf '%s' "$PANEL_MODELS_OVERRIDE" | jq -e \ - 'type == "array" and length > 0 and all(.[]; type == "string" and test("^[a-z0-9][a-z0-9._-]*$"))' > /dev/null; then - echo "::error::panel_models must be a non-empty JSON array of Cursor model ids (e.g. [\"gpt-5.6-sol-xhigh\", \"claude-opus-5-thinking-xhigh\"]); got: $PANEL_MODELS_OVERRIDE" + # Bounded at BOTH ends. The matrix fans out to models x 2 review + # types and GitHub caps a matrix at 256 jobs, so an oversized array + # fails the whole run at startup (permanently red under + # `blocking: true`) — and every extra id is another full + # cursor-agent pass over the diff. 12 is far above any real panel + # (the built-in list is 3) and far below the cap. + # Slurped (-s) and held to exactly ONE document: jq reads a stream, + # so `["a"] ["b"]` or `null ["a"]` would otherwise pass on its last + # value and normalize to two lines, not the one array the matrix + # needs. + if ! printf '%s' "$PANEL_MODELS_OVERRIDE" | jq -e -s \ + 'length == 1 and (.[0] | type == "array" and length > 0 and length <= 12 and all(.[]; type == "string" and test("^[a-z0-9][a-z0-9._-]*$")))' > /dev/null; then + # Echo the rejected value with newlines stripped and truncated. + # This input is caller-settable, and a caller wiring it to a + # variable makes it settable with no PR at all, so a raw + # multi-line echo would let it smuggle ::workflow commands:: into + # a PUBLIC repo's run log. Same reason the `Require a pinned + # workflows_ref` step below strips before echoing. + SAFE_OVERRIDE="$(printf '%s' "$PANEL_MODELS_OVERRIDE" | tr -d '\n\r' | cut -c1-200)" + echo "::error::panel_models must be a JSON array of 1-12 Cursor model ids (e.g. [\"gpt-5.6-sol-xhigh\", \"claude-opus-5-thinking-xhigh\"]); got: $SAFE_OVERRIDE" exit 1 fi - printf '%s' "$PANEL_MODELS_OVERRIDE" | jq -c 'unique' > /tmp/models.json + printf '%s' "$PANEL_MODELS_OVERRIDE" | jq -c -s '.[0] | unique' > /tmp/models.json echo "panel_models override in effect: $(cat /tmp/models.json)" + else + # Logged in BOTH branches so the run says which list produced the + # review. A caller driving panel_models from a variable makes the + # empty case reachable by DELETING that variable, and without this + # line a mid-experiment deletion is indistinguishable in the log + # from one that was never set — the panel quietly reverts and the + # rounds either side of it look the same. + echo "panel_models not set — using the built-in list: $(jq -c . /tmp/models.json)" fi echo "models=$(jq -c . /tmp/models.json)" >> "$GITHUB_OUTPUT" diff --git a/docs/callers/cursor-review.md b/docs/callers/cursor-review.md index 8a4d2283..440916d8 100644 --- a/docs/callers/cursor-review.md +++ b/docs/callers/cursor-review.md @@ -111,7 +111,7 @@ pull-requests: write # posting the consolidated review (and, at the round cap, | Input | Default | Notes | |---|---|---| | `judge_model` | `claude-opus-5-thinking-xhigh` | Consolidates the panel into one review. | -| `panel_models` | `''` | JSON array of model ids replacing the built-in panel list (each runs both review types; preflight validates them against the live catalog). Use for per-repo experiments such as a reasoning-tier A/B. | +| `panel_models` | `''` | JSON array of model ids replacing the built-in panel list (each runs both review types; preflight checks each one exists in the live catalog — existence only, not ZDR). Use for per-repo experiments such as a reasoning-tier A/B — and prefer wiring it to a **variable** rather than a literal, see [Running a model experiment](#running-a-model-experiment). | | `skip_bot_branch_prefixes` | `ci/bump- chore/refresh- auto/refresh-` | Skip the panel for Bot-authored PRs on these branch prefixes (machine pin bumps / refreshes). `''` to review every bot PR. | | `diff_size_cap` | `5000` | Skip review above this diff size. An over-cap PR is not silent — see the gotcha below. Under `blocking: true` it is also not green: an unreviewed PR cannot pass the gate. | | `ignore_comments` | `true` | Discount blank/comment-only lines from the size count (count-only — the panel still sees them). | @@ -132,6 +132,80 @@ pull-requests: write # posting the consolidated review (and, at the round cap, | `max_rounds` | `5` | Cap on review rounds per PR (`0` → no cap). At or over it, no panel runs: the PR is labelled `needs-human-review`, one comment lists the latest round's open findings above the threshold, and the `approve_gate` output is `capped`. Removing the label resets the count. See [round cap](#round-cap-and-the-approve_gate-output). | | `runs_on` | `'"ubuntu-latest"'` | JSON-encoded `runs-on` for `diff-size`, `preflight`, the `review` panel and `consolidate` only — the jobs that hold no write credential. Every other job stays on GitHub-hosted `ubuntu-latest`. A self-hosted pool is fine only if it is one-job-per-fresh-pod, identity-free and private-network-isolated (those jobs run models with shell over PR code), on linux/x64 with bash, git, curl, jq, python3, gh, tar and GNU coreutils. Empty falls back to the default, so `${{ vars.CURSOR_REVIEW_RUNS_ON }}` is safe while the variable is unset; e.g. `'["self-hosted", "linux", "x64"]'`. | +## Running a model experiment + +`panel_models` replaces the built-in panel list. Wire it to a **variable** +rather than a literal, so a reasoning-tier or model-version A/B needs no PR and +no SHA-bump across the fleet: + +```yaml + with: + panel_models: ${{ vars.CURSOR_PANEL_MODELS }} +``` + +An unset variable is exactly today's behaviour — the reusable reads empty as +"use the built-in list" — so the wiring is inert until you set it, and deleting +the variable (or every variable) falls back rather than failing. The run log +names which list is in effect either way, so a deletion mid-experiment is +visible instead of looking like a round that was never overridden. + +**Any override forfeits drift coverage, variable included.** +`catalog-drift.py` parses the panel pins out of the heredoc in +`cursor-review.yml`, so while `panel_models` is set the weekly check keeps +auditing the built-in list while the panel runs yours. A variable is not in the +tracked tree at all — no review, no grep, no diff. Watch the catalog yourself +for the duration of an experiment. + +What the variable buys over a literal is therefore not auditing — it is the +revert, and where the fallback lands. Unset, the panel runs the built-in list, +which *is* the drift-audited one, so an experiment ends with a `gh variable +delete` and reverts back into coverage. A caller literal **is** the fallback: +there is nothing to delete, it needs a PR plus a fleet SHA bump to change, and +it holds its snapshot unaudited while the shared default moves on without it. +So prefer the variable, and treat a set variable as temporary by default. + +A **repo** variable overrides an **org** variable of the same name, so one repo +can trial a list the rest of the org is not on, and the org value can be set +later without touching any caller. Note what an org-level value means: it +retargets every enrolled caller at once, and panel composition then lives in +mutable state outside the SHA-pinned tree every caller otherwise pins, with the +run log as the only record of which models judged a given PR. + +```bash +gh variable set CURSOR_PANEL_MODELS --repo / \ + --body '["gpt-5.6-sol-max","claude-opus-5-thinking-xhigh","kimi-k3-high"]' +``` + +Two things to get right: + +**Use Cursor catalog ids, with the tier suffix.** `claude-opus-5-thinking-xhigh`, +not `claude-opus-5`. If your org keeps model slugs in variables for workflows +that call a model API directly, those are a *different namespace* — an API slug +has no tier and the preflight's live-catalog check will reject it. Keep the two +apart rather than pointing this input at them. + +**Change one axis at a time.** A model version and a reasoning tier are separate +variables; moving both at once leaves the result unattributable. Note also that +tier names are not uniform across families — some ship `-thinking-*` tiers and +some do not — so the "same" tier of a newer version may not exist. + +Preflight checks that every id **exists** in the live catalog and fails loud +before any cell spends, so a typo or a delisted id costs nothing but the +preflight job. It is an existence check only — it does **not** validate ZDR +eligibility. The match is on the id as a token, so an id whose catalog line +reads `NO ZDR` passes like any other; NO-ZDR detection lives only in +`catalog-drift.py`, which never sees an overridden list. On a panel that reviews +private-repo diffs, confirm ZDR against the catalog yourself before setting the +variable, and re-check it before a long-running experiment — a model can be +reclassified while your override is in force. + +`judge_model` is **not** a good fit for the same treatment: its input carries a +real default, and a caller that passes an explicitly-empty value gets `''` +rather than that default — so a variable-wired caller would have to restate the +default and then drift from it. (`catalog-drift.py` also parses that default out +of the workflow.) Overriding the judge is better done with a literal, or by +changing the default in the reusable. + ## Gotchas **The label must be applied by a GitHub App token, not `GITHUB_TOKEN`.** Events