Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 44 additions & 0 deletions .github/workflows/ci-cursor-review.yml
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,50 @@ jobs:
with:
# Keep the assets ref in lock-step with the `uses:` ref above.
workflows_ref: 0eac47eef787b9cf9c9ee7c0e8249d470e70e8af
# Panel model override, as a VARIABLE so a reasoning-tier A/B needs no PR
# and no fleet bump — set it, watch a few rounds, unset it to revert. The
# reusable's `panel_models` treats empty as "use the built-in list", so an
# unset variable is exactly today's behaviour; a repo variable overrides an
# org one of the same name, so this can be set org-wide later without
# touching this file.
#
# The trade-off this buys, stated plainly: panel composition becomes
# mutable state OUTSIDE the SHA-pinned tree the header above argues for.
# While the variable is set, nothing in review, CODEOWNERS or git records
# which models judged a PR — the run log is the only trace — and anyone
# with variable write can narrow the panel to one cheap model. At ORG
# level it retargets every enrolled caller at once. That write is
# admin-only, not open to anyone who can push, which is why a variable is
# acceptable here at all; keep it repo-level while experimenting, and
# delete it rather than leaving a stale list in force.
#
# The value is a JSON array of CURSOR CATALOG ids — tier suffix and all
# (`claude-opus-5-thinking-xhigh`, not `claude-opus-5`). It is deliberately
# NOT the org's `CLAUDE_MODEL_*` variables: those hold Anthropic API slugs,
# which the preflight's live-catalog check would reject. Keep the two
# namespaces apart.
#
# Preflight checks that every id EXISTS in the live catalog and fails loud
# before any cell spends, so a typo costs nothing. It does NOT check ZDR
# eligibility — an id on a catalog line reading `NO ZDR` matches like any
# other, and `catalog-drift.py` (which does detect NO-ZDR) parses the
# built-in heredoc and never sees an overridden list. Confirm ZDR yourself
# before setting this.
#
# The documented first experiment is a SINGLE-model tier drop: the
# Anthropic cell from `-max` to `-xhigh` (the judge already made that move
# in #243 on spend grounds), with the other two cells held fixed so the
# result is attributable:
# gh variable set CURSOR_PANEL_MODELS \
# --repo Comfy-Org/github-workflows \
# --body '["gpt-5.6-sol-max","claude-opus-5-thinking-xhigh","kimi-k3-high"]'
Comment thread
mattmillerai marked this conversation as resolved.
Resolved
#
# `judge_model` is deliberately NOT wired the same way. Its input carries a
# real default that catalog-drift.py parses out of cursor-review.yml, and a
# caller passing an explicitly-empty value gets '' rather than that default
# — so the override would have to restate the default here and drift from
# it. Changing that belongs in the reusable, not in a caller.
panel_models: ${{ vars.CURSOR_PANEL_MODELS }}
Comment thread
mattmillerai marked this conversation as resolved.
Resolved
Comment thread
mattmillerai marked this conversation as resolved.
Resolved
# No vendored/generated content in this repo; defensive excludes only.
# These live in extra_generated_globs, not diff_excludes: since BE-7633
# diff_excludes leaves the reviewed diff but is still COUNTED against
Expand Down
33 changes: 29 additions & 4 deletions .github/workflows/cursor-review.yml
Original file line number Diff line number Diff line change
Expand Up @@ -1966,13 +1966,38 @@ jobs:
# Caller override. Must be a non-empty JSON array of catalog-shaped
# ids; the shape check here and the live-catalog check below both
# fail loud, so a bad override costs zero cells.
if ! printf '%s' "$PANEL_MODELS_OVERRIDE" | jq -e \
'type == "array" and length > 0 and all(.[]; type == "string" and test("^[a-z0-9][a-z0-9._-]*$"))' > /dev/null; then
echo "::error::panel_models must be a non-empty JSON array of Cursor model ids (e.g. [\"gpt-5.6-sol-xhigh\", \"claude-opus-5-thinking-xhigh\"]); got: $PANEL_MODELS_OVERRIDE"
# Bounded at BOTH ends. The matrix fans out to models x 2 review
# types and GitHub caps a matrix at 256 jobs, so an oversized array
# fails the whole run at startup (permanently red under
# `blocking: true`) — and every extra id is another full
# cursor-agent pass over the diff. 12 is far above any real panel
# (the built-in list is 3) and far below the cap.
# Slurped (-s) and held to exactly ONE document: jq reads a stream,
# so `["a"] ["b"]` or `null ["a"]` would otherwise pass on its last
# value and normalize to two lines, not the one array the matrix
# needs.
if ! printf '%s' "$PANEL_MODELS_OVERRIDE" | jq -e -s \
'length == 1 and (.[0] | type == "array" and length > 0 and length <= 12 and all(.[]; type == "string" and test("^[a-z0-9][a-z0-9._-]*$")))' > /dev/null; then
# Echo the rejected value with newlines stripped and truncated.
# This input is caller-settable, and a caller wiring it to a
# variable makes it settable with no PR at all, so a raw
# multi-line echo would let it smuggle ::workflow commands:: into
# a PUBLIC repo's run log. Same reason the `Require a pinned
# workflows_ref` step below strips before echoing.
SAFE_OVERRIDE="$(printf '%s' "$PANEL_MODELS_OVERRIDE" | tr -d '\n\r' | cut -c1-200)"
echo "::error::panel_models must be a JSON array of 1-12 Cursor model ids (e.g. [\"gpt-5.6-sol-xhigh\", \"claude-opus-5-thinking-xhigh\"]); got: $SAFE_OVERRIDE"
exit 1
fi
printf '%s' "$PANEL_MODELS_OVERRIDE" | jq -c 'unique' > /tmp/models.json
printf '%s' "$PANEL_MODELS_OVERRIDE" | jq -c -s '.[0] | unique' > /tmp/models.json
echo "panel_models override in effect: $(cat /tmp/models.json)"
else
Comment thread
mattmillerai marked this conversation as resolved.
Resolved
# Logged in BOTH branches so the run says which list produced the
# review. A caller driving panel_models from a variable makes the
# empty case reachable by DELETING that variable, and without this
# line a mid-experiment deletion is indistinguishable in the log
# from one that was never set — the panel quietly reverts and the
# rounds either side of it look the same.
echo "panel_models not set — using the built-in list: $(jq -c . /tmp/models.json)"
fi
echo "models=$(jq -c . /tmp/models.json)" >> "$GITHUB_OUTPUT"

Expand Down
76 changes: 75 additions & 1 deletion docs/callers/cursor-review.md
Original file line number Diff line number Diff line change
Expand Up @@ -111,7 +111,7 @@ pull-requests: write # posting the consolidated review (and, at the round cap,
| Input | Default | Notes |
|---|---|---|
| `judge_model` | `claude-opus-5-thinking-xhigh` | Consolidates the panel into one review. |
| `panel_models` | `''` | JSON array of model ids replacing the built-in panel list (each runs both review types; preflight validates them against the live catalog). Use for per-repo experiments such as a reasoning-tier A/B. |
| `panel_models` | `''` | JSON array of model ids replacing the built-in panel list (each runs both review types; preflight checks each one exists in the live catalog — existence only, not ZDR). Use for per-repo experiments such as a reasoning-tier A/B — and prefer wiring it to a **variable** rather than a literal, see [Running a model experiment](#running-a-model-experiment). |
| `skip_bot_branch_prefixes` | `ci/bump- chore/refresh- auto/refresh-` | Skip the panel for Bot-authored PRs on these branch prefixes (machine pin bumps / refreshes). `''` to review every bot PR. |
| `diff_size_cap` | `5000` | Skip review above this diff size. An over-cap PR is not silent — see the gotcha below. Under `blocking: true` it is also not green: an unreviewed PR cannot pass the gate. |
| `ignore_comments` | `true` | Discount blank/comment-only lines from the size count (count-only — the panel still sees them). |
Expand All @@ -132,6 +132,80 @@ pull-requests: write # posting the consolidated review (and, at the round cap,
| `max_rounds` | `5` | Cap on review rounds per PR (`0` → no cap). At or over it, no panel runs: the PR is labelled `needs-human-review`, one comment lists the latest round's open findings above the threshold, and the `approve_gate` output is `capped`. Removing the label resets the count. See [round cap](#round-cap-and-the-approve_gate-output). |
| `runs_on` | `'"ubuntu-latest"'` | JSON-encoded `runs-on` for `diff-size`, `preflight`, the `review` panel and `consolidate` only — the jobs that hold no write credential. Every other job stays on GitHub-hosted `ubuntu-latest`. A self-hosted pool is fine only if it is one-job-per-fresh-pod, identity-free and private-network-isolated (those jobs run models with shell over PR code), on linux/x64 with bash, git, curl, jq, python3, gh, tar and GNU coreutils. Empty falls back to the default, so `${{ vars.CURSOR_REVIEW_RUNS_ON }}` is safe while the variable is unset; e.g. `'["self-hosted", "linux", "x64"]'`. |

## Running a model experiment

`panel_models` replaces the built-in panel list. Wire it to a **variable**
rather than a literal, so a reasoning-tier or model-version A/B needs no PR and
no SHA-bump across the fleet:

```yaml
with:
panel_models: ${{ vars.CURSOR_PANEL_MODELS }}
```

An unset variable is exactly today's behaviour — the reusable reads empty as
"use the built-in list" — so the wiring is inert until you set it, and deleting
the variable (or every variable) falls back rather than failing. The run log
names which list is in effect either way, so a deletion mid-experiment is
visible instead of looking like a round that was never overridden.

**Any override forfeits drift coverage, variable included.**
`catalog-drift.py` parses the panel pins out of the heredoc in
`cursor-review.yml`, so while `panel_models` is set the weekly check keeps
auditing the built-in list while the panel runs yours. A variable is not in the
tracked tree at all — no review, no grep, no diff. Watch the catalog yourself
for the duration of an experiment.

What the variable buys over a literal is therefore not auditing — it is the
revert, and where the fallback lands. Unset, the panel runs the built-in list,
which *is* the drift-audited one, so an experiment ends with a `gh variable
delete` and reverts back into coverage. A caller literal **is** the fallback:
there is nothing to delete, it needs a PR plus a fleet SHA bump to change, and
it holds its snapshot unaudited while the shared default moves on without it.
So prefer the variable, and treat a set variable as temporary by default.

A **repo** variable overrides an **org** variable of the same name, so one repo
can trial a list the rest of the org is not on, and the org value can be set
later without touching any caller. Note what an org-level value means: it
retargets every enrolled caller at once, and panel composition then lives in
mutable state outside the SHA-pinned tree every caller otherwise pins, with the
run log as the only record of which models judged a given PR.

```bash
gh variable set CURSOR_PANEL_MODELS --repo <owner>/<repo> \
--body '["gpt-5.6-sol-max","claude-opus-5-thinking-xhigh","kimi-k3-high"]'
```

Two things to get right:

**Use Cursor catalog ids, with the tier suffix.** `claude-opus-5-thinking-xhigh`,
not `claude-opus-5`. If your org keeps model slugs in variables for workflows
that call a model API directly, those are a *different namespace* — an API slug
has no tier and the preflight's live-catalog check will reject it. Keep the two
apart rather than pointing this input at them.

**Change one axis at a time.** A model version and a reasoning tier are separate
variables; moving both at once leaves the result unattributable. Note also that
tier names are not uniform across families — some ship `-thinking-*` tiers and
some do not — so the "same" tier of a newer version may not exist.

Preflight checks that every id **exists** in the live catalog and fails loud
before any cell spends, so a typo or a delisted id costs nothing but the
preflight job. It is an existence check only — it does **not** validate ZDR
eligibility. The match is on the id as a token, so an id whose catalog line
reads `NO ZDR` passes like any other; NO-ZDR detection lives only in
`catalog-drift.py`, which never sees an overridden list. On a panel that reviews
private-repo diffs, confirm ZDR against the catalog yourself before setting the
variable, and re-check it before a long-running experiment — a model can be
reclassified while your override is in force.

`judge_model` is **not** a good fit for the same treatment: its input carries a
real default, and a caller that passes an explicitly-empty value gets `''`
rather than that default — so a variable-wired caller would have to restate the
default and then drift from it. (`catalog-drift.py` also parses that default out
of the workflow.) Overriding the judge is better done with a literal, or by
changing the default in the reusable.

## Gotchas

**The label must be applied by a GitHub App token, not `GITHUB_TOKEN`.** Events
Expand Down
Loading