Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 16 additions & 1 deletion .github/cursor-review/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -183,13 +183,28 @@ The build is then **verified, not trusted**, and against the reviewed diff's own
|---|---|
| OpenAI | `gpt-5.6-sol-xhigh` |
| Anthropic | `claude-opus-5-5-xhigh` |
| Moonshot | `kimi-k3-high` |
| Moonshot | `kimi-k3-max` |

Gemini 3.1 Pro was dropped on 2026-08-27 after a spend review: across 1,450
judge-kept findings its two cells were the sole raiser of 2.5% of findings and
3 of 213 critical/high ones, at ~9% of every run's cost. Kimi moved from `-max`
to `-high` and the judge from `-max` to `-xhigh` in the same review.

**Kimi moved back to `-max` on 2026-10-09.** The `-high` tier turned
capacity-bound — cells dying in ~25s with `RetriableError:
[resource_exhausted]`, 32% of cells red on one consumer, and the in-step retry does
not rescue a sustained condition. Re-running the attribution above over 1,158
judge-kept findings (82 PRs), normalised per finding raised: Kimi is the sole
raiser of 1.05 findings per PR at **both** tiers and uniquely caught 3 of 58
critical/high at `-max`, against Gemini's 0.33 per PR and 0 of 58 — so the lane
is worth about three times what the dropped Gemini lane was, and the fix is the
tier, not removal. This gives back the spend the `-high` notch was for.
Reliability at `-max` is not re-established under current conditions (0 of 36
red, but measured 2026-10-07, and the tiers may share a pool); if it goes the
same way, the attribution says drop to a two-lab Cursor panel rather than
substitute Gemini or Grok — Grok `4.7-high`, tried 2026-10-09, raised 0 unique
findings in 17 while timing out on 11 of 20 cells.

The Anthropic cell moved from Opus 5 `-thinking-max` to **Opus 5.5 `-xhigh`** on
2026-10-07 — a newer model one reasoning notch down, on the same spend
reasoning. Note that Opus 5.5 ships no `-thinking-` variants in the Cursor
Expand Down
2 changes: 1 addition & 1 deletion .github/cursor-review/prompt-judge.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ two fields on no other finding.
- `body`: concise (1-3 sentences). Do NOT prefix the body with a
severity word or emoji; the severity field drives the rendered badge. END
with attribution like
`_Raised by 3 of N reviewers (gpt-5.6-sol-xhigh adversarial, claude-opus-5-5-xhigh edge-case, kimi-k3-high adversarial)._`
`_Raised by 3 of N reviewers (gpt-5.6-sol-xhigh adversarial, claude-opus-5-5-xhigh edge-case, kimi-k3-max adversarial)._`
where N is the number of panel cells listed in the PANEL FINDINGS block.

Order findings most-severe first. If no findings rise to the bar, submit an
Expand Down
47 changes: 40 additions & 7 deletions .github/workflows/cursor-review.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2029,9 +2029,9 @@ jobs:
# nearest is a plain tier id. That plain line IS the thinking model:
# Cursor labels non-thinking explicitly where it applies (every plain
# Haiku 5.5 entry reads "No Thinking") and no Opus 5.5 or Sonnet 5.5
# entry carries that marker. Moonshot: Kimi K3 at the `high` tier —
# same model as the former `kimi-k3-max` pin, one reasoning notch
# lower; no NO-ZDR marker.
# entry carries that marker. Moonshot: Kimi K3 at the `max` tier —
# RESTORED 2026-10-09 after the `-high` tier proved capacity-bound; see
# the note below. No NO-ZDR marker.
#
# Gemini 3.1 Pro was removed 2026-08-27. Over 1,450 judge-kept
# findings (172 PRs), its two cells were the sole raiser of 2.5% of
Expand All @@ -2042,8 +2042,41 @@ jobs:
# moved from Opus 5 `-thinking-max` to Opus 5.5 `-xhigh` on
# 2026-10-07 for the same reason: a newer model one reasoning notch
# down. That is deliberately TWO axes at once (version + tier) — it
# is a default change, not an attribution experiment. The
# `panel_models` input — or the CURSOR_PANEL_MODELS variable, see the
# is a default change, not an attribution experiment.
#
# Kimi went BACK to `-max` on 2026-10-09. The `-high` tier turned
Comment thread
mattmillerai marked this conversation as resolved.
Resolved
# capacity-bound, not slow: cells died in ~25 seconds with
# `RetriableError: [resource_exhausted]` after three internal
# reconnects, zero findings, leg red, `Panel integrity` red with it.
# Measured on one consumer over 56 panels: 24 of 76 `kimi-k3-high`
# cells red (32%), 19 of them this error, and 12 of 22 on the last day.
# It is not a property of one repo — the same model threw the same error
# the same day on two unrelated consumers — and #388's in-step retry
# does not rescue it, because the condition is sustained rather than
# transient: both cells on github-workflows#392's own panel exhausted
# all three attempts.
#
# `-max` rather than dropping the lane, because the lane EARNS its
# place. Re-running the 2026-08-27 attribution over 1,158 judge-kept
# findings (82 PRs), normalised per finding raised so panel size does
# not flatter anyone: Kimi was the sole raiser of 17% of what it raised
# at `-max` and 27% at `-high` — 1.05 unique findings per PR at BOTH
# tiers — and uniquely caught 3 of 58 critical/high at `-max`. Compare
# Gemini, cut for exactly this reason: 12% and 0.33 per PR, 0 of 58
# critical/high. Dropping Kimi would cost roughly three times what
# dropping Gemini did.
#
# What this trades away is the spend the tier drop was for, so it is a
# deliberate reversal of part of #243, not an oversight. The reliability
# it buys is also NOT established at this tier under present conditions:
# `-max` was 0 of 36 cells red, but that was measured on 2026-10-07,
# before the capacity trouble, and the two tiers may well share a pool.
# If `-max` goes the same way, the evidence above says drop to a
# two-lab Cursor panel rather than substitute Gemini or Grok — Grok
# `4.7-high` was tried on 2026-10-09 and raised 0 unique findings in 17
# while timing out on 11 of 20 cells.
#
# The `panel_models` input — or the CURSOR_PANEL_MODELS variable, see the
# env block below — overrides this list for A/B experiments.
#
# The OpenAI cell moved from `gpt-5.6-sol-max` to `gpt-5.6-sol-xhigh`:
Expand Down Expand Up @@ -2087,7 +2120,7 @@ jobs:
[
"gpt-5.6-sol-xhigh",
"claude-opus-5-5-xhigh",
"kimi-k3-high"
"kimi-k3-max"
Comment thread
mattmillerai marked this conversation as resolved.
Resolved
]
JSON
if [ -n "$(printf '%s' "$PANEL_MODELS_OVERRIDE" | tr -d '[:space:]')" ]; then
Expand Down Expand Up @@ -4458,7 +4491,7 @@ jobs:
# not submit` in the `review` job is the per-cell half. This job is what a
# gate should read, because a matrix leg's context name carries the model id
# and therefore CHANGES whenever the panel list does — a required check on
# `edge-case (kimi-k3-high)` bricks every PR the day that pin moves.
# `edge-case (kimi-k3-max)` bricks every PR the day that pin moves.
#
# ADVISORY by default, exactly like `Blocking gate`: red here fails no other
# job IN THIS WORKFLOW, gates no merge, and does not gate `post-review` — a
Expand Down
6 changes: 3 additions & 3 deletions docs/callers/cursor-review.md
Original file line number Diff line number Diff line change
Expand Up @@ -143,11 +143,11 @@ pull-requests: write # posting the consolidated review (and, at the round cap,
```bash
# one repo
gh variable set CURSOR_PANEL_MODELS --repo <owner>/<repo> \
--body '["gpt-5.6-sol-xhigh","claude-opus-5-5-max","kimi-k3-high"]'
--body '["gpt-5.6-sol-xhigh","claude-opus-5-5-max","kimi-k3-max"]'

# or the whole fleet at once (visibility must include private repos)
gh variable set CURSOR_PANEL_MODELS --org <org> --visibility all \
--body '["gpt-5.6-sol-xhigh","claude-opus-5-5-max","kimi-k3-high"]'
--body '["gpt-5.6-sol-xhigh","claude-opus-5-5-max","kimi-k3-max"]'
```

These examples deliberately differ from the built-in list by exactly one axis:
Expand Down Expand Up @@ -382,7 +382,7 @@ The remaining shapes to expect before you require it:
`cancel-in-progress` caller above that red lands on the head SHA that was
superseded, not on the new one.
* **Do not require a leg check instead.** A panel cell's context name carries
the model id (`edge-case (kimi-k3-high)`), so it changes whenever the panel
the model id (`edge-case (kimi-k3-max)`), so it changes whenever the panel
list does — and a required check whose name no longer exists blocks every PR
in the repo. `Panel integrity` is stable by design.
* **It is red, not skipped, when the decision itself failed.** Panel integrity
Expand Down
Loading