diff --git a/.github/cursor-review/README.md b/.github/cursor-review/README.md index 6350cd32..a77a3b27 100644 --- a/.github/cursor-review/README.md +++ b/.github/cursor-review/README.md @@ -183,13 +183,28 @@ The build is then **verified, not trusted**, and against the reviewed diff's own |---|---| | OpenAI | `gpt-5.6-sol-xhigh` | | Anthropic | `claude-opus-5-5-xhigh` | -| Moonshot | `kimi-k3-high` | +| Moonshot | `kimi-k3-max` | Gemini 3.1 Pro was dropped on 2026-08-27 after a spend review: across 1,450 judge-kept findings its two cells were the sole raiser of 2.5% of findings and 3 of 213 critical/high ones, at ~9% of every run's cost. Kimi moved from `-max` to `-high` and the judge from `-max` to `-xhigh` in the same review. +**Kimi moved back to `-max` on 2026-10-09.** The `-high` tier turned +capacity-bound — cells dying in ~25s with `RetriableError: +[resource_exhausted]`, 32% of cells red on one consumer, and the in-step retry does +not rescue a sustained condition. Re-running the attribution above over 1,158 +judge-kept findings (82 PRs), normalised per finding raised: Kimi is the sole +raiser of 1.05 findings per PR at **both** tiers and uniquely caught 3 of 58 +critical/high at `-max`, against Gemini's 0.33 per PR and 0 of 58 — so the lane +is worth about three times what the dropped Gemini lane was, and the fix is the +tier, not removal. This gives back the spend the `-high` notch was for. +Reliability at `-max` is not re-established under current conditions (0 of 36 +red, but measured 2026-10-07, and the tiers may share a pool); if it goes the +same way, the attribution says drop to a two-lab Cursor panel rather than +substitute Gemini or Grok — Grok `4.7-high`, tried 2026-10-09, raised 0 unique +findings in 17 while timing out on 11 of 20 cells. + The Anthropic cell moved from Opus 5 `-thinking-max` to **Opus 5.5 `-xhigh`** on 2026-10-07 — a newer model one reasoning notch down, on the same spend reasoning. Note that Opus 5.5 ships no `-thinking-` variants in the Cursor diff --git a/.github/cursor-review/prompt-judge.md b/.github/cursor-review/prompt-judge.md index 8991162f..27185ea7 100644 --- a/.github/cursor-review/prompt-judge.md +++ b/.github/cursor-review/prompt-judge.md @@ -75,7 +75,7 @@ two fields on no other finding. - `body`: concise (1-3 sentences). Do NOT prefix the body with a severity word or emoji; the severity field drives the rendered badge. END with attribution like - `_Raised by 3 of N reviewers (gpt-5.6-sol-xhigh adversarial, claude-opus-5-5-xhigh edge-case, kimi-k3-high adversarial)._` + `_Raised by 3 of N reviewers (gpt-5.6-sol-xhigh adversarial, claude-opus-5-5-xhigh edge-case, kimi-k3-max adversarial)._` where N is the number of panel cells listed in the PANEL FINDINGS block. Order findings most-severe first. If no findings rise to the bar, submit an diff --git a/.github/workflows/cursor-review.yml b/.github/workflows/cursor-review.yml index f6467082..fe78b301 100644 --- a/.github/workflows/cursor-review.yml +++ b/.github/workflows/cursor-review.yml @@ -2029,9 +2029,9 @@ jobs: # nearest is a plain tier id. That plain line IS the thinking model: # Cursor labels non-thinking explicitly where it applies (every plain # Haiku 5.5 entry reads "No Thinking") and no Opus 5.5 or Sonnet 5.5 - # entry carries that marker. Moonshot: Kimi K3 at the `high` tier — - # same model as the former `kimi-k3-max` pin, one reasoning notch - # lower; no NO-ZDR marker. + # entry carries that marker. Moonshot: Kimi K3 at the `max` tier — + # RESTORED 2026-10-09 after the `-high` tier proved capacity-bound; see + # the note below. No NO-ZDR marker. # # Gemini 3.1 Pro was removed 2026-08-27. Over 1,450 judge-kept # findings (172 PRs), its two cells were the sole raiser of 2.5% of @@ -2042,8 +2042,41 @@ jobs: # moved from Opus 5 `-thinking-max` to Opus 5.5 `-xhigh` on # 2026-10-07 for the same reason: a newer model one reasoning notch # down. That is deliberately TWO axes at once (version + tier) — it - # is a default change, not an attribution experiment. The - # `panel_models` input — or the CURSOR_PANEL_MODELS variable, see the + # is a default change, not an attribution experiment. + # + # Kimi went BACK to `-max` on 2026-10-09. The `-high` tier turned + # capacity-bound, not slow: cells died in ~25 seconds with + # `RetriableError: [resource_exhausted]` after three internal + # reconnects, zero findings, leg red, `Panel integrity` red with it. + # Measured on one consumer over 56 panels: 24 of 76 `kimi-k3-high` + # cells red (32%), 19 of them this error, and 12 of 22 on the last day. + # It is not a property of one repo — the same model threw the same error + # the same day on two unrelated consumers — and #388's in-step retry + # does not rescue it, because the condition is sustained rather than + # transient: both cells on github-workflows#392's own panel exhausted + # all three attempts. + # + # `-max` rather than dropping the lane, because the lane EARNS its + # place. Re-running the 2026-08-27 attribution over 1,158 judge-kept + # findings (82 PRs), normalised per finding raised so panel size does + # not flatter anyone: Kimi was the sole raiser of 17% of what it raised + # at `-max` and 27% at `-high` — 1.05 unique findings per PR at BOTH + # tiers — and uniquely caught 3 of 58 critical/high at `-max`. Compare + # Gemini, cut for exactly this reason: 12% and 0.33 per PR, 0 of 58 + # critical/high. Dropping Kimi would cost roughly three times what + # dropping Gemini did. + # + # What this trades away is the spend the tier drop was for, so it is a + # deliberate reversal of part of #243, not an oversight. The reliability + # it buys is also NOT established at this tier under present conditions: + # `-max` was 0 of 36 cells red, but that was measured on 2026-10-07, + # before the capacity trouble, and the two tiers may well share a pool. + # If `-max` goes the same way, the evidence above says drop to a + # two-lab Cursor panel rather than substitute Gemini or Grok — Grok + # `4.7-high` was tried on 2026-10-09 and raised 0 unique findings in 17 + # while timing out on 11 of 20 cells. + # + # The `panel_models` input — or the CURSOR_PANEL_MODELS variable, see the # env block below — overrides this list for A/B experiments. # # The OpenAI cell moved from `gpt-5.6-sol-max` to `gpt-5.6-sol-xhigh`: @@ -2087,7 +2120,7 @@ jobs: [ "gpt-5.6-sol-xhigh", "claude-opus-5-5-xhigh", - "kimi-k3-high" + "kimi-k3-max" ] JSON if [ -n "$(printf '%s' "$PANEL_MODELS_OVERRIDE" | tr -d '[:space:]')" ]; then @@ -4458,7 +4491,7 @@ jobs: # not submit` in the `review` job is the per-cell half. This job is what a # gate should read, because a matrix leg's context name carries the model id # and therefore CHANGES whenever the panel list does — a required check on - # `edge-case (kimi-k3-high)` bricks every PR the day that pin moves. + # `edge-case (kimi-k3-max)` bricks every PR the day that pin moves. # # ADVISORY by default, exactly like `Blocking gate`: red here fails no other # job IN THIS WORKFLOW, gates no merge, and does not gate `post-review` — a diff --git a/docs/callers/cursor-review.md b/docs/callers/cursor-review.md index 60d58c1a..29092c7b 100644 --- a/docs/callers/cursor-review.md +++ b/docs/callers/cursor-review.md @@ -143,11 +143,11 @@ pull-requests: write # posting the consolidated review (and, at the round cap, ```bash # one repo gh variable set CURSOR_PANEL_MODELS --repo / \ - --body '["gpt-5.6-sol-xhigh","claude-opus-5-5-max","kimi-k3-high"]' + --body '["gpt-5.6-sol-xhigh","claude-opus-5-5-max","kimi-k3-max"]' # or the whole fleet at once (visibility must include private repos) gh variable set CURSOR_PANEL_MODELS --org --visibility all \ - --body '["gpt-5.6-sol-xhigh","claude-opus-5-5-max","kimi-k3-high"]' + --body '["gpt-5.6-sol-xhigh","claude-opus-5-5-max","kimi-k3-max"]' ``` These examples deliberately differ from the built-in list by exactly one axis: @@ -382,7 +382,7 @@ The remaining shapes to expect before you require it: `cancel-in-progress` caller above that red lands on the head SHA that was superseded, not on the new one. * **Do not require a leg check instead.** A panel cell's context name carries - the model id (`edge-case (kimi-k3-high)`), so it changes whenever the panel + the model id (`edge-case (kimi-k3-max)`), so it changes whenever the panel list does — and a required check whose name no longer exists blocks every PR in the repo. `Panel integrity` is stable by design. * **It is red, not skipped, when the decision itself failed.** Panel integrity