Skip to content

fix(cursor-review): retry a panel cell on Cursor's transient resource_exhausted error - #388

Merged
mattmillerai merged 3 commits into
mainfrom
be-19849-cursor-review-retry-resource-exhausted
Oct 9, 2026
Merged

mattmillerai merged 3 commits into
mainfrom
be-19849-cursor-review-retry-resource-exhausted

Conversation

@mattmillerai

@mattmillerai mattmillerai commented Oct 9, 2026 •

Copy link
Copy Markdown
Member

ELI-5

Sometimes Cursor says "I'm too busy right now, try again" (RetriableError: [resource_exhausted]) about 25 seconds into a review cell. Today the cell gives up straight away and the panel posts with reviewers missing. With this change the cell waits a bit and tries again, at most twice, before it gives up.

What changed

  • stop-on-submit.py (the wrapper around each panel cell's cursor-agent) gets a retry loop:
    • Trigger: the agent exits by itself with a code > 0, and the last 64 KiB of its stderr contains [resource_exhausted] or RetriableError. stderr passes through the wrapper and is copied on to its own stderr byte for byte, so /tmp/review-stderr.txt gets the same content as before. stdout is inherited untouched.
    • Prompt replay: stdin (< /tmp/prompt.txt) is rewound to its starting offset before every retry; a stdin that cannot seek rules the retry out.
    • Backoff: --retry-delays 30,90, so at most 2 retries.
    • Never re-runs a submitted or partial cell: a retry needs findings.json to parse as an object whose status is not ok and whose findings list is empty. review-output-mcp.py appends each recorded finding, so re-running a cell that already recorded some would duplicate them. Those cells, and an unreadable file, fail exactly as they do today.
    • Fits the step cap: no retry starts more than --retry-window 300 seconds after the wrapper started (delay included). The last attempt therefore still has about 10 of the 15 minutes for a full review. If it runs past that, the step cap kills it as before.
    • Never retried: exit 0, any other error, a death by signal, and a forwarded signal (the step cap or a cancel, including one that arrives during the backoff or while the agent's stderr drains, which ends the wrapper with 128+signal).
    • Logging: each retry prints one ::warning::Reviewer cell <review_type>/<model>: attempt N/3 failed with Cursor's retriable error [resource_exhausted] (exit 1); retrying in 30s. The warning names only the fixed marker, never text the agent printed, and the cell label is sanitized.
  • cursor-review.yml Run cursor review passes --retry-delays 30,90 --retry-window 300 --cell "$REVIEW_TYPE/$MODEL" (it adds REVIEW_TYPE to the step env) and has a comment explaining why. A cell that uses up its retries still ends status=error, and Fail the leg when the cell did not submit still turns it red.
  • README row for stop-on-submit.py updated.

Judgment calls

  • The retry lives in the Python wrapper, not the step shell. That makes it unit-testable against a stub agent, and it keeps the existing signal forwarding and process-group SIGKILL working for each attempt.
  • Only stderr is scanned (changed in review, see Review fixes). The first version scanned stdout too, but stdout is the model's own text, which the PR under review can steer.
  • "No structured findings were submitted" is read strictly as no ok status and no recorded findings, so partial reviews are never duplicated.

Provenance

  • Authored by: agent-work loop
  • Verified:
    • Tests: the full .github/cursor-review/tests suite (1188 tests, Python 3.12) passes on the current head, including test_stop_on_submit.py (41 tests)
    • RetryTest (real wrapper over a stub agent that reads its whole stdin): retry then success, retries used up, every attempt gets the full prompt (also when stdin starts partway in), a pipe on stdin rules the retry out, a marker on stdout isn't retried, the warning starts its own line, other errors, exit 0 and signal death not retried, submitted or partially recorded findings not re-run, retry window, signal during the backoff
    • A signal during the stderr drain exits 128+signal within 2s; this test fails against the previous head
    • Lints: check_workflow_pins.py and check_agents_md.py pass on the current head; check-org-repo-literals.sh, shellcheck and actionlint were run on earlier rounds
    • PR CI was green on the previous head; the current head is awaiting CI
    • Not run: a live panel run against Cursor (it can't be triggered from here)
  • Deviations: none from the acceptance criteria. See Residual for what was not exercised live.

Residual

  • Not checked live against Cursor. None of the six sampled failing ci-cursor-review.yml runs on the consumer repo were re-read here. The matcher is based on the quoted log tail RetriableError: [resource_exhausted] Error. After the next cursor-review bump, check a consumer run with a Kimi (kimi-k3-high) failure for the new ::warning::Reviewer cell … line and a second attempt in the Run cursor review log.
  • This doesn't help if Cursor's capacity stays exhausted for more than about 2 minutes. In that case the cell still ends status=error, as the ticket allows. The model swap and counting the direct-OpenAI cell toward the gate are the separate follow-ups the ticket names.
  • The direct-OpenAI Codex cell (review-openai-direct) and the judge/consolidate step get no retry. Both are out of scope per the ticket.

Review fixes

  • A retry now gets the whole prompt again. The step feeds the prompt as < /tmp/prompt.txt, and every attempt shares that one open file and its read position. The first attempt read it to the end, so a retried cursor-agent got an empty prompt and failed at once (No prompt provided for print mode). The retry could never save a cell. The wrapper now notes where stdin starts and moves it back there before every retry. If stdin can't be rewound (a pipe or a terminal), it doesn't retry and logs one line saying why. A workflow comment and the wiring test now pin the file redirect.
  • The ::warning:: always starts on its own line. When the agent's last stderr write had no trailing newline, the warning was glued onto it and the runner didn't pick it up. The wrapper now starts a new line first when needed.
  • Only stderr can trigger a retry. stdout is the model's own text, which the PR under review can influence, so the marker is no longer looked for there, and stdout goes straight to /tmp/review-raw.txt again, as before this PR. This replaces the "both stdout and stderr are scanned" judgment call above.
  • A signal that lands while the agent's stderr drains is reported as a signal. A step-cap signal that arrived after the agent exited (exit 0, say) but before its stderr finished draining was recorded and then ignored. The wrapper returned the agent's own code, and the drain could still run for its full 5s. Now the drain stops as soon as a signal is pending, and the wrapper exits 128+signal.

…_exhausted error

A cell whose cursor-agent exits non-zero with Cursor's retriable marker
(`RetriableError: [resource_exhausted]`) is re-run by stop-on-submit.py
after 30s, then 90s, before falling through to the pre-seeded status=error
path. Only a cell with no submission and no recorded findings is re-run
(the MCP tool appends findings), no retry starts past 300s so the last
attempt still fits the 15-minute step cap, and each retry emits one
::warning:: naming the cell, attempt and marker. Any other exit, a signal
death or a forwarded signal behaves exactly as before.
@mattmillerai mattmillerai added cursor-review Multi-model cursor review agent-coded Authored by the agent-work loop labels Oct 9, 2026
@coderabbitai

coderabbitai Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Review paused — included plan limit reached

Keep your review moving with free on-demand reviews.

  • Run this review for free

On-demand reviews are free for one more day.

  • Ask an admin to make reviews automatic

Open in CodeRabbit

Reviews can continue after your included limit without a manual trigger. An admin must approve usage-based billing.

Promotion and pricing details

On-demand reviews are free for one more day. After that, they cost $0.25 per reviewed file.

Review limit details

Or wait 5 minutes for your next included review.

Check out review usage here.

Limit details: You’ve used the included review currently available. Your 129 included PR review attempts over the past 7 days set your current allowance at 1 review per hour.

Learn how review limits work.

Review configuration:

⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: ASSERTIVE
  • Plan: Team
  • Run ID: c87b1e43-0ced-46d1-a14b-cb64e9dc8cf0
📥 Commits

Reviewing files that changed from the base of the PR and between 7de68e5 and 454e25e.

📒 Files selected for processing (4)
  • .github/cursor-review/README.md
  • .github/cursor-review/stop-on-submit.py
  • .github/cursor-review/tests/test_stop_on_submit.py
  • .github/workflows/cursor-review.yml
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 Cursor Review — Consolidated panel

Triggered by @mattmillerai.

Found 6 finding(s).

Severity Count
🟠 High 1
🟢 Low 4
⚪ Nit 1

Panel: 5/6 reviewers contributed findings.

Reviewers that did not contribute: kimi-k3-high:adversarial (error)

Comment thread .github/cursor-review/stop-on-submit.py Resolved
Comment thread .github/cursor-review/stop-on-submit.py Outdated
child = subprocess.Popen(command, start_new_session=True,
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
current[0] = child
tees = (Tee(child.stdout, sys.stdout.fileno()),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Low — Both Tee(...) objects are built after the agent is spawned but before the try/finally that SIGKILLs its process group. If either one raises (for example Thread.start() hitting a thread/pid limit, or sys.stdout.fileno() failing), the wrapper exits and leaves the credential-bearing agent group running unsupervised. That breaks the documented rule that every exit path ends with SIGKILL, so an orphan could rewrite findings.json after the upload. Move the tee construction inside the try. Raised by 3 of 6 reviewers (claude-opus-5-5-xhigh adversarial, gpt-5.6-sol-xhigh adversarial, kimi-k3-high edge-case).

Comment thread .github/cursor-review/stop-on-submit.py Outdated
signal_group(child.pid, signal.SIGKILL)
current[0] = None
tails = [tee.drain() for tee in tees]
for stream in (child.stdout, child.stderr):

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Low — drain() can time out when a process that escaped the group (for example a setsid descendant of the --trust shell) still holds a pipe's write end. When it does, the pump thread is still blocked in os.read(fd), and stream.close() frees that fd number anyway. A later loop iteration of that stale thread can then read from a reused fd (the next attempt's pipe, or read_findings' os.open). It would steal the retry's output, so the retry marker or the passthrough bytes go missing, or it would corrupt the findings read. Don't close a stream, or don't retry, while its pump thread is still alive. Or give the thread a select-based loop with a stop flag. Raised by 5 of 6 reviewers (claude-opus-5-5-xhigh adversarial, gpt-5.6-sol-xhigh adversarial, claude-opus-5-5-xhigh edge-case, gpt-5.6-sol-xhigh edge-case, kimi-k3-high edge-case).

Comment thread .github/cursor-review/stop-on-submit.py Outdated
def retriable_marker(*tails):
"""The fixed marker name Cursor's retriable error carries, or None."""
for name, pattern in RETRIABLE:
if any(pattern.search(tail) for tail in tails):

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Low — The marker can match anywhere in the last 64 KiB of either stream. Under --print, stdout is model-written text that can quote the diff under review (this PR contains RetriableError: [resource_exhausted]) or be steered by prompt injection. So any unrelated non-zero exit before findings are recorded gets treated as capacity exhaustion, which costs up to two extra billed runs and 120s of backoff. Anchor the match to Cursor's own error line, for example the final non-empty line of stderr (or of stdout, if that is where the CLI reports it), rather than a free-text search. Raised by 5 of 6 reviewers (claude-opus-5-5-xhigh adversarial, gpt-5.6-sol-xhigh adversarial, claude-opus-5-5-xhigh edge-case, gpt-5.6-sol-xhigh edge-case, kimi-k3-high edge-case).

Comment thread .github/cursor-review/stop-on-submit.py Resolved
Comment thread .github/cursor-review/stop-on-submit.py Resolved
mattmillerai and others added 2 commits October 9, 2026 00:15
Every attempt inherits the wrapper's stdin, the `< /tmp/prompt.txt`
redirect, together with its shared file offset. The first attempt reads
it to EOF, so a retried cursor-agent read an empty prompt and failed at
once ("No prompt provided for print mode"): a retry could never recover
a cell. The wrapper now records stdin's offset at startup and seeks back
to it before every retry. A stdin that cannot seek (a pipe, a tty)
cannot be replayed, so it rules the retry out and the wrapper logs why.

Two smaller fixes from the same review:
- The retry `::warning::` always starts a line of its own. It was glued
  onto the agent's last stderr write when that write had no trailing
  newline, and the runner then did not parse it as a command.
- The retriable marker is matched in the agent's stderr only. stdout is
  the model's own text, which the PR under review can steer, so it is
  inherited as is again instead of being teed and searched.

New tests use a stub agent that reads its whole stdin: every attempt
must get the full prompt (from the offset stdin started at), a pipe on
stdin must rule the retry out, a marker on stdout must not trigger one,
and the warning must start a line after an unterminated stderr. The
retry tests now feed the wrapper a prompt file, as the workflow does,
instead of inheriting the test runner's stdin.
…rr drains

A forwarded signal (the step cap) that arrived after the agent exited by
itself, while its stderr was still draining, was recorded but ignored:
the wrapper returned the agent's own code, possibly 0, instead of
128+signal, and the drain could still run its full 5s past the signal.
The drain now stops as soon as a signal is pending, and run() reports
any received signal as the wrapper's status before considering a retry.

New test: an agent that exits 0 leaving a process outside its group that
holds stderr open; SIGTERM during the drain must exit 143 at once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mattmillerai
mattmillerai merged commit 578c48d into main Oct 9, 2026
8 checks passed
@mattmillerai
mattmillerai deleted the be-19849-cursor-review-retry-resource-exhausted branch October 9, 2026 07:35
mattmillerai added a commit that referenced this pull request Oct 9, 2026
#393)

* fix(cursor-review): move the default Moonshot cell back to kimi-k3-max

`kimi-k3-high` is capacity-bound, so the built-in panel is currently one lane
short on every consumer that has not overridden it — which is most of them.

The cells do not run slow, they die: ~25 seconds in, `RetriableError:
[resource_exhausted]` after three internal reconnects, zero findings, leg red,
`Panel integrity` red with it. Measured on one consumer over 56 panels, 24 of
76 `kimi-k3-high` cells red (32%), 19 of those this error, and 12 of 22 on the
last day. Not a property of one repo: the same model threw the same error the
same day on two unrelated consumers. And #388's in-step retry does not rescue
it, because the condition is sustained rather than transient — both cells on
this repo's own panel for #392 exhausted all three attempts.

`-max` rather than dropping the lane, because the lane earns its place. The
2026-08-27 attribution that cut Gemini was re-run over 1,158 judge-kept
findings (82 PRs), normalised per finding raised so panel size flatters nobody:

  model                   raised  sole  distinct  sole/PR  c/h sole
  kimi-k3-max                229    39       17%     1.05     3/58
  kimi-k3-high                81    22       27%     1.05     0/11
  gemini-3.1-pro              99    12       12%     0.33     0/58

Kimi contributes 1.05 unique findings per PR at BOTH tiers and uniquely caught
3 of 58 critical/high at `-max`. Gemini — cut for exactly this reason — managed
0.33 and 0 of 58. Dropping Kimi would cost about three times what dropping
Gemini did, so the fix is the tier, not removal.

This deliberately gives back the spend the `-high` notch in #243 was for; it is
a partial reversal of that change, not an oversight. And the reliability it
buys is NOT established at this tier under present conditions: `-max` was 0 of
36 cells red, but that was 2026-10-07, before the capacity trouble, and the two
tiers may share a pool. Recorded in the workflow so the next person does not
have to rediscover it, along with what the evidence says to do if `-max` goes
the same way: drop to a two-lab Cursor panel rather than substitute Gemini or
Grok. Grok `4.7-high` was tried on 2026-10-09 and raised 0 unique findings in
17 while timing out on 11 of 20 cells.

Default change, so it reaches every consumer on the next SHA bump; a repo that
wants the old tier back sets CURSOR_PANEL_MODELS. README panel table and
history updated alongside.

Tests: cursor-review 1257, workflow-pins 551 + pin lint, agents-md — all green.
The historical-era fixtures and `test_catalog_drift.py`'s REAL_PANEL pin their
own eras by design and are deliberately untouched.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* docs(cursor-review): rejoin the panel_models sentence the Kimi note split

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(cursor-review): move the remaining kimi-k3-high examples to the -max pin

The CURSOR_PANEL_MODELS copy-paste examples would otherwise change two
axes and silently put a fleet back on the capacity-bound -high tier.
Also reword the README's 32% figure to match the source data.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent-coded Authored by the agent-work loop cursor-review Multi-model cursor review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants