Repository navigation
fix(cursor-review): withhold auto-approve when a reviewer leg failed but every cell artifact reads ok - #391
Merged
Conversation
…but every cell artifact reads ok A prompt-injected review cell can upload a forged `status: ok` record under another cell's `findings-<type>-<model>` artifact name first; the honest leg's upload then 409s and goes red, but consolidate read the forgery and the approve gate could still pass, since nothing read the leg results. `Fail the leg when the cell did not submit` reds every non-ok cell, so a review matrix that did not succeed while every counted cell artifact reads ok is inconsistent. aggregate-panel.py now takes the matrices' results (`needs.review.result`, `needs.review-openai-direct.result` when the direct lane counts) and writes `panel_inconsistent`; consolidate exposes it and writes it into the consolidated file; auto-approve treats it as a non-retryable untrusted reason; Panel integrity fails naming it. A record whose model/review_type disagree with its artifact name now counts as `status: mismatch`. No new permission: consolidate stays contents: read.
Contributor
There was a problem hiding this comment.
🔍 Cursor Review — Consolidated panel
Triggered by @mattmillerai.
Found 7 finding(s).
| Severity | Count |
|---|---|
| 🟡 Medium | 4 |
| 🟢 Low | 2 |
| ⚪ Nit | 1 |
Panel: 6/6 reviewers contributed findings.
…tput, and flag phantom/mismatched cells Review round on the panel_inconsistent check: - auto-approve's decide now takes `--panel-inconsistent` from `needs.consolidate.outputs.panel_inconsistent` (written by aggregate-panel.py before the `--trust` judge ran), OR-ed with the consolidated file's copy, so the judge cannot clear it by rewriting /tmp/panel.json. Unrecognised values fail closed. - A counted record that is `mismatch` now also sets panel_inconsistent. - A cell keyed to no (model, type) the matrix runs is `mismatch`, so a phantom `error` cannot explain a red leg away (or be tolerated) and a phantom `ok` cannot complete a review type. - A mismatched record is re-keyed to its artifact name's slot (claim kept as claimed_model / claimed_review_type), so it no longer skews ok_count/total. - `--review-result` is required, so a dropped flag fails instead of reading as success. - The card for this cause no longer says a re-run lands the same way; it says a re-run clears a post-review flake, not a forgery. - Docs state the per-matrix limits plainly: any other non-ok cell explains a red leg away, and a forger able to delete and re-upload leaves none. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Member
Author
|
🤖 The reviews loop filed Linear follow-up ticket(s) for review thread(s) deferred as out of scope for this PR:
The following carry
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ELI-5
Each reviewer leg uploads a findings file under its own name. A hijacked reviewer can upload a fake "all clear" under another leg's name first. The real leg then can't upload, so its check goes red, but the panel used to read the fake file and could still auto-approve. Now, if a reviewer leg is red while every file reads "ok", the round is marked untrusted. Auto-approve holds off and Panel integrity says why. The review itself is still posted.
What changed
Invariant:
Fail the leg when the cell did not submitturns every non-okcell red. So areviewmatrix whose result is notsuccess, while EVERY counted cell artifact readsok, is inconsistent. Some leg failed after its agent ran: either an upload 409 (the forgery signature) or a post-review infra failure. In both cases the count can't be trusted.aggregate-panel.py: new--review-result(required, so a dropped flag fails the step instead of reading as success) and--direct-result(defaultskipped).aggregate()now also returnsinconsistentand writespanel_inconsistent=true|falsetoGITHUB_OUTPUT, with a::warning::when it is true. The direct lane is checked only when it counts. Cell records are not changed, so the judge still reads every finding.findings-[direct-]<type>-<model>is the cell's identity. If a record'smodel/review_typedisagree with that name, the name doesn't parse, or the record names no cell the matrix runs (a phantom), its status becomesmismatch. That is notokand not the tolerableerror, so it always counts as a failed reviewer. A countedmismatchalso setspanel_inconsistent. The record is re-keyed to its artifact's slot and the claim is kept asclaimed_model/claimed_review_type, so it can't occupy or double-count a slot.cursor-review.yml:Aggregate panel findingsnow passesREVIEW_RESULT: ${{ needs.review.result }}andDIRECT_RESULT: ${{ needs.review-openai-direct.result }}.consolidateexposes thepanel_inconsistentoutput.Build consolidated findings filewrites"panel_inconsistent"intoconsolidated.json. It reads the value from the step output, not/tmp/panel.json, because the judge runs in the same job after aggregation.Panel integritycounts it as a failing cause with the error message from the spec.consolidatecomment that said the direct job's result "is never read" has been corrected.auto-approve.py:decide()anddecide_gate()takepanel_inconsistent=False. When it is true they addREASON_PANEL_INCONSISTENT, which makes the gateuntrusted.cmd_decidetakes--panel-inconsistentfromneeds.consolidate.outputs.panel_inconsistent, the pre-judge step output. The file's copy is only OR-ed in, so a judge rewriting/tmp/panel.jsoncan't clear it. An unrecognised value fails closed.STRUCTURAL_CAUSES, so the card asks for a human. It is not inRETRYABLE_CAUSES: a fresh round runs the same PR content, so a forger forges again, and auto-retry can't tell that from a flake. The card says a re-run clears a post-review flake but not a forgery.cursor-approve.md, the last sentence of the "incidental backstop" paragraph now states the new rule and its remaining gap.cursor-review.mdgets theuntrustedgate row, the no-decision list and a new "Panel inconsistent" row in the Panel integrity table. The cursor-review README has the matching notes.No new permission:
consolidatestayscontents: read.Provenance
.github/cursor-review/tests: 1257 tests, all green..github/cursor-approve/testsand.github/workflow-pins/tests: green.check_workflow_pins.pyOK,check-org-repo-literals.shOK,check_agents_md.pypassed (2 pre-existing warnings),git diff --checkclean.--panel-inconsistentfails 2 decide tests.error/okcells, mismatch re-keying, the required flag, the pre-judge flag beating the file, the card text and the workflow wiring.actions: readfrom callers.Residual
actions: readto close; proposed as a follow-up): any other non-okcell in the same matrix explains a red leg away. That can be an ordinary step-cap timeout or the forger's own error, and underapprove_max_failed_reviewers> 0 it is tolerated. A forger able to delete the honest upload and re-upload under its name leaves no red leg at all. The docs say both plainly.okand the leg then failed for an infra reason (for example the artifact upload itself failed), the round is nowuntrustedand asks for a human instead of auto-approving. This is intentional: such a count can't be told apart from a forgery. It has not been measured against real run history.