Skip to content

fix(cursor-review): withhold auto-approve when a reviewer leg failed but every cell artifact reads ok - #391

Merged
mattmillerai merged 2 commits into
mainfrom
matt/be-19886-panel-inconsistent
Oct 9, 2026
Merged

mattmillerai merged 2 commits into
mainfrom
matt/be-19886-panel-inconsistent

Conversation

@mattmillerai

@mattmillerai mattmillerai commented Oct 9, 2026 •

Copy link
Copy Markdown
Member

ELI-5

Each reviewer leg uploads a findings file under its own name. A hijacked reviewer can upload a fake "all clear" under another leg's name first. The real leg then can't upload, so its check goes red, but the panel used to read the fake file and could still auto-approve. Now, if a reviewer leg is red while every file reads "ok", the round is marked untrusted. Auto-approve holds off and Panel integrity says why. The review itself is still posted.

What changed

Invariant: Fail the leg when the cell did not submit turns every non-ok cell red. So a review matrix whose result is not success, while EVERY counted cell artifact reads ok, is inconsistent. Some leg failed after its agent ran: either an upload 409 (the forgery signature) or a post-review infra failure. In both cases the count can't be trusted.

  • aggregate-panel.py: new --review-result (required, so a dropped flag fails the step instead of reading as success) and --direct-result (default skipped). aggregate() now also returns inconsistent and writes panel_inconsistent=true|false to GITHUB_OUTPUT, with a ::warning:: when it is true. The direct lane is checked only when it counts. Cell records are not changed, so the judge still reads every finding.
    • Name vs record check: the artifact directory name findings-[direct-]<type>-<model> is the cell's identity. If a record's model/review_type disagree with that name, the name doesn't parse, or the record names no cell the matrix runs (a phantom), its status becomes mismatch. That is not ok and not the tolerable error, so it always counts as a failed reviewer. A counted mismatch also sets panel_inconsistent. The record is re-keyed to its artifact's slot and the claim is kept as claimed_model/claimed_review_type, so it can't occupy or double-count a slot.
  • cursor-review.yml:
    • Aggregate panel findings now passes REVIEW_RESULT: ${{ needs.review.result }} and DIRECT_RESULT: ${{ needs.review-openai-direct.result }}.
    • consolidate exposes the panel_inconsistent output.
    • Build consolidated findings file writes "panel_inconsistent" into consolidated.json. It reads the value from the step output, not /tmp/panel.json, because the judge runs in the same job after aggregation.
    • Panel integrity counts it as a failing cause with the error message from the spec.
    • The consolidate comment that said the direct job's result "is never read" has been corrected.
  • auto-approve.py:
    • decide() and decide_gate() take panel_inconsistent=False. When it is true they add REASON_PANEL_INCONSISTENT, which makes the gate untrusted.
    • cmd_decide takes --panel-inconsistent from needs.consolidate.outputs.panel_inconsistent, the pre-judge step output. The file's copy is only OR-ed in, so a judge rewriting /tmp/panel.json can't clear it. An unrecognised value fails closed.
    • The reason is in STRUCTURAL_CAUSES, so the card asks for a human. It is not in RETRYABLE_CAUSES: a fresh round runs the same PR content, so a forger forges again, and auto-retry can't tell that from a flake. The card says a re-run clears a post-review flake but not a forgery.
  • Docs: in cursor-approve.md, the last sentence of the "incidental backstop" paragraph now states the new rule and its remaining gap. cursor-review.md gets the untrusted gate row, the no-decision list and a new "Panel inconsistent" row in the Panel integrity table. The cursor-review README has the matching notes.

No new permission: consolidate stays contents: read.

Provenance

  • Authored by: agent-work loop
  • Verified:
    • Tests: .github/cursor-review/tests: 1257 tests, all green. .github/cursor-approve/tests and .github/workflow-pins/tests: green.
    • Lints: check_workflow_pins.py OK, check-org-repo-literals.sh OK, check_agents_md.py passed (2 pre-existing warnings), git diff --check clean.
    • Mutation-tested: dropping the phantom-cell check fails 2 aggregate tests. Ignoring --panel-inconsistent fails 2 decide tests.
    • Tests cover the ticket cases (a)–(f), the direct lane, cancelled matrices, missing and unparseable artifacts, phantom error/ok cells, mismatch re-keying, the required flag, the pre-judge flag beating the file, the card text and the workflow wiring.
    • Not run: a live workflow run. The forged-upload 409 path can't be reproduced locally and is covered by fixture tests only.
  • Deviations:
    • Per-leg attribution, needed to close the residuals below, is out of scope by design: it needs actions: read from callers.
    • Beyond the plan: README notes and a Panel integrity cause-table row, to keep the docs in sync.

Residual

  • The leg check is per matrix (needs actions: read to close; proposed as a follow-up): any other non-ok cell in the same matrix explains a red leg away. That can be an ordinary step-cap timeout or the forger's own error, and under approve_max_failed_reviewers > 0 it is tolerated. A forger able to delete the honest upload and re-upload under its name leaves no red leg at all. The docs say both plainly.
  • Accepted false positive: if a leg's agent submitted ok and the leg then failed for an infra reason (for example the artifact upload itself failed), the round is now untrusted and asks for a human instead of auto-approving. This is intentional: such a count can't be told apart from a forgery. It has not been measured against real run history.

…but every cell artifact reads ok

A prompt-injected review cell can upload a forged `status: ok` record under
another cell's `findings-<type>-<model>` artifact name first; the honest
leg's upload then 409s and goes red, but consolidate read the forgery and
the approve gate could still pass, since nothing read the leg results.

`Fail the leg when the cell did not submit` reds every non-ok cell, so a
review matrix that did not succeed while every counted cell artifact reads
ok is inconsistent. aggregate-panel.py now takes the matrices' results
(`needs.review.result`, `needs.review-openai-direct.result` when the direct
lane counts) and writes `panel_inconsistent`; consolidate exposes it and
writes it into the consolidated file; auto-approve treats it as a
non-retryable untrusted reason; Panel integrity fails naming it. A record
whose model/review_type disagree with its artifact name now counts as
`status: mismatch`. No new permission: consolidate stays contents: read.
@mattmillerai mattmillerai added cursor-review Multi-model cursor review agent-coded Authored by the agent-work loop labels Oct 9, 2026
@coderabbitai

coderabbitai Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: ASSERTIVE
  • Plan: Team
  • Run ID: 51086741-d2cf-4628-8c1a-f92a9e685f4f

📥 Commits

Reviewing files that changed from the base of the PR and between 67dcd0b and 745e901.


📒 Files selected for processing (9)
  • .github/cursor-review/README.md
  • .github/cursor-review/aggregate-panel.py
  • .github/cursor-review/auto-approve.py
  • .github/cursor-review/tests/test_aggregate_panel.py
  • .github/cursor-review/tests/test_auto_approve.py
  • .github/cursor-review/tests/test_workflow_panel_integrity.py
  • .github/workflows/cursor-review.yml
  • docs/callers/cursor-approve.md
  • docs/callers/cursor-review.md

Included review availability: This review used your included allowance. 0 included reviews remain after this review. Your included PR review attempts over the past 7 days set your current allowance at 1 review per hour. Your free on-demand review promotion remains active until October 9, 2026 at 6:00 PM UTC.



📝 Walkthrough

Walkthrough

The panel aggregator now checks artifact identities against artifact names and compares counted cell statuses with reviewer matrix results. The workflow passes a panel_inconsistent flag to auto-approval and Panel integrity. Auto-approval treats that flag as a structural trust failure.

Changes

Panel integrity and approval

Layer / File(s) Summary
Artifact and matrix consistency checks
.github/cursor-review/aggregate-panel.py, .github/cursor-review/tests/test_aggregate_panel.py, .github/cursor-review/README.md
The aggregator marks identity mismatches and unexpected records, checks counted cell statuses against matrix results, and writes panel_inconsistent with the existing counts. Tests cover failed or missing artifacts, identity mismatches, advisory direct cells, and CLI output.
Workflow output and integrity checks
.github/workflows/cursor-review.yml, .github/cursor-review/tests/test_workflow_panel_integrity.py
The consolidate job exposes and carries panel_inconsistent to auto-approval and Panel integrity. Panel integrity reports an error when the flag is true.
Auto-approval response to inconsistency
.github/cursor-review/auto-approve.py, .github/cursor-review/tests/test_auto_approve.py, docs/callers/cursor-approve.md, docs/callers/cursor-review.md
The approval gate treats panel inconsistency as a structural trust failure, withholds a decision, and provides a human next step. Tests and caller guides describe the outcomes and stated limits.

Sequence Diagram(s)

sequenceDiagram
  participant Aggregator as aggregate-panel.py
  participant Consolidate as consolidate job
  participant Approval as Auto-approve decision
  participant Integrity as Panel integrity
  Aggregator->>Consolidate: Write panel_inconsistent output
  Consolidate->>Approval: Pass panel_inconsistent
  Consolidate->>Integrity: Pass panel_inconsistent
Loading

Priority: ➖ Normal

Merge Risk: ⚪ Minimal · up to 745e9

The supplied evidence establishes no new issue requiring a merge hold; normal workflow checks remain appropriate.

🚥 Pre-merge checks | ✅ 2
✅ Passed checks (2 passed)
Check name Status Explanation
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.


✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR

✨ Simplify code
  • Commit to this branch
  • Create a new PR


  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 Cursor Review — Consolidated panel

Triggered by @mattmillerai.

Found 7 finding(s).

Severity Count
🟡 Medium 4
🟢 Low 2
⚪ Nit 1

Panel: 6/6 reviewers contributed findings.

Comment thread .github/cursor-review/auto-approve.py Resolved
Comment thread .github/cursor-review/auto-approve.py Resolved
…tput, and flag phantom/mismatched cells

Review round on the panel_inconsistent check:

- auto-approve's decide now takes `--panel-inconsistent` from
  `needs.consolidate.outputs.panel_inconsistent` (written by aggregate-panel.py
  before the `--trust` judge ran), OR-ed with the consolidated file's copy, so
  the judge cannot clear it by rewriting /tmp/panel.json. Unrecognised values
  fail closed.
- A counted record that is `mismatch` now also sets panel_inconsistent.
- A cell keyed to no (model, type) the matrix runs is `mismatch`, so a phantom
  `error` cannot explain a red leg away (or be tolerated) and a phantom `ok`
  cannot complete a review type.
- A mismatched record is re-keyed to its artifact name's slot (claim kept as
  claimed_model / claimed_review_type), so it no longer skews ok_count/total.
- `--review-result` is required, so a dropped flag fails instead of reading as
  success.
- The card for this cause no longer says a re-run lands the same way; it says
  a re-run clears a post-review flake, not a forgery.
- Docs state the per-matrix limits plainly: any other non-ok cell explains a
  red leg away, and a forger able to delete and re-upload leaves none.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mattmillerai

Copy link
Copy Markdown
Member Author

🤖 The reviews loop filed Linear follow-up ticket(s) for review thread(s) deferred as out of scope for this PR:

  • BE-19889 — Attribute each cursor-review reviewer leg's conclusion to its own findings artifact — filed as agent-spike (premise unverified)

The following carry agent-spike instead of agent-ok because their reachability claim was not backed by evidence (BE-5378) — the claim is investigated before any code is written, and "the premise does not hold" is a valid, successful outcome:

  • Attribute each cursor-review reviewer leg's conclusion to its own findings artifact — no reachability block in the proposal

@mattmillerai
mattmillerai merged commit 3484514 into main Oct 9, 2026
8 checks passed
@mattmillerai
mattmillerai deleted the matt/be-19886-panel-inconsistent branch October 9, 2026 10:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent-coded Authored by the agent-work loop cursor-review Multi-model cursor review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants