Repository navigation
feat(cursor-review): gate on the direct-OpenAI cells when they replace the Cursor OpenAI lane - #387
Conversation
…e the Cursor OpenAI lane With openai_direct_replaces_cursor_openai (the default once openai_direct_model is set), preflight drops the Cursor gpt-* cells and the OpenAI lane is reviewed only by the direct Codex cell. That cell was advisory and adversarial-only, so the gating panel shrank to two labs (4 cells) and a single lab outage withheld auto-approve. - review-openai-direct is now a 2-entry matrix (adversarial, edge-case) using the same prompt-<review_type>.md as the Cursor cells; artifacts are findings-direct-<review_type>-<model>. - preflight emits openai_direct_counts. When true, the direct job drops its job-level continue-on-error and gains the "Fail the leg when the cell did not submit" step, and Aggregate panel findings counts both direct cells in expected/present/ok_count/total (a missing one is synthesised as error). - Side by side (replace false) the direct cells stay advisory, as before. - The aggregation moves into aggregate-panel.py so it is unit-tested; counted direct cells reach the panel metadata tagged `direct`, and the review's "did not contribute" line names them, e.g. `gpt-x:edge-case (direct, error)`.
|
Warning Review paused — included plan limit reachedKeep your review moving with free on-demand reviews.
On-demand reviews are free for one more day.
Reviews can continue after your included limit without a manual trigger. An admin must approve usage-based billing. Promotion and pricing detailsOn-demand reviews are free for one more day. After that, they cost $0.25 per reviewed file. Review limit detailsOr wait 33 seconds for your next included review. Limit details: You’ve used the included review currently available. Your 129 included PR review attempts over the past 7 days set your current allowance at 1 review per hour. Review configuration: ⚙️ Run configuration
📒 Files selected for processing (7)
Comment |
There was a problem hiding this comment.
🔍 Cursor Review — Consolidated panel
Triggered by @mattmillerai.
Found 3 finding(s).
| Severity | Count |
|---|---|
| 🟡 Medium | 1 |
| 🟢 Low | 2 |
Panel: 4/6 reviewers contributed findings.
Reviewers that did not contribute: kimi-k3-high:adversarial (error), kimi-k3-high:edge-case (error)
|
|
||
| def fill_missing(cells, models, error): | ||
| """Append a status=error record for every expected (model, type) with no cell.""" | ||
| present = {(c.get("model"), c.get("review_type")) for c in cells} |
There was a problem hiding this comment.
🟡 Medium — fill_missing puts (c.get("model"), c.get("review_type")) into a set without checking their types, and load_cells only checks that the record is a dict. A cell's findings.json can be written by its own agent, so a value like "model": [] or {} raises TypeError: unhashable type. That aborts Aggregate panel findings and the whole consolidate job, which throws away every other cell's findings when it should only cost that one cell its slot. In load_cells, either drop records whose model/review_type are not strings (treat them like unreadable artifacts) or coerce those fields to None. Raised by 3 of 6 reviewers (claude-opus-5-5-xhigh adversarial, claude-opus-5-5-xhigh edge-case, gpt-5.6-sol-xhigh edge-case).
| fi | ||
| echo "openai_direct=${openai_direct}" >> "$GITHUB_OUTPUT" | ||
| openai_direct_counts=false | ||
| if [ "$openai_direct" = "true" ] \ |
There was a problem hiding this comment.
🟢 Low — openai_direct_counts=true is set before checking whether any gpt-* id was actually dropped. In the "panel list carries no Cursor OpenAI id to replace; panel unchanged" branch (for example, a panel_models override with no gpt-* id), the direct cells replace nothing but still gate Panel integrity, the auto-approve gate and the rollup. That adds an OpenAI-API failure point the docs don't describe: they say the cells gate only "like the Cursor cells they replace". Move the assignment inside the branch where dropped is non-empty. Raised by 2 of 6 reviewers (claude-opus-5-5-xhigh adversarial, claude-opus-5-5-xhigh edge-case).
| if not counts: | ||
| c["advisory"] = True | ||
| counted = panel + direct if counts else panel | ||
| ok = sum(1 for c in counted if c.get("status") == "ok") |
There was a problem hiding this comment.
🟢 Low — In side-by-side mode (--direct-counts false), ok leaves out the advisory direct cells, but Build judge prompt also uses ok_count != '0' to decide whether any reviewer contributed. If every Cursor cell fails while an advisory direct cell succeeds, the judge is skipped and the direct cell's findings are never adjudicated. Consider giving the judge step a separate "any cell ok" signal that includes advisory cells, and keep ok_count/total gate-only. Raised by 1 of 6 reviewers (gpt-5.6-sol-xhigh edge-case).
Execute the chain that makes the direct-OpenAI cells gate when they replace the Cursor OpenAI lane: preflight's `Define panel models`, consolidate's `Aggregate panel findings`, and the direct leg's seed and leg check, each run from the workflow's own script and env. These one-line edits used to pass every suite while quietly turning the cells advisory again, and each now fails the new tests: deleting preflight's `openai_direct_counts=true`, hard-coding `--direct-counts "false"`, blanking `DIRECT_MODEL`, and turning the leg check's `exit 1` into `exit 0`. Also correct the job comment on what the advisory setup cost: a failed direct cell was free against approve_max_failed_reviewers (a lab outage withholds approval either way). Pluralize the input descriptions, and list aggregate-panel.py in the cursor-review README.
…op-on-submit.py row
ELI-5
When a repo sends its OpenAI review straight to OpenAI instead of through Cursor, that direct reviewer used to be a "guest": its notes reached the judge, but it was never counted when deciding whether the panel was complete enough to auto-approve. It also only did one of the two passes. So turning it on quietly shrank the voting panel from three labs to two. Now the direct reviewer does both passes and, when it replaces the Cursor OpenAI reviewer, it votes like the one it replaced.
What changes
review-openai-directis a 2-entry matrix (adversarial,edge-case). Each leg builds its prompt from the sameprompt-<review_type>.mdthe Cursor cells use. Artifacts arefindings-direct-<review_type>-<model>. Check names are<review_type> direct (<model>), so the adversarial leg keeps its existing context name.openai_direct_counts, set when the direct cell is in effect andopenai_direct_replaces_cursor_openaiis true. When it is set:continue-on-errorevaluates tofalse;Fail the leg when the cell did not submitstep runs after the upload (same contract as the Cursor step of that name), so a failed direct cell is red in the rollup;Aggregate panel findingsputs both direct cells inexpected,present,ok_countandtotal. A missing artifact is filled in asstatus=error, just as it is for a Cursor cell. The panel line reads e.g.6/6.false) is unchanged. The direct cells stay advisory: they are never counted, never red, and kept out of the panel metadata auto-approve reads..github/cursor-review/aggregate-panel.pyso it can be unit-tested. Both markers (direct_api,advisory) are popped from every cell's own record and set only by the script, so a cell still cannot excuse itself from the count.panelmetadata taggeddirect.post-review.pynames a failed one in the "Reviewers that did not contribute" line asgpt-5.6-sol:edge-case (direct, error). The judge input carries each direct cell's ownreview_type.openai_direct_model,openai_direct_replaces_cursor_openai),.github/cursor-review/README.mdanddocs/callers/cursor-review.mddescribe the new gating.Judgment call: "replace is true" is read as "
openai_directis in effect AND the replace input is true". That includes the edge case where the panel list had nogpt-*id to drop. In that case the direct cells are still the run's only OpenAI lane, so they count.Self-review notes
continue-on-error: ${{ needs.preflight.outputs.openai_direct_counts != 'true' }}. GitHub allowsneedsandmatrixin that expression, and actionlint reports nothing new on this file versusmain(only the existingjob.workflow_sha/ SC2018-19 notices). If the output is ever empty, the expression evaluates totrue, which is today's advisory behaviour. So the failure direction is "does not gate", never "spuriously red".consolidatestill gates on!cancelled()and reads the direct job only for ordering. A failed or absent direct cell is counted from its artifact, or from the missing artifact.Provenance
test_aggregate_panel.py, which covers replace=true with both direct cells ok (6/6), one direct error (5/6), a missing direct artifact (synthesised as an error), replace=false (advisory, 4/4), marker forgery, the(direct, error)summary line, andpanel_gatecounting a direct failure.test_workflow_panel_integrity.py's aggregate-step pin to follow the counting into the script.check_workflow_pins.py,check_agents_md.py, org-repo-literal lint: passcursor-review.yml: no new findings vsmainResidual
continue-on-error, and the red leg on a failed direct cell. The first consumer run after the SHA bump will show it. Check that the review postsPanel: 6/6and shows twodirect (gpt-…)legs. Also check that a direct error turns its leg red and appears as(direct, error).Panel: 2/4alongsideDirect-API cell (advisory, not counted): status=ok) was not re-read. I have no access to those run logs, so the fix is checked against the aggregation code path and unit tests, not those runs.OPENAI_API_KEYspend limit may need to raise it.approve_max_failed_reviewers, and moving another lab to a direct-API path.Review follow-ups
approve_max_failed_reviewers; before this PR it was free. Callers with a 1-failure allowance therefore gate on the full panel again: a failed direct cell uses up that allowance, just as a failed Cursor cell does. A whole-lab outage is two failed cells, so it withholds approval under that allowance both before and after this change.ci-cursor-review.yml) does not setopenai_direct_model, and it runs the reusable workflow pinned to a main SHA.DirectCellsGateWhenTheyReplaceTestintest_workflow_panel_integrity.py). They execute the real step scripts, from preflight'sDefine panel modelsthroughAggregate panel findingsto the direct leg check. Four one-line edits used to pass every suite while making the cells advisory again: deleting preflight'sopenai_direct_counts=true, hard-coding--direct-counts "false", blankingDIRECT_MODEL, and changing the leg check'sexit 1toexit 0. Each was applied to a scratch copy, and each now fails these tests.aggregate-panel.pyin the README file table. Verified: the cursor-review suite (1194 tests), the cursor-approve suite, the workflow-pins suite withcheck_workflow_pins.py, and the org-repo-literal lint all pass; actionlint shows no new findings.