Skip to content

feat(cursor-review): gate on the direct-OpenAI cells when they replace the Cursor OpenAI lane - #387

Merged
mattmillerai merged 3 commits into
mainfrom
matt/be-19850-direct-openai-gating
Oct 9, 2026
Merged

mattmillerai merged 3 commits into
mainfrom
matt/be-19850-direct-openai-gating

Conversation

@mattmillerai

@mattmillerai mattmillerai commented Oct 9, 2026 •

Copy link
Copy Markdown
Member

ELI-5

When a repo sends its OpenAI review straight to OpenAI instead of through Cursor, that direct reviewer used to be a "guest": its notes reached the judge, but it was never counted when deciding whether the panel was complete enough to auto-approve. It also only did one of the two passes. So turning it on quietly shrank the voting panel from three labs to two. Now the direct reviewer does both passes and, when it replaces the Cursor OpenAI reviewer, it votes like the one it replaced.

What changes

  • review-openai-direct is a 2-entry matrix (adversarial, edge-case). Each leg builds its prompt from the same prompt-<review_type>.md the Cursor cells use. Artifacts are findings-direct-<review_type>-<model>. Check names are <review_type> direct (<model>), so the adversarial leg keeps its existing context name.
  • preflight emits openai_direct_counts, set when the direct cell is in effect and openai_direct_replaces_cursor_openai is true. When it is set:
    • the job's continue-on-error evaluates to false;
    • a new Fail the leg when the cell did not submit step runs after the upload (same contract as the Cursor step of that name), so a failed direct cell is red in the rollup;
    • Aggregate panel findings puts both direct cells in expected, present, ok_count and total. A missing artifact is filled in as status=error, just as it is for a Cursor cell. The panel line reads e.g. 6/6.
  • Side-by-side mode (replace false) is unchanged. The direct cells stay advisory: they are never counted, never red, and kept out of the panel metadata auto-approve reads.
  • Aggregation moved from inline Python into .github/cursor-review/aggregate-panel.py so it can be unit-tested. Both markers (direct_api, advisory) are popped from every cell's own record and set only by the script, so a cell still cannot excuse itself from the count.
  • Counted direct cells enter the panel metadata tagged direct. post-review.py names a failed one in the "Reviewers that did not contribute" line as gpt-5.6-sol:edge-case (direct, error). The judge input carries each direct cell's own review_type.
  • Docs updated: the input descriptions (openai_direct_model, openai_direct_replaces_cursor_openai), .github/cursor-review/README.md and docs/callers/cursor-review.md describe the new gating.

Judgment call: "replace is true" is read as "openai_direct is in effect AND the replace input is true". That includes the edge case where the panel list had no gpt-* id to drop. In that case the direct cells are still the run's only OpenAI lane, so they count.

Self-review notes

  • Riskiest line: the job-level continue-on-error: ${{ needs.preflight.outputs.openai_direct_counts != 'true' }}. GitHub allows needs and matrix in that expression, and actionlint reports nothing new on this file versus main (only the existing job.workflow_sha / SC2018-19 notices). If the output is ever empty, the expression evaluates to true, which is today's advisory behaviour. So the failure direction is "does not gate", never "spuriously red".
  • A red direct leg cannot block consolidation. consolidate still gates on !cancelled() and reads the direct job only for ordering. A failed or absent direct cell is counted from its artifact, or from the missing artifact.
  • Cost: with replace on, each run makes one extra Codex call (the edge-case pass), which the ticket asks for.

Provenance

  • Authored by: agent-work loop
  • Verified:
    • Tests: cursor-review script suite all green (1184 tests). This includes the new test_aggregate_panel.py, which covers replace=true with both direct cells ok (6/6), one direct error (5/6), a missing direct artifact (synthesised as an error), replace=false (advisory, 4/4), marker forgery, the (direct, error) summary line, and panel_gate counting a direct failure.
    • Updated test_workflow_panel_integrity.py's aggregate-step pin to follow the counting into the script.
    • workflow-pins suite + check_workflow_pins.py, check_agents_md.py, org-repo-literal lint: pass
    • actionlint on cursor-review.yml: no new findings vs main
    • Not run: an end-to-end Actions run of the reusable workflow (it only executes from a caller after merge + bump)
  • Deviations: none against the acceptance criteria

Residual

  • No live run yet. None of the new behaviour has been exercised in a real Actions run: the 2-leg direct matrix, the expression-valued job continue-on-error, and the red leg on a failed direct cell. The first consumer run after the SHA bump will show it. Check that the review posts Panel: 6/6 and shows two direct (gpt-…) legs. Also check that a direct error turns its leg red and appears as (direct, error).
  • Ticket evidence not reproduced from here. The ~220 / 18 cloud-run sample (consolidate logs showing Panel: 2/4 alongside Direct-API cell (advisory, not counted): status=ok) was not re-read. I have no access to those run logs, so the fix is checked against the aggregation code path and unit tests, not those runs.
  • Cost not measured. The extra edge-case Codex call per run doubles direct-API spend on the OpenAI lane. Callers on a tight OPENAI_API_KEY spend limit may need to raise it.
  • Out of scope per the ticket: retrying the direct cell on transient OpenAI errors, more than one direct model, changing the Kimi model or any caller's approve_max_failed_reviewers, and moving another lab to a direct-API path.

Review follow-ups

  • A failed direct cell now counts. With replace on, a failed direct cell counts against approve_max_failed_reviewers; before this PR it was free. Callers with a 1-failure allowance therefore gate on the full panel again: a failed direct cell uses up that allowance, just as a failed Cursor cell does. A whole-lab outage is two failed cells, so it withholds approval under that allowance both before and after this change.
  • Not exercised here. This repo's own panel run has not taken the new path: its caller (ci-cursor-review.yml) does not set openai_direct_model, and it runs the reusable workflow pinned to a main SHA.
  • Wiring tests added (DirectCellsGateWhenTheyReplaceTest in test_workflow_panel_integrity.py). They execute the real step scripts, from preflight's Define panel models through Aggregate panel findings to the direct leg check. Four one-line edits used to pass every suite while making the cells advisory again: deleting preflight's openai_direct_counts=true, hard-coding --direct-counts "false", blanking DIRECT_MODEL, and changing the leg check's exit 1 to exit 0. Each was applied to a scratch copy, and each now fails these tests.
  • Also: corrected the job comment's lab-outage rationale, made the input descriptions plural, and listed aggregate-panel.py in the README file table. Verified: the cursor-review suite (1194 tests), the cursor-approve suite, the workflow-pins suite with check_workflow_pins.py, and the org-repo-literal lint all pass; actionlint shows no new findings.

…e the Cursor OpenAI lane

With openai_direct_replaces_cursor_openai (the default once
openai_direct_model is set), preflight drops the Cursor gpt-* cells and the
OpenAI lane is reviewed only by the direct Codex cell. That cell was advisory
and adversarial-only, so the gating panel shrank to two labs (4 cells) and a
single lab outage withheld auto-approve.

- review-openai-direct is now a 2-entry matrix (adversarial, edge-case)
  using the same prompt-<review_type>.md as the Cursor cells; artifacts are
  findings-direct-<review_type>-<model>.
- preflight emits openai_direct_counts. When true, the direct job drops its
  job-level continue-on-error and gains the "Fail the leg when the cell did
  not submit" step, and Aggregate panel findings counts both direct cells in
  expected/present/ok_count/total (a missing one is synthesised as error).
- Side by side (replace false) the direct cells stay advisory, as before.
- The aggregation moves into aggregate-panel.py so it is unit-tested; counted
  direct cells reach the panel metadata tagged `direct`, and the review's
  "did not contribute" line names them, e.g. `gpt-x:edge-case (direct, error)`.
@mattmillerai mattmillerai added cursor-review Multi-model cursor review agent-coded Authored by the agent-work loop labels Oct 9, 2026
@coderabbitai

coderabbitai Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Review paused — included plan limit reached

Keep your review moving with free on-demand reviews.

  • Run this review for free

On-demand reviews are free for one more day.

  • Ask an admin to make reviews automatic

Open in CodeRabbit

Reviews can continue after your included limit without a manual trigger. An admin must approve usage-based billing.

Promotion and pricing details

On-demand reviews are free for one more day. After that, they cost $0.25 per reviewed file.

Review limit details

Or wait 33 seconds for your next included review.

Check out review usage here.

Limit details: You’ve used the included review currently available. Your 129 included PR review attempts over the past 7 days set your current allowance at 1 review per hour.

Learn how review limits work.

Review configuration:

⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: ASSERTIVE
  • Plan: Team
  • Run ID: a0b18a20-98db-4f81-9d79-18a858e0c5b7
📥 Commits

Reviewing files that changed from the base of the PR and between 7de68e5 and 36ae64c.

📒 Files selected for processing (7)
  • .github/cursor-review/README.md
  • .github/cursor-review/aggregate-panel.py
  • .github/cursor-review/post-review.py
  • .github/cursor-review/tests/test_aggregate_panel.py
  • .github/cursor-review/tests/test_workflow_panel_integrity.py
  • .github/workflows/cursor-review.yml
  • docs/callers/cursor-review.md
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 Cursor Review — Consolidated panel

Triggered by @mattmillerai.

Found 3 finding(s).

Severity Count
🟡 Medium 1
🟢 Low 2

Panel: 4/6 reviewers contributed findings.

Reviewers that did not contribute: kimi-k3-high:adversarial (error), kimi-k3-high:edge-case (error)


def fill_missing(cells, models, error):
"""Append a status=error record for every expected (model, type) with no cell."""
present = {(c.get("model"), c.get("review_type")) for c in cells}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium — fill_missing puts (c.get("model"), c.get("review_type")) into a set without checking their types, and load_cells only checks that the record is a dict. A cell's findings.json can be written by its own agent, so a value like "model": [] or {} raises TypeError: unhashable type. That aborts Aggregate panel findings and the whole consolidate job, which throws away every other cell's findings when it should only cost that one cell its slot. In load_cells, either drop records whose model/review_type are not strings (treat them like unreadable artifacts) or coerce those fields to None. Raised by 3 of 6 reviewers (claude-opus-5-5-xhigh adversarial, claude-opus-5-5-xhigh edge-case, gpt-5.6-sol-xhigh edge-case).

fi
echo "openai_direct=${openai_direct}" >> "$GITHUB_OUTPUT"
openai_direct_counts=false
if [ "$openai_direct" = "true" ] \

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Low — openai_direct_counts=true is set before checking whether any gpt-* id was actually dropped. In the "panel list carries no Cursor OpenAI id to replace; panel unchanged" branch (for example, a panel_models override with no gpt-* id), the direct cells replace nothing but still gate Panel integrity, the auto-approve gate and the rollup. That adds an OpenAI-API failure point the docs don't describe: they say the cells gate only "like the Cursor cells they replace". Move the assignment inside the branch where dropped is non-empty. Raised by 2 of 6 reviewers (claude-opus-5-5-xhigh adversarial, claude-opus-5-5-xhigh edge-case).

if not counts:
c["advisory"] = True
counted = panel + direct if counts else panel
ok = sum(1 for c in counted if c.get("status") == "ok")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Low — In side-by-side mode (--direct-counts false), ok leaves out the advisory direct cells, but Build judge prompt also uses ok_count != '0' to decide whether any reviewer contributed. If every Cursor cell fails while an advisory direct cell succeeds, the judge is skipped and the direct cell's findings are never adjudicated. Consider giving the judge step a separate "any cell ok" signal that includes advisory cells, and keep ok_count/total gate-only. Raised by 1 of 6 reviewers (gpt-5.6-sol-xhigh edge-case).

Execute the chain that makes the direct-OpenAI cells gate when they
replace the Cursor OpenAI lane: preflight's `Define panel models`,
consolidate's `Aggregate panel findings`, and the direct leg's seed and
leg check, each run from the workflow's own script and env. These
one-line edits used to pass every suite while quietly turning the cells
advisory again, and each now fails the new tests: deleting preflight's
`openai_direct_counts=true`, hard-coding `--direct-counts "false"`,
blanking `DIRECT_MODEL`, and turning the leg check's `exit 1` into
`exit 0`.

Also correct the job comment on what the advisory setup cost: a failed
direct cell was free against approve_max_failed_reviewers (a lab outage
withholds approval either way). Pluralize the input descriptions, and
list aggregate-panel.py in the cursor-review README.
@mattmillerai
mattmillerai merged commit c37492c into main Oct 9, 2026
8 checks passed
@mattmillerai
mattmillerai deleted the matt/be-19850-direct-openai-gating branch October 9, 2026 08:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent-coded Authored by the agent-work loop cursor-review Multi-model cursor review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants