Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion .github/cursor-review/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -242,7 +242,7 @@ the consolidated review's panel table. See [Panel integrity](#panel-integrity).

| File | Role |
|---|---|
| [`aggregate-panel.py`](aggregate-panel.py) | Run by `consolidate`'s **Aggregate panel findings** step. Combines the cells' `findings-*` artifacts into the `panel.json` the judge reads, adds a `status=error` record for every expected cell whose artifact never arrived, and writes the `ok_count` / `total` / `panel_inconsistent` that `Panel integrity` and auto-approve read (`panel_inconsistent`: a reviewer matrix did not succeed although every counted cell artifact reads `ok`, or a counted record is `status: mismatch` — it disagrees with its artifact name or names no cell the panel runs). The direct-OpenAI cells (`findings-direct-*`) count like Cursor cells when they replace the Cursor OpenAI lane (`openai_direct_replaces_cursor_openai: true`, so a failed one also counts against `approve_max_failed_reviewers`) and stay advisory side by side. Only this script sets the `direct_api` / `advisory` markers, so a cell's own record cannot excuse itself from the count. |
| [`aggregate-panel.py`](aggregate-panel.py) | Run by `consolidate`'s **Aggregate panel findings** step. Combines the cells' `findings-*` artifacts into the `panel.json` the judge reads, adds a `status=error` record for every expected cell whose artifact never arrived, and writes the `ok_count` / `total` / `panel_inconsistent` that `Panel integrity` and auto-approve read (`panel_inconsistent`: a reviewer matrix did not succeed although every counted cell artifact reads `ok`, or a counted record is `status: mismatch` — it disagrees with its artifact name or names no cell the panel runs). The direct-OpenAI cells (`findings-direct-*`) count like Cursor cells when they replace the Cursor OpenAI lane (`openai_direct_replaces_cursor_openai: true`, so a failed one also counts against `approve_max_failed_reviewers`) and stay advisory side by side; the direct-Anthropic cells do the same under `anthropic_direct_replaces_cursor_anthropic`, each artifact assigned to its lab by model id and counted only by that lab's flag. Only this script sets the `direct_api` / `advisory` markers, so a cell's own record cannot excuse itself from the count. |
| [`prompt-adversarial.md`](prompt-adversarial.md) | Prompt for the security/reliability review pass. |
| [`prompt-edge-case.md`](prompt-edge-case.md) | Prompt for the correctness/logic review pass. |
| [`prompt-judge.md`](prompt-judge.md) | Prompt the judge model uses to consolidate panel findings into one review. |
Expand Down Expand Up @@ -327,6 +327,7 @@ jobs:
|---|---|---|---|
| Secret | `CURSOR_API_KEY` | **yes** | Bills the panel + judge `cursor-agent` calls. |
| Secret | `OPENAI_API_KEY` | no | Only with `openai_direct_model`: bills the optional direct-OpenAI cells. Use a project-scoped service-account key with a spend limit. |
| Secret | `ANTHROPIC_API_KEY` | no | Only with `anthropic_direct_model`: bills the optional direct-Anthropic cells. Use a key scoped to its own Anthropic workspace with a spend limit, never a personal key. |
| Secret | `SLACK_BOT_TOKEN` | no | Enables start/complete DMs to the triggerer. |
| Variable | `CURSOR_REVIEW_DM_EMAIL_MAP` | no | Maps GitHub logins → emails for Slack DM lookup. |

Expand Down Expand Up @@ -391,6 +392,9 @@ All optional except `workflows_ref` (required, no default) — pass them under
| `panel_models` | `''` (built-in list) | JSON array of Cursor model ids that **replaces** the panel list; each still runs both review types and is validated against the live catalog by preflight. For per-repo experiments (e.g. `-xhigh` vs `-max` tiers). |
| `openai_direct_model` | `''` (off) | OPT-IN. An OpenAI model id reviewed through the OpenAI API directly (pinned Codex CLI, `package.json` here) as two cells (**adversarial** + **edge-case**, the Cursor cells' prompts) on GitHub-hosted runners, by default **replacing** the Cursor `gpt-*` cells (next row). Needs the `OPENAI_API_KEY` secret and a plain id (`[A-Za-z0-9._-]`); without either, preflight warns and leaves the cells off and the panel unchanged. Not part of `panel_models` (not in the Cursor catalog). Whether the cells gate depends on the next row. |
| `openai_direct_replaces_cursor_openai` | `true` | When the direct cells are in effect, drop the Cursor `gpt-*` cell(s) so the OpenAI lane is reviewed once — the direct cells then **gate** like the Cursor cells they replace: counted in `Panel integrity` and the auto-approve gate, a failed one red in the rollup and named `(direct, <status>)` in the review's "did not contribute" line. `false` runs both for a side-by-side backend comparison, with the direct cells **advisory** (findings reach the judge; never counted, never red). Preflight fails rather than leave the Cursor panel empty. |
| `anthropic_direct_model` | `''` (off) | OPT-IN. The Anthropic twin of `openai_direct_model`: an Anthropic model id (e.g. `claude-opus-5-5`) reviewed through the Anthropic API directly (pinned Claude Code CLI, `package.json` here) as two cells on GitHub-hosted runners, by default **replacing** the Cursor `claude-*` cells. Needs the `ANTHROPIC_API_KEY` secret and a plain id other than `openai_direct_model`'s; without any of those, preflight warns and leaves the cells off and the panel unchanged. The agent also reads a **read-only PR checkout** (symlinks removed) with Read, Grep and Glob only — no Bash or code-running tool — confined to it by `--restricted`, with `/proc` and `/sys` also denied by explicit deny rules. This panel reviews private-repo diffs and the workflow cannot check the Anthropic organization's data-retention terms: callers on private repos must use a key from an organization with zero data retention. Each cell also uploads a `usage-direct-<review_type>-<model>` artifact (kept 30 days) with the API's token counts for that cell, so cost can be measured without an Admin API key. |
| `anthropic_direct_effort` | `xhigh` | `--effort` for the direct Anthropic cells: `low`, `medium`, `high`, `xhigh` or `max`; anything else warns and falls back to `xhigh`. |
| `anthropic_direct_replaces_cursor_anthropic` | `true` | As `openai_direct_replaces_cursor_openai`, for the Cursor `claude-*` cells. Each lab counts by its own flag. Never touches `judge_model`. |
| `cell_timeout_minutes` | `15` | Wall-clock cap on one reviewer cell's agent step (Cursor or direct). Over it the cell exits 130, uploads `status: error` with no findings, reddens its leg and fails `Panel integrity`. Shared by the cell's in-step retries, so it bounds attempts + backoff. Derives three more caps that cannot be set alone: cell job = this + 15, `Run judge` step = max(30, this × 2), judge job = that + 10. At the default: 15 / 30 / 30 / 40, today's values exactly. Preflight bounds it 10–45. |
| `skip_bot_branch_prefixes` | `ci/bump- chore/refresh- auto/refresh-` | Skip the panel when the PR author is a Bot **and** the head branch starts with one of these prefixes (machine pin bumps / catalog refreshes). `''` reviews every bot PR. |
| `diff_size_cap` | `5000` | Max counted changed lines (after generated-file exclusion and comment discounting); larger PRs are skipped. |
Expand Down
67 changes: 53 additions & 14 deletions .github/cursor-review/aggregate-panel.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,12 @@
reach the judge, but they stay out of `present` (so one cannot mask a missing
Cursor cell with the same id) and out of ok_count/total.

The role is chosen PER LAB: the OpenAI cells (`--direct-model`,
`--direct-counts`, `--direct-result`, the original flags) and the Anthropic
ones (`--anthropic-direct-*`) each count only by their own lab's flag. A direct
artifact is assigned to a lab by its model id; one matching neither lab is
unexpected — a mismatch when any lab counts, advisory otherwise.

Two markers are set HERE and only here — a cell's own record is model-adjacent
output and must never be able to excuse itself from the count: `direct_api`
(this cell came through the direct backend) and `advisory` (keep it out of the
Expand Down Expand Up @@ -61,7 +67,9 @@
Stdlib only. Usage:
aggregate-panel.py --panel-dir /tmp/panel --models '<json list>' \
[--direct-model <id> --direct-counts true|false] \
[--anthropic-direct-model <id> --anthropic-direct-counts true|false] \
--review-result <result> [--direct-result <result>] \
[--anthropic-direct-result <result>] \
--out /tmp/panel.json [--github-output "$GITHUB_OUTPUT"]
"""

Expand Down Expand Up @@ -171,7 +179,9 @@ def _all_ok(cells):


def aggregate(panel_dir, models, direct_model="", direct_counts=False,
review_result="success", direct_result="skipped"):
review_result="success", direct_result="skipped",
anthropic_direct_model="", anthropic_direct_counts=False,
anthropic_direct_result="skipped"):
"""Return (cells, ok_count, total, inconsistent).

`cells` is everything the judge reads; `inconsistent` is either
Expand All @@ -180,22 +190,43 @@ def aggregate(panel_dir, models, direct_model="", direct_counts=False,
panel, direct = load_cells(panel_dir)
flag_unexpected(panel, models)
fill_missing(panel, models, "Cell findings artifact was not uploaded.")
counts = bool(direct_counts and direct_model)
if counts:
flag_unexpected(direct, [direct_model])
# Same contract as a Cursor cell: absent means failed, never "not run".
fill_missing(direct, [direct_model], "Direct-API cell findings artifact was not uploaded.")
# (model, counts, matrix result) per lab; OpenAI first, the original flags.
labs = [
(direct_model, bool(direct_counts and direct_model), direct_result),
(anthropic_direct_model, bool(anthropic_direct_counts and anthropic_direct_model),
anthropic_direct_result),
]
by_lab = [[] for _ in labs]
unassigned = []
for c in direct:
for i, (model, _, _) in enumerate(labs):
if model and c.get("model") == model:
Comment thread
mattmillerai marked this conversation as resolved.
Resolved
by_lab[i].append(c)
break
else:
unassigned.append(c)
any_counts = any(counts for _, counts, _ in labs)
if any_counts:
# Same as a Cursor cell: a direct artifact no lab runs is a mismatch.
flag_unexpected(unassigned, [])
counted = list(panel)
inconsistent = review_result != "success" and _all_ok(panel)
for (model, counts, result), cells in zip(labs, by_lab):
if counts:
# Same contract as a Cursor cell: absent means failed, never "not run".
fill_missing(cells, [model], "Direct-API cell findings artifact was not uploaded.")
counted += cells
inconsistent = inconsistent or (result not in ("success", "skipped") and _all_ok(cells))
if any_counts:
counted += unassigned
direct = [c for cells in by_lab for c in cells] + unassigned
counted_ids = {id(c) for c in counted}
for c in direct:
c["direct_api"] = True
if not counts:
if id(c) not in counted_ids:
c["advisory"] = True
counted = panel + direct if counts else panel
ok = sum(1 for c in counted if c.get("status") == "ok")
inconsistent = (
(review_result != "success" and _all_ok(panel))
or (counts and direct_result not in ("success", "skipped") and _all_ok(direct))
or any(c.get("status") == MISMATCH_STATUS for c in counted)
)
inconsistent = inconsistent or any(c.get("status") == MISMATCH_STATUS for c in counted)
return panel + direct, ok, len(counted), inconsistent


Expand All @@ -213,22 +244,30 @@ def main(argv=None):
# Required, so a dropped flag fails the step rather than reading as success.
p.add_argument("--review-result", required=True, help="needs.review.result")
p.add_argument("--direct-result", default="skipped", help="needs.review-openai-direct.result")
p.add_argument("--anthropic-direct-model", default="")
p.add_argument("--anthropic-direct-counts", default="false")
p.add_argument("--anthropic-direct-result", default="skipped",
help="needs.review-anthropic-direct.result")
p.add_argument("--out", required=True)
p.add_argument("--github-output", default="")
args = p.parse_args(argv)

counts = args.direct_counts.strip().lower() == "true"
review_result = args.review_result.strip()
direct_result = args.direct_result.strip()
anthropic_counts = args.anthropic_direct_counts.strip().lower() == "true"
anthropic_result = args.anthropic_direct_result.strip()
cells, ok, total, inconsistent = aggregate(
args.panel_dir, json.loads(args.models), args.direct_model.strip(), counts,
review_result, direct_result,
args.anthropic_direct_model.strip(), anthropic_counts, anthropic_result,
)
print(f"Panel: {ok}/{total} cells contributed findings.")
if inconsistent:
print(
f"::warning::A counted findings artifact may not be its own cell's: a reviewer matrix "
f"did not succeed (review={_log_safe(review_result)}, direct={_log_safe(direct_result)}) "
f"did not succeed (review={_log_safe(review_result)}, direct={_log_safe(direct_result)}, "
f"anthropic_direct={_log_safe(anthropic_result)}) "
f"although every counted cell artifact of it reads ok, or a record is "
f"status={MISMATCH_STATUS}. The panel count is untrusted (panel_inconsistent=true)."
)
Expand Down
5 changes: 3 additions & 2 deletions .github/cursor-review/package.json
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
{
"name": "comfy-cursor-review-direct-cli",
"private": true,
"description": "Manifest whose ONLY job is to be the single, Dependabot-visible source of truth for the @openai/codex CLI version that cursor-review.yml's optional direct-API review cell installs. Nothing is ever `npm install`ed from this directory: the job reads the pin out of this file and runs `npm install -g @openai/codex@<pin>`. It is a real npm manifest rather than a workflow `env:` constant purely so Dependabot's npm ecosystem can see the pin and open a bump PR — an inline `npm install -g pkg@1.2.3` inside a `run:` step is invisible to every ecosystem, which is how the groom CLI pin silently rotted (see .github/groom/package.json, the precedent this mirrors). Keep the version EXACT (no ^ or ~): the agent CLI is executable supply chain for a step that reads untrusted repo content, and the job rejects a non-exact pin at run time.",
"description": "Manifest whose ONLY job is to be the single, Dependabot-visible source of truth for the CLI versions cursor-review.yml's optional direct-API review cells install: @openai/codex (the direct OpenAI cells) and @anthropic-ai/claude-code (the direct Anthropic cells). Nothing is ever `npm install`ed from this directory: the job reads the pin out of this file and runs `npm install -g <package>@<pin>`. It is a real npm manifest rather than a workflow `env:` constant purely so Dependabot's npm ecosystem can see the pin and open a bump PR — an inline `npm install -g pkg@1.2.3` inside a `run:` step is invisible to every ecosystem, which is how the groom CLI pin silently rotted (see .github/groom/package.json, the precedent this mirrors). Keep the version EXACT (no ^ or ~): the agent CLI is executable supply chain for a step that reads untrusted repo content, and the job rejects a non-exact pin at run time.",
"dependencies": {
"@openai/codex": "0.162.0"
"@openai/codex": "0.162.0",
"@anthropic-ai/claude-code": "2.1.286"
}
}
Loading
Loading