Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions .github/cursor-review/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -217,6 +217,7 @@ the consolidated review's panel table. See [Panel integrity](#panel-integrity).

| File | Role |
|---|---|
| [`aggregate-panel.py`](aggregate-panel.py) | Run by `consolidate`'s **Aggregate panel findings** step. Combines the cells' `findings-*` artifacts into the `panel.json` the judge reads, adds a `status=error` record for every expected cell whose artifact never arrived, and writes the `ok_count` / `total` that `Panel integrity` reads. The direct-OpenAI cells (`findings-direct-*`) count like Cursor cells when they replace the Cursor OpenAI lane (`openai_direct_replaces_cursor_openai: true`, so a failed one also counts against `approve_max_failed_reviewers`) and stay advisory side by side. Only this script sets the `direct_api` / `advisory` markers, so a cell's own record cannot excuse itself from the count. |
| [`prompt-adversarial.md`](prompt-adversarial.md) | Prompt for the security/reliability review pass. |
| [`prompt-edge-case.md`](prompt-edge-case.md) | Prompt for the correctness/logic review pass. |
| [`prompt-judge.md`](prompt-judge.md) | Prompt the judge model uses to consolidate panel findings into one review. |
Expand Down Expand Up @@ -300,7 +301,7 @@ jobs:
| Kind | Name | Required | Purpose |
|---|---|---|---|
| Secret | `CURSOR_API_KEY` | **yes** | Bills the panel + judge `cursor-agent` calls. |
| Secret | `OPENAI_API_KEY` | no | Only with `openai_direct_model`: bills the optional direct-OpenAI cell. Use a project-scoped service-account key with a spend limit. |
| Secret | `OPENAI_API_KEY` | no | Only with `openai_direct_model`: bills the optional direct-OpenAI cells. Use a project-scoped service-account key with a spend limit. |
| Secret | `SLACK_BOT_TOKEN` | no | Enables start/complete DMs to the triggerer. |
| Variable | `CURSOR_REVIEW_DM_EMAIL_MAP` | no | Maps GitHub logins → emails for Slack DM lookup. |

Expand Down Expand Up @@ -363,8 +364,8 @@ All optional except `workflows_ref` (required, no default) — pass them under
|---|---|---|
| `judge_model` | `claude-opus-5-5-xhigh` | Model that consolidates panel findings. |
| `panel_models` | `''` (built-in list) | JSON array of Cursor model ids that **replaces** the panel list; each still runs both review types and is validated against the live catalog by preflight. For per-repo experiments (e.g. `-xhigh` vs `-max` tiers). |
| `openai_direct_model` | `''` (off) | OPT-IN. An OpenAI model id reviewed through the OpenAI API directly (pinned Codex CLI, `package.json` here) as ONE **adversarial** cell on GitHub-hosted runners, by default **replacing** the Cursor `gpt-*` cell (next row). Needs the `OPENAI_API_KEY` secret and a plain id (`[A-Za-z0-9._-]`); without either, preflight warns and leaves the cell off and the panel unchanged. Not part of `panel_models` (not in the Cursor catalog); advisory — the leg never fails the run and is not counted in `Panel integrity` / the auto-approve gate. |
| `openai_direct_replaces_cursor_openai` | `true` | When the direct cell is in effect, drop the Cursor `gpt-*` cell(s) so the OpenAI lane is reviewed once. `false` runs both for a side-by-side backend comparison. Preflight fails rather than leave the Cursor panel empty. |
| `openai_direct_model` | `''` (off) | OPT-IN. An OpenAI model id reviewed through the OpenAI API directly (pinned Codex CLI, `package.json` here) as two cells (**adversarial** + **edge-case**, the Cursor cells' prompts) on GitHub-hosted runners, by default **replacing** the Cursor `gpt-*` cells (next row). Needs the `OPENAI_API_KEY` secret and a plain id (`[A-Za-z0-9._-]`); without either, preflight warns and leaves the cells off and the panel unchanged. Not part of `panel_models` (not in the Cursor catalog). Whether the cells gate depends on the next row. |
| `openai_direct_replaces_cursor_openai` | `true` | When the direct cells are in effect, drop the Cursor `gpt-*` cell(s) so the OpenAI lane is reviewed once — the direct cells then **gate** like the Cursor cells they replace: counted in `Panel integrity` and the auto-approve gate, a failed one red in the rollup and named `(direct, <status>)` in the review's "did not contribute" line. `false` runs both for a side-by-side backend comparison, with the direct cells **advisory** (findings reach the judge; never counted, never red). Preflight fails rather than leave the Cursor panel empty. |
| `skip_bot_branch_prefixes` | `ci/bump- chore/refresh- auto/refresh-` | Skip the panel when the PR author is a Bot **and** the head branch starts with one of these prefixes (machine pin bumps / catalog refreshes). `''` reviews every bot PR. |
| `diff_size_cap` | `5000` | Max counted changed lines (after generated-file exclusion and comment discounting); larger PRs are skipped. |
| `ignore_comments` | `true` | Discount blank/comment-only lines from the size count (count-only; the panel still sees them). |
Expand Down
132 changes: 132 additions & 0 deletions .github/cursor-review/aggregate-panel.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,132 @@
#!/usr/bin/env python3
"""Combine the panel cells' findings artifacts into one panel.json.

Run by `consolidate`'s `Aggregate panel findings` step, after
`Download panel findings` has unpacked every `findings-*` artifact under one
directory. Each artifact directory holds one findings.json. The output keeps the
cell metadata the judge and `Build consolidated findings file` read, and the
`ok_count` / `total` pair is what `Panel integrity` and auto-approve gate on.

The opt-in direct-API cells (`findings-direct-<review_type>-<model>`) take one
of two roles, chosen by `--direct-counts`:

- counted (`openai_direct_replaces_cursor_openai: true`): the direct cells ARE
the panel's OpenAI lane, so they join the expected set and ok_count/total
exactly like a Cursor cell — a missing or errored one is a failed reviewer.
- advisory (replace false, the side-by-side comparison): their findings still
reach the judge, but they stay out of `present` (so one cannot mask a missing
Cursor cell with the same id) and out of ok_count/total.

Two markers are set HERE and only here — a cell's own record is model-adjacent
output and must never be able to excuse itself from the count: `direct_api`
(this cell came through the direct backend) and `advisory` (keep it out of the
panel metadata auto-approve counts failures from).

Stdlib only. Usage:
aggregate-panel.py --panel-dir /tmp/panel --models '<json list>' \
[--direct-model <id> --direct-counts true|false] \
--out /tmp/panel.json [--github-output "$GITHUB_OUTPUT"]
"""

import argparse
import glob
import json
import os
import re
import sys

REVIEW_TYPES = ("adversarial", "edge-case")
DIRECT_PREFIX = "findings-direct-"
MARKERS = ("direct_api", "advisory")


def load_cells(panel_dir):
"""(cursor_cells, direct_cells) read from `<panel_dir>/findings-*/findings.json`."""
cursor, direct = [], []
for path in sorted(glob.glob(os.path.join(panel_dir, "findings-*", "findings.json"))):
try:
with open(path, encoding="utf-8") as f:
cell = json.load(f)
except (OSError, ValueError):
continue
if not isinstance(cell, dict):
continue
for marker in MARKERS:
cell.pop(marker, None)
if os.path.basename(os.path.dirname(path)).startswith(DIRECT_PREFIX):
direct.append(cell)
else:
cursor.append(cell)
return cursor, direct


def fill_missing(cells, models, error):
"""Append a status=error record for every expected (model, type) with no cell."""
present = {(c.get("model"), c.get("review_type")) for c in cells}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium — fill_missing puts (c.get("model"), c.get("review_type")) into a set without checking their types, and load_cells only checks that the record is a dict. A cell's findings.json can be written by its own agent, so a value like "model": [] or {} raises TypeError: unhashable type. That aborts Aggregate panel findings and the whole consolidate job, which throws away every other cell's findings when it should only cost that one cell its slot. In load_cells, either drop records whose model/review_type are not strings (treat them like unreadable artifacts) or coerce those fields to None. Raised by 3 of 6 reviewers (claude-opus-5-5-xhigh adversarial, claude-opus-5-5-xhigh edge-case, gpt-5.6-sol-xhigh edge-case).

expected = {(m, t) for m in models for t in REVIEW_TYPES}
for model, review_type in sorted(expected - present):
cells.append({
"model": model,
"review_type": review_type,
"status": "error",
"error": error,
"findings": [],
})


def aggregate(panel_dir, models, direct_model="", direct_counts=False):
"""Return (cells, ok_count, total). `cells` is everything the judge reads."""
panel, direct = load_cells(panel_dir)
fill_missing(panel, models, "Cell findings artifact was not uploaded.")
counts = bool(direct_counts and direct_model)
if counts:
# Same contract as a Cursor cell: absent means failed, never "not run".
fill_missing(direct, [direct_model], "Direct-API cell findings artifact was not uploaded.")
for c in direct:
c["direct_api"] = True
if not counts:
c["advisory"] = True
counted = panel + direct if counts else panel
ok = sum(1 for c in counted if c.get("status") == "ok")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Low — In side-by-side mode (--direct-counts false), ok leaves out the advisory direct cells, but Build judge prompt also uses ok_count != '0' to decide whether any reviewer contributed. If every Cursor cell fails while an advisory direct cell succeeds, the judge is skipped and the direct cell's findings are never adjudicated. Consider giving the judge step a separate "any cell ok" signal that includes advisory cells, and keep ok_count/total gate-only. Raised by 1 of 6 reviewers (gpt-5.6-sol-xhigh edge-case).

return panel + direct, ok, len(counted)


def _log_safe(value):
"""A cell-written value made safe for a ::workflow command::-parsing log line."""
return re.sub(r"[^A-Za-z0-9_.-]", "_", str(value))[:32] or "?"


def main(argv=None):
p = argparse.ArgumentParser(description=__doc__.splitlines()[0])
p.add_argument("--panel-dir", required=True)
p.add_argument("--models", required=True, help="JSON list of Cursor panel model ids")
p.add_argument("--direct-model", default="")
p.add_argument("--direct-counts", default="false")
p.add_argument("--out", required=True)
p.add_argument("--github-output", default="")
args = p.parse_args(argv)

counts = args.direct_counts.strip().lower() == "true"
cells, ok, total = aggregate(
args.panel_dir, json.loads(args.models), args.direct_model.strip(), counts
)
print(f"Panel: {ok}/{total} cells contributed findings.")
for c in cells:
if c.get("direct_api"):
role = "advisory, not counted" if c.get("advisory") else "counted"
print(
f"Direct-API cell {_log_safe(c.get('review_type'))} ({role}): "
f"status={_log_safe(c.get('status'))}."
)

with open(args.out, "w", encoding="utf-8") as f:
json.dump(cells, f, indent=2)
if args.github_output:
with open(args.github_output, "a", encoding="utf-8") as g:
g.write(f"ok_count={ok}\n")
g.write(f"total={total}\n")
return 0


if __name__ == "__main__":
sys.exit(main())
6 changes: 5 additions & 1 deletion .github/cursor-review/post-review.py
Original file line number Diff line number Diff line change
Expand Up @@ -2427,8 +2427,12 @@ def build_panel_summary(panel: list[dict]) -> str:
failed = [c for c in panel if c.get("status") != "ok"]
parts = [f"_Panel: {ok}/{len(panel)} reviewers contributed findings._"]
if failed:
# A direct-API cell (counted only when it replaces the Cursor OpenAI
# lane) is named with its backend, so a reader can tell it from a
# Cursor cell of the same model id.
names = ", ".join(
f"{c.get('model','?')}:{c.get('review_type','?')} ({c.get('status','?')})"
f"{c.get('model','?')}:{c.get('review_type','?')} "
f"({'direct, ' if c.get('direct') is True else ''}{c.get('status','?')})"
for c in failed
)
parts.append(f"_Reviewers that did not contribute: {names}_")
Expand Down
Loading
Loading