diff --git a/.github/cursor-review/README.md b/.github/cursor-review/README.md index 2528bbe8..58f73c7e 100644 --- a/.github/cursor-review/README.md +++ b/.github/cursor-review/README.md @@ -217,6 +217,7 @@ the consolidated review's panel table. See [Panel integrity](#panel-integrity). | File | Role | |---|---| +| [`aggregate-panel.py`](aggregate-panel.py) | Run by `consolidate`'s **Aggregate panel findings** step. Combines the cells' `findings-*` artifacts into the `panel.json` the judge reads, adds a `status=error` record for every expected cell whose artifact never arrived, and writes the `ok_count` / `total` that `Panel integrity` reads. The direct-OpenAI cells (`findings-direct-*`) count like Cursor cells when they replace the Cursor OpenAI lane (`openai_direct_replaces_cursor_openai: true`, so a failed one also counts against `approve_max_failed_reviewers`) and stay advisory side by side. Only this script sets the `direct_api` / `advisory` markers, so a cell's own record cannot excuse itself from the count. | | [`prompt-adversarial.md`](prompt-adversarial.md) | Prompt for the security/reliability review pass. | | [`prompt-edge-case.md`](prompt-edge-case.md) | Prompt for the correctness/logic review pass. | | [`prompt-judge.md`](prompt-judge.md) | Prompt the judge model uses to consolidate panel findings into one review. | @@ -300,7 +301,7 @@ jobs: | Kind | Name | Required | Purpose | |---|---|---|---| | Secret | `CURSOR_API_KEY` | **yes** | Bills the panel + judge `cursor-agent` calls. | -| Secret | `OPENAI_API_KEY` | no | Only with `openai_direct_model`: bills the optional direct-OpenAI cell. Use a project-scoped service-account key with a spend limit. | +| Secret | `OPENAI_API_KEY` | no | Only with `openai_direct_model`: bills the optional direct-OpenAI cells. Use a project-scoped service-account key with a spend limit. | | Secret | `SLACK_BOT_TOKEN` | no | Enables start/complete DMs to the triggerer. | | Variable | `CURSOR_REVIEW_DM_EMAIL_MAP` | no | Maps GitHub logins → emails for Slack DM lookup. | @@ -363,8 +364,8 @@ All optional except `workflows_ref` (required, no default) — pass them under |---|---|---| | `judge_model` | `claude-opus-5-5-xhigh` | Model that consolidates panel findings. | | `panel_models` | `''` (built-in list) | JSON array of Cursor model ids that **replaces** the panel list; each still runs both review types and is validated against the live catalog by preflight. For per-repo experiments (e.g. `-xhigh` vs `-max` tiers). | -| `openai_direct_model` | `''` (off) | OPT-IN. An OpenAI model id reviewed through the OpenAI API directly (pinned Codex CLI, `package.json` here) as ONE **adversarial** cell on GitHub-hosted runners, by default **replacing** the Cursor `gpt-*` cell (next row). Needs the `OPENAI_API_KEY` secret and a plain id (`[A-Za-z0-9._-]`); without either, preflight warns and leaves the cell off and the panel unchanged. Not part of `panel_models` (not in the Cursor catalog); advisory — the leg never fails the run and is not counted in `Panel integrity` / the auto-approve gate. | -| `openai_direct_replaces_cursor_openai` | `true` | When the direct cell is in effect, drop the Cursor `gpt-*` cell(s) so the OpenAI lane is reviewed once. `false` runs both for a side-by-side backend comparison. Preflight fails rather than leave the Cursor panel empty. | +| `openai_direct_model` | `''` (off) | OPT-IN. An OpenAI model id reviewed through the OpenAI API directly (pinned Codex CLI, `package.json` here) as two cells (**adversarial** + **edge-case**, the Cursor cells' prompts) on GitHub-hosted runners, by default **replacing** the Cursor `gpt-*` cells (next row). Needs the `OPENAI_API_KEY` secret and a plain id (`[A-Za-z0-9._-]`); without either, preflight warns and leaves the cells off and the panel unchanged. Not part of `panel_models` (not in the Cursor catalog). Whether the cells gate depends on the next row. | +| `openai_direct_replaces_cursor_openai` | `true` | When the direct cells are in effect, drop the Cursor `gpt-*` cell(s) so the OpenAI lane is reviewed once — the direct cells then **gate** like the Cursor cells they replace: counted in `Panel integrity` and the auto-approve gate, a failed one red in the rollup and named `(direct, )` in the review's "did not contribute" line. `false` runs both for a side-by-side backend comparison, with the direct cells **advisory** (findings reach the judge; never counted, never red). Preflight fails rather than leave the Cursor panel empty. | | `skip_bot_branch_prefixes` | `ci/bump- chore/refresh- auto/refresh-` | Skip the panel when the PR author is a Bot **and** the head branch starts with one of these prefixes (machine pin bumps / catalog refreshes). `''` reviews every bot PR. | | `diff_size_cap` | `5000` | Max counted changed lines (after generated-file exclusion and comment discounting); larger PRs are skipped. | | `ignore_comments` | `true` | Discount blank/comment-only lines from the size count (count-only; the panel still sees them). | diff --git a/.github/cursor-review/aggregate-panel.py b/.github/cursor-review/aggregate-panel.py new file mode 100644 index 00000000..0e7d7815 --- /dev/null +++ b/.github/cursor-review/aggregate-panel.py @@ -0,0 +1,132 @@ +#!/usr/bin/env python3 +"""Combine the panel cells' findings artifacts into one panel.json. + +Run by `consolidate`'s `Aggregate panel findings` step, after +`Download panel findings` has unpacked every `findings-*` artifact under one +directory. Each artifact directory holds one findings.json. The output keeps the +cell metadata the judge and `Build consolidated findings file` read, and the +`ok_count` / `total` pair is what `Panel integrity` and auto-approve gate on. + +The opt-in direct-API cells (`findings-direct--`) take one +of two roles, chosen by `--direct-counts`: + +- counted (`openai_direct_replaces_cursor_openai: true`): the direct cells ARE + the panel's OpenAI lane, so they join the expected set and ok_count/total + exactly like a Cursor cell — a missing or errored one is a failed reviewer. +- advisory (replace false, the side-by-side comparison): their findings still + reach the judge, but they stay out of `present` (so one cannot mask a missing + Cursor cell with the same id) and out of ok_count/total. + +Two markers are set HERE and only here — a cell's own record is model-adjacent +output and must never be able to excuse itself from the count: `direct_api` +(this cell came through the direct backend) and `advisory` (keep it out of the +panel metadata auto-approve counts failures from). + +Stdlib only. Usage: + aggregate-panel.py --panel-dir /tmp/panel --models '' \ + [--direct-model --direct-counts true|false] \ + --out /tmp/panel.json [--github-output "$GITHUB_OUTPUT"] +""" + +import argparse +import glob +import json +import os +import re +import sys + +REVIEW_TYPES = ("adversarial", "edge-case") +DIRECT_PREFIX = "findings-direct-" +MARKERS = ("direct_api", "advisory") + + +def load_cells(panel_dir): + """(cursor_cells, direct_cells) read from `/findings-*/findings.json`.""" + cursor, direct = [], [] + for path in sorted(glob.glob(os.path.join(panel_dir, "findings-*", "findings.json"))): + try: + with open(path, encoding="utf-8") as f: + cell = json.load(f) + except (OSError, ValueError): + continue + if not isinstance(cell, dict): + continue + for marker in MARKERS: + cell.pop(marker, None) + if os.path.basename(os.path.dirname(path)).startswith(DIRECT_PREFIX): + direct.append(cell) + else: + cursor.append(cell) + return cursor, direct + + +def fill_missing(cells, models, error): + """Append a status=error record for every expected (model, type) with no cell.""" + present = {(c.get("model"), c.get("review_type")) for c in cells} + expected = {(m, t) for m in models for t in REVIEW_TYPES} + for model, review_type in sorted(expected - present): + cells.append({ + "model": model, + "review_type": review_type, + "status": "error", + "error": error, + "findings": [], + }) + + +def aggregate(panel_dir, models, direct_model="", direct_counts=False): + """Return (cells, ok_count, total). `cells` is everything the judge reads.""" + panel, direct = load_cells(panel_dir) + fill_missing(panel, models, "Cell findings artifact was not uploaded.") + counts = bool(direct_counts and direct_model) + if counts: + # Same contract as a Cursor cell: absent means failed, never "not run". + fill_missing(direct, [direct_model], "Direct-API cell findings artifact was not uploaded.") + for c in direct: + c["direct_api"] = True + if not counts: + c["advisory"] = True + counted = panel + direct if counts else panel + ok = sum(1 for c in counted if c.get("status") == "ok") + return panel + direct, ok, len(counted) + + +def _log_safe(value): + """A cell-written value made safe for a ::workflow command::-parsing log line.""" + return re.sub(r"[^A-Za-z0-9_.-]", "_", str(value))[:32] or "?" + + +def main(argv=None): + p = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + p.add_argument("--panel-dir", required=True) + p.add_argument("--models", required=True, help="JSON list of Cursor panel model ids") + p.add_argument("--direct-model", default="") + p.add_argument("--direct-counts", default="false") + p.add_argument("--out", required=True) + p.add_argument("--github-output", default="") + args = p.parse_args(argv) + + counts = args.direct_counts.strip().lower() == "true" + cells, ok, total = aggregate( + args.panel_dir, json.loads(args.models), args.direct_model.strip(), counts + ) + print(f"Panel: {ok}/{total} cells contributed findings.") + for c in cells: + if c.get("direct_api"): + role = "advisory, not counted" if c.get("advisory") else "counted" + print( + f"Direct-API cell {_log_safe(c.get('review_type'))} ({role}): " + f"status={_log_safe(c.get('status'))}." + ) + + with open(args.out, "w", encoding="utf-8") as f: + json.dump(cells, f, indent=2) + if args.github_output: + with open(args.github_output, "a", encoding="utf-8") as g: + g.write(f"ok_count={ok}\n") + g.write(f"total={total}\n") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/.github/cursor-review/post-review.py b/.github/cursor-review/post-review.py index 718d8557..bde996f2 100644 --- a/.github/cursor-review/post-review.py +++ b/.github/cursor-review/post-review.py @@ -2427,8 +2427,12 @@ def build_panel_summary(panel: list[dict]) -> str: failed = [c for c in panel if c.get("status") != "ok"] parts = [f"_Panel: {ok}/{len(panel)} reviewers contributed findings._"] if failed: + # A direct-API cell (counted only when it replaces the Cursor OpenAI + # lane) is named with its backend, so a reader can tell it from a + # Cursor cell of the same model id. names = ", ".join( - f"{c.get('model','?')}:{c.get('review_type','?')} ({c.get('status','?')})" + f"{c.get('model','?')}:{c.get('review_type','?')} " + f"({'direct, ' if c.get('direct') is True else ''}{c.get('status','?')})" for c in failed ) parts.append(f"_Reviewers that did not contribute: {names}_") diff --git a/.github/cursor-review/tests/test_aggregate_panel.py b/.github/cursor-review/tests/test_aggregate_panel.py new file mode 100644 index 00000000..1cc6c9b8 --- /dev/null +++ b/.github/cursor-review/tests/test_aggregate_panel.py @@ -0,0 +1,239 @@ +#!/usr/bin/env python3 +"""Tests for aggregate-panel.py — which panel cells count toward the gate. + +The direct-API OpenAI cells (`findings-direct--`) gate only +when they REPLACE the Cursor OpenAI lane; side by side they stay advisory. A +regression in either direction is invisible in a diff: counting advisory cells +lets a comparison run withhold approval, and NOT counting replacing cells +takes the OpenAI lane out of the gate, so a failed one costs nothing against +`approve_max_failed_reviewers`. The workflow wiring that feeds this script its +flags is executed in test_workflow_panel_integrity.py. + +Also pins the two downstream readers of the markers this script sets: the +`direct` tag in post-review.py's "did not contribute" line, and auto-approve's +`panel_gate` counting a failed direct cell like any other. + +Run: python3 -m unittest discover -s .github/cursor-review/tests -p 'test_*.py' +""" + +import importlib.util +import json +import os +import subprocess +import sys +import tempfile +import unittest + +HERE = os.path.dirname(os.path.abspath(__file__)) +ROOT = os.path.dirname(HERE) +SCRIPT = os.path.join(ROOT, "aggregate-panel.py") + + +def _load(name, filename): + spec = importlib.util.spec_from_file_location(name, os.path.join(ROOT, filename)) + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +AGG = _load("aggregate_panel", "aggregate-panel.py") +PR = _load("post_review", "post-review.py") +AA = _load("auto_approve", "auto-approve.py") + +CURSOR_MODELS = ["claude-opus", "kimi"] +DIRECT = "gpt-5.6-sol" + + +def cell(model, review_type, status="ok", **extra): + return {"model": model, "review_type": review_type, "status": status, "findings": [], **extra} + + +class AggregateTest(unittest.TestCase): + def setUp(self): + self._tmp = tempfile.TemporaryDirectory() + self.dir = self._tmp.name + + def tearDown(self): + self._tmp.cleanup() + + def write(self, artifact, record): + os.makedirs(os.path.join(self.dir, artifact)) + with open(os.path.join(self.dir, artifact, "findings.json"), "w", encoding="utf-8") as f: + json.dump(record, f) + + def full_cursor_panel(self): + for model in CURSOR_MODELS: + for review_type in AGG.REVIEW_TYPES: + self.write(f"findings-{review_type}-{model}", cell(model, review_type)) + + def direct(self, review_type, status="ok", **extra): + self.write(f"findings-direct-{review_type}-{DIRECT}", cell(DIRECT, review_type, status, **extra)) + + def run_agg(self, counts): + return AGG.aggregate(self.dir, CURSOR_MODELS, DIRECT, counts) + + def test_replace_true_both_direct_ok_are_counted(self): + self.full_cursor_panel() + self.direct("adversarial") + self.direct("edge-case") + cells, ok, total = self.run_agg(True) + self.assertEqual((ok, total), (6, 6)) + direct = [c for c in cells if c.get("direct_api")] + self.assertEqual(sorted(c["review_type"] for c in direct), ["adversarial", "edge-case"]) + self.assertFalse(any(c.get("advisory") for c in cells)) + + def test_replace_true_one_direct_error_counts_as_failed(self): + self.full_cursor_panel() + self.direct("adversarial") + self.direct("edge-case", status="error") + _, ok, total = self.run_agg(True) + self.assertEqual((ok, total), (5, 6)) + + def test_replace_true_missing_direct_artifact_is_synthesised_as_error(self): + self.full_cursor_panel() + self.direct("adversarial") + cells, ok, total = self.run_agg(True) + self.assertEqual((ok, total), (5, 6)) + missing = [c for c in cells if c.get("direct_api") and c["review_type"] == "edge-case"] + self.assertEqual(len(missing), 1) + self.assertEqual(missing[0]["status"], "error") + self.assertEqual(missing[0]["model"], DIRECT) + + def test_replace_false_direct_cells_are_advisory_and_uncounted(self): + self.full_cursor_panel() + self.direct("adversarial", status="error") + self.direct("edge-case") + cells, ok, total = self.run_agg(False) + self.assertEqual((ok, total), (4, 4)) + direct = [c for c in cells if c.get("direct_api")] + self.assertEqual(len(direct), 2) + self.assertTrue(all(c.get("advisory") is True for c in direct)) + + def test_replace_false_missing_direct_cell_is_not_synthesised(self): + self.full_cursor_panel() + cells, ok, total = self.run_agg(False) + self.assertEqual((ok, total), (4, 4)) + self.assertFalse(any(c.get("direct_api") for c in cells)) + + def test_advisory_direct_cell_cannot_mask_a_missing_cursor_cell(self): + # Side by side the Cursor cell may share the direct id; the direct + # record must not stand in for a Cursor cell that never uploaded. + self.write("findings-adversarial-claude-opus", cell("claude-opus", "adversarial")) + self.write(f"findings-direct-edge-case-{DIRECT}", cell("claude-opus", "edge-case")) + cells, ok, total = AGG.aggregate(self.dir, ["claude-opus"], DIRECT, False) + self.assertEqual((ok, total), (1, 2)) + + def test_a_cell_cannot_set_its_own_markers(self): + self.write("findings-adversarial-kimi", cell("kimi", "adversarial", "error", advisory=True, direct_api=True)) + self.direct("adversarial", status="error", advisory=True) + cells, ok, total = AGG.aggregate(self.dir, ["kimi"], DIRECT, True) + cursor = [c for c in cells if c["model"] == "kimi" and c["review_type"] == "adversarial"] + self.assertNotIn("advisory", cursor[0]) + self.assertNotIn("direct_api", cursor[0]) + self.assertFalse(any(c.get("advisory") for c in cells)) + self.assertEqual((ok, total), (0, 4)) + + def test_counts_without_a_model_falls_back_to_advisory(self): + self.full_cursor_panel() + self.direct("adversarial") + _, ok, total = AGG.aggregate(self.dir, CURSOR_MODELS, "", True) + self.assertEqual((ok, total), (4, 4)) + + def test_unreadable_artifacts_are_skipped(self): + os.makedirs(os.path.join(self.dir, "findings-adversarial-kimi")) + with open(os.path.join(self.dir, "findings-adversarial-kimi", "findings.json"), "wb") as f: + f.write(b"\xff\xfe not json") + cells, ok, total = AGG.aggregate(self.dir, ["kimi"], "", False) + self.assertEqual((ok, total), (0, 2)) + + def test_cli_writes_panel_and_outputs(self): + self.full_cursor_panel() + self.direct("adversarial") + self.direct("edge-case", status="error") + out = os.path.join(self.dir, "panel.json") + gh_out = os.path.join(self.dir, "gh-output") + result = subprocess.run( + [sys.executable, SCRIPT, "--panel-dir", self.dir, "--models", json.dumps(CURSOR_MODELS), + "--direct-model", DIRECT, "--direct-counts", "true", "--out", out, "--github-output", gh_out], + capture_output=True, text=True, check=True, + ) + self.assertIn("Panel: 5/6 cells contributed findings.", result.stdout) + self.assertIn("Direct-API cell edge-case (counted): status=error.", result.stdout) + with open(gh_out, encoding="utf-8") as f: + self.assertEqual(f.read(), "ok_count=5\ntotal=6\n") + with open(out, encoding="utf-8") as f: + self.assertEqual(len(json.load(f)), 6) + + +class DownstreamTest(unittest.TestCase): + def test_summary_names_a_failed_direct_cell_with_its_backend(self): + panel = [ + cell("claude-opus", "adversarial"), + {"model": DIRECT, "review_type": "edge-case", "status": "error", "direct": True}, + {"model": "kimi", "review_type": "adversarial", "status": "error"}, + ] + summary = PR.build_panel_summary(panel) + self.assertIn("_Panel: 1/3 reviewers contributed findings._", summary) + self.assertIn(f"{DIRECT}:edge-case (direct, error)", summary) + self.assertIn("kimi:adversarial (error)", summary) + + def test_auto_approve_counts_a_failed_direct_cell(self): + panel = [ + cell(m, t) for m in CURSOR_MODELS for t in AGG.REVIEW_TYPES + ] + [ + {"model": DIRECT, "review_type": "adversarial", "status": "ok", "direct": True}, + {"model": DIRECT, "review_type": "edge-case", "status": "error", "direct": True}, + ] + reason, tolerated = AA.panel_gate(panel, max_failed=0) + self.assertEqual(reason, "1/6 panel reviewers did not complete") + reason, tolerated = AA.panel_gate(panel, max_failed=1) + self.assertIsNone(reason) + self.assertEqual(len(tolerated), 1) + + +class DirectJobShapeTest(unittest.TestCase): + """The workflow half: none of these would fail a lint if dropped.""" + + @classmethod + def setUpClass(cls): + sys.path.insert(0, HERE) + try: + import test_workflow_panel_integrity as wpi + finally: + sys.path.remove(HERE) + jobs = wpi.split_jobs(wpi.read_workflow()) + cls.direct = "\n".join(wpi.code_lines(jobs["review-openai-direct"])) + cls.consolidate = "\n".join(wpi.code_lines(jobs["consolidate"])) + cls.preflight = "\n".join(wpi.code_lines(jobs["preflight"])) + + def test_runs_both_review_types_with_their_own_prompt(self): + self.assertIn("review_type: [adversarial, edge-case]", self.direct) + self.assertIn('--prompt "$CURSOR_REVIEW_ASSETS/prompt-${REVIEW_TYPE}.md"', self.direct) + self.assertNotIn("prompt-adversarial.md", self.direct) + self.assertNotIn("--review-type adversarial", self.direct) + + def test_artifact_is_named_per_review_type(self): + self.assertIn( + "name: findings-direct-${{ matrix.review_type }}-${{ inputs.openai_direct_model }}", + self.direct, + ) + + def test_failures_are_swallowed_only_when_advisory(self): + self.assertIn( + "continue-on-error: ${{ needs.preflight.outputs.openai_direct_counts != 'true' }}", + self.direct, + ) + self.assertNotIn("continue-on-error: true\n timeout-minutes", self.direct) + self.assertIn( + "if: always() && needs.preflight.outputs.openai_direct_counts == 'true'", self.direct + ) + + def test_preflight_and_consolidate_carry_the_role(self): + self.assertIn("openai_direct_counts: ${{ steps.models.outputs.openai_direct_counts }}", self.preflight) + self.assertIn("DIRECT_COUNTS: ${{ needs.preflight.outputs.openai_direct_counts }}", self.consolidate) + self.assertIn('if c.get("advisory") is True:', self.consolidate) + self.assertIn('meta["direct"] = True', self.consolidate) + + +if __name__ == "__main__": + unittest.main() diff --git a/.github/cursor-review/tests/test_workflow_panel_integrity.py b/.github/cursor-review/tests/test_workflow_panel_integrity.py index 5d3ebac4..a0e42308 100644 --- a/.github/cursor-review/tests/test_workflow_panel_integrity.py +++ b/.github/cursor-review/tests/test_workflow_panel_integrity.py @@ -26,10 +26,12 @@ Run: python3 .github/cursor-review/tests/test_workflow_panel_integrity.py """ +import json import os import re import shutil import subprocess +import tempfile import unittest WORKFLOW = os.path.normpath( @@ -297,8 +299,15 @@ def test_the_panel_counts_are_job_outputs(self): ) def test_the_aggregate_step_still_writes_both_counts(self): - for written in ('g.write(f"ok_count={ok}\\n")', 'g.write(f"total={len(panel)}\\n")'): - self.assertIn(written, self.consolidate) + # The counting lives in aggregate-panel.py (behaviour tested in + # test_aggregate_panel.py); the step must still hand it GITHUB_OUTPUT. + self.assertIn('aggregate-panel.py"', self.consolidate) + self.assertIn('--github-output "$GITHUB_OUTPUT"', self.consolidate) + script = os.path.join(os.path.dirname(WORKFLOW), "..", "cursor-review", "aggregate-panel.py") + with open(script, encoding="utf-8") as f: + source = f.read() + for written in ('g.write(f"ok_count={ok}\\n")', 'g.write(f"total={total}\\n")'): + self.assertIn(written, source) class PanelIntegrityJobTest(unittest.TestCase): @@ -638,39 +647,44 @@ def test_the_deliberate_skips_are_still_skips(self): self.assertIn(gate, self.condition, f"`{PANEL_JOB}` lost gate `{gate}`") -NEEDS_REF = re.compile(r"needs\.([A-Za-z0-9_-]+)\.(result|outputs\.[A-Za-z0-9_]+)") +CONTEXT_REF = re.compile(r"\b(?:needs|inputs|steps|vars|matrix)(?:\.[A-Za-z0-9_-]+)+") -def evaluate_if(condition, needs): - """Evaluate a job `if:` built only from `always()`, `needs.*` reads, string - literals, `==`/`!=`, `&&`/`||` and parentheses. +def evaluate(expression, context): + """Evaluate an expression built only from context reads, string literals, + `always()`, `==`/`!=`, `&&`/`||` and parentheses. - `needs` maps a job to `{"result": ..., "outputs": {...}}`. A job absent from - it, or an output it does not declare, reads as '' — which is exactly what - GitHub hands a FAILED or SKIPPED job's outputs. Anything outside that small - grammar fails the translation loudly rather than evaluating to a guess. + `context` maps a dotted read (`needs.gate.result`, `inputs.`, + `steps..outputs.`, `matrix.`) to its value. A read it does + not list is '' — exactly what GitHub hands back for a FAILED or SKIPPED + job's outputs. `&&`/`||` return an operand, in GitHub as in Python, so + `a == 'true' && inputs.b || ''` yields a string. Anything outside that + small grammar fails the translation loudly rather than evaluating to a + guess. """ - expr = condition.strip() + expr = expression.strip() if expr.startswith("${{") and expr.endswith("}}"): expr = expr[3:-2].strip() - - def read(match): - job, field = match.group(1), match.group(2) - entry = needs.get(job, {}) - if field == "result": - value = entry.get("result", "") - else: - value = entry.get("outputs", {}).get(field.split(".", 1)[1], "") - return repr(value) - - py = NEEDS_REF.sub(read, expr) + py = CONTEXT_REF.sub(lambda match: repr(context.get(match.group(0), "")), expr) py = py.replace("always()", "True").replace("&&", " and ").replace("||", " or ") - leftover = re.sub(r"'[^']*'|\b(True|and|or)\b|==|!=|[()\s]", "", py) + leftover = re.sub(r"'[^']*'|\b(True|False|and|or)\b|==|!=|[()\s]", "", py) if leftover: # pragma: no cover - only on a grammar this helper does not know - raise AssertionError("cannot evaluate `if:` remainder %r" % leftover) + raise AssertionError("cannot evaluate expression remainder %r" % leftover) return eval(py, {"__builtins__": {}}, {}) # noqa: S307 - grammar checked above +def evaluate_if(condition, needs): + """Evaluate a job `if:` against `needs`, which maps a job to + `{"result": ..., "outputs": {...}}`. A job absent from it, or an output it + does not declare, reads as '' (see `evaluate`).""" + context = {} + for job, entry in needs.items(): + context["needs.%s.result" % job] = entry.get("result", "") + for name, value in entry.get("outputs", {}).items(): + context["needs.%s.outputs.%s" % (job, name)] = value + return evaluate(condition, context) + + class RoundCapIsADecisionJobTest(unittest.TestCase): """`round-cap` (the `max_rounds` stop) sits upstream of `diff-size`. @@ -882,5 +896,298 @@ def test_every_value_the_step_echoes_goes_through_it(self): self.assertIn("tr -d", self.definition, "flatten() no longer strips newlines") +DIRECT_JOB = "review-openai-direct" +PANEL_MODELS_STEP = "Define panel models" +AGGREGATE_STEP = "Aggregate panel findings" +SEED_STEP = "Seed default findings artifact" +REVIEW_TYPES = ("adversarial", "edge-case") +ASSETS = os.path.normpath(os.path.join(os.path.dirname(WORKFLOW), "..", "cursor-review")) +EXPRESSION = re.compile(r"\$\{\{(.*?)\}\}") + + +def step_scalar(step_lines, key): + """The scalar value of a step-level ` :`, or None when absent.""" + prefix = " %s:" % key + for line in code_lines(step_lines): + if line.startswith(prefix): + return line[len(prefix):].strip() + return None + + +def block_mapping(lines, header, indent): + """{key: raw value} for the one-line `key: value` entries under `header`. + + `header` is the whole line opening the block (` env:` in a step, + ` outputs:` in a job) and its entries sit `indent` spaces deep; the block + ends at the first line indented less than that. + """ + entry = re.compile(r"^%s([A-Za-z0-9_-]+):\s*(.*)$" % (" " * indent)) + mapping, inside = {}, False + for line in code_lines(lines): + if not inside: + inside = line == header + elif line.strip(): + if len(line) - len(line.lstrip()) < indent: + break + match = entry.match(line) + if match: + mapping[match.group(1)] = match.group(2).strip() + return mapping + + +def render(value, context): + """A one-line YAML value as the runner hands it on: unquoted, and every + `${{ … }}` in it replaced by its value (a boolean becomes `true`/`false`).""" + if len(value) >= 2 and value[0] == value[-1] and value[0] in "'\"": + value = value[1:-1] + + def substitute(match): + result = evaluate(match.group(1), context) + if isinstance(result, bool): + return "true" if result else "false" + return str(result) + + return EXPRESSION.sub(substitute, value) + + +def run_block(step_lines): + """A step's `run: |` script, de-indented the way YAML hands it to bash.""" + script, inside = [], False + for line in step_lines: + if not inside: + inside = line == " run: |" + elif line.strip() and not line.startswith(" " * 10): + break + else: + script.append(line[10:]) + if not inside: + raise AssertionError("the step has no `run: |` block") + return "\n".join(script).rstrip("\n") + "\n" + + +def run_step(step_lines, context, workdir): + """Execute a step for real: its `env:` rendered from `context`, its script + run as the runner runs it (`bash -e`) with `/tmp/` re-rooted under + `workdir`. Returns (CompletedProcess, {output: value} it wrote).""" + script = run_block(step_lines) + if "/tmp/" not in script: # pragma: no cover - re-rooting would be a no-op + raise AssertionError("the step no longer works under /tmp/; re-root its new path here") + output = os.path.join(workdir, "github-output") + open(output, "w", encoding="utf-8").close() + env = {"PATH": os.environ.get("PATH", ""), "GITHUB_OUTPUT": output, "CURSOR_REVIEW_ASSETS": ASSETS} + for name, value in block_mapping(step_lines, " env:", 10).items(): + env[name] = render(value, context) + done = subprocess.run( + [shutil.which("bash"), "-e", "-c", script.replace("/tmp/", workdir + "/")], + env=env, + cwd=workdir, + capture_output=True, + text=True, + timeout=60, + ) + outputs = {} + with open(output, encoding="utf-8") as f: + for line in f: + key, _, value = line.rstrip("\n").partition("=") + outputs[key] = value + return done, outputs + + +class DirectCellsGateWhenTheyReplaceTest(unittest.TestCase): + """With `openai_direct_replaces_cursor_openai` on, the direct-API cells ARE + the panel's OpenAI lane, so they must gate like the Cursor cells they + replace: counted in ok_count/total, and a leg that goes red when its cell + did not submit. + + That rests on a few one-line links across three jobs: `preflight` setting + `openai_direct_counts=true`, `Aggregate panel findings` handing that flag + and the direct model id to aggregate-panel.py, and the direct leg check's + `exit 1`. Break any one and the cells quietly fall back to advisory — a + failed one costs nothing against `approve_max_failed_reviewers` and its leg + stays green — while the workflow still parses and lints, and + test_aggregate_panel.py, which calls the script with the flags already set, + still passes. So the chain is EXECUTED here, from the inputs to the counts + and the leg's exit status, rather than pattern-matched. + """ + + DIRECT_MODEL = "gpt-6.1-sol" + + @classmethod + def setUpClass(cls): + for tool in ("bash", "jq", "python3"): + if shutil.which(tool) is None: + raise AssertionError(f"`{tool}` is required to execute the workflow steps") + jobs = split_jobs(read_workflow()) + cls.preflight = jobs["preflight"] + cls.review = jobs[MATRIX_JOB] + cls.direct = jobs[DIRECT_JOB] + cls.consolidate = jobs["consolidate"] + + def setUp(self): + tmp = tempfile.TemporaryDirectory() + self.addCleanup(tmp.cleanup) + self.tmp = tmp.name + + def step(self, job_lines, name): + body = step_named(job_lines, name) + self.assertIsNotNone(body, f"step `{name}` is gone") + return body + + def preflight_outputs(self, replace, key_present="true"): + """`needs.preflight.outputs.*` once `Define panel models` has run.""" + done, step_outputs = run_step( + self.step(self.preflight, PANEL_MODELS_STEP), + { + "inputs.openai_direct_model": self.DIRECT_MODEL, + "inputs.openai_direct_replaces_cursor_openai": replace, + "needs.gate.outputs.openai_key_present": key_present, + }, + tempfile.mkdtemp(dir=self.tmp), + ) + self.assertEqual(done.returncode, 0, done.stdout + done.stderr) + steps = {"steps.models.outputs.%s" % k: v for k, v in step_outputs.items()} + return { + "needs.preflight.outputs.%s" % name: render(value, steps) + for name, value in block_mapping(self.preflight, " outputs:", 6).items() + } + + def downstream(self, replace): + """The context the direct legs and `consolidate` evaluate against.""" + context = self.preflight_outputs(replace) + context["inputs.openai_direct_model"] = self.DIRECT_MODEL + return context + + def upload(self, panel, job_lines, cell_context, model, review_type, status): + """Lay one cell's record out as `Download panel findings` unpacks it: + a directory per artifact, named by that leg's own upload step.""" + name = block_mapping(self.step(job_lines, UPLOAD_STEP), " with:", 10)["name"] + artifact = os.path.join(panel, render(name, cell_context)) + os.makedirs(artifact) + with open(os.path.join(artifact, "findings.json"), "w", encoding="utf-8") as f: + json.dump({"model": model, "review_type": review_type, "status": status, "findings": []}, f) + + def aggregate(self, context, direct_statuses): + """(ok_count, total, Cursor cell count) from `Aggregate panel findings` + over every Cursor cell `ok` plus the direct cells given.""" + workdir = tempfile.mkdtemp(dir=self.tmp) + panel = os.path.join(workdir, "panel") + models = json.loads(context["needs.preflight.outputs.models"]) + for model in models: + for rt in REVIEW_TYPES: + self.upload(panel, self.review, {"matrix.model": model, "matrix.review_type": rt}, model, rt, "ok") + for rt, status in direct_statuses.items(): + self.upload(panel, self.direct, {**context, "matrix.review_type": rt}, self.DIRECT_MODEL, rt, status) + done, outputs = run_step(self.step(self.consolidate, AGGREGATE_STEP), context, workdir) + self.assertEqual(done.returncode, 0, done.stdout + done.stderr) + return int(outputs["ok_count"]), int(outputs["total"]), 2 * len(models) + + def leg(self, context, review_type, submitted): + """The direct leg check for one cell, after `Seed default findings + artifact` and — when `submitted` — the `ok` record a submission writes + over the seed. None when the step is skipped, else its process.""" + context = {**context, "matrix.review_type": review_type} + workdir = tempfile.mkdtemp(dir=self.tmp) + done, _ = run_step(self.step(self.direct, SEED_STEP), context, workdir) + self.assertEqual(done.returncode, 0, done.stdout + done.stderr) + if submitted: + # The seed's path, re-rooted the same way run_step re-roots /tmp/. + with open(os.path.join(workdir, "findings-out", "findings.json"), "w", encoding="utf-8") as f: + json.dump({"model": self.DIRECT_MODEL, "review_type": review_type, "status": "ok", "findings": []}, f) + check = self.step(self.direct, LEG_STEP) + if not evaluate(step_scalar(check, "if"), context): + return None + return run_step(check, context, workdir)[0] + + def test_preflight_marks_replacing_cells_as_counted(self): + replacing = self.preflight_outputs(replace=True) + self.assertEqual(replacing["needs.preflight.outputs.openai_direct"], "true") + self.assertEqual( + replacing["needs.preflight.outputs.openai_direct_counts"], + "true", + "preflight no longer marks REPLACING direct cells as counted, so every " + "reader downstream treats them as advisory", + ) + self.assertEqual( + [m for m in json.loads(replacing["needs.preflight.outputs.models"]) if m.startswith("gpt-")], + [], + "the Cursor OpenAI cells were not dropped from the panel", + ) + for label, outputs in ( + ("side by side", self.preflight_outputs(replace=False)), + ("no OPENAI_API_KEY", self.preflight_outputs(replace=True, key_present="false")), + ): + with self.subTest(label): + self.assertEqual(outputs["needs.preflight.outputs.openai_direct_counts"], "false") + + def test_a_failed_replacing_cell_counts_against_the_panel(self): + ok, total, cursor = self.aggregate( + self.downstream(replace=True), {"adversarial": "ok", "edge-case": "error"} + ) + self.assertEqual( + (ok, total), + (cursor + 1, cursor + 2), + "the replacing direct cells are missing from ok_count/total, so a failed " + "one is free against approve_max_failed_reviewers", + ) + + def test_side_by_side_cells_stay_out_of_the_count(self): + ok, total, cursor = self.aggregate( + self.downstream(replace=False), {"adversarial": "ok", "edge-case": "error"} + ) + self.assertEqual( + (ok, total), + (cursor, cursor), + "side-by-side direct cells are counted, so a comparison run can withhold approval", + ) + + def test_a_replacing_cell_that_did_not_submit_fails_its_leg(self): + context = self.downstream(replace=True) + self.assertEqual( + render(job_scalar(self.direct, "continue-on-error") or "false", context), + "false", + f"`{DIRECT_JOB}` absorbs its own failure while it REPLACES the Cursor OpenAI lane", + ) + for review_type in REVIEW_TYPES: + with self.subTest(review_type): + failed = self.leg(context, review_type, submitted=False) + self.assertIsNotNone(failed, f"`{LEG_STEP}` is skipped on a replacing cell") + self.assertNotEqual( + failed.returncode, 0, f"`{LEG_STEP}` passed a direct cell that never submitted" + ) + self.assertIn("::error::", failed.stdout) + passed = self.leg(context, review_type, submitted=True) + self.assertEqual(passed.returncode, 0, passed.stdout + passed.stderr) + + def test_a_side_by_side_leg_stays_advisory(self): + context = self.downstream(replace=False) + self.assertEqual(render(job_scalar(self.direct, "continue-on-error") or "false", context), "true") + self.assertIsNone( + self.leg(context, "adversarial", submitted=False), + f"`{LEG_STEP}` fails an ADVISORY side-by-side cell", + ) + + def test_the_leg_check_follows_the_upload_and_spares_its_sibling(self): + # Same shape as the `review` job's pins above: after the upload, so the + # errored record still reaches the judge; never absorbed itself; and a + # red adversarial leg must not cancel the edge-case one. + order = step_order(self.direct) + self.assertIn(UPLOAD_STEP, order) + self.assertIn(LEG_STEP, order) + self.assertLess( + order.index(UPLOAD_STEP), + order.index(LEG_STEP), + f"`{LEG_STEP}` must come AFTER `{UPLOAD_STEP}` in `{DIRECT_JOB}`", + ) + self.assertIsNone( + step_scalar(self.step(self.direct, LEG_STEP), "continue-on-error"), + f"`{DIRECT_JOB}`'s `{LEG_STEP}` carries continue-on-error and can no longer turn the leg red", + ) + self.assertIn( + " fail-fast: false", + code_lines(self.direct), + f"`{DIRECT_JOB}` lost `fail-fast: false`: one red direct leg would cancel the other", + ) + + if __name__ == "__main__": unittest.main(verbosity=2) diff --git a/.github/workflows/cursor-review.yml b/.github/workflows/cursor-review.yml index 4437bd25..36838693 100644 --- a/.github/workflows/cursor-review.yml +++ b/.github/workflows/cursor-review.yml @@ -280,33 +280,37 @@ on: openai_direct_model: description: >- OPT-IN. An OpenAI model id (e.g. `gpt-6.1-sol`) reviewed through the - OpenAI API directly, via the Codex CLI, as ONE adversarial cell. + OpenAI API directly, via the Codex CLI, as two cells — adversarial + and edge-case, the same passes and prompts a Cursor model gets. Empty (default) = the job is skipped and nothing changes. Requires the `OPENAI_API_KEY` secret and a plain id (`[A-Za-z0-9._-]`); set - without either, preflight warns, the cell stays off and the panel is + without either, preflight warns, the cells stay off and the panel is unchanged. Deliberately NOT part of `panel_models`: that list is validated against the live CURSOR catalog, which a direct-API id is not in. - Advisory — its findings reach the judge like any other cell's, but the - leg never fails the run and is not counted in `Panel integrity` or - the auto-approve gate. By default it REPLACES the Cursor OpenAI cell - rather than adding to it; see `openai_direct_replaces_cursor_openai`. + By default the cells REPLACE the Cursor OpenAI cells rather than + adding to them, and then GATE like them; see + `openai_direct_replaces_cursor_openai`. type: string required: false default: '' openai_direct_replaces_cursor_openai: description: >- - When the direct cell is in effect (`openai_direct_model` set, a plain - id, and its key present), drop the Cursor-hosted OpenAI cell + When the direct cells are in effect (`openai_direct_model` set, a + plain id, and its key present), drop the Cursor-hosted OpenAI cells from the panel so the lane is reviewed once rather than twice. Default - true: the direct cell REPLACES it. Matching is by vendor prefix + true: the direct cells REPLACE them, and so gate exactly as the Cursor + cells would — counted in `Panel integrity` and the auto-approve gate + (`approve_max_failed_reviewers`), a failed one red in the rollup and + named in the review's "did not contribute" line. Matching is by vendor prefix (`gpt-`) against the resolved panel list, which is how every OpenAI id in Cursor's catalog is spelled today — the other labs' ids are untouched. Set false to run both for a side-by-side comparison of the - two backends on identical diffs, which is worth doing once before - trusting the replacement. Refuses to leave the Cursor panel empty: if + two backends on identical diffs; the direct cells are then ADVISORY — + their findings reach the judge, but the legs never fail the run and + are not counted. Refuses to leave the Cursor panel empty: if every remaining id would be dropped, preflight fails rather than - silently shrinking the panel to a single direct cell. + silently shrinking the panel to the direct cells alone. type: boolean required: false default: true @@ -1988,6 +1992,11 @@ jobs: # input, so a misconfigured opt-in neither drops the Cursor OpenAI cell # nor schedules a cell that can only error. openai_direct: ${{ steps.models.outputs.openai_direct }} + # 'true' when the direct-API cells are in effect AND replace the Cursor + # OpenAI lane: they are then that lane, so they gate exactly like Cursor + # cells (counted in ok_count/total, leg red on failure). Side by side + # ('false') they stay advisory. + openai_direct_counts: ${{ steps.models.outputs.openai_direct_counts }} steps: - name: Define panel models id: models @@ -2137,8 +2146,10 @@ jobs: fi fi echo "openai_direct=${openai_direct}" >> "$GITHUB_OUTPUT" + openai_direct_counts=false if [ "$openai_direct" = "true" ] \ && [ "${OPENAI_DIRECT_REPLACES:-true}" = "true" ]; then + openai_direct_counts=true # ascii_downcase so a differently-cased id cannot slip past the # vendor match and leave a duplicate OpenAI cell running. dropped="$(jq -c '[.[] | select((ascii_downcase | startswith("gpt-")))]' /tmp/models.json)" @@ -2154,6 +2165,7 @@ jobs: echo "direct-API cell is on, but the panel list carries no Cursor OpenAI id to replace; panel unchanged." fi fi + echo "openai_direct_counts=${openai_direct_counts}" >> "$GITHUB_OUTPUT" echo "models=$(jq -c . /tmp/models.json)" >> "$GITHUB_OUTPUT" - name: Require a pinned workflows_ref @@ -2739,9 +2751,10 @@ jobs: echo "Reviewer cell ${REVIEW_TYPE}/${MODEL} submitted its findings." review-openai-direct: - # An OPT-IN adversarial cell that reaches its model through the OpenAI API - # directly, via the Codex CLI, instead of through Cursor. By default it - # REPLACES the Cursor-hosted OpenAI cell (preflight drops that id from the + # An OPT-IN pair of cells (adversarial + edge-case, the same two passes and + # prompts as a Cursor model) that reach their model through the OpenAI API + # directly, via the Codex CLI, instead of through Cursor. By default they + # REPLACE the Cursor-hosted OpenAI cells (preflight drops that id from the # panel list) so the lane is reviewed once, not twice; set # `openai_direct_replaces_cursor_openai: false` to run both for a # side-by-side comparison. Default OFF: @@ -2762,9 +2775,7 @@ jobs: # (`findings-direct-...`) is read alongside the others, and `consolidate` # lists this job in `needs:` so it never downloads before the upload # lands. The `direct-` prefix is what `Aggregate panel findings` keys on - # to keep this cell OUT of the Cursor panel's `expected` set and its - # ok_count/total — the numbers `Panel integrity` and the auto-approve gate - # read — so an opt-in cell can inform the judge without ever gating. + # to tell these cells apart from the Cursor ones (aggregate-panel.py). # # Credential posture: `contents: read` only, no write-scoped token (see # SECURITY.md), and the only PR-authored content the agent sees is the @@ -2774,12 +2785,21 @@ jobs: # read-only sandbox could still read), and the sandboxed shell's env is # cut to the core set so the agent cannot echo it back. # - # ADVISORY on purpose: `continue-on-error` at the JOB level, so no step in - # here — install, version check, prompt build, upload — can turn the - # caller's run red while this backend is still being characterised. + # Gating follows the role (preflight's `openai_direct_counts`): + # - REPLACING the Cursor OpenAI lane: these cells ARE that lane, so they + # gate exactly like a Cursor cell — counted in ok_count/total, which + # `Panel integrity` and the auto-approve gate read, and the leg goes red + # when the cell did not submit. While they were advisory, a failed + # direct cell cost nothing against `approve_max_failed_reviewers`, so a + # run could approve with the OpenAI lane silent; counted, it spends that + # allowance like a failed Cursor cell, so the gate covers the full panel + # again. + # - SIDE BY SIDE (replace false): ADVISORY, as when the backend was first + # characterised — job-level `continue-on-error`, never counted — since + # the Cursor OpenAI cells already gate that lane. # - # v1 scope: adversarial only (not edge-case), one model, one round. - name: 'adversarial direct (${{ inputs.openai_direct_model }})' + # Scope: one model, one round, no retry. + name: '${{ matrix.review_type }} direct (${{ inputs.openai_direct_model }})' needs: [gate, diff-size, preflight, ledger] if: >- needs.preflight.outputs.openai_direct == 'true' @@ -2790,12 +2810,18 @@ jobs: # OPENAI_API_KEY, and a pool is where a prompt-injected model cell may # have run. Only CURSOR_API_KEY-only, read-only jobs may follow `runs_on`. runs-on: ubuntu-latest - continue-on-error: true + # Advisory only side by side; when replacing, a failed cell is as red as a + # failed Cursor cell. + continue-on-error: ${{ needs.preflight.outputs.openai_direct_counts != 'true' }} # Matches the `review` job's cap so the judge's budget reasoning (see # `review`) holds for this cell too; the agent step caps lower, at 15. timeout-minutes: 30 permissions: contents: read + strategy: + fail-fast: false + matrix: + review_type: [adversarial, edge-case] steps: - name: Seed default findings artifact # Same contract as the `review` job: pre-seed status=error so the cell @@ -2803,13 +2829,14 @@ jobs: # MCP output tool overwrites this on success. env: MODEL: ${{ inputs.openai_direct_model }} + REVIEW_TYPE: ${{ matrix.review_type }} run: | mkdir -p /tmp/findings-out python3 - <<'PY' import json, os record = { "model": os.environ["MODEL"], - "review_type": "adversarial", + "review_type": os.environ["REVIEW_TYPE"], "status": "error", "error": "Direct-API cell did not reach the submission step.", "findings": [], @@ -2964,17 +2991,18 @@ jobs: if: steps.key.outputs.present == 'true' # Byte-identical construction to the Cursor cells' `Build prompt`, so a # finding difference is attributable to the model or the backend rather - # than to the prompt: the same prompt-adversarial.md with the same + # than to the prompt: the same prompt-.md with the same # prior-review ledger spliced in, the same marker stripped and # re-emitted by fence-diff.py, the same new-hunks block from round 2 # on, and a fresh per-prompt nonce minted into a shell variable (never # a step `env:`) so a leaked marker from one cell cannot forge a fence # in another. See that step for the full rationale. env: + REVIEW_TYPE: ${{ matrix.review_type }} LEDGER_ROUNDS: ${{ needs.ledger.outputs.rounds }} run: | python3 "$CURSOR_REVIEW_ASSETS/build-ledger.py" splice \ - --prompt "$CURSOR_REVIEW_ASSETS/prompt-adversarial.md" \ + --prompt "$CURSOR_REVIEW_ASSETS/prompt-${REVIEW_TYPE}.md" \ --marker '=== BEGIN DIFF ===' \ --insert /tmp/ledger/ledger.md \ --out /tmp/prompt-head.txt @@ -3007,6 +3035,7 @@ jobs: timeout-minutes: 15 env: MODEL: ${{ inputs.openai_direct_model }} + REVIEW_TYPE: ${{ matrix.review_type }} # `codex exec` reads its API key from this variable, so the key lives # only in this step's process env: never argv (visible in the process # list), never a file (`codex login` writes ~/.codex/auth.json, which @@ -3025,7 +3054,7 @@ jobs: --mode reviewer \ --out /tmp/findings-out/findings.json \ --model "$MODEL" \ - --review-type adversarial \ + --review-type "$REVIEW_TYPE" \ --init # The SAME MCP submission tool the Cursor cells use, so the findings # schema the judge reads is produced by one implementation for both @@ -3040,7 +3069,7 @@ jobs: "--mode", "reviewer", "--out", "/tmp/findings-out/findings.json", "--model", os.environ["MODEL"], - "--review-type", "adversarial", + "--review-type", os.environ["REVIEW_TYPE"], ])) PY )" @@ -3058,6 +3087,8 @@ jobs: if: always() env: MODEL: ${{ inputs.openai_direct_model }} + REVIEW_TYPE: ${{ matrix.review_type }} + COUNTS: ${{ needs.preflight.outputs.openai_direct_counts }} # Status and a count only — NOT the findings verbatim, unlike the Cursor # cells: the run log is public, and finding bodies are model output # steered by PR-authored text. The full record still reaches the judge @@ -3079,10 +3110,14 @@ jobs: PY ) echo "Direct-API cell: status=${status}, ${count} finding(s)." - if [ "$status" != "ok" ]; then - echo "::warning::Direct-API cell adversarial/${MODEL} did not submit (status=$status). Its record still reaches the judge; this leg stays green because the cell is advisory." + if [ "$status" = "ok" ]; then + echo "Direct-API cell ${REVIEW_TYPE}/${MODEL} submitted its findings." + elif [ "$COUNTS" = "true" ]; then + # Failed AFTER the upload below, by `Fail the leg when the cell did + # not submit`, so the errored record still reaches the judge. + echo "Direct-API cell ${REVIEW_TYPE}/${MODEL} did not submit (status=$status); it replaces the Cursor OpenAI lane, so it counts as a failed reviewer." else - echo "Direct-API cell adversarial/${MODEL} submitted its findings." + echo "::warning::Direct-API cell ${REVIEW_TYPE}/${MODEL} did not submit (status=$status). Its record still reaches the judge; this leg stays green because the cell is advisory." fi - name: Upload findings artifact @@ -3093,11 +3128,46 @@ jobs: # tell it apart. The id is charset-checked in preflight. uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 with: - name: findings-direct-adversarial-${{ inputs.openai_direct_model }} + name: findings-direct-${{ matrix.review_type }}-${{ inputs.openai_direct_model }} path: /tmp/findings-out/findings.json if-no-files-found: error retention-days: 7 + - name: Fail the leg when the cell did not submit + # Only when these cells REPLACE the Cursor OpenAI lane — the same + # rollup-visibility contract as the `review` job's step of this name + # (BE-15554), and like it deliberately AFTER the upload, so the errored + # record still reaches the judge and the consolidated review still + # posts. Side by side the cell is advisory and this step is skipped. + # Same availability-not-attestation scope as that step: the status is + # read from a file this cell's own agent could write. + if: always() && needs.preflight.outputs.openai_direct_counts == 'true' + env: + MODEL: ${{ inputs.openai_direct_model }} + REVIEW_TYPE: ${{ matrix.review_type }} + run: | + # Read defensively and sanitized for a ::workflow command:: line, as + # in the `review` job (`ValueError` covers invalid UTF-8 too). + status=$(python3 - <<'PY' + import json, re + + try: + with open("/tmp/findings-out/findings.json", encoding="utf-8") as f: + cell = json.load(f) + except (OSError, ValueError): + cell = None + value = cell.get("status") if isinstance(cell, dict) else None + if not isinstance(value, str): + value = "error" + print(re.sub(r"[^A-Za-z0-9_.-]", "_", value)[:32] or "error") + PY + ) + if [ "$status" != "ok" ]; then + echo "::error::Direct-API reviewer cell ${REVIEW_TYPE}/${MODEL} did not submit a review (status=$status). It replaces the Cursor OpenAI lane, so it counts as a failed reviewer; its findings artifact was uploaded and the panel still consolidates." + exit 1 + fi + echo "Direct-API reviewer cell ${REVIEW_TYPE}/${MODEL} submitted its findings." + consolidate: # Adjudicates the panel cell artifacts into one consolidated review. # Calls cursor-agent one more time as the judge, with prompt-judge.md. @@ -3107,9 +3177,10 @@ jobs: # "Credential isolation" note for why the judge must not share a job with # a write-scoped token. name: Consolidate panel - # `review-openai-direct` is here only for ORDERING — its artifact must land + # `review-openai-direct` is here only for ORDERING — its artifacts must land # before `Download panel findings` runs. Its result is never read: when it - # is skipped (the default) or fails, `!cancelled()` below still runs. + # is skipped (the default) or fails, `!cancelled()` below still runs, and + # a failed cell is counted from its artifact (or its absence) instead. needs: [gate, diff-size, preflight, review, review-openai-direct, ledger] # `!cancelled()`, not `always()`: the judge is the most expensive call in # the workflow, and the only job that does anything with its payload, @@ -3316,68 +3387,21 @@ jobs: id: aggregate env: PANEL_MODELS: ${{ needs.preflight.outputs.models }} + # The direct-API cells COUNT only when they replace the Cursor OpenAI + # lane; side by side they stay advisory. See aggregate-panel.py. + DIRECT_MODEL: ${{ needs.preflight.outputs.openai_direct == 'true' && inputs.openai_direct_model || '' }} + DIRECT_COUNTS: ${{ needs.preflight.outputs.openai_direct_counts }} run: | # Each artifact directory contains one findings.json. Combine into - # a single panel.json array preserving cell metadata. - python3 - <<'PY' - import json, os, glob - - panel = [] - # The opt-in direct-API cell (`findings-direct-*`). Its findings go to - # the judge like any cell's, but it is NOT a Cursor panel cell: it is - # kept out of `present` (so it cannot mask a missing Cursor cell with - # the same id) and out of ok_count/total, which `Panel integrity` and - # the auto-approve gate read — an advisory cell must not gate. - direct = [] - for path in sorted(glob.glob("/tmp/panel/findings-*/findings.json")): - try: - with open(path, encoding="utf-8") as f: - cell = json.load(f) - except (OSError, json.JSONDecodeError): - continue - if isinstance(cell, dict): - if os.path.basename(os.path.dirname(path)).startswith("findings-direct-"): - direct.append(cell) - else: - # Only THIS step may set the marker: a Cursor cell's own - # record must never be able to excuse itself from the count. - cell.pop("direct_api", None) - panel.append(cell) - - expected = { - (model, review_type) - for model in json.loads(os.environ["PANEL_MODELS"]) - for review_type in ("adversarial", "edge-case") - } - present = { - (cell.get("model"), cell.get("review_type")) - for cell in panel - } - for model, review_type in sorted(expected - present): - panel.append({ - "model": model, - "review_type": review_type, - "status": "error", - "error": "Cell findings artifact was not uploaded.", - "findings": [], - }) - - ok = sum(1 for c in panel if c.get("status") == "ok") - print(f"Panel: {ok}/{len(panel)} cells contributed findings.") - for c in direct: - print(f"Direct-API cell (advisory, not counted): status={c.get('status')}.") - # Read by `Build consolidated findings file`, which keeps these - # out of the `panel` metadata auto-approve counts failures from. - c["direct_api"] = True - - with open("/tmp/panel.json", "w", encoding="utf-8") as f: - json.dump(panel + direct, f, indent=2) - - # Emit metadata for the next step to gate on. - with open(os.environ["GITHUB_OUTPUT"], "a") as g: - g.write(f"ok_count={ok}\n") - g.write(f"total={len(panel)}\n") - PY + # a single panel.json array preserving cell metadata, and emit the + # ok_count/total `Panel integrity` and auto-approve gate on. + python3 "$CURSOR_REVIEW_ASSETS/aggregate-panel.py" \ + --panel-dir /tmp/panel \ + --models "$PANEL_MODELS" \ + --direct-model "$DIRECT_MODEL" \ + --direct-counts "${DIRECT_COUNTS:-false}" \ + --out /tmp/panel.json \ + --github-output "$GITHUB_OUTPUT" - name: Build judge prompt if: steps.aggregate.outputs.ok_count != '0' @@ -3498,14 +3522,18 @@ jobs: with open("/tmp/panel.json", encoding="utf-8") as f: panel = json.load(f) - # Cursor panel cells only: auto-approve counts failed reviewers from - # this list, and the advisory direct-API cell must not gate. Its - # findings still reach the degraded fallback below via `panel`. - panel_meta = [ - {"model": c.get("model"), "review_type": c.get("review_type"), "status": c.get("status")} - for c in panel - if c.get("direct_api") is not True - ] + # Counted cells only: auto-approve counts failed reviewers from this + # list, so an ADVISORY direct-API cell must not appear here. A counted + # one does, tagged `direct` so the posted review names its backend. + # Advisory findings still reach the degraded fallback via `panel`. + panel_meta = [] + for c in panel: + if c.get("advisory") is True: + continue + meta = {"model": c.get("model"), "review_type": c.get("review_type"), "status": c.get("status")} + if c.get("direct_api") is True: + meta["direct"] = True + panel_meta.append(meta) with open("/tmp/judge-findings.json", encoding="utf-8") as f: judge = json.load(f) diff --git a/docs/callers/cursor-review.md b/docs/callers/cursor-review.md index 6148a5d7..0cf264b2 100644 --- a/docs/callers/cursor-review.md +++ b/docs/callers/cursor-review.md @@ -113,8 +113,8 @@ pull-requests: write # posting the consolidated review (and, at the round cap, |---|---|---| | `judge_model` | `claude-opus-5-5-xhigh` | Consolidates the panel into one review. | | `panel_models` | `''` | JSON array of model ids replacing the built-in panel list (each runs both review types; preflight checks each one exists in the live catalog — existence only, not ZDR). Use for per-repo experiments such as a reasoning-tier A/B — and prefer wiring it to a **variable** rather than a literal, see [Running a model experiment](#running-a-model-experiment). | -| `openai_direct_model` | `''` | Opt-in. An OpenAI model id reviewed directly through the OpenAI API (pinned Codex CLI) as one **adversarial** cell, always on GitHub-hosted `ubuntu-latest`. By default it **replaces** the Cursor-hosted OpenAI (`gpt-*`) panel cell — see the next row. Requires `secrets.OPENAI_API_KEY` and a plain id (`[A-Za-z0-9._-]`); without either, preflight warns, the cell stays off and the panel is unchanged. Advisory: its findings reach the judge, but the leg never fails the run and is not counted in `Panel integrity` or the auto-approve gate. | -| `openai_direct_replaces_cursor_openai` | `true` | Only with `openai_direct_model` (and its key) in effect: drop the Cursor `gpt-*` cell(s) from the panel so the OpenAI lane is reviewed once. `false` runs both, for a side-by-side comparison of the two backends. Preflight fails rather than leave the Cursor panel empty. | +| `openai_direct_model` | `''` | Opt-in. An OpenAI model id reviewed directly through the OpenAI API (pinned Codex CLI) as two cells — **adversarial** and **edge-case**, with the same prompts the Cursor cells use — always on GitHub-hosted `ubuntu-latest`. By default they **replace** the Cursor-hosted OpenAI (`gpt-*`) panel cells — see the next row, which also decides whether they gate. Requires `secrets.OPENAI_API_KEY` and a plain id (`[A-Za-z0-9._-]`); without either, preflight warns, the cells stay off and the panel is unchanged. | +| `openai_direct_replaces_cursor_openai` | `true` | Only with `openai_direct_model` (and its key) in effect: drop the Cursor `gpt-*` cell(s) from the panel so the OpenAI lane is reviewed once. The direct cells then **gate** exactly like the Cursor cells they replace: they count toward `Panel integrity` and the auto-approve gate (`approve_max_failed_reviewers`), a failed one turns its leg red, and the review's "did not contribute" line names it, e.g. `gpt-5.6-sol:edge-case (direct, error)`. `false` runs both, for a side-by-side comparison of the two backends; the direct cells are then **advisory** — their findings reach the judge, but they are never counted and their legs never fail the run. Preflight fails rather than leave the Cursor panel empty. | | `skip_bot_branch_prefixes` | `ci/bump- chore/refresh- auto/refresh-` | Skip the panel for Bot-authored PRs on these branch prefixes (machine pin bumps / refreshes). `''` to review every bot PR. | | `diff_size_cap` | `5000` | Skip review above this diff size. An over-cap PR is not silent — see the gotcha below. Under `blocking: true` it is also not green: an unreviewed PR cannot pass the gate. | | `ignore_comments` | `true` | Discount blank/comment-only lines from the size count (count-only — the panel still sees them). |