Add BC PR-Review engine arm with BCQuality ref override

[gggdttt]

What

Adds the BC PR-Review engine arm to BC-Bench's code-review category: instead of BC-Bench's own review pipeline, this arm runs the engine's production reviewer (Invoke-LocalReview.ps1 from microsoft/BC-ALAgents) against the patched worktree and scores its findings with BC-Bench's existing CodeReviewPipeline. This lets us benchmark the real engine + BCQuality policy head-to-head with the Copilot arm.

The arm is opt-in and code-review only; with it off, nothing changes.

Changes

• src/bcbench/evaluate/codereview_engine.py (new) — run_engine_review(...), the engine adapter:
  1. Commits the patched worktree and captures the pre-patch HEAD as the diff base.
  2. Provisions a disposable BCQuality checkout — copies BCQUALITY_ROOT (minus .git) if set, otherwise clones microsoft/BCQuality at BCQUALITY_REF (default main). A disposable copy is required because the engine's filter step deletes files in place, so it must never point at a live clone. The clone token is passed via an auth header (never in the remote URL).
  3. Runs Invoke-LocalReview.ps1 -RepoPath -BaseRef -BCQualityRoot -OutputDir [-Model].
  4. Reads the engine's _review-report.json from OutputDir and normalizes each finding (location.file, location.line/range.start-line, range.end-line, message) into the flat review.json the scorer already understands.
  5. Maps BCQuality severity to the gold scale, mirroring the production engine: blocker→critical, major→high, minor→medium, info→low.
• src/bcbench/commands/evaluate.py — wires the arm into the existing bcbench evaluate copilot command via a new --engine-scripts-dir option (env ENGINE_SCRIPTS_DIR). When set (code-review only), the pipeline routes through run_engine_review and labels the agent "BC PR-Review Engine"; otherwise the Copilot path is unchanged.
• .github/workflows/copilot-evaluation.yml — activates the arm on CI when category == 'code-review' and BCQUALITY_REF != ''. It checks out microsoft/BC-ALAgents at ENGINE_REF (1.15.4) into engine/ and passes --engine-scripts-dir engine/agents/ALReviewAgent/scripts. Default (empty BCQUALITY_REF) keeps the current Copilot code-review.
• tests/test_codereview.py — adds TestEngineFindingMapping (severity mapping, line/range fallback, skipping incomplete findings, unknown-severity passthrough).

How to run the engine arm

• CI: on a branch, set BCQUALITY_REF (branch/tag/SHA) in the workflow env and dispatch with category: code-review. This pins the exact BCQuality content the arm reviews with, without editing the engine.
• Local: ENGINE_SCRIPTS_DIR=<...>/agents/ALReviewAgent/scripts bcbench evaluate copilot <entry> --category code-review [--engine-scripts-dir ...], optionally with BCQUALITY_ROOT to reuse a local checkout.

Validation

• End-to-end normalization verified against a real engine _review-report.json (4 findings → 4 review.json comments, severities mapped correctly).
• pytest tests/test_codereview.py — 64 passed; ruff clean.

Review note (open question)

In CI the run step exposes only GH_TOKEN (= github.token). If microsoft/BCQuality is private, the adapter's clone (step 2 above) may need a token with cross-repo read — e.g. ENGINE_REPO_TOKEN / BCQUALITY_REPO_TOKEN exported into that step. Today it falls back to github.token; flagging in case an explicit token needs to be threaded through when BCQUALITY_ROOT isn't provided.

[Copilot]

Pull request overview

Adds a BC PR-review engine evaluation path and workflow integration.

Changes:

• Adds engine invocation and findings normalization.
• Routes code-review evaluations through the engine.
• Adds conditional engine checkout and execution in CI.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

File	Description
src/bcbench/evaluate/codereview_engine.py	Implements the engine adapter.
src/bcbench/commands/evaluate.py	Adds engine routing to evaluation CLI.
.github/workflows/copilot-evaluation.yml	Configures the engine evaluation arm.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (2)

src/bcbench/commands/evaluate.py:90

• This branch accepts --engine-scripts-dir with every evaluation category even though the engine only produces code-review output. For bug-fix, test-generation, or nl2al, the selected category pipeline will subsequently look for a patch or answer that the engine never creates. Reject non-code-review categories before running the pipeline.

    if engine_scripts_dir:
        context.category.pipeline.execute(

src/bcbench/evaluate/codereview_engine.py:219

• The tests cover only finding-field mapping; none exercise the new subprocess orchestration introduced here. Add mocked tests for BCQuality ref/repository selection, the base/patch commit passed to PowerShell, timeout/failure handling, and selection of the engine's normalized artifact so regressions do not require a live engine run to detect.

    bcquality_root = _prepare_bcquality(output_dir)
    base_ref = _commit_patched_worktree(repo_path)

    started_at = time.monotonic()
    _run_local_review(local_review_script, repo_path, bcquality_root, engine_output_dir, base_ref, model)

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 4 comments.

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (2)

src/bcbench/commands/evaluate.py:86

• This makes every code-review invocation select the engine because code_review_uses_engine() defaults to True, even when BCQUALITY_REF is empty and --engine-scripts-dir was not supplied. In the default workflow path the run therefore clones/runs the engine while the job labels and summarizes it as GitHub Copilot CLI, contradicting the PR's opt-in contract and corrupting baseline results. Make the engine selection explicit (for example, activate it when the scripts option is supplied or when an explicit selector is set) and keep the no-option default on Copilot; the helper, CLI routing, tests, and workflow condition need to use the same rule.

    use_engine = category == EvaluationCategory.CODE_REVIEW and code_review_uses_engine()

src/bcbench/agent/engine.py:373

• If the engine times out, _run_local_review raises AgentTimeoutError without metrics or configuration. EvaluationPipeline.execute copies those exception fields into the saved timeout result, so this path loses the elapsed time plus engine_ref/bcquality_sha, preventing the failed run from being reproduced or attributed to the tested inputs. Attach the metadata before propagating the timeout, as the existing Copilot runner does.

    started_at = time.monotonic()
    _run_local_review(local_review_script, repo_path, bcquality_root, engine_output_dir, base_ref, model)
    execution_time = time.monotonic() - started_at

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (1)

src/bcbench/agent/engine.py:178

• copytree includes uncommitted and untracked BCQuality changes, but the recorded bcquality_sha identifies only HEAD. Distinct local policy contents can therefore receive the same experiment identity and be merged by summary grouping, making the reported configuration non-reproducible. Either require a clean checkout or record an identity derived from the copied content (or an explicit dirty marker).

    local = os.environ.get("BCQUALITY_ROOT")
    if local:
        shutil.copytree(local, dest, ignore=shutil.ignore_patterns(".git"))
        return dest, _git_head(Path(local))

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (3)

src/bcbench/agent/engine.py:74

• This makes the engine the default for every code-review run, contradicting the PR contract that the arm is opt-in and that existing Copilot reviews remain unchanged when it is off. A plain bcbench evaluate copilot ... --category code-review now clones two repositories and runs the engine unless users discover an undocumented opt-out environment variable, so baseline results silently change. Please make activation explicit (for example, a non-empty BCQUALITY_REF or --engine-scripts-dir) and retain Copilot as the default.

    return os.environ.get("BCBENCH_CODE_REVIEW_AGENT", "engine").strip().lower() != "copilot"

.github/workflows/copilot-evaluation.yml:119

• The workflow is missing the documented category == code-review && BCQUALITY_REF != '' activation gate: this labels every code-review dispatch as the engine, including the default blank-ref run that the PR description says should remain Copilot. Gate both engine selection and this label on inputs.bcquality-ref != ''; otherwise default benchmark runs are silently reassigned to a different arm.

          $label = if ('${{ inputs.category }}' -eq 'code-review') { 'BC PR-Review Engine' } else { 'GitHub Copilot CLI' }
          "label=$label" | Out-File -FilePath $env:GITHUB_OUTPUT -Append

src/bcbench/agent/engine.py:279

• Unlike the existing Copilot/Claude/BCal runners, this timeout exception carries neither AgentMetrics nor ExperimentConfiguration. EvaluationPipeline.execute copies those fields from AgentTimeoutError into the saved timeout result, so timed-out engine runs lose their execution time, custom_agent, resolved engine ref, and BCQuality SHA—precisely the metadata needed to attribute and reproduce the run. Catch/rethrow at the orchestration level with the elapsed metrics and already-resolved configuration attached.

    except subprocess.TimeoutExpired as exc:
        raise AgentTimeoutError(f"PR-review engine timed out after {_ENGINE_TIMEOUT_SECONDS}s") from exc

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

src/bcbench/agent/engine.py:74

• The arm is described as opt-in, with blank BCQUALITY_REF preserving the existing Copilot review, but this default routes every code-review invocation through the engine. As a result, an unchanged bcbench evaluate copilot --category code-review now clones two repositories and requires PowerShell instead of running Copilot directly; the workflow also labels every code-review run as engine-backed. Please either restore the documented opt-in condition consistently across the CLI/workflow or update the PR contract if this baseline replacement is intentional.

    return os.environ.get("BCBENCH_CODE_REVIEW_AGENT", "engine").strip().lower() != "copilot"

src/bcbench/agent/engine.py:178

• For a git-backed BCQUALITY_ROOT, copytree includes uncommitted edits but bcquality_sha records only HEAD. The result can therefore claim a reproducible BCQuality commit even though different content was reviewed, which is especially likely when this override is used to test a local policy change. Detect and record dirty content (or require a clean checkout) rather than reporting the clean HEAD as the reviewed content.

    local = os.environ.get("BCQUALITY_ROOT")
    if local:
        shutil.copytree(local, dest, ignore=shutil.ignore_patterns(".git"))
        return dest, _git_head(Path(local))