Support plugins changes from private branchs

[haoranpb]

Why

Team needs to be able to test their changes on the plugins before merging. Currently, Copilot CLI only supports loading from the default branch (requires merging).

How

Instead of relying on the marketplace and things like copilot marketplace add, we orchestrate clone the repo containing plugins ourselves and load them using --plugin-dir.

---

Also bundled in some other changes like moving hardcoded path into config and set the plugin dir under BC-Bench repo.

[Copilot]

Pull request overview

Reworks plugin experiments to load local or revision-pinned GitHub plugins directly through --plugin-dir.

Changes:

• Adds typed plugin configuration and GitHub revision cloning.
• Integrates session-scoped plugins into Copilot and Claude runners.
• Moves generated plugins outside evaluated repositories and updates tests/docs.

Reviewed changes

Copilot reviewed 22 out of 23 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
.gitignore	Ignores runtime plugin storage.
EXPERIMENT.md	Documents the new plugin model.
src/bcbench/agent/claude/agent.py	Loads resolved plugins in Claude.
src/bcbench/agent/copilot/agent.py	Loads resolved plugins in Copilot.
src/bcbench/agent/shared/__init__.py	Exports plugin resolution.
src/bcbench/agent/shared/config.yaml	Defines example plugin entries.
src/bcbench/agent/shared/lsp.py	Uses centralized plugin storage.
src/bcbench/agent/shared/plugin.py	Resolves local and GitHub plugins.
src/bcbench/agent/shared/plugins/bcbench-example/.claude-plugin/marketplace.json	Removes the obsolete marketplace.
src/bcbench/agent/shared/plugins/bcbench-example/.claude-plugin/plugin.json	Renames the example plugin.
src/bcbench/agent/shared/plugins/bcbench-example/bcbench-example-plugin/skills/bcbench-example/SKILL.md	Removes the nested example skill.
src/bcbench/agent/shared/plugins/bcbench-example/skills/bcbench-example/SKILL.md	Adds the direct example skill.
src/bcbench/commands/evaluate.py	Updates sample experiment metadata.
src/bcbench/config.py	Centralizes plugin paths and manifest location.
src/bcbench/operations/__init__.py	Updates exported operations.
src/bcbench/operations/git_operations.py	Adds revision-based cloning through gh.
src/bcbench/operations/plugin_operations.py	Removes marketplace installation logic.
src/bcbench/types.py	Adds validated plugin configuration.
tests/conftest.py	Adds isolated plugin storage fixture.
tests/test_agent_plugins.py	Tests plugin validation and resolution.
tests/test_git_operations.py	Tests revision cloning.
tests/test_lsp_config.py	Tests external plugin-root behavior.
tests/test_plugin_operations.py	Removes marketplace operation tests.

[Copilot]

Pull request overview

Copilot reviewed 22 out of 23 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (5)

src/bcbench/agent/shared/plugin.py:74

• plugin.name is used directly as a deletion target: values such as .., ../NAV, or an absolute path escape .bcbench, and clone_repo_at_revision calls rmtree on that destination before cloning. Constrain the destination to one immediate child of the configured plugin root before invoking the destructive clone.

            clone_dir: Path = _config.paths.plugin_root / plugin.name
            clone_repo_at_revision(str(plugin.repo), str(plugin.revision), clone_dir)

src/bcbench/agent/shared/plugin.py:75

• The documented GitHub path contract says it is relative to the clone, but an absolute path discards clone_dir and .. can escape it. If the external location contains a manifest, the run loads unrelated local content while recording it as the requested GitHub revision. Resolve the path and reject anything outside the clone.

            plugin_dir = clone_dir / plugin.path

src/bcbench/agent/shared/plugin.py:74

• Clone destinations are keyed only by name, while the configuration permits multiple entries with the same name and different revisions. Resolving the second entry deletes/replaces the first clone, but both records remain and the first --plugin-dir then points into the second revision (or a missing subpath). Reject duplicate enabled names or derive isolated destinations per entry; also reserve the generated al-lsp-plugin directory name.

            clone_dir: Path = _config.paths.plugin_root / plugin.name
            clone_repo_at_revision(str(plugin.repo), str(plugin.revision), clone_dir)

src/bcbench/agent/copilot/agent.py:87

• The previous runner-level plugin test was deleted, and the replacement tests stop at path resolution; none verifies that Copilot receives every resolved plugin via --plugin-dir. Add a mocked run_copilot_agent test asserting the generated subprocess arguments and recorded plugin list, including multiple plugins.

        cmd_args.extend(f"--plugin-dir={plugin_dir}" for plugin_dir in plugins.values())

src/bcbench/agent/claude/agent.py:82

• The previous runner-level plugin test was deleted, and no replacement verifies that Claude receives resolved plugins via --plugin-dir. Add a mocked run_claude_code test asserting the subprocess arguments and ExperimentConfiguration.plugins, including multiple plugin directories.

        cmd_args.extend(f"--plugin-dir={plugin_dir}" for plugin_dir in plugins.values())

[Copilot]

Pull request overview

Copilot reviewed 25 out of 26 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (5)

src/bcbench/agent/shared/plugin.py:64

• A config that omits the optional plugins section now raises KeyError; the replaced setup path explicitly treated that case as no plugins. Preserve the no-plugin default so older/minimal configs continue to run.

    plugins = [PluginConfig(**entry) for entry in agent_config["plugins"] if entry.get("enabled", False)]

src/bcbench/agent/shared/plugin.py:87

• Clone storage is keyed only by name, but the model permits two enabled entries with the same name at different revisions. Resolving the second entry deletes and replaces the first clone, while the returned mapping still records and passes both revisions, so the experiment metadata claims code was loaded that is no longer present. Reject duplicate GitHub names or include the revision in an independently confined destination.

    clone_dir: Path = (plugin_root / plugin.name).resolve()

src/bcbench/types.py:58

• This slug constraint rejects valid GitHub repositories whose names contain dots (for example, owner/plugin.js), so those repositories cannot be used as plugin sources. Allow GitHub-valid repository-name characters rather than carrying over the narrower dataset regex.

type RepoSlug = Annotated[str, StringConstraints(pattern=r"^[a-zA-Z0-9_-]+/[a-zA-Z0-9_-]+$")]

src/bcbench/agent/copilot/agent.py:87

• The runner's new plugin integration is untested: the previous runner-level plugin test was deleted, while the replacement tests stop at path resolution and never assert that Copilot receives each --plugin-dir or records the matching keys. Add a runner test covering this command assembly.

        cmd_args.extend(f"--plugin-dir={plugin_dir}" for plugin_dir in plugins.values())

src/bcbench/agent/claude/agent.py:82

• The runner's new plugin integration is untested: the previous runner-level plugin test was deleted, while the replacement tests stop at path resolution and never assert that Claude receives each --plugin-dir or records the matching keys. Add a runner test covering this command assembly.

        cmd_args.extend(f"--plugin-dir={plugin_dir}" for plugin_dir in plugins.values())

[Copilot]

Pull request overview

Copilot reviewed 25 out of 26 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (3)

src/bcbench/agent/shared/plugin.py:87

• GitHub clone destinations are keyed only by plugin.name, even though PluginConfig allows multiple enabled entries with the same name and this root also stores the generated al-lsp-plugin. clone_repo_at_revision removes an existing destination, so a later duplicate (or a configured plugin named al-lsp-plugin) replaces the earlier/generated plugin; the returned paths then all load the last clone while the experiment records the others as enabled. Use a separate namespace keyed by repository/revision and reject duplicate records or reserved names before cloning.

    clone_dir: Path = (plugin_root / plugin.name).resolve()

src/bcbench/agent/copilot/agent.py:87

• The previous runner-level plugin test was deleted, but no replacement verifies that Copilot receives every resolved plugin directory and records the corresponding keys. Please add a mocked run_copilot_agent test covering resolve_config_plugins, repeated --plugin-dir arguments, and ExperimentConfiguration.plugins; otherwise a wiring regression here can pass all resolver-only tests while plugins are never loaded.

        cmd_args.extend(f"--plugin-dir={plugin_dir}" for plugin_dir in plugins.values())

src/bcbench/agent/claude/agent.py:82

• The previous runner-level plugin test was deleted, but no replacement verifies that Claude receives every resolved plugin directory and records the corresponding keys. Please add a mocked run_claude_code test covering resolve_config_plugins, repeated --plugin-dir arguments, and ExperimentConfiguration.plugins; resolver tests alone do not exercise this integration.

        cmd_args.extend(f"--plugin-dir={plugin_dir}" for plugin_dir in plugins.values())

[Copilot]

Pull request overview

Copilot reviewed 25 out of 26 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (3)

src/bcbench/agent/shared/config.yaml:119

• This GitHub entry omits the required path field, so test_every_shipped_github_entry_parses fails with a Pydantic validation error before any evaluation can run. The pinned BCQuality revision has its plugin manifest at the repository root, so set the path to ".".

    revision: "ad8ccde5953fd6ce072e3b59049406783066ed60"

src/bcbench/agent/copilot/agent.py:87

• The previous plugin tests covered the Copilot runner wiring, but that coverage was deleted and the replacement tests stop at path resolution. Add a runner test that mocks resolve_config_plugins and asserts both the repeated --plugin-dir arguments and ExperimentConfiguration.plugins; otherwise the PR's main integration can regress while all new tests still pass.

        cmd_args.extend(f"--plugin-dir={plugin_dir}" for plugin_dir in plugins.values())

src/bcbench/agent/claude/agent.py:82

• The previous plugin tests covered the Claude runner wiring, but that coverage was deleted and the replacement tests stop at path resolution. Add a runner test that mocks resolve_config_plugins and asserts both the repeated --plugin-dir arguments and ExperimentConfiguration.plugins; otherwise the PR's main integration can regress while all new tests still pass.

        cmd_args.extend(f"--plugin-dir={plugin_dir}" for plugin_dir in plugins.values())

[Copilot]

Pull request overview

Copilot reviewed 25 out of 26 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (4)

src/bcbench/types.py:58

• RepoSlug rejects valid GitHub repository names containing periods (for example, owner/.github or owner/plugin.v2). Since plugin configuration is documented as accepting GitHub owner/repo slugs, these repositories currently fail Pydantic validation before cloning. Allow . in the repository component.

type RepoSlug = Annotated[str, StringConstraints(pattern=r"^[a-zA-Z0-9_-]+/[a-zA-Z0-9_-]+$")]

src/bcbench/agent/shared/plugin.py:90

• The clone destination is keyed only by plugin.name in the same root used by generated plugins. Two enabled GitHub entries with the same name but different revisions therefore return distinct records pointing to one directory; the second clone deletes the first, so both CLI arguments load the second revision. A configured plugin named al-lsp-plugin likewise replaces the generated LSP plugin. Put config clones in a separate namespace and include the revision (or reject duplicate/reserved names).

    plugin_root: Path = _config.paths.plugin_root.resolve()
    clone_dir: Path = (plugin_root / plugin.name).resolve()
    if clone_dir.parent != plugin_root:
        raise AgentError(f"Plugin '{plugin.name}': name must be a single directory directly under {plugin_root}")
    return clone_dir

src/bcbench/agent/copilot/agent.py:87

• The previous runner integration test was deleted, and the replacement tests only exercise plugin resolution; none verifies that resolved plugins are actually added to the Copilot command. Please retain a mocked run_copilot_agent test asserting each resolved path becomes a repeatable --plugin-dir argument and its record is captured in ExperimentConfiguration.

        cmd_args.extend(f"--plugin-dir={plugin_dir}" for plugin_dir in plugins.values())

src/bcbench/agent/claude/agent.py:82

• The previous Claude runner integration test was deleted, and the new resolver tests do not verify this command wiring. Add a mocked run_claude_code test asserting that all resolved paths are passed as repeatable --plugin-dir arguments and that their records are stored in ExperimentConfiguration; otherwise a CLI assembly regression can leave resolution tests green while no plugin is loaded.

        cmd_args.extend(f"--plugin-dir={plugin_dir}" for plugin_dir in plugins.values())