feat(benchmark): extract BEAM and LongMemEval into standalone plugins

[xyf2020]

Summary

LME and BEAM benchmark implementations and presets currently ship with the core package. This change separates benchmark-specific code from the core while preserving evaluation behavior.

• Move benchmark Steps, prompts, and presets into independently installable reme-lme and reme-beam plugins.
• Keep the shared BaseAgenticAnswerStep in reme.steps.benchmark.base_agentic_answer.
• Add Application(plugin_packages=...) so benchmark runners can load local plugin sources without installing their distributions.
• Preserve existing Job names, prompts, model environment variables, and configuration override behavior.
• Add plugin registration, configuration, and runner tests; update CI and migration documentation.

Related issue

None linked.

Contract and data impact

• No public configuration, schema, CLI, endpoint, streaming, or workspace-layout contract changes
• No user-owned memory files are deleted or rewritten
• Derived indexes, catalogs, graphs, caches, and metadata remain rebuildable

Compatibility changes and migration:

• lme / lme.yaml and beam / beam.yaml are now plugin-provided presets. CLI users must install the corresponding plugin:

  reme plugins install ./plugins/lme --editable
  reme plugins install ./plugins/beam --editable

• Repository benchmark runners load local plugin sources directly. ReMe and runtime dependencies must still be installed. Custom configurations using extends should reference preset file paths when the plugins are not installed.
• Replace imports from reme.steps.benchmark.lme and .beam with reme_lme.steps and reme_beam.steps. The shared base remains available from reme.steps.benchmark.
• plugin_packages is a Python-only source mapping, not serialized configuration. It loads only explicitly enabled plugins and preserves application-local registry isolation.
• Enabling a plugin alone does not add evaluation Jobs or change the default service configuration.

Request/response schemas, endpoints, streaming behavior, and workspace layouts are unchanged.

Validation

• Focused tests pass
• Unit tests pass, or omitted tests are explained below
• pre-commit run --all-files passes, or omitted checks are explained below
• Frontend checks were run when reme_studio/ changed

Checks run:

PYTHONPATH=plugins/lme/src:plugins/beam/src:. python -m pytest \
  tests/unit/test_plugin.py \
  tests/unit/test_injected_job_kwargs.py \
  plugins/lme/tests plugins/beam/tests -q

Result: 66 passed. Coverage includes plugin opt-in behavior, registry isolation, preset compatibility, mock answer/judge execution, and fresh-process runner loading without installed plugin entry points.

git diff --check upstream/main...HEAD

Result: passed.

The full unit suite and pre-commit run --all-files were not run locally. Direct Black and Flake8 checks could not run because those modules were unavailable in the active Python environment. The final deletion commit's hooks passed Pyroma and skipped file checks because that commit contained only deletions; this does not validate the full PR.

Live model evaluations were not run because they require external services and datasets. Frontend checks are not applicable: reme_studio/ is unchanged.

Checklist

• I reviewed the diff for unrelated changes and sensitive data
• Tests cover intentional behavior changes
• Defaults, schemas, and concise documentation were updated together when required
• Long-lived clients, tasks, services, and executors follow the application lifecycle

Screenshots or additional notes

No UI changes or new long-lived resources.

The three intermediate source bundles were removed in the final commit and are absent from the net PR diff. Dataset files, user workspaces, and evaluation results are unchanged.

[jinliyl]

LGTM

[jinliyl]

这里建议在 reme_beam/__init__.py 和 reme_lme/__init__.py 中显式导出各自的公共 Step。当前 README 写的是自定义 Python 调用方应从 reme_beam / reme_lme 导入插件 Step，但两个 __init__.py 目前只有 docstring，实际执行 from reme_beam import BeamAutoMemoryStep 或 from reme_lme import LmeAutoMemoryStep 会报 ImportError。显式导出后，文档声明的公共导入路径才能成立；judge Step 如果也属于公共接口，也请一并考虑从对应插件包导出。