Skip to content

Make benchmark setup, harness integration, and scoring discoverable - #4

Merged
zomux merged 2 commits into
developfrom
chore/benchmark-onboarding
Sep 30, 2026
Merged

zomux merged 2 commits into
developfrom
chore/benchmark-onboarding

Conversation

@zomux

@zomux zomux commented Sep 30, 2026

Copy link
Copy Markdown
Contributor

New contributors currently encounter Qwen-only onboarding, machine-specific launchers, and mandatory vLLM installation even when they only need an HTTP model endpoint. This change gives model and harness authors a documented path from setup to one task, a full suite, offline success-rate reporting, and GitHub issue submission.

  • Add a concise repository map, benchmark quickstart, custom-agent contract, scoring guide, and server setup guide. Preserve the previous READMEs as historical references and retain existing dataset paths.
  • Split API-runner and optional local-inference dependencies; add environment-based example configurations, configurable OpenAI-compatible endpoints/authentication, and a module:Class adapter hook.
  • Share task-specific verifier selection between the runner and offline SR reporting. Augmented variants now use their adjacent verifier and retain distinct trajectory filenames. The SR report exposes missing tasks and rejects duplicate/invalid attempts rather than publishing an incomplete full-suite score.
  • Give the existing CCE script an explicit trajectory input and environment-based judge settings, removing its embedded credential and machine-specific log root. Its metric calculation is unchanged.

Validation: 21 offline tests passed on Python 3.12 and 3.14, covering config expansion, endpoint/auth routing, custom adapters, main/augmented verifier selection, output preservation, SR coverage and errors, and CCE entry points without model calls. Runner/scorer help commands, Python compilation, local documentation links, and git diff --check passed. Added a GitHub Actions workflow for the offline checks.

No live game build or paid model/judge evaluation was performed. Augmented early stopping changes because the previous runner selected main-task criteria; historical trajectories need review before comparison. The docs explicitly distinguish repository verifier scores from paper-protocol certification, and document that a validated reproduction of the paper's PSR remains unavailable in this checkout.

@zomux
zomux marked this pull request as ready for review September 30, 2026 13:40
@zomux
zomux merged commit e193520 into develop Sep 30, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant