Skip to content

test(benchmarks): establish a common-main 34-task baseline - #50

Merged
minixalpha merged 15 commits into
mainfrom
benchmark/dev34-main-baseline-20261012
Oct 14, 2026
Merged

minixalpha merged 15 commits into
mainfrom
benchmark/dev34-main-baseline-20261012

Conversation

@minixalpha

@minixalpha minixalpha commented Oct 12, 2026 •

Copy link
Copy Markdown
Owner

The development pool previously combined results from separate runs and agent versions. This PR establishes one fresh trial for each of all 34 tasks on main 2eb99b608dfb39464d17d109a4a8f11afda607a9: 28 passes and six scored failures (82.35%), with zero model-phase setup errors, verifier errors, or verifier retries.

  • Add frozen configurations, audited results, a complete report, and failure diagnoses. Retain the earlier twelve-task baseline separately; no historical model outputs contribute to the 34-task score.
  • Add tested orchestration and verified caches for original Docker images, immutable APT indexes, and uv/CPython release archives. Preserve task images, package origins, official tests, runtime source, and budgets. Record failed preflights separately; QEMU's dependency profile passes, while the other 33 remain not_configured.
  • Flag one passing tensor-parallelism trial's external benchmark revision-metadata access. Preserve raw rewards and disclose incomplete accounting for two trials. Publish only derived findings and hashes; raw logs containing credentials remain local. This development-pool result is not a held-out or exposure-free evaluation.
  • Record the result and frozen prompt in the development notes, regenerate the English notes from the Chinese source, and align both Harbor READMEs. The Chinese note retains the full English prompt and Chinese translation together.

Validation: 167 Harbor workflow tests; Python 3.13/3.14 and Harbor CI; original dependency-download integration probes; source, wheel, image, configuration, and frozen-evidence checks; documentation links, prompt SHA256, and credential scan.

Full report

No changelog entry is added because this is internal benchmark tooling and experiment documentation. No research-note files changed. The agent and adapter runtime are unchanged; this PR does not publish a release.

@minixalpha
minixalpha marked this pull request as ready for review October 14, 2026 13:59
@minixalpha
minixalpha merged commit 56a9e38 into main Oct 14, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant