Repository navigation
docs(benchmarks): record the v0.9.0 pilot baseline - #44
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Record the frozen v0.9.0 pilot20 baseline for 0.10.0 development: 18/20 passed, 2 scored failures, 0 unscored trials. Add the predeclared configurations, all per-task results and failure analysis, reproduction instructions, and synchronized Chinese/English 0.10.x development notes.
The benchmark wheels exactly match the PyPI release. Each task ran once, and all verifiers completed on their first attempt. Both failures ended with
response_truncated. The known estimated cost is $1.974266; usage from two recovered Scheme stream interruptions is missing, so this is a partial subtotal. Agent setup was allowed 1,800 seconds instead of the historical 360-second default, preserving native work and per-attempt verifier limits. Preflight passed all 20 installations and QEMU's registered profile; 19 verifier profiles remain explicitlynot_configured.Validation: 20/20 native ATIF trajectories, 21 release source hashes, 152 unchanged task-file hashes, preflight/full-run task checksums, five runner configurations, accounting totals, and 21 local documentation links verified. CI passes on Python 3.13, Python 3.14, and Harbor workflow contracts. This PR changes benchmark records/configuration and documentation only; raw logs remain in ignored local jobs directories.