Repository navigation
CBMC: Push performance data to gh-pages - #1494
Merged
Merged
Conversation
CI posts a CBMC runtime report on each PR. report.py compares the runtime of each proof configuration (proof name, `dm`, solver) with a baseline stored on the gh-pages branch. Baselines written before configurations existed have a single entry per proof name, timed with the default solver. classify_proof() fell back to that name-only entry for every configuration without an exact match. A secondary solver was therefore compared with the default solver's time and flagged as a regression. For example, poly_caddq_native with bitwuzla showed 112s against 3s, which is the time of the default z3 configuration. Use the name-only entry only for default configurations. Other configurations without an exact baseline entry are now reported as `new`. Signed-off-by: Hanno Becker <beckphan@amazon.co.uk>
PR reports compare CBMC runtimes with baselines on gh-pages
(dev/cbmc[-reduce-ram]/mldsa-{44,65,87}.json). report.py wrote them in
CI runs on main. Since 529a89d ("CI: Do not run on pushes to main"),
CI runs on PRs and in the merge queue only, and the baselines are
still those of 2026-07-03, from before per-configuration timings.
The merge queue tests the exact commit that main is then fast-forwarded
to, so its timings are the right baseline. When a batch of PRs merges
as one push, the run for the tip covers the whole batch.
- Each CBMC job writes its result file at its gh-pages path under
$RUNNER_TEMP/cbmc-results. In merge queue runs it uploads that tree
as artifact cbmc-results-mldsa-<set>[-reduce-ram]. A failing proof
fails the step before the upload.
- cbmc_publish.yml runs on each push to main. It finds the merge queue
run of all.yml for the pushed commit, downloads its artifacts into a
gh-pages checkout, and pushes them as one commit with the workflow
token. If bench.yml pushed first, it rebases and retries once. For a
commit that reached main outside the merge queue, it warns and skips.
The workflow pushes with git instead of the Contents API that
push_to_gh_pages() used. A result file now has 3010 entries (215
proofs, 2 data models, 7 solver profiles), about 570 KiB. Its base64
encoding is too long for the single `-F content=...` argument of
`gh api`: Linux limits one argument to 128 KiB.
Remove push_to_gh_pages() and the refs/heads/main branch of report.py
main() that called it.
Signed-off-by: Hanno Becker <beckphan@amazon.co.uk>
mkannwischer
approved these changes
Oct 25, 2026
mkannwischer
left a comment
Contributor
There was a problem hiding this comment.
Thanks for fixing this @hanno-becker. LGTM (from looking at the code - let's try it out).
mkannwischer
enabled auto-merge
October 25, 2026 14:46
Contributor
CBMC Results (ML-DSA-44, REDUCE-RAM)
Full Results (397 configurations)
|
Contributor
CBMC Results (ML-DSA-44)
Full Results (397 configurations)
|
Contributor
CBMC Results (ML-DSA-65)
Full Results (397 configurations)
|
Contributor
CBMC Results (ML-DSA-87, REDUCE-RAM)
Full Results (397 configurations)
|
Contributor
CBMC Results (ML-DSA-65, REDUCE-RAM)
Full Results (397 configurations)
|
Contributor
CBMC Results (ML-DSA-87)
Full Results (397 configurations)
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When we switched to using a merge queue in July, we stopped pushing CBMC performance results to Github pages. In consequence, all CBMC performance comparisons displayed in PRs since then are stale.
The reason is that the job that used to push results to gh-pages only does so when run on behalf of
origin/main. With the switch to merge queues, we stopped running CI upon merge toorigin/main(because the CI already ran in the merge queue, which is atoporigin/main).This PR resurrects CBMC performance data on gh-pages, as follows: Upon a successful run of the CBMC job on the merge queue, the job pushes an artifact with the performance data. Upon merge to
origin/main, a new job picks up this artifact and pushes it to gh-pages. This separation is to prevent a successful CBMC job in an eventually failing merge queue run updating the gh-pages.