Repository navigation
CBMC: Push performance data to gh-pages - #1961
Merged
Merged
Conversation
CI posts a CBMC runtime report for each pull request. report.py compares every proof configuration, identified by its proof name, data model, and solver, with the baseline stored on gh-pages. Older baselines contain only one entry per proof and record the default configuration. classify_proof() used that name-only entry whenever it could not find an exact configuration match. A secondary solver was therefore compared with the default solver runtime and could be reported as a false regression. Use name-only entries only for results marked as default. Report other configurations without an exact baseline as new. Ports pq-code-package/mldsa-native#1494. Signed-off-by: Hanno Becker <beckphan@amazon.co.uk>
Pull-request reports compare CBMC runtimes with baselines on gh-pages
(dev/cbmc/mlkem-{512,768,1024}.json). report.py previously wrote them
from CI runs on main. Since 116a592 ("CI: Do not run on pushes to
main"), CBMC runs only for pull requests and merge groups, so the
baselines have not been updated since 2026-08-03 and still contain one
result per proof.
The merge queue tests the exact commit that main then receives, so its
results are the right baseline. Each CBMC job now maps k=2, 3, or 4 to
the corresponding ML-KEM name, writes the result at its gh-pages path,
and uploads that tree from merge-group runs.
Run cbmc_publish.yml on each push to main. It finds the all.yml merge
queue run for that commit, downloads all CBMC artifacts into a gh-pages
checkout, and pushes them as one commit. Rebase and retry once if the
benchmark workflow changed gh-pages first; warn and skip commits that
did not pass through the merge queue.
Remove the Contents API upload from report.py. Configured result files
no longer need to be base64-encoded into a single gh command argument.
Ports pq-code-package/mldsa-native#1494.
Signed-off-by: Hanno Becker <beckphan@amazon.co.uk>
Contributor
CBMC Results (ML-KEM-768)
Full Results (379 configurations)
|
Contributor
CBMC Results (ML-KEM-1024)
Full Results (379 configurations)
|
Contributor
CBMC Results (ML-KEM-512)
Full Results (379 configurations)
|
mkannwischer
approved these changes
Oct 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Port of pq-code-package/mldsa-native#1494