Repository navigation
refactor(benchmarks): measure fixed workloads in milliseconds - #323
Merged
Merged
Conversation
Member
Author
Fixed-workload baselineBaseline for commit 70c2117, measured on 2026-11-04. This is a fresh full run of the committed benchmark definitions: 207 scenarios, 279 scenario/implementation combinations, three repetitions each. The results below retain all three measurements, including the first run and outliers. Environment: Apple M4 Max, 36 GiB RAM, macOS 26.6.2 (arm64); Reproduce at the same revision with: cargo x bench -- --repeat 3 --rawThe round counts are the committed defaults, with no automatic calibration or round override. Each round executes the complete workload closure; input sizes and inner batches are unchanged. The clock includes fixture construction, per-round input cloning where needed, result destruction, and cleanup. XOR peers share rounds and rotate execution order; matched-build cases check each constructed filter's expected seed directly. 200 of 207 reference medians fall between 100 and 999 ms. Shorter results are retained as measured; equivalent faster strategies are not resized independently. Different scenario rows can have different work sizes, so the ranges below are a sizing overview, not a throughput ranking. This establishes a baseline for future revisions, not a before/after speedup claim or a timing threshold for CI. Old Divan numbers use different timing boundaries and should not be compared directly.
All elapsed times below are integer milliseconds. bloom: 3 scenarios, 3 implementation rows
countmin: 7 scenarios, 7 implementation rows
cpc: 6 scenarios, 6 implementation rows
frequencies: 2 scenarios, 2 implementation rows
hll: 22 scenarios, 22 implementation rows
kll: 12 scenarios, 12 implementation rows
req: 8 scenarios, 8 implementation rows
tdigest: 89 scenarios, 89 implementation rows
theta: 2 scenarios, 2 implementation rows
tuple: 2 scenarios, 2 implementation rows
xor: 54 scenarios, 126 implementation rows
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replace Divan's per-operation measurements with fixed workloads sized around hundreds of milliseconds, following Benchmark In Milliseconds. Each revision runs the same amount of work, so improvements reduce elapsed time instead of changing the workload size.
The runner measures fixture construction through cleanup and reports integer-millisecond medians, full ranges, and optional raw samples. XOR comparisons share round counts and rotate execution order across repetitions. Matched-build cases validate the resulting seeds directly, preserving equal construction attempt counts without building another implementation inside the measurement. All 279 existing operation, input, and implementation combinations are retained across 207 scenarios.
Add filtering, exact selection, repetition and round overrides, listing, compile-only, and smoke-test support through
cargo x bench. Document timing boundaries and baseline maintenance, and run smoke tests in CI without timing assertions. These measurements establish a new baseline because setup and destruction are now included.Validation:
cargo x checkcargo x testcargo x lintin a clean source checkoutcargo x bench -- --test: all 279 combinations passcargo +1.86.0 check --package benchmarks --all-targets --locked