refactor(benchmarks): measure fixed workloads in milliseconds

.github/workflows/ci.yml

@@ -54,6 +54,8 @@ jobs:
         run: cargo x lint
       - name: Check feature matrix
         run: cargo x check
+      - name: Exercise benchmark workloads
+        run: cargo x bench -- --test
 
   msrv:
     name: Resolve MSRV

CONTRIBUTING.md

@@ -67,14 +67,42 @@ Benchmark:
 cargo x bench
 ```
 
-Benchmarks live in the standalone `benchmarks` crate. Files are grouped first by sketch and then by workload, for example `benchmarks/cpc/serde.rs` and `benchmarks/tdigest/update.rs`. Keep distinct workloads in separate modules so Divan reports stable names such as `cpc::serde::serialize`.
+Benchmarks live in the standalone `benchmarks` crate. Files are grouped first by sketch and then by workload, for example `benchmarks/cpc/serde.rs` and `benchmarks/tdigest/update.rs`. Each workload module registers its scenarios and fixed round counts with the [runner](benchmarks/harness.rs). Keep distinct workloads in separate modules and use names such as `cpc::serde::serialize/200` that identify the sketch, operation, and input.
 
-To run only one sketch or workload, pass a Divan filter directly:
+Compile, list, or smoke-test the benchmarks:
 
 ```shell
-cargo bench --package benchmarks --bench benchmarks -- cpc::serde
+cargo x bench --no-run
+cargo x bench -- --list
+cargo x bench -- --test
 ```
 
+Smoke mode executes every selected implementation for one outer round, preserving input sizes, inner loops, and correctness checks. CI runs this mode without asserting elapsed times. The runner's tests are included in `cargo x test`.
+
+To select workloads and inspect repeated measurements:
+
+```shell
+cargo x bench -- cpc::serde --repeat 10 --raw
+cargo x bench -- xor::matched_build::8 --repeat 5
+cargo x bench -- 'kll::query::quantile/0.5' --exact --rounds 100000
+```
+
+Filters are substrings; a scenario must match all supplied filters. Use `--exact` with one complete name to select only that scenario. A filter that matches nothing fails. Runner options can also be passed directly with `cargo bench --package benchmarks --bench benchmarks -- ...`.
+
+### Fixed workloads and timing
+
+[Benchmark In Milliseconds](https://matklad.github.io/2026/10/05/benchmark-milliseconds.html) recommends sizing useful work to take roughly 300 ms. Choose that work size on a reference revision, then preserve it across comparisons. The runner never calibrates rounds automatically: an optimization should reduce elapsed time for the same work. The initial counts were sized against the library at `5f080f9` using the timing boundaries below. Hardware and toolchains affect duration, so 300 ms is a sizing guideline, not a performance assertion. The eight-million-key XOR cases retain one full build per round.
+
+Each repetition starts the clock before fixture construction and stops after cleanup. A fixture is built once per repetition and may be reused across its fixed rounds. Updates rebuild the sketch each round; read-only queries reuse a prepared sketch; destructive operations clone their inputs each round. Input generation, cloning, allocations, output destruction, and fixture validation all count toward elapsed time. No allocation profiler is installed. These measurements establish a new baseline and are not directly comparable to the old Divan timings that excluded setup or destruction.
+
+A round is one invocation of the workload closure, not necessarily one item. For example, hash-based updates process 10,000 inputs, parameterized updates process the named input size, XOR queries visit every key in the selected present or absent set, and aggregation rounds process the entire specified group of partial states. Keep inner batch sizes and sketch parameters stable when adjusting outer rounds; changing cardinality can change the algorithmic path.
+
+The default is five repetitions. Output reports the median and full minimum-to-maximum range in integer milliseconds. `--raw` includes every elapsed time in measurement order, including the first run and outliers. XOR tables interleave implementations and rotate the first implementation across repetitions, using identical rounds and input parameters. `vs ref` divides each median by the first implementation's median before rounding; `0.80x` means 20% less elapsed time for that workload. `--baseline-only` runs just the first implementation.
+
+XOR build tables compare DataSketches' distinct-hash and deduplicating entry points with xorf on distinct inputs. The `matched_build` scenarios additionally align the seed sequence and check each resulting seed against the expected first or second attempt, including a case requiring a second attempt. Query tables use the same key sets and fingerprint widths. These comparisons include each implementation's fixture construction and cleanup. Different scenario rows may have different round counts and are not a throughput ranking; use identical rounds when comparing equivalent work across rows.
+
+To add or resize a scenario, select a reference revision and run it with an explicit `--rounds` value and several repetitions. Choose a convenient fixed count that puts the reference in the hundreds of milliseconds, then record it in the workload registration. Give all implementations of that scenario the same count. Keep it fixed for later revisions and faster peers. When inputs, rounds, or timing boundaries change, rerun both sides under the new definition. Inspect the measured range before drawing conclusions; changing rounds alone does not eliminate noise.
+
 ## Public API documentation
 
 - Describe types with noun phrases and API behavior with third-person present-tense verbs such as `Creates`, `Updates`, and `Returns`.

Cargo.toml

@@ -33,7 +33,6 @@ datasketches = { path = "datasketches" }
 
 # Crates.io dependencies
 clap = { version = "4.6.5", features = ["derive"] }
-divan = { version = "0.1.21" }
 googletest = { version = "0.14.3" }
 insta = { version = "1.48.0" }
 cargo_metadata = { version = "0.23.1" }

benchmarks/Cargo.toml

@@ -22,6 +22,9 @@ publish = false
 edition.workspace = true
 rust-version.workspace = true
 
+[dependencies]
+clap = { workspace = true }
+
 [dev-dependencies]
 xorf = { version = "=0.13.0", default-features = false }
 datasketches = { workspace = true, features = [
@@ -37,9 +40,12 @@ datasketches = { workspace = true, features = [
   "tuple",
   "xor",
 ] }
-divan = { workspace = true }
 rand = { workspace = true }
 
+[lib]
+path = "harness.rs"
+bench = false
+
 [[bench]]
 harness = false
 name = "benchmarks"

benchmarks/bloom/mod.rs

@@ -16,3 +16,7 @@
 // under the License.
 
 mod update;
+
+pub fn register(runner: &mut benchmarks::Runner) {
+    update::register(runner);
+}

benchmarks/bloom/update.rs

@@ -15,18 +15,18 @@
 // specific language governing permissions and limitations
 // under the License.
 
+use std::hint::black_box;
+
+use benchmarks::Bencher;
+use benchmarks::Runner;
 use datasketches::bloom::BloomFilterBuilder;
 use datasketches::hash::value::raw_bytes;
-use divan::Bencher;
-use divan::black_box;
-use divan::counter::ItemsCount;
 
 use crate::hash_inputs::ITEMS;
 use crate::hash_inputs::bytes_32_values;
 
-#[divan::bench]
 fn u64(bencher: Bencher) {
-    bencher.counter(ItemsCount::new(ITEMS)).bench_local(|| {
+    bencher.bench_local(|| {
         let mut filter = BloomFilterBuilder::with_accuracy(ITEMS as u64, 0.01)
             .build()
             .unwrap();
@@ -37,11 +37,10 @@ fn u64(bencher: Bencher) {
     });
 }
 
-#[divan::bench]
 fn bytes_32(bencher: Bencher) {
     let values = bytes_32_values();
 
-    bencher.counter(ItemsCount::new(ITEMS)).bench_local(|| {
+    bencher.bench_local(|| {
         let mut filter = BloomFilterBuilder::with_accuracy(ITEMS as u64, 0.01)
             .build()
             .unwrap();
@@ -52,7 +51,6 @@ fn bytes_32(bencher: Bencher) {
     });
 }
 
-#[divan::bench]
 fn contains_and_insert_present_u64(bencher: Bencher) {
     let mut populated = BloomFilterBuilder::with_accuracy(ITEMS as u64, 0.01)
         .build()
@@ -61,13 +59,21 @@ fn contains_and_insert_present_u64(bencher: Bencher) {
         populated.insert(value);
     }
 
-    bencher
-        .counter(ItemsCount::new(ITEMS))
-        .with_inputs(|| populated.clone())
-        .bench_local_values(|mut filter| {
-            for value in 0..ITEMS as u64 {
-                black_box(filter.contains_and_insert(black_box(&value)));
-            }
-            filter
-        });
+    bencher.bench_local(|| {
+        let mut filter = populated.clone();
+        for value in 0..ITEMS as u64 {
+            black_box(filter.contains_and_insert(black_box(&value)));
+        }
+        filter
+    });
+}
+
+pub fn register(runner: &mut Runner) {
+    runner.case("bloom::update::u64", 2_000, u64);
+    runner.case("bloom::update::bytes_32", 1_000, bytes_32);
+    runner.case(
+        "bloom::update::contains_and_insert_present_u64",
+        2_000,
+        contains_and_insert_present_u64,
+    );
 }

benchmarks/countmin/decay.rs

@@ -15,18 +15,28 @@
 // specific language governing permissions and limitations
 // under the License.
 
+use std::hint::black_box;
+
+use benchmarks::Bencher;
+use benchmarks::Runner;
 use datasketches::countmin::CountMinSketch;
-use divan::Bencher;
-use divan::black_box;
 
-#[divan::bench(args = [0.5, 0.99, 1.0])]
 fn u64(bencher: Bencher, factor: f64) {
     let mut sketch = CountMinSketch::<u64>::new(4, 16_384).unwrap();
     for item in 0..128_u64 {
         sketch.update_with_weight(item, (1 << 53) + item);
     }
 
-    bencher
-        .with_inputs(|| sketch.clone())
-        .bench_local_refs(|sketch| black_box(sketch).decay(black_box(factor)));
+    bencher.bench_local(|| {
+        let mut sketch = sketch.clone();
+        black_box(&mut sketch).decay(black_box(factor))
+    });
+}
+
+pub fn register(runner: &mut Runner) {
+    for &(factor, rounds) in &[(0.5, 20_000), (0.99, 10_000), (1.0, 50_000)] {
+        runner.case(format!("countmin::decay::u64/{factor:?}"), rounds, |b| {
+            u64(b, factor)
+        });
+    }
 }

benchmarks/countmin/estimate.rs

@@ -15,23 +15,23 @@
 // specific language governing permissions and limitations
 // under the License.
 
+use std::hint::black_box;
+
+use benchmarks::Bencher;
+use benchmarks::Runner;
 use datasketches::countmin::CountMinSketch;
 use datasketches::hash::value::raw_bytes;
-use divan::Bencher;
-use divan::black_box;
-use divan::counter::ItemsCount;
 
 use crate::hash_inputs::ITEMS;
 use crate::hash_inputs::bytes_32_values;
 
-#[divan::bench]
 fn u64(bencher: Bencher) {
     let mut sketch = CountMinSketch::<u64>::new(4, 16_384).unwrap();
     for value in 0..ITEMS as u64 {
         sketch.update(value);
     }
 
-    bencher.counter(ItemsCount::new(ITEMS)).bench_local(|| {
+    bencher.bench_local(|| {
         let mut total = 0u64;
         for value in 0..ITEMS as u64 {
             total = total.wrapping_add(sketch.estimate(black_box(value)));
@@ -40,19 +40,23 @@ fn u64(bencher: Bencher) {
     });
 }
 
-#[divan::bench]
 fn bytes_32(bencher: Bencher) {
     let values = bytes_32_values();
     let mut sketch = CountMinSketch::<u64>::new(4, 16_384).unwrap();
     for value in &values {
         sketch.update(raw_bytes::from_slice(value));
     }
 
-    bencher.counter(ItemsCount::new(ITEMS)).bench_local(|| {
+    bencher.bench_local(|| {
         let mut total = 0u64;
         for value in &values {
             total = total.wrapping_add(sketch.estimate(raw_bytes::from_slice(black_box(value))));
         }
         black_box(total)
     });
 }
+
+pub fn register(runner: &mut Runner) {
+    runner.case("countmin::estimate::u64", 2_000, u64);
+    runner.case("countmin::estimate::bytes_32", 1_000, bytes_32);
+}

benchmarks/countmin/mod.rs

@@ -18,3 +18,9 @@
 mod decay;
 mod estimate;
 mod update;
+
+pub fn register(runner: &mut benchmarks::Runner) {
+    decay::register(runner);
+    estimate::register(runner);
+    update::register(runner);
+}