micromeasure

micromeasure is a microbenchmark harness for Rust for systems work where timing alone is not enough information.

It is aimed at very focused operations where you care about instruction count, branch predictor behaviour, cache misses, and operation latency. It now also supports GPU benchmarks via pluggable measurement backends and per-sample custom metrics.

It grew out of the needs of my mooR project, where many of the interesting questions were about tiny operations and internal data-structure mechanics. The goal was to measure things like:

• what changed in the instruction count for something as small as 1 + 1 on a custom value type?
• did a small change in an internal data structure alter branch predictor behaviour?
• did cache misses move for a tight lookup or mutation path?
• did a micro-operation get noisier even if mean elapsed time barely moved?

That means:

• direct Linux perf counter (PMU) integration as a first-class feature
• simple hand-written microbench drivers, not macro-heavy harness structure
• output that emphasizes instruction count, branch behaviour, cache misses, and timing together
• benchmark binaries that can be filtered and run directly during systems work
• persisted raw samples so you can compare a current run against the last compatible run immediately

If micromeasure is useful in your work, consider sponsoring development on GitHub Sponsors. I am also available for consulting in systems engineering, profiling and performance tuning, and Rust development (10 years at Google, 25+ years in software development). If this project is useful or interesting for your team, feel free to reach out.

... but why not Criterion?

Criterion is a strong general-purpose Rust benchmarking library. It has the bulk of ecosystem mindshare, better polished statistical analysis, and a mature presentation story. This crate is narrower.

Use this crate when:

• you are tuning very small operations and want PMU-derived metrics beside latency/throughput
• you want to inspect instruction count, branch misses, cache misses, and timing in one report
• you are working on internal value operations, cache lookups, symbol tables, allocators, or similar hot paths
• you want a small custom benchmark binary that you control directly
• you want immediate "last run vs this run" output from persisted sample data

Use Criterion when:

• you want a polished general-purpose benchmark framework
• you want richer out-of-the-box statistical analysis and reporting
• you want HTML reports and the broader Criterion workflow
• PMU metrics are not the main reason you are benchmarking

One concrete difference is Linux perf integration. Criterion perf integrations are generally measurement plugins, which means a given run tends to be centred on one selected perf event. The use case here is different: collect timing, throughput, and multiple PMU-derived metrics together in one run so you can see whether a tiny operation changed latency, instruction count, branch misses, and cache-miss behaviour at the same time.

It's possible I missed the right knobs to make criterion do what I need, but this has suited me well-ish, so far, so I am sharing it.

Current focus

micromeasure currently emphasizes:

• timing and throughput
• explicit throughput units per measured operation, so reports can say lines/s, bytes/s, rows/s, not just generic ops
• median, p95, MAD, coefficient of variation, and outlier counts
• coordinated concurrent microbenchmarks with the same sample/report pipeline
• Linux perf counters when available
• graceful fallback to timing-only runs when PMU access is unavailable
• persisted benchmark reports with per-sample throughput and latency series
• side-by-side comparison against the latest compatible saved report
• GPU benchmarking with pluggable measurement backends (CUDA event timing, custom metrics)
• per-sample custom metrics (e.g. cuda_event_ms, tflops, host_overhead_ms) with aggregation and JSON persistence
• measurement domain tagging (Cpu, Gpu, Mixed) that suppresses or relabels CPU-PMU diagnostics for GPU work

Wiring It Up

Add micromeasure as a dev-dependency:

[dev-dependencies]
micromeasure = "0.7"

Then add a custom bench target in your Cargo.toml:

[[bench]]
name = "basic"
harness = false

That bench target usually lives at benches/basic.rs.

For the bench entrypoint itself, prefer the shared launcher instead of hand-rolling argument parsing, report printing, and result persistence in every benchmark binary.

Example

For a standalone example in this repository, run:

cargo run --example basic --release

For a concurrent workload example, run:

cargo run --example concurrent_scenario --release

For a concurrent workload example with bench-defined event counters, run:

cargo run --example concurrent_counters --release

For an example that reports domain throughput like lines/s, run:

cargo run --example throughput_units --release

For an example that combines fluent factory(...) and throughput configuration, run:

cargo run --example factory_builder --release

For GPU benchmarking examples, run:

cargo run --example gpu_domain --release        # measurement domain: suppress CPU-PMU diagnostics
cargo run --example custom_metrics --release     # per-sample custom metrics with bench_sample()
cargo run --example custom_backend --release     # pluggable MeasurementBackend (simulated CUDA events)

In a consuming crate, you would usually run your benchmark with:

cargo bench --bench basic

Example output:

use micromeasure::{NoContext, Throughput, benchmark_main, black_box};

fn add_bench(_ctx: &mut NoContext, chunk_size: usize, _chunk_num: usize) {
    let mut acc = black_box(0_u64);
    let limit = black_box(chunk_size as u64);
    for i in 0..limit {
        acc = acc.wrapping_add(black_box(i));
    }
    black_box(acc);
}

benchmark_main!(|runner| {
    runner.group::<NoContext>("Arithmetic", |g| {
        g.throughput(Throughput::per_operation(8, "bytes"))
            .bench("add_loop", add_bench);
    });
});

If one measured operation represents something other than a single logical op, declare it on the group or benchmark. For example, if each measured operation compiles 1000 lines of code, use g.throughput(Throughput::per_operation(1000, "lines")) and the report will render throughput as lines/s.

Group configuration is fluent. That means you can apply shared benchmark setup once and then run multiple benches beneath it, for example:

runner.group::<MyContext>("Parser", |g| {
    g.throughput(Throughput::per_operation(4096, "bytes"))
        .factory(&|| MyContext::prepare_input())
        .bench("parse_config", parse_config);
});

Concurrent groups can also set sample duration fluently:

runner.concurrent_group::<SharedState>("Contention", |g| {
    g.sample_duration(Duration::from_millis(50))
        .throughput(Throughput::per_operation(1000, "lines"))
        .bench("compile_under_lock", &workers);
});

benchmark_main! handles:

• parsing an optional benchmark filter from the command line
• constructing BenchmarkRunner with that filter
• printing the session summary
• saving the report to the default location

If you want a custom suite name, custom filter help text, or custom runtime options, use run_benchmark_main(BenchmarkMainOptions { ... }, |runner| { ... }) instead.

Runtime Configuration

You can configure the benchmark runtime behavior (warm-up duration, target benchmark duration, and sample counts) either on the BenchmarkRunner or via BenchmarkMainOptions.

benchmark_main!(|runner| {
    let runtime = BenchmarkRuntimeOptions {
        warm_up_duration: Duration::from_millis(500),
        benchmark_duration: Duration::from_secs(2),
        ..BenchmarkRuntimeOptions::default()
    };

    runner
        .set_runtime(runtime)
        .group::<NoContext>("Arithmetic", |g| {
            g.bench("add_loop", add_bench);
        });
});

Default values are:

• warm_up_duration: 1 second
• benchmark_duration: 5 seconds
• min_samples: 20
• max_samples: 100

Concurrent Benchmarks

micromeasure can also benchmark coordinated concurrent workloads while still using the same sample-driven measurement pipeline as the single-threaded path.

That means concurrent benchmarks still get:

• the usual sample count and calibration flow
• the usual timing statistics
• Linux PMU counters when available
• persisted BenchmarkResult data and normal session summaries

The difference is the shape of one sample: instead of one function running on one thread, a sample runs multiple worker roles against shared state for a fixed sample window.

Use this when the thing you care about only shows up under contention, for example:

• cache misses caused by reader/writer interference
• branch miss behaviour in optimistic retry loops
• lock or latch implementations under mixed access patterns

The concurrent API is centered on:

• ConcurrentBenchContext
• ConcurrentWorker
• ConcurrentBenchControl
• ConcurrentWorkerResult
• BenchmarkRunner::concurrent_group(...)

See examples/concurrent_scenario.rs for a complete reader/writer contention benchmark using ConcurrentBenchContext and BenchmarkRunner::concurrent_group(...).

If a concurrent benchmark needs to report scenario-specific events such as retries, failed try-locks, dropped work, or backoffs, workers can return ConcurrentWorkerResult instead of just an operation count. These event counters are intended to be:

• worker-local plain integers in the hot loop
• packaged once at the end of the sample
• aggregated by worker role after join

That keeps event reporting out of the measured hot path. The framework reports them under each worker role as bench event counters, including total count, per-operation rate, and per-second rate.

See examples/concurrent_counters.rs for a complete concurrent benchmark that reports bench-defined event counters.

In concurrent output, worker-role tables are the primary view. Each worker role gets the same stats table shape as the normal benchmark path, including throughput, latency, and PMU-derived metrics like instructions/op, branch misses, and cache misses.

The workers combined section at the bottom is a whole-scenario aggregate. It is mainly useful as the PMU view of the entire interacting workload; the worker-role tables are usually the more meaningful place to interpret throughput and latency.

GPU Benchmarks

micromeasure can benchmark GPU work (e.g. cuBLASLt GEMM, CUDA streams) alongside CPU microbenchmarks. Three features work together to make GPU output less misleading:

MeasurementDomain (g.measurement_domain(MeasurementDomain::Gpu)):

• Gpu: suppresses CPU-PMU bottleneck diagnostics (the host thread's counters describe launch/sync orchestration, not the GPU kernel) and relabels the PMU coverage line as host PMU (orchestration).
• Mixed: emits CPU-PMU diagnostics with a [host] prefix.
• Cpu: unchanged historic behaviour (the default).

MeasurementBackend (g.backend(|| Box::new(MyCudaBackend::new()))):

• Pluggable measurement window around the bench closure. The runner calls begin() / end() / collect() per sample.
• Default on Linux is LinuxPerfBackend (perf-event group + individual-counter fallback). On other platforms it falls back to WallClockBackend (timing-only).
• A CUDA event adapter (implemented in consuming code) records cudaEventRecord in begin() / end(), computes elapsed in collect(), and pushes cuda_event_ms and host_overhead_ms as custom metrics. No CUDA dependency in micromeasure itself.
• The backend's measurement_label() (e.g. "timing + CUDA events") is shown in the Measurement row.

Per-sample custom metrics (g.bench_sample(name, f)):

• The bench function returns a BenchSampleResult { operations, metrics } instead of ().
• MetricValue carries name, value, unit, an optional section, an optional display_name, and a format hint (Number or Integer for IDs/counts).
• The runner aggregates per (section, name, unit) across samples into MetricSummary (mean, median, p95, min, max, contributing sample count) and renders a custom metrics: table.
• Bench-function metrics and backend-pushed metrics are merged into one table.
• JSON persistence works through the existing BenchmarkStats serde derive.

Diagnostic replay metrics (g.diagnostic_pass(f)):

• Runs once after the normal timing samples, using the calibrated chunk size.
• Metrics returned by the diagnostic pass are merged into the same custom metrics table.
• DiagnosticResult::new(section) applies section as the default for metrics that do not set their own section.
• The diagnostic pass does not contribute to latency, throughput, CV, or outlier statistics, so it can collect invasive counters without contaminating normal timing.
• Use g.diagnostic_samples(n) to repeat noisy diagnostic counters.
• The diagnostic pass is fallible; failures are reported as diagnostic metrics rather than timing failures.

fn my_gpu_bench(ctx: &mut GpuContext, chunk_size: usize, _chunk_num: usize) -> BenchSampleResult {
    let device_s = ctx.run_kernel(chunk_size);
    BenchSampleResult::operations(chunk_size as u64)
        .push_metric(
            MetricValue::duration_ms("cuda_event_ms", Duration::from_secs_f64(device_s))
                .with_display_name("CUDA event time"),
        )
        .push_metric(
            MetricValue::throughput_tflops("tflops", 12345, device_s)
                .with_display_name("TFLOP/s"),
        )
}

fn my_gpu_counter_replay(
    ctx: &mut GpuContext,
    chunk_size: usize,
    _chunk_num: usize,
) -> Result<DiagnosticResult, DiagnosticError> {
    ctx.run_kernel_under_profiler(chunk_size)
}

benchmark_main!(|runner| {
    runner.group::<GpuContext>("cuBLASLt FP4", |g| {
        g.throughput(Throughput::bytes(8))
            .measurement_domain(MeasurementDomain::Gpu)
            .backend(|| Box::new(CudaEventBackend::new()))
            .diagnostic_samples(3)
            .diagnostic_pass(my_gpu_counter_replay)
            .bench_sample("fp4_gemm", my_gpu_bench);
    });
});

See examples/gpu_domain.rs, examples/custom_metrics.rs, and examples/custom_backend.rs for complete runnable examples.

Linux-first, and why

This crate is strongly Linux-specific, as its main differentiator is direct integration with Linux perf events and PMU counters. The timing side of the crate is portable enough, but the most important measurements here are things like:

• instructions retired
• branch instructions
• branch misses
• cache misses

If those counters are not available, the crate still runs and still reports timing data, but you are only getting part of what it is designed for.

If your primary goal is portable benchmarking across platforms, Criterion is usually the better fit.

It is probably feasible to add similar support for other platforms over time, including Darwin/macOS, but that work has not been done here. I do not have a Mac to develop and validate that path myself, so the crate is currently designed and tested with Linux as the primary target.

Enabling perf counters on Linux

On many Linux systems, unprivileged access to perf events is restricted by kernel.perf_event_paranoid.

The common cases are:

• -1 or 0: broad access
• 1 or 2: common developer-friendly settings
• 3 or 4: often too restrictive for useful PMU access in normal user sessions

You can inspect the current setting with:

cat /proc/sys/kernel/perf_event_paranoid

To lower it temporarily until reboot:

sudo sysctl kernel.perf_event_paranoid=2

To make it persistent:

echo 'kernel.perf_event_paranoid=2' | sudo tee /etc/sysctl.d/99-micromeasure.conf
sudo sysctl --system

Depending on your environment, you may also need one of:

• CAP_PERFMON
• CAP_SYS_ADMIN
• a container/runtime configuration that allows perf_event_open

This matters in containers, CI environments, and some locked-down distributions where the kernel setting alone is not enough.

When PMU access is unavailable, the crate will fall back to timing-only measurement and tell you that it has done so.

What this crate is not

• not a replacement for Criterion in the general case
• not intended for large end-to-end benchmark suites
• not trying to hide measurement mechanics behind a lot of framework structure
• not a cross-platform PMU abstraction layer

Origin

This crate started inside mooR, a multithreaded MOO server and transactional object database/runtime. Its benchmark harness grew out of performance work on tiny VM & DB operations, value manipulation, caches, symbol handling, string processing, and other internal systems paths where the interesting behaviour was often below the level of a conventional application benchmark.

That origin explains the design bias:

• systems-level microbenchmarks
• direct execution from custom bench binaries
• PMU-aware analysis
• immediate regression visibility while iterating on low-level code

License

micromeasure is licensed under the Apache License, Version 2.0. See LICENSE.

Unless explicitly stated otherwise, any contribution intentionally submitted for inclusion in this project is contributed under the same license.

Contributing

Contributions are welcome, especially around:

• better statistical analysis and comparison reporting
• improved presentation and terminal output
• additional platform backends for non-Linux systems

If you find defects, I am very interested in hearing about them.

I am not a professional statistician. It's possible my code is lying to you. If so, please tell me.