Skip to content

README.md

Latest commit

 

History

History
466 lines (346 loc) · 15.3 KB

README.md

File metadata and controls

466 lines (346 loc) · 15.3 KB
Raw

Benchmark Suite

Single source of truth for benchmark operation (human + agent).

Scope

The benchmark workflow is intentionally reduced to three run types:

  • MICRO (Google Benchmark micro suite)
  • MACRO-SMALL (macro suite at 10k rows)
  • MACRO-LARGE (macro suite at 500k rows)

Legacy compare/leaderboard/report-latest flows are removed. Reporting and comparison are handled by benchmark/report.py.

Current macro dataset catalog contains 15 profiles, including string-heavy reference workloads (event_log, iot_fleet, financial_orders) and the 84-column measurement_campaign profile for Static vs Flexible parity investigation.

Macro runs execute mode families across storage (flexible|static) and codec (dense|zoh|delta) depending on CLI filters.

Build

cmake --preset ninja-release
cmake --build --preset ninja-release-build -j$(nproc) --target bench_macro_datasets bench_micro_types

Direct Macro CLI Quick Start

Use the macro executable directly when you want fast local iteration on profile/scenario selection.

# Show CLI overview, options, and examples
build/ninja-release/bin/bench_macro_datasets --help

# Run one profile with selected scenarios
build/ninja-release/bin/bench_macro_datasets \
  --profile=rtl_waveform \
  --rows=10000 \
  --scenario=baseline,sparse_columns_k1

# Force narrow-terminal summary output
build/ninja-release/bin/bench_macro_datasets \
  --size=S \
  --summary=compact \
  --output=macro_small_results.json

Summary modes:

  • --summary=full (default): full metrics table
  • --summary=compact: terminal-friendly compact table (fits narrow terminals)

Discovery helpers:

  • --list prints all profile names
  • --list-scenarios prints all scenario IDs

Timing modes:

  • Default: timed read loops include per-row validation (regenerate expected row + compare). Good for regression A/B within this suite — but the numbers understate pure decode speed several-fold.
  • --no-validate: pure decode throughput (no expected-row work in the timed loop). Use for absolute claims and cross-format comparisons.
  • Note: ZoH/Delta modes use generateTimeSeries data while CSV/Dense use the volatile generate data — the compression-vs-CSV column therefore compares against a CSV of a different dataset (footnoted in generated reports).

Measurement Methodology & Noise Floor

Quantified on the reference machine (Ryzen 9 9950X3D) — full study in docs/archive/NOISE_FLOOR_2026-07-12.md:

Regime Read noise (median CV) Write noise Use for
8-way parallel sweep 5.5–8.5 % (p90 to 20 %) 7–9 % (p90 to 23 %) coverage smoke, regressions > 15–20 %
Solo, pinned, interleaved A/B ~1–2 % ~5–7 % (binary-layout luck) verdicts in the ±10 % band

Rules of thumb:

  1. Judge on per-mode medians across profiles × scenarios, never on single metrics (single-metric deltas below the p90 CV are noise).
  2. Reads are the reliable metric; writes carry ±5–7 % binary-layout noise even between semantically identical builds.
  3. Prefer M/L sizes for gating (S is 1.5–2× noisier); run ≥ 3 repetitions and use medians (warm-up median penalty is ≤ 3 %, but ~10 % of metrics show a 10–23 % slower first repetition).
  4. Parallel-sweep numbers are 7–14 % below solo numbers — never mix regimes without same-regime anchors.
  5. Micro suite: gate only on aggregate ops (≥ ~10 ns: Visit, Serialize, CsvWriteRow); the sub-ns Get_*/Set_* accessors swing ±30 % run-to-run.

Codec Parity Investigation (bench_codec_compare)

Standalone tool for cross-product comparison of all file codecs × row codecs × layout types (Flexible vs Static). Uses interleaved iterations to neutralize thermal throttling. Not driven by run.py — intended for targeted investigation.

# Build
cmake --build --preset ninja-release-build --target bench_codec_compare -j$(nproc)

# Full parity run (84-column measurement_campaign profile)
./build/ninja-release/bin/bench_codec_compare \
    --rows=50000 --iterations=5 --profile=measurement_campaign \
    --storage=both --json=tmp/parity_investigation.json

# Flexible-only (skip Static candidates)
./build/ninja-release/bin/bench_codec_compare \
    --rows=10000 --iterations=3 --profile=mixed_generic \
    --storage=flexible

# Static-only
./build/ninja-release/bin/bench_codec_compare \
    --rows=10000 --iterations=3 --profile=measurement_campaign \
    --storage=static

Options:

Flag Default Description
--rows=N 10000 Rows per write/read cycle
--iterations=N 5 Interleaved iterations per candidate
--profile=NAME|all all Single profile or all profiles
--storage=both|flexible|static both Layout filter
--json=PATH (none) Write JSON results to file

Static candidates appear with (S) suffix in output and are auto-skipped for profiles without a compile-time LayoutStatic definition.

When JSON output is produced via the orchestrated report pipeline, report.py generates a Static Parity table pairing Flexible/Static results per codec and flagging anomalies where Static is slower than Flexible.

Streamlined CLI

All benchmark workflows are accessible through a single entry point:

python3 benchmark/run.py <subcommand> [options]

Subcommands

Command Purpose
wip Run benchmarks on the current working-tree build (default workflow)
baseline Clone a git ref, build, and benchmark into a clean baseline directory
compare Generate a markdown comparison report between two run directories
interleaved Run head-to-head interleaved pairs and produce a comparison

wip flags

  • --type=MICRO,MACRO-SMALL,MACRO-LARGE (comma-separated, default MACRO-SMALL)
  • --repetitions=<N> (default 1)
  • --pin=NONE|CPU<id> (default NONE, example CPU2)
  • --git=<label> (default WIP, controls results bucket naming)
  • --results=<path> (default benchmark/results/<hostname>/<git>)
  • --no-build / --no-report
  • --languages=python,csharp (optional language lanes)
  • --macro-profile=<name> (pass-through to bench_macro_datasets --profile)
  • --macro-scenario=<csv> (pass-through to bench_macro_datasets --scenario)
  • --macro-storage=both|flexible|static (default both)
  • --macro-codec=both|dense|zoh|delta|primary (default both)
    • primary = dense + zoh + delta (skip CSV-only)
  • --detail (show per-profile breakdown table in operator summary)

Operator Summary Output

The terminal operator summary includes:

  • Wall-clock timing: total benchmark execution time per run type
  • Profile coverage: N/M column shows how many profiles contributed to each mode's median (Static modes only run on profiles with no-copy generators: mixed_generic, sparse_events)
  • Noise flags: a ~ marker appears when stdev exceeds the median (CV > 1.0), indicating high across-profile variance — consider re-running with --pin or more --repetitions
  • Per-profile breakdown (--detail): write rows/s for every profile × mode combination

Standard Runs

# Default developer run
python3 benchmark/run.py wip

# Micro benchmark pinned to CPU2 (target <5 min)
python3 benchmark/run.py wip --type=MICRO --pin=CPU2

# Full campaign (all three types)
python3 benchmark/run.py wip --type=MICRO,MACRO-SMALL,MACRO-LARGE

# Isolated macro family: flexible + dense only
python3 benchmark/run.py wip \
  --type=MACRO-SMALL \
  --macro-storage=flexible \
  --macro-codec=dense

# Isolated macro family: static + ZoH only
python3 benchmark/run.py wip \
  --type=MACRO-SMALL \
  --macro-storage=static \
  --macro-codec=zoh

# Delta codec only (flexible + static)
python3 benchmark/run.py wip \
  --type=MACRO-SMALL \
  --macro-codec=delta

# All BCSV codecs, skip CSV (dense + zoh + delta)
python3 benchmark/run.py wip \
  --type=MACRO-SMALL \
  --macro-codec=primary

# Include Python language lane in the same run directory/report
python3 benchmark/run.py wip --type=MACRO-SMALL --languages=python

# Clean baseline run bucket
python3 benchmark/run.py wip --type=MACRO-LARGE --git=clean

Baseline + WIP Recipe (Recommended)

Use this flow for reproducible comparison between a clean commit baseline and current workspace changes.

# 1) Purge old results
rm -rf benchmark/results && mkdir -p benchmark/results

# 2) Clean baseline from a fresh clone at the selected git ref
python3 benchmark/run.py baseline \
  --git-ref HEAD \
  --types MICRO,MACRO-SMALL,MACRO-LARGE \
  --repetitions 5 \
  --temp-root /tmp/bcsv \
  --results-root benchmark/results/$(hostname) \
  --pin NONE

# 3) Current workspace run (WIP) compared against latest clean baseline
python3 benchmark/run.py wip \
  --type=MICRO,MACRO-SMALL,MACRO-LARGE \
  --repetitions=5 \
  --pin=NONE \
  --git=WIP \
  --results=benchmark/results/$(hostname)/WIP \
  --no-build

Interleaved Pair Comparison (Operator Utility)

For alternating baseline/candidate campaigns (for example 5 interleaved pairs), generate a single pair-wise + aggregate markdown summary:

python3 benchmark/interleaved_compare.py \
  --baseline-root benchmark/results/$(hostname)/clean_interleaved \
  --candidate-root benchmark/results/$(hostname)/wip_interleaved \
  --expected-pairs 5 \
  --run-type MACRO-SMALL

True Interleaved Pair Runner (Head-to-Head)

For strict interleaving semantics (for repetition -> for benchmark type -> run A/B pair), use the unified CLI's interleaved subcommand:

python3 benchmark/run.py interleaved \
  --baseline-bin /tmp/bcsv/<git-hash>/build/ninja-release/bin \
  --candidate-bin build/ninja-release/bin \
  --baseline-label git-<git-hash>-clean \
  --candidate-label WIP \
  --types MICRO,MACRO-SMALL \
  --repetitions 5 \
  --results-root benchmark/results/$(hostname)/interleaved_h2h

Or use the individual scripts directly:

python3 benchmark/run_interleaved_pairs.py \
  --baseline-bin /tmp/bcsv/<git-hash>/build/ninja-release/bin \
  --candidate-bin build/ninja-release/bin \
  --baseline-label git-<git-hash>-clean \
  --candidate-label WIP \
  --types MICRO,MACRO-SMALL \
  --repetitions 5

Then compare the pair roots:

python3 benchmark/interleaved_compare.py \
  --baseline-root benchmark/results/$(hostname)/interleaved_h2h/git-<git-hash>-clean \
  --candidate-root benchmark/results/$(hostname)/interleaved_h2h/WIP \
  --expected-pairs 5 \
  --run-type MACRO-SMALL

Default output:

  • benchmark/results/<hostname>/interleaved_5x_comparison.md

The script also prints a concise terminal summary (median/min/max deltas per metric) so operators get immediate feedback without opening the markdown file. Scores and comparison matrices are computed only from workloads present in both baseline and candidate runs; excluded workloads are listed explicitly in the report.

Delta conventions used by comparison reports:

  • Macro write/read deltas are expressed as execution-time change (derived from throughput). Positive means WIP is slower; negative means WIP is faster.
  • Micro deltas are expressed as latency change (real_time ns). Positive means WIP is slower; negative means WIP is faster.
  • Compression/file-size deltas use (WIP - baseline) / baseline. Positive means WIP output is larger.

Supported run types for --run-type:

  • MACRO-SMALL
  • MACRO-LARGE
  • MICRO (latency deltas, using micro_results.json)

Temp clone location defaults:

  • Linux: /tmp/bcsv/<git-hash>
  • Windows: C:/temp/bcsv/<git-hash>

Runtime Targets

  • MACRO-SMALL should complete in < 3 minutes.
  • MACRO-LARGE should complete in < 60 minutes.
  • MICRO should run pinned to CPU2 and complete in < 5 minutes.

Isolated Profiling Runs (Linux, non-mixing)

Use isolated runs to avoid mixing strategy families in one result set.

# 1) Dense-only, flexible
python3 benchmark/run.py wip \
  --type=MACRO-SMALL \
  --macro-storage=flexible \
  --macro-codec=dense \
  --pin=CPU2

# 2) Dense-only, static
python3 benchmark/run.py wip \
  --type=MACRO-SMALL \
  --macro-storage=static \
  --macro-codec=dense \
  --pin=CPU2

# 3) ZoH-only, flexible
python3 benchmark/run.py wip \
  --type=MACRO-SMALL \
  --macro-storage=flexible \
  --macro-codec=zoh \
  --pin=CPU2

# 4) ZoH-only, static
python3 benchmark/run.py wip \
  --type=MACRO-SMALL \
  --macro-storage=static \
  --macro-codec=zoh \
  --pin=CPU2

For hotspot attribution, run profiler passes on a single isolated family at a time.

Reporting

run.py wip calls benchmark/report.py automatically (unless --no-report).

Manual usage:

# Auto-compare against latest git-clean run if available
python3 benchmark/run.py compare benchmark/results/<hostname>/WIP/<run_timestamp>

# Explicit baseline comparison
python3 benchmark/run.py compare <candidate_run_dir> --baseline <baseline_run_dir>

# Full report with detail tables
python3 benchmark/run.py compare <run_dir> --baseline <baseline_dir> --full

Generated summary report name:

  • report_<git_label>_<timestamp>.md

Report content is intentionally concise:

  • Host information
  • Condensed Performance Matrix (with ~ noise markers when CV > 1.0)
  • Condensed Performance Matrix Comparison
  • Baseline staleness warning (when baseline is > 7 days old)

Outputs

Each run directory contains:

  • platform.json
  • manifest.json
  • macro_small_results.json (when MACRO-SMALL selected)
  • macro_large_results.json (when MACRO-LARGE selected)
  • macro_results.json (compatibility merged macro view)
  • micro_results.json (when micro selected)
  • macro_small_stdout.log, macro_small_stderr.log
  • macro_large_stdout.log, macro_large_stderr.log
  • micro_stdout.log, micro_stderr.log (when micro selected)
  • condensed_metrics.json
  • report.md and timestamped report_<label>_<timestamp>.md

For --repetitions > 1, raw per-repetition artifacts are stored under repeats/run_<N>/ and top-level macro_*_results.json / micro_results.json contain median-aggregated results across repetitions.

Skipped results (e.g., Static modes on profiles without no-copy generators) are automatically stripped from stored JSON files. Only status: "ok" rows are retained; a skipped_count field records how many were removed.

External CSV Benchmark Policy

3rd-party CSV benchmarks are treated as one-time documented reference measurements and are not part of the default orchestrator flow.

Reference Workloads

Open-data reference mapping and cache/download workflow:

  • benchmark/REFERENCE_WORKLOADS.md
  • benchmark/reference_workloads.json
  • benchmark/fetch_reference_datasets.py

Fetch datasets to non-versioned cache (tmp/reference_datasets):

python3 benchmark/fetch_reference_datasets.py

Dedicated Python Benchmarks

Python lane is intentionally separate from the C++ orchestrator and writes schema-compatible macro rows:

python3 python/benchmarks/run_pybcsv_benchmarks.py --size=S

Output default:

  • benchmark/results/<hostname>/python/py_macro_results_<timestamp>.json

Dedicated C# Benchmarks (Standalone .NET)

Build native C API target, then run the .NET benchmark harness:

cmake --preset ninja-release
cmake --build --preset ninja-release-build -j$(nproc) --target bcsv_c_api
export LD_LIBRARY_PATH="$PWD/build/ninja-release/lib:$LD_LIBRARY_PATH"
dotnet run --project csharp/benchmarks/Bcsv.Benchmarks.csproj -- --size=S

Output default:

  • benchmark/results/<hostname>/csharp/cs_macro_results_<timestamp>.json