Skip to content

sketch-bench: a trials axis for accuracy (fresh, relabel, reseed) - #236

Merged
GordonYuanyc merged 2 commits into
mainfrom
sketchbench-trials-axis
Nov 4, 2026
Merged

GordonYuanyc merged 2 commits into
mainfrom
sketchbench-trials-axis

Conversation

@GordonYuanyc

@GordonYuanyc GordonYuanyc commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

Adds --trials T, --trial-start S and --trial-mode fresh|relabel|reseed to approxbench sketchbench, so many independent accuracy trials run in one process instead of one process per trial. Trial t writes the same record whatever S and T are. Each record carries bench.trial and bench.trial_mode, the per-trial sketch config and the per-trial dataset descriptor. --flat writes one line per trial. --per-group-out gains a trial column, and all trials go to one file. A failed trial names the trial and its derived seeds. A run asking only for accuracy now skips the time-based warmup.

  • fresh: every distribution seed and the burst seed become mix(seed, t) for t >= 1, and the key columns are offset.
  • relabel: the table is drawn once and only the key columns are offset (integers and f64 + t*2^40, strings prefixed "t:"). The membership comparator's absent probes are offset too.
  • reseed: the table is drawn once and the row's seed field becomes mix(base, t). Only rows whose wrapper uses the seed accept it: kll-percall, kll-cdf, kll-dynamic, hydra-kll, hydra-kll-cdf, coco, uniform-sampling, nitro-count-min, nitro-count-sketch (all lib).
  • datagen: new ColumnSpec.key_offset (a number, or a prefix for strings), applied after generation and serialized only when set. mix_connected_bits is renamed splitmix64 and made public so mix reuses it.

Decisions:

  • f64 key columns get the t*2^40 offset too. A key the offset does not move exactly (any non-integer f64 key) is refused, naming the column.
  • No truth mapping under relabel. The table is generated once, then cloned and offset per trial, and rows::measurements recomputes the truth. reseed reuses the table the same way.
  • The per-trial offset is stored in the descriptor's key_offset, so --trials 1 --spec <descriptor> reproduces a trial.
  • mix() is documented in the --trial-mode help text.
  • No --trial-threads.

Acceptance:

  1. 94 rows in --list-impls. 3 measure no accuracy. I compared --trials 1 against main f31707d on each row's query (and merge where the row has it) accuracy, workload and sketch_config:
    • 74 rows are identical.
    • 8 seeded rows (kll-percall, kll-cdf, kll-dynamic, hydra-kll, hydra-kll-cdf, coco, nitro-count-min, nitro-count-sketch, all lib) give different numbers on two runs of main itself when no seed is set. With seed=7 they match main exactly.
    • 9 rows also give different numbers on two runs of main itself: hydra-cms/lib, hydra-cs/lib, hydra-univmon-entropy/lib and /polars, univmon-cardinality/lib, univmon-entropy/lib and /oxide, univmon-two-tier-cardinality/lib, univmon-two-tier-entropy/lib. The branch values fall within main's own run-to-run spread.
  2. Prefix property: aqpbm-cli/tests/trials.rs (a_later_start_reproduces_the_same_trials).
  3. relabel keeps stream_items, stream_distinct and stream_l2, and Count-Min aae_top1 differs across trials (relabel_keeps_the_stream_and_moves_the_collisions).
  4. reseed: KLL rank_err_* differ across trials and are identical on a rerun. Count-Min under reseed is rejected with the list of seeded rows.
  5. A fresh trial matches a --trials 1 --spec <its descriptor> run for CMS (i64) and Bloom (string).
  6. hll lg_k=14, N = 2^16, 1024 trials: 276.8 s in one process vs 262.1 s as 1024 single-trial processes. Both give identical records. One process is not faster. About 200 ms of each trial is the query measurement itself: the cardinality comparator asks the sketch 4096 times. Process start costs about 20 ms. Before this change, a process that did not set BENCH_WARMUP_SECS=0 also paid the 10 s default warmup.
  7. cargo fmt --check, cargo clippy --workspace --all-targets --all-features -- -D warnings and cargo test --workspace pass.

Different from the spec:

  • UnivMonQParams has seed_index, not seed, so univmon-q rows are not reseed-able.
  • --trials with --workload-spec is rejected.
  • A key column that already carries key_offset is rejected under fresh/relabel.
  • reseed is checked per (variant, library): the oxide KLL rows (no seed) and the polars baselines (seed ignored) are rejected.
  • Every trial option counts as a trial run, --trials 1 --trial-start S included, so the accuracy-only checks apply to it.

🤖 Generated with Claude Code

GordonYuanyc and others added 2 commits November 3, 2026 22:40
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… lazy absent probes, one trial loop)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@GordonYuanyc
GordonYuanyc merged commit 68f6514 into main Nov 4, 2026
3 of 4 checks passed
@GordonYuanyc
GordonYuanyc deleted the sketchbench-trials-axis branch November 4, 2026 05:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant