Repository navigation
sketch-bench: a trials axis for accuracy (fresh, relabel, reseed) - #236
Merged
Merged
Conversation
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… lazy absent probes, one trial loop) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
--trials T,--trial-start Sand--trial-mode fresh|relabel|reseedtoapproxbench sketchbench, so many independent accuracy trials run in one process instead of one process per trial. Trial t writes the same record whatever S and T are. Each record carriesbench.trialandbench.trial_mode, the per-trial sketch config and the per-trial dataset descriptor.--flatwrites one line per trial.--per-group-outgains atrialcolumn, and all trials go to one file. A failed trial names the trial and its derived seeds. A run asking only for accuracy now skips the time-based warmup.fresh: every distribution seed and the burst seed become mix(seed, t) for t >= 1, and the key columns are offset.relabel: the table is drawn once and only the key columns are offset (integers and f64 + t*2^40, strings prefixed"t:"). The membership comparator's absent probes are offset too.reseed: the table is drawn once and the row'sseedfield becomes mix(base, t). Only rows whose wrapper uses the seed accept it: kll-percall, kll-cdf, kll-dynamic, hydra-kll, hydra-kll-cdf, coco, uniform-sampling, nitro-count-min, nitro-count-sketch (alllib).ColumnSpec.key_offset(a number, or a prefix for strings), applied after generation and serialized only when set.mix_connected_bitsis renamedsplitmix64and made public somixreuses it.Decisions:
relabel. The table is generated once, then cloned and offset per trial, androws::measurementsrecomputes the truth.reseedreuses the table the same way.key_offset, so--trials 1 --spec <descriptor>reproduces a trial.mix()is documented in the--trial-modehelp text.--trial-threads.Acceptance:
--list-impls. 3 measure no accuracy. I compared--trials 1against main f31707d on each row's query (and merge where the row has it) accuracy, workload and sketch_config:seed=7they match main exactly.aqpbm-cli/tests/trials.rs(a_later_start_reproduces_the_same_trials).relabelkeepsstream_items,stream_distinctandstream_l2, and Count-Minaae_top1differs across trials (relabel_keeps_the_stream_and_moves_the_collisions).reseed: KLLrank_err_*differ across trials and are identical on a rerun. Count-Min underreseedis rejected with the list of seeded rows.freshtrial matches a--trials 1 --spec <its descriptor>run for CMS (i64) and Bloom (string).hlllg_k=14, N = 2^16, 1024 trials: 276.8 s in one process vs 262.1 s as 1024 single-trial processes. Both give identical records. One process is not faster. About 200 ms of each trial is the query measurement itself: the cardinality comparator asks the sketch 4096 times. Process start costs about 20 ms. Before this change, a process that did not setBENCH_WARMUP_SECS=0also paid the 10 s default warmup.cargo fmt --check,cargo clippy --workspace --all-targets --all-features -- -D warningsandcargo test --workspacepass.Different from the spec:
UnivMonQParamshasseed_index, notseed, so univmon-q rows are notreseed-able.--trialswith--workload-specis rejected.key_offsetis rejected underfresh/relabel.reseedis checked per (variant, library): the oxide KLL rows (no seed) and the polars baselines (seed ignored) are rejected.--trials 1 --trial-start Sincluded, so the accuracy-only checks apply to it.🤖 Generated with Claude Code