Repository navigation
feat(analytics): sketch analytics for every sketch and framework - #262
Merged
Merged
Conversation
A full column is stored as u16, so a wider sketch folded distinct columns together. CountMinSketchFold::new and CountSketchFold::new now panic above MAX_FULL_COLS. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dimensions_for sized slices from the approximation e^(-n/c) and capped them at BLOOM_MAX_BITS, so it could return a filter above the target. It now widens until the exact rate (1 - (1 - 1/c)^n)^r is at most the target and returns None when that needs more than BLOOM_MAX_BITS; with_capacity panics there. predicted_fpp uses the exact rate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With Nitro sampling enabled, Total sums sampled counters and is an unbiased estimate, so the family row marks it AsIfBuiltOverUnion. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The proof bounds the heap estimator's variance by 1.35 F0^2/k and the docs rounded it up to 2. The UnivMon and UnivMonTwoTier pages now state 1.35/(k eps^2), and the cardinality tests check it (2000-trial runs pass). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adds the analytics module with the trait each sketch implements, the error-bound types, and the shared formula helpers (P_med, est(d), h(k), lg). StatisticKind gains QuantileInterpolated, answered by DDSketch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
analytics::ops::count(f) returns the ops f performed on this thread. Vector2D, the fixed matrices, BitMatrix and the median helper count their cell reads and writes; without the feature the counting compiles to nothing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SketchAnalytics for CountMinSketch, CountSketch, CountSketchWithL2Norm, and the experimental CountMinHll and MicroCm, with memory, op-count, error and sizing tests in the analytics test target. CountSketch's ItemFrequency bound takes a chosen delta: the largest per-row rate beta with P_med(d, beta) <= delta gives eps = sqrt(q/beta), and params_for picks the odd d <= 19 and width with the fewest counters (d = 5, w ~ 9.5/eps^2 at delta = 0.01). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SketchAnalytics for SpaceSaving, ElasticSketch and CocoSketch, with memory, op-count, error and sizing tests in the analytics test target, and memory_bytes() on all three. SpaceSaving: update 47 ops (from 45), merge 44m + 2m·lg(2m) + 13 (from 36m + 2m·lg(2m) + 17), TopK 1 + 7·min(k, m), memory with the index under churn; Total has no accessor and leaves the table. ElasticSketch: delta = e^-min(d, 5), update 2 + d, TopK H·(5 + d) + 2H·lg H, merge H·(13 + 3d) + 2dw. CocoSketch: update d + 1, merge dw·(d + 2); Total has no accessor. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SketchAnalytics for CountMinSketchWithHeap, CountSketchWithHeap, CountMinSketchWithSpaceSaving, CountSketchWithSpaceSaving, CountMinSketchFold and CountSketchFold, with memory, op-count, error and sizing tests in the analytics test target, op counting in TopKHeap, CommonHeap and the fold cells, and memory_bytes() on the Space-Saving variants and the folds. TopKHeap: h(k) = 8 + 14·floor(log2 k) (from 16 per level), exact for a root-to-leaf descent. Heap variants: merge 2dw + 3 + u·(3 + est(d) + h(k)). Space-Saving variants: update d + 47 (from d + 45), TopK m'·(5 + est(d)) + m'·lg m' (from 7), merge 2dw + 44m + 2m·lg(2m) + 13 (from 2dw + 36m + 2m·lg(2m) + 17). Folds: same-level merge dW/2^F + d·min(W, D)·(2 + E) + min(k, D)·(1 + dE + est(d) + h(k)) (from dW·2^F + k·(dE + est(d) + h(k)), which undercounted), hierarchical merge dW + Σ(dW/2^F_i + 2e_i + k_i·(1 + est(d) + h(k))). TopK error multiplies the per-key δ by D (heap, folds) or min(m, D) (Space-Saving); sizing rows added for ItemFrequency and TopK. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SketchAnalytics for HyperLogLog (Classic, Ertl), HyperLogLogHip, Kmv, SetAggregator and Bloom, with memory, op-count, error and sizing tests in the analytics test target, op counting in the HLL, HIP, Kmv and SetAggregator code paths, memory_bytes() on SetAggregator and Bloom, Bloom::theoretical_memory_bytes and SetAggregator::contains. Classic HLL with max_distinct takes a chosen delta: the smallest eps from max_distinct/(2(m - max_distinct)) to 1 with max over every D of lambda^t/t! at most delta. Bloom params_for minimizes r*c over r <= 20 and power-of-two c. HyperLogLogHip update: 6 (from 4). Kmv: update k + 3 + 6*floor(log2 k) (from 8 per level), query 1 (from 2), merge k*(k + 4 + 6*floor(log2 k)); D alone as data. SetAggregator: Distinct query 0 (a length), merge 2D. Bloom: update r on both paths (from r + 1 / 2r + 1; there is no inserted count), merge 2r*ceil(c/64) (from + 2). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SketchAnalytics for Kll, KllDynamic, DdSketch and the experimental UniformSampling, with memory, op-count, error and sizing tests in the analytics test target, op counting in their code paths, and UniformSampling::memory_bytes(). Kll and KllDynamic: Quantile on the 2.296/k^0.9723 fit and Rank as Absolute(L1) on a fit measured on asap_sketchlib, 2.07/k^0.9646, both with a chosen delta scaled by s(delta) below 0.01 and None for m < 8. A state holds L(n) - 1 to L(n) + 1 levels, so query and merge bounds use L(n) + 1. Kll: update average 15 + lg(c_b + 1) (from 26 + ...), quantile query 3R + R lg R + 2L + lg R + 2 (from 3R + R lg R + lg R), rank R + 2L (from R), merge B lg B + 16R + 8L + 40 (from B lg B + 25R - 11B + 10L + 2(L_a + L_b) + 74). KllDynamic: update average 11 + 2 lg(c_b + 1) (from 5 + lg(c_b + 1) + lg(1.5k), exceeded for small k), merge S + 16R + 8L, queries as Kll. DdSketch: Quantile and QuantileInterpolated only (no sum, min or max), update 4 (from 6), quantile B + 4, interpolated 2B + 7, merge 2(B_a + B_b) + 11. UniformSampling: Sample only, update 3 + lg s + 2s, query 2s, merge 3 + 2(s_a + s_b) + (s_a + s_b) lg(s_a + s_b). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SketchAnalytics for UnivMon, UnivMonTwoTier and UnivMonQ, with op-count instrumentation, memory/ops/error/sizing tests, and UnivMonTwoTier::memory_bytes, UnivMonTwoTier::theoretical_memory_bytes and UnivMonQ::theoretical_memory_bytes. UnivMon and UnivMonTwoTier: Total exact, FrequencyL2 as CountSketchWithL2Norm on layer 0, Distinct at a chosen delta with eps = sqrt(1.35/(k(delta - psi))). Per-layer heap terms recounted with h(k) = 8 + 14 floor(log2 k); queries add the total reads and the per-layer partial sums (4 + 2L); merge counts the candidate set reads and inserts and the flags (3dw + d + 6 + c(22 + 4d + 2d lg d + 28 floor(log2 k))). UnivMonQ: ItemFrequency follows Count Sketch (delta chosen, eps = sqrt(q/beta)), with q the most collision probability over the layers (1/w_j for power-of-two widths). Quantile and Rank bounds need UnivMonQData::heavy_set_empty. Total/Min/Max rows dropped (not in the capability row). BinaryHeap pushes and pops are counted at their sift worst case; update, ItemFrequency query (1 + 2d + d lg d) and a new merge formula follow those counts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SketchAnalytics for Hydra, ExponentialHistogram, EhBucketSketch, ExponentialHistogramUnivMon (experimental), NitroBatch<S: NitroInner>, TumblingWindow<S> and HashSketchEnsemble; Octo as functions over OctoConfig (octo_error, octo_memory_bytes, octo_update_ops). Enum-cell frameworks dispatch per cell kind (HydraCell, EhCell, EnsembleCell); generic ones compose the inner trait through InnerSketch, whose C_in is the inner heap over its cell size. Clones, clears and the frameworks' own grids, bucket lists, pools and sketch lists are op-counted, and memory_bytes() is added where missing. Doc corrections: Hydra update (2^a-1)d(1+U_in), query (d+1)Q_in+2d+d lg d, merge dw(2+M_in), group-total ops, Kll Quantile as Rank; EH bucket-list slots, concrete average update and query ops, FrequencyL2 as the only by-mass readout, no DdSketch Min/Max; EHUnivMon has no ItemFrequency readout; Nitro merge is M_in; Tumbling slot memory, per-boundary 4+2*max_windows+C_in, query C_in+t(1+M_in), no Kll-window bound; Octo parents hash on RegularPath and the HLL parent is Classic. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- SketchAnalytics::theoretical_memory_bytes returns Option<usize>; None when it needs an unknown input with no upper bound (key bytes, label bytes, SetAggregator D and l, Uniform n, EH and EhUnivMon bucket counts). - Tumbling update ops are unbounded when b is unknown. - ElasticSketch key bytes are the passed String's capacity. - HashSketchEnsemble cells carry each matrix's counter bytes. - UnivMonQ clones and decodes reserve the modelled heaps and tables. - Hydra escaped labels own exactly their length. - Costly op-count arguments compile only with the op-count feature. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- SpaceSaving, ElasticSketch, the folds' TopK and Hydra's Kll readouts take a `merged` data input and give a bound only for Some(false). - SpaceSaving merge ops take the merged-in capacity `other_capacity`: 34m + 10m_o + (m + m_o)·lg(m + m_o) + 13, unbounded when unknown. - The Space-Saving matrices' TopK error and sizing use min(m, D), m when D is unknown. - The spec states a unit SpaceSaving update costs at most 47 in any state. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- HLL Classic: the collision bound only when its ε is at most β_m/√m, and sizing returns a precision whose own bound meets the request. - HLL: Flajolet's β_m for m ≤ 128, no bound below p = 4. - Fold sizing caps the fold level at log2 W. - Octo τ_max covers the initial threshold and the clamped band. - UnivMonQ Quantile/Rank read F̂₂ and need F̂₂·M < n²; sizing needs n and F̂₂. - CountSketchWithL2Norm ItemFrequency sizing searches only shapes within 128 hash bits; regular-path Count Sketch sizing keeps w ≥ 1. - DdSketch: α capped at 1/2, α outside (0, 1) and magnitudes outside the indexable range or negative/NaN have no bound. - normal_upper_quantile for Kll's s(δ) at tiny δ. - Kmv rejects NaN and negative δ. - Hydra HyperLogLog Distinct has no bound or sizing. - Kll Rank fit holds for k ≥ 32. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- UnivMonQ: a BinaryHeap pop counts 7 + 5⌊log2 n⌋, mirrored against std; update and merge formulas follow. - TopKHeap::clear counts 2 per resident; heap-matrix and UnivMon merge formulas follow. h(k) notes the digest-collision probe. - Octo τ_min is the smallest threshold a worker holds. - EH HyperLogLog Distinct has no bound; EhData drops window_distinct. - UnivMonQ Quantile/Rank ε adds 1/n for rounding; sizing follows. - Tests: FastPath non-power-of-two collision rate and ε, FastPath sizing check on its own type, heavy targets for Hydra Count-Min and UnivMonQ ItemFrequency, exact never-evicted Elastic residents, UnivMon Distinct at k = 256, UnivMonQ samples on increasing data. - Docs: Count-Min FastPath rate, Count Sketch FastPath sizing, d ≤ 20 cap; CHANGELOG marks the Bloom::dimensions_for and fold-width changes breaking. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements
docs/sketch_analytics.md(#253): every sketch and framework reports its theoretical memory, update/query/merge ops, error bound per statistic, and the parameters that meet a target(ε, δ), for ASAPPlanner to call. ASAPPlanner pins asap_sketchlib by rev, so it moves to this API in its own PR.One commit per step; review commit by commit.
Commits
fix(fold)!CountMinSketchFold::new/CountSketchFold::newpanic forfull_cols > 65536(MAX_FULL_COLS): a full column is au16.fix(bloom)!Bloom::dimensions_forreturnsOption, widens until the exact rate(1 − (1 − 1/c)^n)^rmeets the target, and returnsNonepastBLOOM_MAX_BITS;with_capacitypanics there.predicted_fppuses the exact rate.fix(capability)!CountMinSketchTotal isAsIfBuiltOverUnion, notExact: under Nitro sampling it is an estimate.docs(univmon)1.35/(kε²), the variance the proof derives; the tests check it.feat!renameestimated_memory_bytes→theoretical_memory_byteseverywhere.feat(analytics)traitSketchAnalytics,ErrorBound,ErrorMetric,NormScale,StatisticKind::QuantileInterpolated, shared helpers (P_med,est(d),h(k),lg).feat(analytics)op-countop-countfeature:analytics::ops::count(f)returns the opsfperformed. Off by default and free when off.feat(analytics)× 7tests/analytics/.style(analytics)docs(tests)docs/tests.mdrows for the new tests.fix(analytics)× 4Decisions
δ:ε = √(q/β)withβthe largest per-row rate whereP_med(d, β) ≤ δ.params_forpicks the oddd ≤ 19and width with the fewest counters, which givesd = 5,w ≈ 9.5/ε²atδ = 0.01(ASAPPlanner's per-row 1/3 gaved = 47, past ASAPv1's 20 rows). Contractcount_sketch_l2_median_chosen_row_rate_v1; the heap, Space-Saving, fold, L2-norm and UnivMonQ variants follow it.rank()returns a count, so Rank isAbsolute(L1)with a fit measured on asap_sketchlib:ε = 2.07·s(δ)/k^0.9646at 99%, contractkll_rank_measured_99_v1. Measured: 99th percentile of|rank(x) − true rank|/nover 1000 seeded trials × 9 query points,n = 10^6,m = 8,kfrom 32 to 4096; the regression is1.8647/k^0.9646, and 2.07 makes it an upper envelope of every measuredk.StatisticKindonly. Doc rows for Min, Max and Sum (DDSketch, UnivMonQ) and for Totals without an accessor (CocoSketch, SpaceSaving, UniformSampling) are dropped, matching the capability table.Formulas corrected by counting
The op tests count every access under
op-countand check it against the formula; the doc now states what the code does. The larger corrections:h(k)(aTopKHeapupdate):8 + 16·⌊log2 k⌋→8 + 14·⌊log2 k⌋.44m + 2m·lg(2m) + 13.k + 3 + 6·⌊log2 k⌋. Bloom: updateron both paths (no inserted count).15 + lg(c_b + 1), mergeB·lg B + 16R + 8L + 40; queries countL(n) + 1levels.δise^−min(d, 5): light row 5 shares the heavy part's seed.Other public API
memory_bytes()on the sketches and frameworks that lacked it (CHANGELOG lists them);SpaceSaving,Bloom,UnivMonTwoTierandUnivMonQgain an inherenttheoretical_memory_bytes;SetAggregator::contains.CountMinFastOpsandCountSketchFastOpsrequirememory_bytes.Independent review fixes
Eight reviewers read the diff against the spec without the author's conclusions; the confirmed findings are fixed in the four
fix(analytics)commits:theoretical_memory_bytesreturnsOption<usize>:Nonewhen it needs an input the section gives no bound for (key, label or mean key bytes), where an unknown key length used to count as 0.merged: Option<bool>data input: SpaceSaving, ElasticSketch, the folds' TopK and Hydra's Kll readouts give a bound only for an unmerged state.theoretical_errorreports (HLL Classic), returned parametersnewrejects (fold level forW < 8), or returnedNonetoo early (CountSketchWithL2Norm under the 128-bit hash, DdSketch forε ≥ 1).τ_max/τ_mininclude the initial threshold; UnivMonQ's Quantile/Rank condition readsF̂₂·M < n²instead of a flag; DdSketch checks the indexable range; Kll Rank needsk ≥ 32; HLL uses Flajolet's constants below 128 registers; Hydra's and EH's HLL Distinct have no bound (a standard error mixed with a probabilistic term).BinaryHeap::pop(7 + 5⌊log2 n⌋),TopKHeap::clear(2 per resident), SpaceSaving merges of unequal capacities.cells()) is behind#[cfg(feature = "op-count")], so the default build does none of it.Tests
tests/analytics/(156 tests with all features, 108 withoutop-count): memory againstmemory_bytes(), counted ops against each formula at two or more sizes, error bounds as binomial failure rates over independent trials, and sizing against its own bound. Every error test is mutation-checked: breaking the sketch's update or estimator (a dropped row, an ignored sign, a stale heap, a wrong compaction coin, …) turns it red.Local:
cargo fmt --check;cargo test --test asapv1_golden; clippy with-D warningsfor default,--all-features,octo-runtimeandexperimental;cargo docwith-D warnings;cargo test --test analyticswith default and all features;cargo test --doc; the lib tests of each sketch whose source changed. The pbt and e2e suites run in CI.Not covered
octo_error,octo_memory_bytes,octo_update_ops).🤖 Generated with Claude Code