Skip to content

feat: add reproducible UDDSketch quantiles - #310

Draft
tisonkun wants to merge 3 commits into
apache:mainfrom
tisonkun:codex/uddsketch
Draft

tisonkun wants to merge 3 commits into
apache:mainfrom
tisonkun:codex/uddsketch

Conversation

@tisonkun

@tisonkun tisonkun commented Oct 27, 2026 •

Copy link
Copy Markdown
Member

Adds UddSketch for reproducible quantile aggregation over finite signed f64 values, with weighted updates, merging, and serialization.

Design

Based on UDDSketch (2020) and its mergeability analysis (2021), the implementation uses logarithmic buckets and uniformly coarsens them to stay within a configured bucket budget.

  • A fixed, monotone Q52 logarithm approximation gives deterministic initial bucket indices. Translating the smallest positive subnormal to index zero makes integer right shifts form the collapse hierarchy. Mirrored negative buckets and a dedicated zero bucket cover the finite f64 domain with a minimum budget of three.
  • Each bucket stores its count and observed extrema. Their harmonic mean supplies the representative value, and the extrema determine a conservative relative-value error bound that includes floating-point rounding.
  • The UDD1 format stores a 32-byte header and 32 bytes per bucket. Decoding validates counts, ordering, extrema, and bucket membership.

Guarantees

For identical parameters and mapping versions, states built through updates, merges, and serialization roundtrips produce identical answers and bytes across input orders, partitions, and merge trees. Every build reaches the same smallest collapse level that fits the budget; integer parent mapping and exact count/extrema combination preserve this property during merging.

Queries approximate the lower discrete quantile at floor(q * (n - 1)), with exact endpoints and exact zero results. relative_error() reports the final bound on abs(estimate - value) / abs(value) for nonzero quantiles. The configured initial accuracy guides bucket spacing; uniform collapse can increase the final bound.

Validation and performance

cargo x check, cargo x test (800 tests including doctests), and cargo x lint pass. MSRV and release-mode checks also passed. Regression tests cover mapping boundaries, subnormals, reciprocal correction, bucket lookup, large weights, merging, error bounds, and serialization.

Across seven distributions, 128 input/partition/merge layouts, and six query ranks, the 100,001-row experiment produced identical UDDSketch states and answers, with every measured error within its reported bound.

Local release benchmarks on Apple M4 Max, macOS 26.6.2, rustc 1.101.0-nightly; medians of 15 measured rounds after two warmup rounds:

Lognormal workload Build Decode Serialized size
1M values, 512 buckets, accuracy 0.01 8.07 ns/value 3.21 µs/state 13,856 bytes
1M values, 2,048 buckets, accuracy 0.001 11.29 ns/value 14.30 µs/state 63,296 bytes

The included comparison harness reports timing, size, accuracy, and merge stability:

cargo run --release -p benchmarks --example quantile_comparison
cargo bench -p benchmarks --bench benchmarks -- uddsketch

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant