Repository navigation
perf(hll): reduce union copying and defer dense estimator rebuilds - #317
Merged
Merged
Conversation
Adopting an HLL8 sketch into an empty union previously allocated a zeroed register array, merged the source registers, and rebuilt the zero count and KxQ caches. Recreating a union for each small partial therefore scanned the entire accumulated dense sketch even when no downsampling was needed. Clone the source array and its estimator when its precision fits within the union's maximum. This also covers a smaller source precision and the reverse-merge path that adopts a dense input before replaying coupons. Representation conversions and actual downsampling retain their existing paths. Add regression coverage for subsequent updates after copying both HIP and composite estimates. Add benchmarks for empty-union adoption, repeated union construction, persistent small-partial merges, and dense-partial merges at lg_k 12 and 18. Validation: all 46 HLL integration tests pass. Release benchmarks confirm that dense-input adoption and repeated union construction avoid the previous estimator-rebuild cost; persistent small-partial merges remain on their existing path.
Aggregation engines that emit an HLL union state currently call `to_sketch(HllType::Hll8).serialize()`. That first clones the union's retained storage and then copies it again into the serialized image. Add HllUnion::serialize to encode its HLL8 gadget directly. Document that the bytes represent a sketch with the effective precision, not a saved union configuration, and match the existing to_sketch serialization. Together with update_value and update, this supports aggregators that accept both raw values and partial sketches through one persistent union. Exercise empty, LIST, SET, and dense results from HLL4, HLL6, and HLL8 inputs, including reduced precision, continued updates after decoding, and reuse after reset. Compare direct serialization against the existing path in benchmarks, including allocation counts. Validation: all 16 HLL serialization tests and all 10 HLL union doctests pass. In the lg_k 18 dense-result benchmark, direct serialization removes one register-array allocation and copy, with a median of about 4 us compared with about 8 us through an intermediate sketch.
Every dense merge previously scanned the resulting register array again to recompute the zero count and KxQ sums. Aggregations that merge many partials before reading a result paid that reconstruction cost after every input, even though the intermediate estimates were never used. Keep the HLL8 register-derived state in a OnceLock and invalidate it after bulk merges. An absent cache also records that HIP history is unavailable. Estimates, bounds, serialization, and equality materialize the cache when needed, preserving shared-reference queries and thread safety. Scalar updates maintain an initialized cache or keep an already invalidated cache deferred. Copying preserves cache state, while restoring a valid HIP history after downsampling rebuilds the fields needed by later updates. Check emptiness directly from registers when the cache is absent so union bookkeeping does not force a rebuild between merges. Packed HLL4/HLL6 inputs invalidate the same destination cache after their register writes. Cover mixed representations and precisions, queries interleaved with scalar updates, copying a result before its first query, and concurrent shared queries. Extend cache reconstruction coverage across the KxQ split and maximum six-bit register value. Benchmarks include a final estimate, queries after every merge, and downsampled dense partials. Validation: cargo x check, cargo x test, and cargo x lint pass. All 48 HLL integration tests also pass on Rust 1.86.0. On an Apple M4 Max, the release benchmark merging 16 dense partials at lg_k 18 and then querying the result has a median of 0.284 ms versus 4.91 ms on the base revision. Querying after every input retains the per-merge reconstruction cost, as expected.
Describe the new public capability without register-encoding details or intermediate-copy mechanics. Keep the output-format contract in the API documentation, where callers can find it when choosing how to serialize a union result. Validation: cargo x lint passes in a clean worktree containing the same change; git diff --check passes.
This was referenced Nov 4, 2026
Dense unions need to defer rebuilding register-derived statistics across merges without treating HIP update history as reconstructible cache data. Keep that distinction in the shared Estimator instead of an Array8-specific CachedState wrapper. Store estimate history in state and keep zero counts and KxQ sums together in RegisterStatistics. Estimator owns their lazy initialization, invalidation, scalar maintenance, and HIP restoration for all three HLL representations. OnceLock preserves shared queries without requiring mutable access. HLL4 retains its packing minimum and minimum count as storage metadata, independently of the logical zero-register count. Remove Array8's cache lifecycle handling and Array6's separate zero count. Keep scalar update paths inline and expand their benchmarks to cover HLL4, HLL6, and HLL8. Add coverage for restoring HIP after deferred reconstruction and for estimates across HLL4 minimum shifts and serialization round trips. On aarch64, the shared statistics increase HllSketch's inline size from 88 to 104 bytes. The serialized format is unchanged. A benchmark of 16 dense merges at lg_k=18 followed by a query retains deferred reconstruction, with median time decreasing from approximately 288 us to 252 us. Validation: cargo x check, cargo x test, cargo x lint, cross-language serialization fixtures, Rust 1.86 HLL tests, and HLL update/union benchmarks.
Keep dense registers and their estimation state in a single HllArray owner. Queries resolve statistics from that owner's registers, and bulk mutations invalidate statistics before writing, including when an input iterator panics partway through a merge. Select statistics storage by register representation. HLL4 and HLL6 maintain KxQ eagerly and use their existing zero-count metadata; only HLL8 uses OnceLock for deferred reconstruction. This restores the 88-byte HllSketch layout on aarch64 without changing the serialized representation. Route dense coupon batches directly to their stable register representation instead of dispatching through the sketch mode for each coupon. Cover HIP restoration, occupied-register bounds, and interrupted bulk writes at the owning-array boundary.
Revert the shared estimator and owning-array refactors to keep the deferred reconstruction implementation specific to Array8. Packed representations continue to maintain their estimator fields directly without a common statistics-storage abstraction. Retain direct HLL8 result serialization, fast adoption of dense HLL8 inputs, and deferred reconstruction across dense merges. This reverts commits 0b2e638 and 6eb6363. The resulting tree matches fb2c360.
Name the HLL8 zero-count and estimator bundle EstimationData and its OnceLock field estimation. Align the accessor, reconstruction helper, and local variables with the estimation data they provide. Use state for Estimator's EstimateState field. These naming changes retain the Array8-local deferred reconstruction and direct estimator maintenance in HLL4 and HLL6. Validation: cargo x check, cargo x test, cargo x lint, and Rust 1.86 HLL tests.
When a dense input lowers a union's effective precision, the existing HLL8 gadget was cloned into a temporary Mode solely to dispatch the register downsampling operation. Borrow its register slice and call the Array8 operation directly, removing a full register allocation and copy. The operation still invalidates HIP and defers rebuilding estimation fields until they are needed. Rename EstimationData to Estimation and destructure its fields at query sites to make access to the estimator and zero-register count clearer. Validation: cargo x check, cargo x test, and cargo x lint.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Repeated HLL unions can spend most of their time copying a dense accumulator and rebuilding its estimator, even when each incoming partial is small. Persistent unions also rebuild register-derived caches after every dense input, and serializing a union currently requires cloning an intermediate sketch.
This addresses the upstream costs identified in apache/datafusion-comet#7341 and complements apache/datafusion-comet#7345.
Changes
The implementation is split into three commits:
HllUnion::serialize(). Emit the same HLL8 sketch bytes asto_sketch(HllType::Hll8).serialize()without the intermediate sketch allocation and copy. The image retains the effective precision, not the union's maximum configuration.OnceLockcontaining the zero count and estimator. Queries rebuild it on demand. Scalar updates maintain an initialized cache or leave an invalidated cache deferred. Copies retain that state, and restoring HIP after downsampling rebuilds the fields needed by subsequent updates. Shared-reference queries and thread safety are preserved.Dense merges still read the input registers. The benefit of deferral is avoiding repeated reconstruction when several inputs arrive before a query or serialization. Callers that query after every input still pay that reconstruction cost.
Performance
Release benchmarks on an Apple M4 Max with Rust
1.101.0-nightly (8006091eb 2026-10-17), 100 samples, medians. The baseline isbb6ba57with the same union benchmarks added. Merge benchmarks include a final estimate and HLL8 result extraction.On this PR's head, serializing an 80,000-value HLL8 union at lg_k 18 takes 4.00 µs directly versus 7.66 µs through
to_sketch, removing one register-array allocation and copy. These are library benchmarks, not end-to-end Comet measurements.Reproduce with:
Validation
cargo x prepare-testdatacargo x checkcargo x testcargo x lintcargo +1.86.0 test -p tests-integration --test hll_test— all 48 tests pass.Regression coverage includes copied HIP/composite state followed by updates; direct serialization across LIST, SET, and HLL modes and input representations; downsampling; query timing interleaved with scalar updates; copying an unqueried merged result; concurrent shared queries; and the KxQ split at register values 31/32 and the maximum six-bit value. Existing cross-language serialization tests also pass.