Skip to content

feat(hll): support all output types in union serialization #318

Closed as not planned

Description

@tisonkun

Problem

PR #317 adds HllUnion::serialize(), which directly emits an HLL8 result. The union accepts HLL4, HLL6, and HLL8 inputs, but callers choosing HLL4 or HLL6 output still need union.to_sketch(hll_type).serialize(), allocating an intermediate sketch. Applications persisting or transferring results should be able to choose the register representation when serializing directly.

Suggested approach

  • Define an output-type selection API consistent with HllUnion::to_sketch(HllType); document the output representation and effective precision.
  • Share encoding logic with sketch serialization. Keep the HLL8 fast path and implement HLL4/HLL6 output without constructing a complete temporary sketch merely to serialize it.
  • Account for the packing and auxiliary-entry work required by HLL4/HLL6; direct serialization need not eliminate all scratch storage.

Start with datasketches/src/hll/union.rs, array4.rs, array6.rs, and array8.rs. This is a follow-up to #317; its current no-argument method is the starting point for the API discussion.

Validation

Compare each output type against to_sketch(type).serialize() for empty, LIST, SET, and dense results, including downsampling, merged estimator state, and HLL4 auxiliary entries. Verify round trips and existing cross-language fixtures. Extend benchmarks/hll/serde.rs to measure time and allocations for all three output types.

Follow CONTRIBUTING.md for changelog entries, integration-test placement, serialization fixtures, and the required checks.

Activity

  1. tisonkun commented on Nov 4, 2026

    @tisonkun
    MemberAuthor

    No need for this API unless we find any performance improvement potential.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions