[zzylol]

Part of ProjectASAP/asap_sketchlib#250. Stacked on the DDSketch ASAPv1 codec PR #90 (branch asapv1-only/ddsketch); base this PR on that branch. Mirrors ProjectASAP/asap_sketchlib#252.

Why

asap_sketchlib drops sum, min and max from the DDSketch state. A sum or an extreme is a separate exact aggregate, not a DDSketch readout, and a delta (bucket indices and counts) cannot rebuild them: a delta-fed receiver ended with different scalars than the producer. Per Q-VER the payload change takes a new kind id, 05 01. The Go codec has to follow, or it would encode a kind Rust no longer writes and reject the kind Rust now writes.

What

• asapv1.KindDDSketch is 05 01. The new asapv1.KindDDSketchWithScalars is 05 00 (retired: decoded, never encoded). The registry grows from 37 to 38 ids.
• DDSketch.MarshalASAPv1 writes 05 01: metadata_version 1 and alpha, payload [counts, offset, negative_counts, negative_offset, zero_count]. It no longer switches between metadata versions 1 and 2.
• DDSketch.UnmarshalASAPv1 accepts 05 01, and 05 00 with its eight-field payload. The three scalars are read as float64 and dropped without any check.
• DDSketch is buckets only. Quantile(q) reads the bucket representative of the nearest rank ceil(q*n), clamped to 1..n, for every q. If count ever exceeded the buckets, it returns the last occupied bucket of any sign; before, it returned the last positive one, or 0 when the positive store was empty. q = 0 and q = 1 are within alpha of the true extremes, and nothing is clamped to [min, max].
• Deleted, because no non-test code reads them once the scalars are gone:
  • the fields sum, min and max;
  • the methods Min() and Max(), read only by tests;
  • addToSum and resetScalarsIfEmpty;
  • the scalar decode rules in checkState;
  • the representative cache (reps, repEntry, repCacheSize, cachedValue, fillValue), which existed only to feed sum in addOneFast.
• asapv1_golden moves from a70fa1a to 08440a1 (ddsketch: fixtures for the buckets-only kinds 05 01 / 05 81 sketchlib-golden-bytes#4). That span is 19 golden commits and includes the delta, admission and seed-derivation fixtures.

How

• sketches/DDSketch/asapv1.go: one five-field encode. The decoder uses asapv1.Split and checks the kind against both ids, so it can choose 5 or 8 fields. checkState(counts, offset, negCounts, negOffset, zeroCount) keeps the span, empty-at-offset-0 and total-overflow rules on both encode and decode.
• sketches/DDSketch/DDSketch.go: Update, UpdateGOS, Merge, AddToBucket, addOneFast, ResetBucket, DrainBuckets, Clear, Clone and the constructors no longer touch the scalars. Quantile keeps the negative, then zero, then positive walk.

Key code interfaces added / modified

wire/asapv1/kind.go

KindDDSketch            KindID = "\x05\x01" // was "\x05\x00"
KindDDSketchWithScalars KindID = "\x05\x00" // new: retired kind with sum/min/max; decoded, never encoded

sketches/DDSketch/asapv1.go, sketches/DDSketch/DDSketch.go

// Encodes kind 05 01: alpha in the metadata; payload
// [counts, offset, negative_counts, negative_offset, zero_count].
func (d *DDSketch) MarshalASAPv1() ([]byte, error) // was 05 00, metadata v1 (5 fields) or v2 (8 fields) with sum/min/max
// Decodes 05 01, or the retired 05 00 with sum/min/max read and dropped.
func (d *DDSketch) UnmarshalASAPv1(b []byte) error // was 05 00 only, with scalar order rules
// Nearest-rank bucket representative for every q in [0, 1].
func (d *DDSketch) Quantile(q float64) (float64, bool) // was: q==0 / q==1 returned the exact min / max, others clamped to [min, max]
// func (d *DDSketch) Min() (float64, bool) // removed
// func (d *DDSketch) Max() (float64, bool) // removed

Usage

d := ddsketch.New(0.01)
for _, v := range []float64{12, 15.5, 230, 1200} {
	d.Update(v)
}
b, _ := d.MarshalASAPv1() // kind 05 01, five-field payload
var got ddsketch.DDSketch
_ = got.UnmarshalASAPv1(b)    // also accepts retired 05 00 bytes
hi, _ := got.Quantile(1)      // highest bucket's representative, within 1% of 1200

Before this PR

The Go DDSketch kept sum / min / max, encoded them under 05 00, and returned exact extremes at q = 0 / 1.

After this PR

The Go DDSketch is buckets only and encodes 05 01, byte for byte as Rust does. Retired 05 00 bytes still decode, and re-encode as their ddsketch_buckets_* twins.

Verification

• Commits 2d75744 (codec) and 1d5ca3d (Quantile fallback). go test ./sketches/DDSketch/: 37 top-level tests pass. Failing first: before this change, TestASAPv1Golden failed on the 05 01 fixtures. New or rewritten tests:
  • TestASAPv1Golden: byte-for-byte on ddsketch_buckets_{empty,positive,signed}_a001.
  • TestASAPv1RetiredFixturesDecodeToBucketTwins: ddsketch_* (05 00) decode and re-encode as the ddsketch_buckets_* bytes.
  • TestASAPv1RetiredScalarsAreDropped: NaN, out-of-order and empty-style scalars under 05 00 still decode.
  • TestQuantileEndpointsAreWithinAlpha, TestBucketFedSketchMatchesTheProducer.
  • TestQuantilePastTheBucketsReadsTheLastOccupiedBucket: the fallback.
  • TestSignedFixtureQuantilesMatchRust: compared with the Rust values at q = 0, 0.05, 0.1, 0.4, 0.75, 0.9 and 1, from get_value_at_quantile on ddsketch_buckets_signed_a001.
  • TestASAPv1RejectsCraftedBytes: 05 01 vs 05 00 field counts, metadata version 2 now rejected.
• go test ./wire/asapv1/ -run TestRegistry: passes (38 ids).
• go test ./sketch_framework/OctoSketch/ ./sketch_framework/ExponentialHistogram/: pass. go vet ./...: clean.
• Not passing, and not from this change: wire/asapv1's TestGoldenFixtures and TestEveryFixtureIsWellFormed fail at golden 08440a1 on HLL / Count-Min / Count Sketch bytes (the seed_derivation key) and on delta kinds (01 81, 02 80, 05 80, 05 81, ...). Neither is in this branch's registry, because feat(ddsketch)!: ASAPv1 codec with signed values; drop the protobuf, msgpack, delta and gob formats #90's base predates main's delta-kind registry. With the golden pointer at feat(ddsketch)!: ASAPv1 codec with signed values; drop the protobuf, msgpack, delta and gob formats #90's a70fa1a, the same package passes. tests/cross_language needs XTEST_DIR.

Merge order

1. ddsketch: fixtures for the buckets-only kinds 05 01 / 05 81 sketchlib-golden-bytes#4.
2. The Go main registry PR (branch ddsketch/kind-0501).
3. feat(ddsketch)!: ASAPv1 codec with signed values; drop the protobuf, msgpack, delta and gob formats #90 (asapv1-only/ddsketch) rebased onto Go main, taking main's wire/asapv1/kind.go (the registry / kindEntry shape). That brings in 05 01 / 05 81 / 05 00 / 05 80 and the delta kinds.
4. The codec branch (ddsketch/buckets-0501) restacked on the rebased feat(ddsketch)!: ASAPv1 codec with signed values; drop the protobuf, msgpack, delta and gob formats #90, with its kind.go / kind_test.go hunks dropped.

Limitations

• No 05 81 / 05 80 here: feat(ddsketch)!: ASAPv1 codec with signed values; drop the protobuf, msgpack, delta and gob formats #90's base has no delta kinds and no Go DDSketch delta codec. The Go main registry PR registers them (see Merge order).
• Merge after ddsketch: fixtures for the buckets-only kinds 05 01 / 05 81 sketchlib-golden-bytes#4. Then move the submodule pointer to the merged commit.

🤖 Generated with Claude Code