feat(analytics): sketch analytics for every sketch and framework

CHANGELOG.md

@@ -12,6 +12,32 @@ signals a backwards-compatible change.
 
 ### Added
 
+- **Sketch analytics** (`asap_sketchlib::analytics`): the
+  `SketchAnalytics` trait, with `ErrorBound`, `ErrorMetric` and `NormScale`
+  re-exported at the root, gives a sketch type's theoretical memory, update,
+  query and merge ops, error bound per statistic, and the parameters that
+  meet a target `(ε, δ)`, as `docs/sketch_analytics.md` defines them.
+- **`StatisticKind::QuantileInterpolated`**, answered by `DDSketch`.
+- **`memory_bytes()`** on `CountMinHll`, `MicroCm`, `SpaceSaving`,
+  `ElasticSketch`, `CocoSketch`, `CountMinSketchWithSpaceSaving`,
+  `CountSketchWithSpaceSaving`, `CountMinSketchFold`, `CountSketchFold`,
+  `SetAggregator`, `Bloom`, `UniformSampling`, `UnivMonTwoTier`, `Hydra`,
+  `HydraCounter`, `ExponentialHistogram`, `EhBucketSketch`,
+  `ExponentialHistogramUnivMon`, `UnivMonPool`, `NitroBatch` over a
+  Count-Min or Count Sketch, `TumblingWindow`, `HashSketchEnsemble` and
+  `EnsembleSketch`, and `SpaceSaving::theoretical_memory_bytes`,
+  `Bloom::theoretical_memory_bytes(rows, cols)`,
+  `UnivMonTwoTier::theoretical_memory_bytes` and
+  `UnivMonQ::theoretical_memory_bytes(config)`, which also bounds a cloned
+  or decoded `UnivMonQ`.
+- **`SetAggregator::contains`**: whether a key is in the set.
+- **`TumblingWindowSketch::tumbling_memory_bytes` and `tumbling_cells`**,
+  provided methods `TumblingWindow::memory_bytes` and the `op-count`
+  feature read; the built-in window sketches override both.
+- **`op-count` feature**: `analytics::ops::count(f)` returns the ops `f`
+  performed on this thread, one per counter, register, bucket or item read or
+  written, for testing the analytics op formulas. Off by default, and free
+  when off.
 - **Family-capability table** (`asap_sketchlib::capability`, re-exported at
   the root): `SketchAlgorithm::capability()` gives, for each public sketch
   family, the `StatisticKind`s one state answers, whether states merge and
@@ -31,20 +57,20 @@ signals a backwards-compatible change.
   outside `[0, 1]`. Within `alpha` relative when both neighbours are
   indexable and share a sign; an absolute bound across zero. The legacy
   portable `DdSketch` had it before 0.4.0.
-- **`estimated_memory_bytes(rows, cols)`** on `CountMinSketch` and
+- **`theoretical_memory_bytes(rows, cols)`** on `CountMinSketch` and
   `CountSketch`: the `memory_bytes` of a sketch with those dimensions, the
   counter width taken from the type.
-- **`Kll::estimated_memory_bytes(k, m, n)`**: an upper bound on the bytes a
+- **`Kll::theoretical_memory_bytes(k, m, n)`**: an upper bound on the bytes a
   sketch of `k` and `m` holds after `n` items, merges and a cached quantile.
-- **`DdSketch::memory_bytes`**, and **`DdSketch::estimated_memory_bytes(alpha,
+- **`DdSketch::memory_bytes`**, and **`DdSketch::theoretical_memory_bytes(alpha,
   n, min_magnitude, max_magnitude, negatives)`**: an upper bound on the bytes
   after `n` values in that magnitude range, in any order and through merges.
-- **`memory_bytes` and `estimated_memory_bytes(precision)`** on
+- **`memory_bytes` and `theoretical_memory_bytes(precision)`** on
   `HyperLogLogImpl` and `HyperLogLogHipImpl`: the struct plus `2^precision`
   register bytes, the precision taken at run time.
-- **`Kmv::estimated_memory_bytes(k, distinct)`**: the bytes a sketch of `k`
+- **`Kmv::theoretical_memory_bytes(k, distinct)`**: the bytes a sketch of `k`
   holds after `distinct` distinct inputs, an upper bound for fewer.
-- **`memory_bytes` and `estimated_memory_bytes`** on `CountSketchWithL2Norm`
+- **`memory_bytes` and `theoretical_memory_bytes`** on `CountSketchWithL2Norm`
   (`rows, cols`) and `UnivMon` (`heap_size, rows, cols, layers, key`): the
   pyramid's estimate is the steady state once every layer's heap has churned.
 - **`CountMinSketch::total()`**: the total admitted update weight, row 0's
@@ -53,6 +79,12 @@ signals a backwards-compatible change.
   estimates, by estimate descending, ties by flow id.
 ### Changed
 
+- **Breaking: `CountMinFastOps` and `CountSketchFastOps` require
+  `memory_bytes`**, which `HashSketchEnsemble::memory_bytes` reads.
+- **`estimated_memory_bytes` is renamed `theoretical_memory_bytes`** on
+  every sketch that has it (`CountMinSketch`, `CountSketch`,
+  `CountSketchWithL2Norm`, the WithHeap sketches, `TopKHeap`, `HyperLogLog`,
+  `HyperLogLogHip`, `Kmv`, `Kll`, `DdSketch`, `UnivMon`). Breaking.
 - **`Kmv` builds without the `experimental` feature**, with its ASAPv1 codec.
 - **Breaking: `DdSketch` no longer keeps `sum`, `min` or `max`.** The state is
   the two bucket stores and the zero count. `sum()`, `min()` and `max()` are
@@ -92,7 +124,7 @@ signals a backwards-compatible change.
   saturated and then decreased now reports its true `l2` instead of an
   understatement; `l2`, `l2_squared`, UnivMon's `estimate_l2` and its
   terminal-mode heavy-hitter threshold change only in that case. Each row's
-  accumulator is 8 bytes wider (`memory_bytes`, `estimated_memory_bytes`).
+  accumulator is 8 bytes wider (`memory_bytes`, `theoretical_memory_bytes`).
   `update_weighted_with_hash_without_l2` and `as_storage_mut`, the two ways
   to move counters without the accumulator, are removed; use
   `update_weighted_with_hash`.
@@ -123,30 +155,45 @@ signals a backwards-compatible change.
 
 ### Docs
 
+- **UnivMon's distinct-count bound is `1.35/(kε²)`**, the variance the
+  proof derives, in place of the looser `2/(kε²)`, in the UnivMon and
+  UnivMonTwoTier API pages; the tests check the tighter bound.
 - **A receiver of `f64` deltas matches the producer only up to rounding.**
   For Count-Min, Count Sketch and the two heap kinds, the wire format
   (§3.23), the `DeltaCounter` docs and the API pages now state the bound;
   `i32` / `i64` receivers match exactly. Tests cover both. No behavior change.
 
 ### Fixed
 
+- **Breaking: `CountMinSketchFold::new` and `CountSketchFold::new` panic
+  for `full_cols` above 65536** (`MAX_FULL_COLS`). A full column is stored as
+  `u16`, and a wider sketch folded distinct columns together.
+- **`CountMinSketch` Total is `AsIfBuiltOverUnion` in the capability
+  table**, not `Exact`: with Nitro sampling enabled it is an unbiased
+  estimate.
+- **Breaking: `Bloom::dimensions_for` meets its target.** It returns
+  `Option<(usize, usize)>`, widens the slices until the exact rate
+  `(1 - (1 - 1/cols)^n)^rows` is at most the target, and returns `None` when
+  that needs more than `BLOOM_MAX_BITS`, where it used to return the widest
+  slices that fit. `with_capacity` panics there. `predicted_fpp` uses the
+  exact rate in place of `(1 - e^(-n/cols))^rows`.
 - **`Kll::update` regains most of its pre-0.4.0 speed.** With the lazily grown
   buffer, `compress_while_updating` was inlined into the insert path, which
   then no longer inlined into callers; a stream of updates ran about 10%
   slower. Compaction is kept out of line again: within 4% of 0.3.x on a
   loaded host, at parity on a quiet one.
-- **`TopKHeap::estimated_memory_bytes` and the WithHeap estimates saturate**
+- **`TopKHeap::theoretical_memory_bytes` and the WithHeap estimates saturate**
   at `usize::MAX` for shapes past the address space instead of overflowing.
 - **Decoded top-k heaps reserve as built ones do.** The ASAPv1 state
   decoders of `CountMinSketchWithHeap`, `CountSketchWithHeap`, `UnivMon`,
   `UnivMonTwoTier` and `Kmv` give the heap the slots a built heap with the
   same entries holds, so a decoded sketch grows within
-  `estimated_memory_bytes`. Delta decoders still reserve for their entries
+  `theoretical_memory_bytes`. Delta decoders still reserve for their entries
   alone.
 - **Clones and applied deltas keep the heap's capacity.** `CommonHeap` and
   `TopKHeap` clones keep their source's slots and index buckets, and
   `merge_delta` on the WithHeap sketches refills the sketch's own heap, so
-  both grow within `estimated_memory_bytes`.
+  both grow within `theoretical_memory_bytes`.
 ### Added
 
 - **Checked f64 updates and heap rescoring for the Count-Min and Count Sketch

Cargo.toml

@@ -34,6 +34,8 @@ proptest = "1.11.0"
 default = []
 experimental = []
 octo-runtime = ["core_affinity", "crossbeam-channel"]
+# Counts the ops each call performs, for the analytics op-count tests.
+op-count = []
 
 [lib]
 crate-type = ["rlib"]

README.md

@@ -212,6 +212,7 @@ cargo test --all-features
 - `--all-features` enables every Cargo feature, so all feature-gated code is compiled and tested. The features include:
   - `experimental` — enables sketches and APIs that are still under development and may change without notice.
   - `octo-runtime` — enables the Octo multi-threaded runtime (pulls in `core_affinity` and `crossbeam-channel`).
+  - `op-count` — counts the ops each call performs, for testing the [sketch analytics](docs/sketch_analytics.md) op formulas.
 
 To build or test with a specific feature:
 

docs/api/api_bloom.md

@@ -22,18 +22,18 @@ AND.
 ```rust
 fn with_capacity(expected_items: usize, target_fpp: f64) -> Self
 fn with_dimensions(rows: usize, cols: usize) -> Self
-fn dimensions_for(expected_items: usize, target_fpp: f64) -> (usize, usize)
+fn dimensions_for(expected_items: usize, target_fpp: f64) -> Option<(usize, usize)>
 
 const BLOOM_MAX_SLICES: usize   // MATRIX_MAX_ROWS (20), the most rows ASAPv1 carries
 const BLOOM_MAX_BITS: usize     // 2^31, the sizing ceiling
 ```
 
 `with_capacity` takes `k = round(log2(1/p))` slices, **capped at
 `BLOOM_MAX_SLICES`**, then solves for the slice width that hits `p` with
-exactly that many slices — `cols = -n / ln(1 - p^(1/k))` — and rounds it up to a
-power of two. The rounding keeps the column fold free of modulo bias, so the
-delivered rate lands under the target rather than over it. `dimensions_for`
-reports the choice without allocating.
+exactly that many slices — `cols = -n / ln(1 - p^(1/k))` — rounds it up to a
+power of two, and doubles it until the exact rate `(1 - (1 - 1/cols)^n)^k` is at
+most `p`. The power-of-two width keeps the column fold free of modulo bias.
+`dimensions_for` reports the choice without allocating.
 
 The cap is the reason the width is solved for `k` rather than taken as the
 `k`-optimal split `m/k`. A filter has at most 20 slices, the most an ASAPv1
@@ -42,9 +42,9 @@ bits — about 20% more at `p = 1e-12` — and delivers the target, where the
 `k`-optimal split would have delivered `2^-20` no matter what was asked for.
 
 `with_capacity` and `dimensions_for` **panic on a NaN or infinite
-`target_fpp`**; a finite value outside `(0, 1)` is clamped into it. A target
-needing more than `BLOOM_MAX_BITS` gets the widest slices that fit, and
-`predicted_fpp` then reports the rate those slices deliver.
+`target_fpp`**; a finite value outside `(0, 1)` is clamped into it. For a target
+needing more than `BLOOM_MAX_BITS`, `dimensions_for` returns `None` and
+`with_capacity` panics.
 
 `Default` is 7 x 65536 — 56 KiB of packed bits — which holds 20k distinct keys
 at about 1 false positive in 11,500.
@@ -83,7 +83,7 @@ fn size_in_bytes(&self) -> usize
 fn is_empty(&self) -> bool
 ```
 
-`predicted_fpp` is the model, `(1 - e^(-n/cols))^rows`. `estimated_fpp` reads
+`predicted_fpp` is the model, `(1 - (1 - 1/cols)^n)^rows`. `estimated_fpp` reads
 the bits actually set, so it is the one to trust on a filter whose distinct
 count is unknown.
 

docs/api/api_common_heap.md

@@ -45,10 +45,10 @@ fn len(&self) -> usize
 fn is_empty(&self) -> bool
 fn capacity(&self) -> usize
 fn memory_bytes(&self) -> usize
-fn estimated_memory_bytes(heap_size: usize, key: &DataInput) -> usize
+fn theoretical_memory_bytes(heap_size: usize, key: &DataInput) -> usize
 ```
 
-`memory_bytes` is the bytes the heap holds now. `estimated_memory_bytes` is the
+`memory_bytes` is the bytes the heap holds now. `theoretical_memory_bytes` is the
 bytes a heap of capacity `heap_size` holds in steady state once keys shaped like
 `key` (owned string or byte length; 0 for a number) churn through it, for
 pricing a heap before building one. A heap holds at most this, and exactly this

docs/api/api_count_min_sketch_with_heap.md

@@ -60,11 +60,11 @@ fn heap(&self) -> &TopKHeap<C>
 
 ```rust
 fn memory_bytes(&self) -> usize
-fn estimated_memory_bytes(rows: usize, cols: usize, heap_size: usize, key: &DataInput) -> usize // Vector2D storage
+fn theoretical_memory_bytes(rows: usize, cols: usize, heap_size: usize, key: &DataInput) -> usize // Vector2D storage
 ```
 
 `memory_bytes` is what the instance holds now: the struct, `rows * cols`
-counters and `TopKHeap::memory_bytes`. `estimated_memory_bytes` prices a
+counters and `TopKHeap::memory_bytes`. `theoretical_memory_bytes` prices a
 `new(rows, cols, heap_size)` whose heap is full of keys shaped like `key`.
 
 ## Merge

docs/api/api_count_sketch_with_heap.md

@@ -61,11 +61,11 @@ fn heap(&self) -> &TopKHeap<C>
 
 ```rust
 fn memory_bytes(&self) -> usize
-fn estimated_memory_bytes(rows: usize, cols: usize, heap_size: usize, key: &DataInput) -> usize // Vector2D storage
+fn theoretical_memory_bytes(rows: usize, cols: usize, heap_size: usize, key: &DataInput) -> usize // Vector2D storage
 ```
 
 `memory_bytes` is what the instance holds now: the struct, `rows * cols`
-counters and `TopKHeap::memory_bytes`. `estimated_memory_bytes` prices a
+counters and `TopKHeap::memory_bytes`. `theoretical_memory_bytes` prices a
 `new(rows, cols, heap_size)` whose heap is full of keys shaped like `key`.
 
 ## Merge

docs/api/api_univmon.md

@@ -191,13 +191,13 @@ so the layer increments form a martingale difference sequence: unconditionally
 expectation by `F0 / 2^j` on low layers, and on high layers by
 `(F0 / 2^j)^3 / (5 k^2)`, using `(x-k)^+ <= x(x-1)(x-2)/(5 k^2)` for `k >= 8`.
 Splitting at `2^j` in `[0.45, 0.9) * F0/k` gives
-`Var[Y*_0] <= 1.35 F0^2/k <= 2 F0^2/k`.
+`Var[Y*_0] <= 1.35 F0^2/k`.
 
 Let `C` be the event that the top heap is complete (`top_layer_complete()`),
 and `Y = estimate_cardinality()`. On `C`, `Y = Y*_0`, so Chebyshev gives the
 **joint** bound
 
-`P(|Y - F0| > epsilon F0 AND C) <= d`, where `d = min(1, 2/(k epsilon^2))`.
+`P(|Y - F0| > epsilon F0 AND C) <= d`, where `d = min(1, 1.35/(k epsilon^2))`.
 
 Completeness does not imply conditional unbiasedness. With `F0 = 9`, `k = 8`,
 `L = 2`, any layer-0 heap misses one key. Of 512 equally likely next-bit

docs/api/api_univmon_two_tier.md

@@ -54,7 +54,7 @@ Count Sketch F2 estimate in standard mode, so its `(ε, δ)` bound uses layer 0'
 dimensions (the elephant ones whenever `elephant_layers >= 1`); in terminal mode
 it sums the strata, each held to its own layer's dimensions.
 `estimate_cardinality()` counts every heap key; it misses by more than `ε F0`
-jointly with `top_layer_complete()` with probability at most `2/(kε²)` for
+jointly with `top_layer_complete()` with probability at most `1.35/(kε²)` for
 `k = heap_size >= 8`, independent of the elephant/mouse split.
 
 ## Merge