Skip to content

docs: sketch analytics design - #253

Merged
GordonYuanyc merged 2 commits into
mainfrom
docs/sketch-analytics
Oct 29, 2026
Merged

GordonYuanyc merged 2 commits into
mainfrom
docs/sketch-analytics

Conversation

@GordonYuanyc

@GordonYuanyc GordonYuanyc commented Oct 28, 2026 •

Copy link
Copy Markdown
Collaborator

Design doc: docs/sketch_analytics.md defines the theoretical error, memory and time analytics asap_sketchlib provides for each sketch, as big O and as callable formulas, and the SketchAnalytics trait (theoretical_memory_bytes, theoretical_update_ops, theoretical_query_ops, theoretical_merge_ops, theoretical_error, params_for) that exposes them. Each sketch section maps its symbols to the paper's and to the code's parameter names. Empirical measurements belong to sketch-bench.

Only the doc changes; no code changes.

Where ASAPPlanner's formulas differ today

  • Kmv update is 1 op and memory is 8k bytes; the update scans the kept hashes linearly, and the heap holds up to about 2k slots (KMV update cost is not one operation per row ASAPPlanner#945).
  • Count-Min and Count Sketch with heap update is d + 1; it also re-estimates the key and sifts the heap.
  • A heap slot is 24 bytes; theoretical_memory_bytes counts the entry, digest, index and key bytes.
  • UnivMon update is λ(L)·(d + 1); it also writes row sums, re-estimates the key and updates each layer's heap.
  • Kll update is 1; compaction sorts the bottom level, lg(c_b + 1) on average.
  • Kll memory leaves out merge scratch space and the CDF cache.
  • DdSketch update is 1; it touches six cells.
  • DdSketch memory grows stores in 128-bin chunks; theoretical_memory_bytes uses 2(s + 126) slots per store.
  • A merge is 1 op per input state, whatever the sketch's size.
  • HyperLogLog parameters do not name the estimator, so the Classic bounded contract is also applied to Ertl.
  • HyperLogLog sizing returns precisions asap_sketchlib does not build by default, such as 15 and 17.
  • The Kll contracts apache_datasketches_kll_empirical_99_a9b42755072b and ..._delta_scaled_... become kll_quantile_empirical_99_v1 and kll_quantile_empirical_99_delta_scaled_v1, since the fit is measured on asap_sketchlib.

Open questions

The two at the end of the doc: the Kll rank() contract, and CountSketch sizing under ASAPv1's 20-row limit.

🤖 Generated with Claude Code

GordonYuanyc and others added 2 commits October 29, 2026 13:18
Each sketch's error bound, sizing, memory, and update/query/merge op
counts, as big O and as formulas, plus the SketchAnalytics trait that
exposes them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… counts

Rename the trait functions and the inherent estimated_memory_bytes to
theoretical_*; map each sketch's symbols to the paper and the code's
parameter names; recount ops under one rule and fix the bounds the
technical review found wrong.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@GordonYuanyc
GordonYuanyc force-pushed the docs/sketch-analytics branch from 88ba54a to 7eaf30e Compare October 29, 2026 17:18
@GordonYuanyc
GordonYuanyc marked this pull request as ready for review October 29, 2026 17:18
@GordonYuanyc
GordonYuanyc merged commit ffc65be into main Oct 29, 2026
1 check failed
@GordonYuanyc
GordonYuanyc deleted the docs/sketch-analytics branch October 29, 2026 18:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant