Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,10 @@ All significant changes to this project will be documented in this file.

## Unreleased

### New features

* Add `UddSketch` behind the `uddsketch` feature for bounded quantile summaries of finite signed values, with weighted updates, compatible merges of sketches or serialized states, a queryable final relative-value error bound, and Rust-specific serialization. With identical parameters and mapping versions, states built through updates, merges, and unmodified serialization roundtrips have the same estimates and serialized bytes regardless of input order, partitioning, or merge tree.

### Improvements

* `BloomFilter::insert` is faster for streams of new items when multiple hashes are configured.
Expand Down
9 changes: 8 additions & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,13 @@ To run only one sketch or workload, pass a Divan filter directly:
cargo bench --package benchmarks --bench benchmarks -- cpc::serde
```

The reproducible quantile comparison includes UDDSketch, the current t-digest, and a pinned t-digest 0.5.0 baseline. It prints tagged CSV with timing, memory estimates, serialized sizes, value and rank errors, and permutation/partition/merge-tree stability. Inputs are synthetic; the experiment excludes database scheduling and I/O. Run it with:

```shell
cargo run --release -p benchmarks --example quantile_comparison
QUANTILE_ROWS=1000000 cargo run --release -p benchmarks --example quantile_comparison
```

## Public API documentation

- Describe types with noun phrases and API behavior with third-person present-tense verbs such as `Creates`, `Updates`, and `Returns`.
Expand Down Expand Up @@ -119,7 +126,7 @@ When adding a case to an existing sketch target, add it to the appropriate modul

Cargo automatically discovers `tests-integration/tests/serde_tests.rs`, which aggregates the sketch-specific modules under `tests-integration/tests/serde_tests`.

To add serialization tests for another sketch, add `serde_tests/<sketch>.rs` and its module declaration in `serde_tests.rs`. Shared path handling belongs in `serde_tests.rs`, and serialization fixtures belong in the appropriate subdirectory under `serde_tests`.
To add serialization tests for another sketch, add `serde_tests/<sketch>.rs` and its module declaration in `serde_tests.rs`. UDDSketch uses local Rust-format fixtures in `serde_tests/uddsketch.rs`; it has no cross-language TCK snapshots. Shared path handling belongs in `serde_tests.rs`, and serialization fixtures belong in the appropriate subdirectory under `serde_tests`.

## Manual workflow (without xtask)

Expand Down
11 changes: 9 additions & 2 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,14 +75,15 @@ Enable multiple algorithms by listing their features together, such as `features
| `tdigest` | `TDigestMut`, `TDigest` | Quantile and rank estimation, with high accuracy near distribution tails. |
| `theta` | `ThetaSketch` and set operations | Distinct counts, set expressions, and Jaccard similarity. |
| `tuple` | `TupleSketch` and set operations | Theta-style keys with user-defined summaries attached to retained entries. |
| `uddsketch` | `UddSketch` | Reproducible quantiles with bounded bucket count and a queryable relative-value error bound. |

See the [API documentation](https://docs.rs/datasketches) for configuration, accuracy guarantees, serialization, and examples for each algorithm. Runnable end-to-end scenarios live in [`datasketches/examples`](datasketches/examples).

## Compatibility

The minimum supported Rust version is 1.86.0. The crate currently supports little-endian targets only.

Supported serialization formats are tested with fixtures produced by Apache DataSketches Java, C++, and Go through the [DataSketches TCK](https://github.com/apache/datasketches-tck).
UDDSketch has a Rust-specific versioned format without cross-language compatibility. Other supported serialization formats are tested with fixtures produced by Apache DataSketches Java, C++, and Go through the [DataSketches TCK](https://github.com/apache/datasketches-tck).

Serialization compatibility does not imply that an ordinary Rust `Hash` implementation produces the same update bytes across languages. When sketches must represent the same inputs across implementations, use `hash::value::{raw_bytes, canonical_float, sign_extend, natural_extend}` (and the constructors within those modules) to select matching input encodings. When interoperating with an implementation that skips empty strings, skip them before updating here as well.

Expand Down
9 changes: 9 additions & 0 deletions benchmarks/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,11 @@ datasketches = { workspace = true, features = [
"tdigest",
"theta",
"tuple",
"uddsketch",
] }
# Pin the comparison baseline so results remain reproducible.
datasketches-v05 = { package = "datasketches", version = "=0.5.0", features = [
"tdigest",
] }
divan = { workspace = true }
rand = { workspace = true }
Expand All @@ -43,5 +48,9 @@ harness = false
name = "benchmarks"
path = "main.rs"

[[example]]
name = "quantile_comparison"
path = "uddsketch/comparison.rs"

[lints]
workspace = true
1 change: 1 addition & 0 deletions benchmarks/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ mod req;
mod tdigest;
mod theta;
mod tuple;
mod uddsketch;

fn main() {
divan::main();
Expand Down
Loading
Loading