diff --git a/CHANGELOG.md b/CHANGELOG.md index 48fb9715..be2161b8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -66,6 +66,15 @@ signals a backwards-compatible change. `0x05 0x80` still decode, with the scalars dropped, and are never encoded. The `sketchlib-golden-bytes` submodule gains the `ddsketch_buckets_*` fixtures. +- **Breaking: `Bloom` no longer keeps an insert counter.** The state is the + bit grid alone, and `inserted()` is removed: no membership answer reads it + (`contains`, `fill_ratio` and `estimated_fpp` read the bits, and + `predicted_fpp` takes its distinct count from the caller). A caller that + needs the number of inserts keeps a separate exact count. +- **ASAPv1: Bloom encodes as `0x17 0x01`.** The payload is `[words]`. The + retired `0x17 0x00`, whose payload also carried `inserted`, still decodes, + with the counter dropped, and is never encoded. The `sketchlib-golden-bytes` + submodule gains the `bloom_bits_*` fixtures. - **Count Sketch estimates no longer allocate.** The per-row estimates a median is taken over are collected on the stack instead of in a fresh `Vec`: up to 20 rows for `Vector2D` (the fast and regular paths), diff --git a/asapv1_golden b/asapv1_golden index f92a8cea..a73d634f 160000 --- a/asapv1_golden +++ b/asapv1_golden @@ -1 +1 @@ -Subproject commit f92a8ceacdaf0f5018182a24a253c9e035e31ae7 +Subproject commit a73d634f6341b6e70e79fe64a9e7b9de82e60130 diff --git a/docs/api/api_bloom.md b/docs/api/api_bloom.md index 209455a6..362ef06b 100644 --- a/docs/api/api_bloom.md +++ b/docs/api/api_bloom.md @@ -61,7 +61,8 @@ fn update_batch(&mut self, values: &[DataInput]) ``` One bit per slice. Re-inserting a key sets nothing new, so the fill is a -function of the distinct keys alone while `inserted()` counts every call. +function of the distinct keys alone. The filter keeps no insert counter; a +caller that needs the number of inserts keeps a separate exact count. ## Query @@ -75,7 +76,6 @@ False is exact: the key was never inserted. True is probabilistic. fn predicted_fpp(&self, distinct_items: usize) -> f64 fn estimated_fpp(&self) -> f64 fn fill_ratio(&self) -> f64 -fn inserted(&self) -> u64 fn rows(&self) -> usize fn cols(&self) -> usize fn bit_capacity(&self) -> usize @@ -104,12 +104,15 @@ fn to_asapv1(&self) -> Result, RmpEncodeError> fn from_asapv1(bytes: &[u8]) -> Result ``` -These produce/consume the **ASAPv1** wire envelope (kind `0x17 0x00`) — see the +These produce/consume the **ASAPv1** wire envelope (kind `0x17 0x01`) — see the [ASAPv1 wire format spec](../asapv1_wire_format.md). The impl is bounded on `Mode: BloomMode` and `H: SketchHasher + HashProfile`, so a hasher that declares no profile cannot serialize at all. `rows`, `cols` and `mode` are carried in the -envelope metadata; the payload is just `[words, inserted]`, the packed bit words -row-major plus the insert counter, with the word stride derived from `cols`. +envelope metadata; the payload is just `[words]`, the packed bit words +row-major, with the word stride derived from `cols`. `from_asapv1` also reads +the retired kind `0x17 0x00`, whose payload `[words, inserted]` carried an +insert counter: the counter is dropped, and the filter re-encodes as +`0x17 0x01`. Nothing encodes `0x17 0x00`. The wire covers the geometries `with_capacity` produces: at most `BLOOM_MAX_SLICES` slices, a power-of-two `cols`, and at most `BLOOM_MAX_BITS` diff --git a/docs/asapv1_wire_format.md b/docs/asapv1_wire_format.md index aa090227..5341da28 100644 --- a/docs/asapv1_wire_format.md +++ b/docs/asapv1_wire_format.md @@ -222,7 +222,8 @@ This registry is the master list of algorithms still to design payloads for, and | `0x14 0x00` | EhBucketSketch | Nested-sketch union | Section 3.18 | implemented | | `0x15 0x00` | ExponentialHistogramUnivMon (`Unstable`) | - | TBD | reserved / not designed | | `0x16 0x00` | OctoSketch | - | TBD | reserved / not designed | -| `0x17 0x00` | Bloom | Partitioned | Section 3.4 | implemented | +| `0x17 0x00` | Bloom | Partitioned, with the `inserted` counter | Section 3.4 | retired: decoded (counter dropped), never encoded | +| `0x17 0x01` | Bloom | Partitioned | Section 3.4 | implemented | | `0x18 0x00` | Space-Saving | Stream-Summary | Section 3.5 | implemented | | `0x19 0x00` | Count-Sketch-with-L2-norm (`CountSketchWithL2Norm`) | Count Sketch with L2 accumulators | Section 3.19 | implemented | | `0x1a 0x00` | UnivMon-Q (`UnivMonQ`) | Quantile pyramid | Section 3.20 | implemented | @@ -489,7 +490,7 @@ On the **fast** path one hash serves the whole matrix wherever it can: the layou In the two packed layouts the rows are **bit fields of one hash**, not independent draws. `rows <= 20` is declared and enforced on **both** doors whichever layout a geometry selects, so one geometry rule covers the format. `with_dimensions` is free to build a wider matrix in memory; such a sketch is rejected on **both** sides, so the format never emits bytes it would refuse to read back. -The same `1 <= rows <= 20` bound is declared by **every matrix-shaped payload**, under whichever name that payload's rows carry: Bloom's slices (`0x17 0x00`, §3.4, as `BLOOM_MAX_SLICES`), Count Sketch (`0x04 0x00`, §3.6), CountMinSketchWithHeap (`0x03 0x00`), CountSketchWithHeap (`0x0a 0x00`), CountMinSketchWithSpaceSaving (`0x1e 0x00`) and CountSketchWithSpaceSaving (`0x1f 0x00`) with their base matrix, CountSketchWithL2Norm (`0x19 0x00`), Coco's table (`0x0c 0x00`), Elastic's light layer (`0x0b 0x00`), Hydra's grid and its matrix counters (`0x07 0x01` / `0x07 0x02`), and every UnivMon (`0x10 0x00`) and Two-tier UnivMon (`0x11 0x00`) layer. +The same `1 <= rows <= 20` bound is declared by **every matrix-shaped payload**, under whichever name that payload's rows carry: Bloom's slices (`0x17 0x01`, §3.4, as `BLOOM_MAX_SLICES`), Count Sketch (`0x04 0x00`, §3.6), CountMinSketchWithHeap (`0x03 0x00`), CountSketchWithHeap (`0x0a 0x00`), CountMinSketchWithSpaceSaving (`0x1e 0x00`) and CountSketchWithSpaceSaving (`0x1f 0x00`) with their base matrix, CountSketchWithL2Norm (`0x19 0x00`), Coco's table (`0x0c 0x00`), Elastic's light layer (`0x0b 0x00`), Hydra's grid and its matrix counters (`0x07 0x01` / `0x07 0x02`), and every UnivMon (`0x10 0x00`) and Two-tier UnivMon (`0x11 0x00`) layer. **CountSketchWithL2Norm never draws per row**, and neither do the UnivMon layers built from it. One `hash128_seeded(seed_index, key)` supplies every row: the column bits at `r * ceil_log2(cols)` and the sign at bit `127 - r`. It still declares `rows <= 20`, and it carries the second bound the packed layout makes binding — `(ceil_log2(cols) + 1) * rows <= 128`, the bits one 128-bit hash has to give out (§3.19). @@ -534,19 +535,18 @@ Both KLL variants (the compact fixed-buffer `Kll` and the growable `KllDynamic`) 10. The per-level sizes must not overflow the weighted count. A layout can satisfy rules 4 to 8 and still park many items at a high compactor level, where each carries weight `2^h`; that overflows `count()` / `rank()` / `cdf()` at query time rather than at decode, so it is rejected here. 11. Compact `Kll` only: `len(items) <= compute_max_capacity(k, m)`, checked **before** the buffer is allocated. `KllDynamic` grows its buffer, so it has no such ceiling. -### 3.4: Bloom payload (`0x17 0x00`) +### 3.4: Bloom payload (`0x17 0x01`) `Bloom` is the *partitioned* filter: `rows` slices of `cols` bits over the same `rows x cols` grid Count-Min probes, one bit set per slice. -Its structural parameters — the grid dimensions (`rows` / `cols`) and the column-derivation **mode** (`"fast"` / `"regular"`) — live in the metadata exactly as Count-Min's do, so the payload is the packed bits plus the one counter nothing else determines: +Its structural parameters — the grid dimensions (`rows` / `cols`) and the column-derivation **mode** (`"fast"` / `"regular"`) — live in the metadata exactly as Count-Min's do, so the payload is the packed bits alone: | Pos | Field | Type | Notes | | ----- | ------- | ------ | ------- | | 0 | `words` | array | the bit grid packed into `u64`s, **row-major**; each row padded out to whole words, so the stride is `ceil(cols / 64)` and the length is `rows * ceil(cols / 64)`. Column `c` of row `r` is bit `c mod 64` (bit 0 the least significant) of `words[r * ceil(cols / 64) + floor(c / 64)]` | -| 1 | `inserted` | u64 | `update` calls the filter has seen, duplicates included | `rows` and `cols` are metadata structural params (Section 2), so the payload omits them and the decoder derives the word stride from `cols`. `mode` records `RegularPath` vs `FastPath` for the same reason Count-Min does: the two paths fold a key into different columns on most geometries, so a filter read back on the wrong path answers no about its own members. -`inserted` is **not** derivable from `words` — re-inserting a key sets no new bit — so it is carried. Everything else the filter reports is derived, and none of the three appears: `fill_ratio` and `estimated_fpp` are recomputed from the bits and the dimensions, and `predicted_fpp` reads no bits at all — it is the closed form `(1 - e^(-n/cols))^rows` over the two dimensions and a distinct-item count `n` the caller supplies, which the filter neither stores nor knows. +Everything the filter reports is derived from the bits and the dimensions, and none of it appears: `fill_ratio` and `estimated_fpp` are recomputed from the bits and the dimensions, and `predicted_fpp` reads no bits at all — it is the closed form `(1 - e^(-n/cols))^rows` over the two dimensions and a distinct-item count `n` the caller supplies, which the filter neither stores nor knows. **Metadata vs payload.** The filter's descriptor keys, in canonical wire order (hash-spec group first, then structural params; Go must mirror it verbatim): @@ -557,13 +557,17 @@ seed_list, matrix_seed_index, rows, cols, mode `matrix_seed_index` is the profile's own row seed index, the one Count-Min carries (§2); `rows` / `cols` / `mode` are Count-Min's structural params minus `counter_type`, which a bit grid has no use for. +**No `inserted` counter.** A count of `update` calls is not a membership readout: `contains`, `fill_ratio` and `estimated_fpp` read the bits, and `predicted_fpp` takes its distinct-item count from the caller. A caller that needs the number of inserts keeps a separate exact count beside the filter. + +**The retired `0x17 0x00`.** It carried `inserted` (u64, `update` calls the filter had seen, duplicates included) as a second payload field after `words`. A decoder still accepts it, reads the counter and drops it, and the result re-encodes as `0x17 0x01`. Nothing encodes `0x17 0x00`; per Q-VER the id is never reused. + **Wire-eligible geometries.** The wire covers what `Bloom::with_capacity` produces: `1 <= rows <= 20` (`BLOOM_MAX_SLICES`, defined as `MATRIX_MAX_ROWS`, the seed list length), a **power-of-two** `cols`, and `rows * cols <= 2^31` (`BLOOM_MAX_BITS`). `Bloom::with_dimensions` is free to build a modulo-folded width, which is outside that subset and rejected on **both** sides, so the format never emits bytes it would refuse to read back. This mirrors Count-Min, whose wire covers i32/i64/f64 counters while the in-memory type stays freer (Section 3.2, Section 5). **Decode rules** (all fail closed, per Section 1's decoder rules): -1. `kind_id` is `0x17 0x00`; any other id is rejected. +1. `kind_id` is `0x17 0x01` with the one-field payload `[words]`, or the retired `0x17 0x00` with its two-field payload `[words, inserted]`; any other id, or a payload of the other kind's length, is rejected. 2. The metadata's hash spec and `mode` must equal the target type's own, with `rows` / `cols` echoed back since those are structural and the filter is sized from them. Cross-profile and cross-mode bytes are rejected. 3. `rows` and `cols` are both non-zero, `cols` is a power of two, `rows <= 20`, and `rows * cols` neither overflows nor exceeds `2^31`. Checked **before** anything is sized from the declared dimensions, so a hostile geometry never reaches an allocation. 4. `len(words) == rows * ceil(cols / 64)` exactly, and that product must not overflow. @@ -991,7 +995,7 @@ A `NaN` sample is legal: `update(f64::NAN)` is legal in memory and `total_cmp` g | ----- | ------- | ------ | ------- | | 0 | `hashes` | array | the retained 64-bit digests, **strictly ascending**; `len(hashes) <= k` | -The payload is a **1-element positional array `[hashes]`**, mirroring Count-Min's `[counts]`. How many digests are retained is `len(hashes)` (derived, so not stored), and the estimate is a closed form over `k` and the largest retained digest, so it is not carried either. KMV keeps no insertion counter, no displaced-weight scalar and no eviction ceiling: unlike Bloom's `inserted` or Space-Saving's `discarded_max`, there is no running state the retained set does not already determine. +The payload is a **1-element positional array `[hashes]`**, mirroring Count-Min's `[counts]`. How many digests are retained is `len(hashes)` (derived, so not stored), and the estimate is a closed form over `k` and the largest retained digest, so it is not carried either. KMV keeps no insertion counter, no displaced-weight scalar and no eviction ceiling: unlike Space-Saving's `discarded_max`, there is no running state the retained set does not already determine. **Metadata vs payload.** `k` is the sketch's one sizing parameter, chosen at construction, so per the config-to-metadata rule it belongs in the descriptor. It is carried as a `u32`; a `k` past that field fails to serialize rather than being truncated. Structural-param order is `... canonical_seed_index, k`. @@ -1529,7 +1533,7 @@ A `"f32"` key is float32 (`0xca`); a decoder rejects any other type there (§3.5 *Key and item arrays.* `keys` and `items` take the element type the metadata names — `key_type` for the `OwnedDataInput` keys of §3.5, §3.7, §3.10, §3.15, §3.16, §3.23, §3.24, §3.25 and Hydra's `cell_key_type` (§3.9), per §3.5's table; `item_type` for KLL's `items` (§3.3) and UniformSampling's `values` (§3.13). Four payloads name no such key and are fixed by their kind_id instead: Coco's `keys` (§3.12) and Elastic's `flow_ids` (§3.11) are nil-or-`str`, SetAggregator's `values` (§3.21) and SetAggregatorDelta's `added` / `removed` (§3.22) are `str`, and Coco's `values` are unsigned integers. -*Scalars.* Float scalars are listed above. Booleans are msgpack `true` / `false` (`0xc3` / `0xc2`) — `evictions` and `stale_copies` (§3.11), `candidate_complete` (§3.15, §3.16), `ever_evicted` (§3.20). Every other scalar is an integer under the family/width rule: `inserted` (§3.4), `total` / `discarded_max` (§3.5, §3.24, §3.25), `offset` / `negative_offset` (§3.8, **signed**), `zero_count` (§3.8), `total_seen` / `rng_state` (§3.13), `bucket_size` / `update_mode` (§3.15, §3.16), `count` / `source_id` / `next_sequence` (§3.20). +*Scalars.* Float scalars are listed above. Booleans are msgpack `true` / `false` (`0xc3` / `0xc2`) — `evictions` and `stale_copies` (§3.11), `candidate_complete` (§3.15, §3.16), `ever_evicted` (§3.20). Every other scalar is an integer under the family/width rule: the retired Bloom `0x17 0x00`'s `inserted` (§3.4), `total` / `discarded_max` (§3.5, §3.24, §3.25), `offset` / `negative_offset` (§3.8, **signed**), `zero_count` (§3.8), `total_seen` / `rng_state` (§3.13), `bucket_size` / `update_mode` (§3.15, §3.16), `count` / `source_id` / `next_sequence` (§3.20). *Nil.* A nullable slot is msgpack **nil** (`0xc0`), never an empty string and never a zero: an unoccupied Coco bucket, a free Elastic heavy bucket, an absent UnivMon-Q `min` / `max`. @@ -1614,7 +1618,7 @@ Go MUST validate the profile it reads (same fail-closed intent as Rust): a sketc - **Q-ORDER**: a payload whose in-memory container order does not survive a rebuild **pins an emitted order**, as a cross-language contract. Heaps, arenas and hash maps all have this property, so the rule reaches Space-Saving's triples (§3.5, §3.24, §3.25), the top-k heap entries (§3.7), KMV's digests (§3.14), UniformSampling's entries (§3.13), UnivMon-Q's candidates and occurrences (§3.20), and every layer- and bucket-indexed array. Two sketches holding the same state then emit the same bytes whatever order they were built in, and a decoded sketch re-serializes byte-identically. The heap key comparator is the same everywhere: descending count, ties by heap key order (§3.5). - **Q-CAP**: **a declared capacity never sizes an allocation.** Every structure is built from the run the payload actually carries, and the declared bound is checked against that run first; a payload naming a capacity of `2^32 - 1` with two entries costs two entries. This holds for every `capacity` (§3.5, §3.24, §3.25), every `heap_size`, KMV's `k`, UniformSampling's `total_seen`, UnivMon-Q's `ordered_samples`, and Hydra's grid and counter dimensions. - **Q-KEYTYPE**: `key_type` names the **exact `OwnedDataInput` variant** and is never widened, wherever heap keys reach the wire — Space-Saving (§3.5), CountMinSketchWithHeap and CountSketchWithHeap (§3.7, §3.10), CountMinSketchWithSpaceSaving and CountSketchWithSpaceSaving (§3.24, §3.25), the UnivMon family (§3.15, §3.16) and Hydra's UnivMon counter (§3.9). A container whose keys mix variants has no single name and **fails to serialize**; `OwnedDataInput::I128` / `U128` have no msgpack integer form and are not wire types. An empty container emits `"u64"`, and a decoder rejects an empty container under any other name, so its `key_type` has one value rather than one per producer. That canonicalizes the key naming and nothing else: a Two-tier UnivMon pyramid still has more than one encoding per state, since layouts differing only in an `elephant_layers` at or above `layers` build identical layers, and an all-elephant pyramid carries its unused mouse dimensions on the wire regardless (§3.16). What holds everywhere is the round trip: a decoded container re-serializes byte-identically. -- **Q-DERIVED**: state a decoder can recompute exactly is **not carried**, and state it cannot is. Recomputed: DDSketch's `count` from its stores and `zero_count` (§3.8), `ExponentialHistogram`'s `l2_mass` and `merge_norm` (§3.17), Elastic's `heavy_buckets` from the declared bucket count (§3.11), every heap's digest index and array order, and every summary a sketch answers from its own state. Carried: Elastic's `stale_copies`, UnivMon's `update_mode`, `candidate_complete` and `bucket_size`, UnivMon-Q's `ever_evicted`, `source_id` and `next_sequence`, UniformSampling's `priorities`, `total_seen` and `rng_state`, and CountSketchWithL2Norm's `l2`. Where a cached derived field exists in memory, the encoder rejects a value that disagrees with what the decoder would recompute. State the sketch's own readouts do not need is not carried either: DDSketch's `sum` / `min` / `max` were (`0x05 0x00`) and are now a separate exact aggregate, since nothing a DDSketch answers reads them (§3.8). +- **Q-DERIVED**: state a decoder can recompute exactly is **not carried**, and state it cannot is. Recomputed: DDSketch's `count` from its stores and `zero_count` (§3.8), `ExponentialHistogram`'s `l2_mass` and `merge_norm` (§3.17), Elastic's `heavy_buckets` from the declared bucket count (§3.11), every heap's digest index and array order, and every summary a sketch answers from its own state. Carried: Elastic's `stale_copies`, UnivMon's `update_mode`, `candidate_complete` and `bucket_size`, UnivMon-Q's `ever_evicted`, `source_id` and `next_sequence`, UniformSampling's `priorities`, `total_seen` and `rng_state`, and CountSketchWithL2Norm's `l2`. Where a cached derived field exists in memory, the encoder rejects a value that disagrees with what the decoder would recompute. State the sketch's own readouts do not need is not carried either: DDSketch's `sum` / `min` / `max` were (`0x05 0x00`) and are now a separate exact aggregate, since nothing a DDSketch answers reads them (§3.8), and so was Bloom's `inserted` counter (`0x17 0x00`), now a separate exact count, since no membership answer reads it (§3.4). - **Q-NIL**: an absent slot is msgpack **nil** (`0xc0`), never a sentinel value. An empty string is a reachable key in Coco (§3.12) and a reachable flow id in Elastic (§3.11), so `""` cannot double as "free"; UnivMon-Q's absent extrema are nil for the same reason (§3.20). Go must model these as nullable, never as a zero value standing in for absent. - **Q-RNG**: RNG state is carried **resumable** when the generator is cheap to reproduce in another language, and opaque otherwise. UniformSampling's `rng_state` is one SplitMix64 word with published constants, so a decoded sampler continues the same draw sequence (§3.13); KLL's `coin` carries the compaction RNG's position and its `compaction_seed` is opaque state Go preserves without interpreting (Q-KLL-SEED). - **Q-UNIVMON-SCOPE**: every layer of `0x10 0x00` and `0x11 0x00` is a CountSketchWithL2Norm with no tag, and another per-layer counter needs a new structural param or a new id. `UnivMonPool` is a `Vec` free list of scratch `UnivMon`s — not of pyramids — with nothing an observer could query, so it is not a wire kind and holds no id. diff --git a/docs/message_pack_format.md b/docs/message_pack_format.md index 3d53a256..77c8d157 100644 --- a/docs/message_pack_format.md +++ b/docs/message_pack_format.md @@ -106,9 +106,10 @@ Converted: `src/sketches/kll_dynamic/wire.rs` dynamic → `0x06 0x01`) — both variants share one payload `[levels, items, coin]` and differ only by `kind_id`. KLL never hashes, so its metadata carries no hash-spec group. -- **Bloom** (`src/sketches/bloom/wire.rs`, kind `0x17 0x00`) — the payload is the - bit grid packed into `u64`s plus the insert count; the wire covers the - geometries `Bloom::with_capacity` produces. +- **Bloom** (`src/sketches/bloom/wire.rs`, kind `0x17 0x01`) — the payload is the + bit grid packed into `u64`s; the wire covers the geometries + `Bloom::with_capacity` produces. The retired `0x17 0x00` also carried an + insert count; it still decodes, with the count dropped, and is never encoded. - **Space-Saving** (`src/sketches/space_saving/wire.rs`, kind `0x18 0x00`) — the payload is the `(key, count, error)` triples plus `total` and `discarded_max`; the Stream-Summary's links and arenas are rebuilt on decode. diff --git a/docs/tests.md b/docs/tests.md index cf5f65b0..84150206 100644 --- a/docs/tests.md +++ b/docs/tests.md @@ -655,15 +655,15 @@ Conformance: [`tests/conformance_kit.rs`](../tests/conformance_kit.rs) runs the | test_name | test_description | what_is_tested | | --- | --- | --- | | `an_inserted_key_is_never_reported_absent` | No false negative, on either hash path. | Fills a filter sized for 20,000 keys at a 1% target and verifies every inserted key reports present on both the `RegularPath` and `FastPath` filters, which decode columns differently. | -| `an_empty_filter_rejects_every_probe` | A fresh filter reports nothing present. | Verifies `is_empty` and a zero insert count, and that probes disjoint from the member set are rejected on both hash paths. | +| `an_empty_filter_rejects_every_probe` | A fresh filter reports nothing present. | Verifies `is_empty`, and that probes disjoint from the member set are rejected on both hash paths. | | `the_measured_false_positive_rate_matches_what_the_sizing_predicts` | The delivered rate honours the target and matches `predicted_fpp`. | Across targets `0.1`, `0.01`, and `0.001`, measures the rate over 200,000 disjoint probes and verifies predicted and measured both stay at or under the target and agree within five binomial standard errors. | -| `the_fill_based_estimate_tracks_the_measured_rate` | `estimated_fpp` reads the bits set, not the insert count. | Inserts the member set three times and verifies `estimated_fpp` still tracks the measured rate within a quarter either way although the insert count has tripled. | +| `the_fill_based_estimate_tracks_the_measured_rate` | `estimated_fpp` reads the bits set, so duplicates do not move it. | Inserts the member set three times and verifies `estimated_fpp` tracks the measured rate within a quarter either way, and that a filter fed the member set once holds the same bits and reports the same `estimated_fpp`. | | `repeated_updates_leave_the_bits_unchanged` | Fill is a function of the distinct keys alone. | Builds one filter with single inserts and one with duplicated inserts over the same keys and verifies identical `count_ones` and `fill_ratio`. | -| `a_union_equals_the_filter_of_the_concatenated_stream` | Merge is an exact union, which is what makes the filter shardable. | Splits a stream across two same-geometry filters, merges them, and verifies the same set bits and insert count as a single-pass filter, plus identical answers on members and non-members alike. | +| `a_union_equals_the_filter_of_the_concatenated_stream` | Merge is an exact union, which is what makes the filter shardable. | Splits a stream across two same-geometry filters, merges them, and verifies the same set bits as a single-pass filter, plus identical answers on members and non-members alike. | | `sizing_meets_the_target_and_is_one_power_of_two_from_missing_it` | `dimensions_for` is pinned to its contract rather than to its own expression. | For `(n, p)` pairs from `(1, 0.5)` through `(10_000, 1e-12)`, verifies every slice is a power of two and the row count stays within `BLOOM_MAX_SLICES`, that the chosen geometry's `predicted_fpp(n)` meets the target, and that halving `cols` misses it. | | `degenerate_geometries_still_answer` | Degenerate geometries answer rather than panic. | Verifies a one-bit filter answers and reports an `estimated_fpp` of `1.0` once set, and that a single-row filter still holds every member it was given. | -| `clearing_restores_an_empty_filter` | `clear` restores the constructed state and keeps the dimensions. | After clearing a filled filter, verifies it is empty with a zero insert count, unchanged rows and columns, and no member reported present. | -| `an_asapv1_round_trip_preserves_every_answer` | An ASAPv1 round trip preserves every answer. | Round-trips through `to_asapv1` / `from_asapv1` and verifies matching dimensions, insert count, and set bits, plus identical answers on members and non-members. | +| `clearing_restores_an_empty_filter` | `clear` restores the constructed state and keeps the dimensions. | After clearing a filled filter, verifies it is empty with unchanged rows and columns, and no member reported present. | +| `an_asapv1_round_trip_preserves_every_answer` | An ASAPv1 round trip preserves every answer. | Round-trips through `to_asapv1` / `from_asapv1` and verifies matching dimensions and set bits, plus identical answers on members and non-members. | | `storage_is_one_bit_per_cell` | Packed storage is one bit per cell, not one byte. | For an `8x4096` filter, verifies `bit_capacity` equals `rows * cols` and `size_in_bytes` is one eighth of it. | | `sizing_never_asks_for_more_slices_than_the_seed_list_has` | The slice count stops at the number of seeds the hasher has. | For 10,000 keys at targets `1e-7`, `1e-9`, and `1e-12`, verifies the row count stays within `BLOOM_MAX_SLICES` and that no two rows of the filled filter hold identical bits. | | `sizing_stays_inside_the_allocation_ceiling` | A target the seed list cannot reach yields the widest geometry that fits. | For `expected_items` up to `usize::MAX` and targets `f64::MIN_POSITIVE`, `0.0`, and `2.0`, verifies `cols` is at least one and a power of two, rows fall in `1..=BLOOM_MAX_SLICES`, and `rows * cols` stays within `BLOOM_MAX_BITS`. | @@ -672,11 +672,11 @@ Conformance: [`tests/conformance_kit.rs`](../tests/conformance_kit.rs) runs the | `a_negative_infinite_target_rate_is_rejected` | A negative infinite target rate panics. | Verifies `with_capacity(1_000, f64::NEG_INFINITY)` panics with "target false-positive rate must be finite". | | `the_two_hash_paths_agree_only_where_the_geometry_gives_each_row_its_own_hash` | Whether the paths land on the same bits is a property of the geometry. | Verifies `8 x 65_536` needs 136 hash bits, so both paths fall back to one seeded hash per row and set identical bits, while `7 x 65_536` fits a packed hash and the paths differ; also verifies that second shape is `BLOOM_DEFAULT_ROWS` by `BLOOM_DEFAULT_COLS`. | | `a_filter_cannot_be_decoded_into_the_other_hash_path` | The ASAPv1 metadata carries which path built the filter. | Verifies regular-path bytes fail to decode as `Bloom`, fast-path bytes fail as `Bloom` and as the unannotated `Bloom`, and fast-path bytes decoded back into `Bloom` still report every member present. | -| `a_fast_path_asapv1_round_trip_preserves_every_answer` | The fast path round-trips like the regular one, tag and all. | Round-trips a fast-path filter sized for 20,000 keys at a 1% target through ASAPv1 and verifies matching dimensions, insert count, and set bits, plus identical answers on 20,000 non-member probes. | -| `a_fast_path_union_equals_the_filter_of_the_concatenated_stream` | Merge is an exact union on the fast path too. | Splits 10,000 keys by parity across two `7 x 2^14` fast-path filters, merges them, and verifies the same set bits and insert count as a single-pass filter, with every key reported present. | +| `a_fast_path_asapv1_round_trip_preserves_every_answer` | The fast path round-trips like the regular one, tag and all. | Round-trips a fast-path filter sized for 20,000 keys at a 1% target through ASAPv1 and verifies matching dimensions and set bits, plus identical answers on 20,000 non-member probes. | +| `a_fast_path_union_equals_the_filter_of_the_concatenated_stream` | Merge is an exact union on the fast path too. | Splits 10,000 keys by parity across two `7 x 2^14` fast-path filters, merges them, and verifies the same set bits as a single-pass filter, with every key reported present. | | `merging_filters_of_different_widths_panics` | A merge across slice widths panics. | Verifies merging a `7 x 2^14` filter with a `7 x 2^13` one panics with "bit matrices must have the same dimensions". | | `merging_filters_of_different_slice_counts_panics` | A merge across slice counts panics. | Verifies merging a `7 x 2^14` filter with an `8 x 2^14` one panics with "bit matrices must have the same dimensions". | -| `update_batch_matches_updating_one_at_a_time` | `update_batch` is the loop, not a different filter. | On both hash paths, verifies a filter filled by `update_batch` over 20,000 keys has the same set bits and insert count as one filled key by key. | +| `update_batch_matches_updating_one_at_a_time` | `update_batch` is the loop, not a different filter. | On both hash paths, verifies a filter filled by `update_batch` over 20,000 keys has the same set bits as one filled key by key. | | `with_capacity_builds_exactly_the_geometry_dimensions_for_reports` | The sizing helper describes the filter `with_capacity` actually builds. | Over `expected_items` `0`, `1`, `17`, `1_000` and `250_000` against targets `0.5`, `0.1`, `0.01`, `1e-4` and `1e-6`, verifies the built filter's rows and columns equal the pair `dimensions_for` reported for the same arguments. | | `the_slice_count_comes_from_the_bit_budget_not_the_target_alone` | The slice count tracks `expected_items` as well as the target rate. | Verifies `dimensions_for` chooses `5` slices at `(1, 0.05)`, `8` at `(1, 1/128)` and `14` at `(2, 1e-4)`, each exactly one over `round(log2(1/p))`; that the two agree for `expected_items` `1_000`, `100_000` and `10_000_000` across six targets; and that `(1_000_000, 1e-12)` saturates at `BLOOM_MAX_SLICES`. | | `a_target_outside_the_open_unit_interval_is_clamped_to_the_endpoints` | A target outside `(0, 1)` clamps onto the interval's own endpoints. | Verifies `0.0`, `-0.0`, `-1.0` and `f64::MIN` all size as `f64::MIN_POSITIVE` does, that `1.0`, `1.5` and `f64::MAX` all size as `1.0 - f64::EPSILON` does, and that both endpoint geometries are non-zero with a power-of-two width. | @@ -689,14 +689,14 @@ Conformance: [`tests/conformance_kit.rs`](../tests/conformance_kit.rs) runs the | `both_rate_reporters_raise_the_per_slice_rate_to_the_slice_count` | Both rate reporters use the slice count as the exponent. | For a `BLOOM_MAX_SLICES x 1024` filter holding 400 keys at a fill strictly inside `(0, 1)`, verifies `estimated_fpp` equals `fill_ratio^rows` and `predicted_fpp(400)` equals `(1 - e^(-400/cols))^rows`. | | `an_empty_filter_predicts_nothing_and_a_saturated_one_predicts_everything` | The two ends of the rate scale are reported exactly. | Verifies an empty filter reports a zero fill ratio, a zero `estimated_fpp` and `predicted_fpp(0) == 0`, then fills a `3x8` filter until its fill ratio is `1.0` and verifies `estimated_fpp` is exactly `1.0`. | | `a_single_update_sets_exactly_one_bit_in_every_slice` | Partitioning: one key touches exactly one bit per slice, whatever the width. | Across widths `1`, `2`, `7`, `64`, `100` and `1024` by row counts `1`, `5` and `BLOOM_MAX_SLICES`, verifies one insert sets exactly `rows` bits on both hash paths, that `fill_ratio` equals `rows / (rows * cols)`, and that the key reads present. | -| `the_insert_counter_counts_calls_and_clear_returns_it_to_zero` | `inserted` counts calls rather than distinct keys, and `clear` resets it with the bits. | Verifies five inserts of one key count `5`, a `update_batch` of three more leaves `8`, and that `clear` returns the count, the bits and the fill ratio to zero. | -| `merging_identical_geometries_sums_the_insert_counts` | A merge adds the counts and keeps both streams' members. | Merges two `5x512` filters holding 30 and 40 distinct keys and verifies the merged count is the sum, all 70 members read present, and the geometry is unchanged. | +| `clear_returns_the_filter_to_empty` | `clear` takes the filter back to the empty state. | After five inserts of one key and an `update_batch` of three more, verifies `clear` leaves no bit set, a zero fill ratio, and the repeated key absent. | +| `merging_identical_geometries_unions_the_members` | A merge keeps both streams' members. | Merges two `5x512` filters holding 30 and 40 distinct keys and verifies all 70 members read present and the geometry is unchanged. | | `merging_mismatched_geometries_panics_instead_of_unioning_a_prefix` | A cross-geometry union panics rather than answering no about its own members. | Verifies merging a `5x512` filter with a `5x256` one panics in `BitMatrix::union_from`. | | `a_zero_slice_width_is_rejected_at_construction` | A zero dimension is refused at construction. | Verifies `with_dimensions(4, 0)` panics rather than handing back a filter every query would panic on. | | `bit_capacity_and_packed_size_describe_the_same_grid` | The addressable grid and the packed storage behind it describe one filter. | Across `1x1`, `3x7`, `5x64`, `7x65` and `BLOOM_MAX_SLICES x 1024`, verifies `bit_capacity` is `rows * cols`, `size_in_bytes` is `rows * ceil(cols / 64) * 8`, and that the bytes cover the addressable bits. | -| `bloom_envelope_structure_and_round_trip` | The envelope frames a filter under kind_id `0x17 0x00`, and the bytes are stable across a round trip. | For a `7 x 2^14` regular-path filter holding 5,000 members, verifies the bytes open with the ASAPv1 magic, `envelope::VERSION`, a `kind_id_len` of `2`, and `0x17 0x00`, then that re-serializing the decoded filter reproduces them exactly and that `inserted`, `fill_ratio`, every member, and 20,000 disjoint probes all answer as the source does. | -| `bloom_fast_path_round_trip` | The fast path round-trips byte for byte. | For a `7 x 2^14` fast-path filter holding 5,000 members, verifies the decoded filter re-serializes to the same bytes, keeps `inserted`, and answers identically on the members and on 20,000 disjoint probes. | -| `bloom_empty_round_trip` | A filter with no inserts round-trips. | Verifies `Bloom::::default()` decodes back empty with a zero insert count, unchanged rows and columns, and identical re-serialized bytes. | +| `bloom_envelope_structure_and_round_trip` | The envelope frames a filter under kind_id `0x17 0x01`, and the bytes are stable across a round trip. | For a `7 x 2^14` regular-path filter holding 5,000 members, verifies the bytes open with the ASAPv1 magic, `envelope::VERSION`, a `kind_id_len` of `2`, and `0x17 0x01`, then that re-serializing the decoded filter reproduces them exactly and that `fill_ratio`, every member, and 20,000 disjoint probes all answer as the source does. | +| `bloom_fast_path_round_trip` | The fast path round-trips byte for byte. | For a `7 x 2^14` fast-path filter holding 5,000 members, verifies the decoded filter re-serializes to the same bytes and answers identically on the members and on 20,000 disjoint probes. | +| `bloom_empty_round_trip` | A filter with no inserts round-trips. | Verifies `Bloom::::default()` decodes back empty with unchanged rows and columns, and identical re-serialized bytes. | | `bloom_cross_mode_rejection` | The metadata `mode` turns a cross-mode decode into an error rather than a filter that denies its own members. | For `7 x 2^12` filters on both paths over the same 5,000 members, verifies the two byte strings differ, regular-path bytes fail to decode as `Bloom`, and fast-path bytes fail as `Bloom`. | | `bloom_custom_hasher_profile_round_trips_and_is_self_describing` | The metadata describes the hasher that built the filter rather than a hardcoded profile. | For a `7 x 2^12` filter over a hasher declaring its own `HashProfile`, verifies the bytes round-trip and preserve every answer, differ from the standard-profile filter's bytes over the same members, and are rejected by a standard-profile decode. | | `bloom_metadata_rejects_unknown_keys` | An unexpected metadata key fails closed. | Encodes the ten `BloomMetadata` fields plus a `bogus_field` as a named map and verifies it does not decode as `BloomMetadata`. | @@ -706,6 +706,8 @@ Conformance: [`tests/conformance_kit.rs`](../tests/conformance_kit.rs) runs the | `bloom_rejects_too_many_rows` | More slices than the seed list has entries is outside the wire-eligible subset. | Verifies a crafted `BLOOM_MAX_SLICES + 1` envelope fails with a message naming `BLOOM_MAX_SLICES`, and that the boundary row count itself decodes. | | `bloom_rejects_oversized_bit_capacity` | The capacity rule fires from the declared dimensions, before any allocation is sized from them. | Verifies a crafted envelope declaring `4 x (BLOOM_MAX_BITS / 2)` while carrying two words fails with a message naming `BLOOM_MAX_BITS`. | | `bloom_rejects_word_count_mismatch` | The word stride is derived from `cols`, so a payload that disagrees is refused in either direction. | Verifies 5-word and 7-word payloads for a `3x128` grid both fail with a "words length" complaint, while the correct 6 words decode. | +| `the_retired_kind_decodes_and_re_encodes_without_the_counter` | The retired `0x17 0x00` still decodes, its `inserted` counter dropped. | Crafts a `2x64` regular-path `0x17 0x00` envelope with words `[0b1011, 2^63]` and `inserted` `300`, and verifies the decode keeps those words and re-encodes as `0x17 0x01` with the payload `[words]` alone, byte for byte equal to a `0x17 0x01` envelope crafted from the same words. | +| `each_kind_rejects_the_other_kinds_payload` | Each kind id fixes its payload length. | Verifies a `0x17 0x01` envelope whose payload still carries a counter, and a `0x17 0x00` envelope whose payload lacks one, both fail to decode. | | `bloom_rejects_padding_bits_set` | Bits parked in a row's trailing slack are rejected. | For a `2x8` grid whose bits `8..64` are padding, verifies payloads setting the first padding bit, the last, and one mid-row are each refused with a "row padding" complaint - such bits are unreachable by `get` but counted by `count_ones`, so they would skew `fill_ratio` and `estimated_fpp` while every membership answer looked fine - and that the same words with the padding clear decode to a `fill_ratio` of `0.5`. | | `bloom_with_capacity_round_trip_meets_predicted_fpp` | A filter sized for a realistic target still delivers it after a round trip. | For 50,000 keys at a `0.001` target, verifies the decoded filter keeps the source's dimensions, reports every member present, and delivers a measured rate over 200,000 disjoint probes within five binomial standard errors of its own `predicted_fpp`. | | `a_packed_64_geometry_gives_each_row_its_own_window` | In the packed 64-bit layout each row reads its own hash window. | Verifies `hash_layout_for_matrix(5, 1024)` is `MatrixHashLayout::Packed64`, that a `5x1024` `Bloom` holding keys `0..300` reports every one present, and that the rate measured over the 200,000 probes from `10_000_000` is within five binomial standard errors of `predicted_fpp(300)` and at or under `0.01`, which one shared window across the five slices would exceed. | @@ -713,7 +715,7 @@ Conformance: [`tests/conformance_kit.rs`](../tests/conformance_kit.rs) runs the | `non_power_of_two_widths_answer_membership_on_both_paths` | A width that is not a power of two folds the same bits on insert and on lookup. | For the geometries `1x1`, `3x65`, `7x100`, `5x127`, `9x129` and `5x1_000`, inserts keys `0..200` into a `Bloom` and a `Bloom` built by `with_dimensions` and verifies neither path loses a single member. | | `the_allocation_ceiling_is_the_documented_number` | The allocation ceiling is a documented number, and a target past it settles on the widest geometry that fits. | Verifies `BLOOM_MAX_BITS` is `1 << 31` and that `Bloom::::dimensions_for(1 << 40, 1e-9)` is `(20, 1 << 26)`, the ceiling divided by the slice count and rounded down to a power of two. | | `the_default_filter_has_the_documented_dimensions` | The default filter is the documented geometry. | Verifies `Bloom::::default()` reports rows and columns equal to `BLOOM_DEFAULT_ROWS` and `BLOOM_DEFAULT_COLS`, and that it is `is_empty`. | -| `known_state_matches_golden` | Known bit grids serialize to the golden bytes. | Builds a `Bloom` of 3x128 with words `[1, 2^63, 0, 0xff, u64::MAX, 0x10000]` and `inserted` `300`, and a `Bloom` of 5x32 with words `[1, 0x80000000, 0, 0xffffffff, 0x10000]` and `inserted` `9`, through `BitMatrix::from_words`, and verifies `to_asapv1` equals `bloom_regular_3x128.hex` and `bloom_fast_5x32.hex`. | +| `known_state_matches_golden` | Known bit grids serialize to the golden bytes. | Builds a `Bloom` of 3x128 with words `[1, 2^63, 0, 0xff, u64::MAX, 0x10000]` and a `Bloom` of 5x32 with words `[1, 0x80000000, 0, 0xffffffff, 0x10000]`, through `BitMatrix::from_words`, and verifies `to_asapv1` equals `bloom_bits_regular_3x128.hex` and `bloom_bits_fast_5x32.hex`. | ### Elastic @@ -2057,7 +2059,7 @@ Test file: [`tests/e2e/wire.rs`](../tests/e2e/wire.rs) | test_name | test_description | what_is_tested | | --- | --- | --- | -| `a_bloom_envelope_round_trips_on_both_paths_and_names_its_kind` | A Bloom envelope names kind_id `0x17 0x00` and preserves every answer on both hash paths. | Fills a `Bloom` and a `Bloom`, each `with_capacity(2_000, 0.01)`, with `DataInput::I64(0..2_000)`, verifies both envelopes carry a `kind_id_len` of `2` and the bytes `0x17 0x00`, that each decode reports all 2,000 members present, that the regular decode answers exactly as its source on the 2,000 non-members `10_000..12_000`, and that `inserted()` is unchanged. | +| `a_bloom_envelope_round_trips_on_both_paths_and_names_its_kind` | A Bloom envelope names kind_id `0x17 0x01` and preserves every answer on both hash paths. | Fills a `Bloom` and a `Bloom`, each `with_capacity(2_000, 0.01)`, with `DataInput::I64(0..2_000)`, verifies both envelopes carry a `kind_id_len` of `2` and the bytes `0x17 0x01`, that each decode reports all 2,000 members present, that the regular decode answers exactly as its source on the 2,000 non-members `10_000..12_000`, and that the regular decode sets as many bits as its source. | | `a_coco_envelope_round_trips_and_names_its_kind` | A Coco envelope names kind_id `0x0c 0x00` and preserves every per-key estimate. | Fills `CocoSketch::::with_dimensions_and_seed(4, 512, 0xC0C0)` with the keys `flow-{k}` of a 20,000-draw Zipf stream (domain 1,024, exponent 1.1, seed `0xC1EA_0001`), verifies the envelope's `kind_id_len` of `2` and bytes `0x0c 0x00`, and that `estimate` on the decode equals the source for all 1,024 domain keys. | | `a_ddsketch_envelope_round_trips_and_names_its_kind` | A DDSketch envelope names kind_id `0x05 0x01` and preserves its quantile answers exactly. | Fills `DdSketch::new(0.01)` with `1.0 + v` over `uniform_u64(20_000, 1_000_000, 0x1DE7_0001)`, verifies the envelope's `kind_id_len` of `2` and bytes `0x05 0x01`, and that the decode matches the source on `get_count()`, `alpha()`, and `get_value_at_quantile` at q `0.0`, `0.1`, `0.5`, `0.9` and `1.0`. | | `a_hydra_envelope_round_trips_and_names_the_counter_it_carries` | A Hydra envelope's kind_id names the counter it carries, and the schema and cells survive. | Builds `Hydra::with_schema(4, 512, ["region", "user"], HydraCounter::CountMinSketch(CountMinSketch::, FastPath>::with_dimensions(3, 512)))`, updates each of the nine region/user pairs 40 times with `DataInput::Str("event")`, verifies the `kind_id_len` of `2` and bytes `0x07 0x01`, an unchanged `schema()`, and equal `query_key` answers under `HydraQuery::Frequency` for all nine fully specified keys. | @@ -2079,8 +2081,9 @@ Test file: [`tests/asapv1_golden.rs`](../tests/asapv1_golden.rs) | test_name | test_description | what_is_tested | | --- | --- | --- | -| `bloom_fast_5x32_matches_golden` | A fast-path Bloom whose rows are narrower than a word serializes to its golden bytes. | Verifies a `Bloom` of 5 slices of 32 bits, words `[1, 0x80000000, 0, 0xffffffff, 0x10000]` and `inserted` `9`, decodes from `asapv1_golden/bloom_fast_5x32.hex` with those dimensions and `inserted`, every bit at `(r, c)` equal to bit `c % 64` of word `r * ceil(cols / 64) + c / 64`, and re-serializes identically. | -| `bloom_regular_3x128_matches_golden` | A regular-path Bloom of two words per row serializes to its golden bytes. | Runs the `bloom_fast_5x32_matches_golden` checks for a `Bloom` of 3 slices of 128 bits, words `[1, 2^63, 0, 0xff, u64::MAX, 0x10000]` and `inserted` `300`, against `asapv1_golden/bloom_regular_3x128.hex`. | +| `bloom_bits_fast_5x32_matches_golden` | A fast-path Bloom whose rows are narrower than a word serializes to its golden bytes. | Verifies a `Bloom` of 5 slices of 32 bits, words `[1, 0x80000000, 0, 0xffffffff, 0x10000]`, decodes from `asapv1_golden/bloom_bits_fast_5x32.hex` with those dimensions, every bit at `(r, c)` equal to bit `c % 64` of word `r * ceil(cols / 64) + c / 64`, and re-serializes identically. | +| `bloom_bits_regular_3x128_matches_golden` | A regular-path Bloom of two words per row serializes to its golden bytes. | Runs the `bloom_bits_fast_5x32_matches_golden` checks for a `Bloom` of 3 slices of 128 bits, words `[1, 2^63, 0, 0xff, u64::MAX, 0x10000]`, against `asapv1_golden/bloom_bits_regular_3x128.hex`. | +| `bloom_retired_fixtures_decode_to_their_bits_twins` | Each retired `17 00` Bloom fixture decodes, its `inserted` counter dropped, and re-encodes as its `17 01` twin. | Runs the same grid checks on `asapv1_golden/bloom_regular_3x128.hex` and `asapv1_golden/bloom_fast_5x32.hex` and verifies each re-encodes to `bloom_bits_regular_3x128.hex` and `bloom_bits_fast_5x32.hex` respectively. | | `cms_f64_fast_2x3_matches_golden` | A known f64 fast-path Count-Min matrix serializes to its checked-in golden bytes. | Verifies `CountMinSketch::, FastPath>::from_storage` over the `2x3` matrix `[[0.0, 1.5, 2.25], [3.75, 4.125, 5.0625]]` serializes to `asapv1_golden/cms_f64_fast_2x3.hex`, and that decoding that fixture recovers the six cells in row-major order and re-serializes to the same bytes. | | `cms_i64_regular_2x3_admission_matches_golden` | A sampled Count-Min state closes its metadata with `admission_rate`. | Verifies the `cms_i64_regular_2x3` matrix with `with_admission_rate(0.1)` serializes to `asapv1_golden/cms_i64_regular_2x3_admission.hex`, and that decoding the fixture recovers the six cells, `admission_rate()` `0.1`, and a byte-identical re-serialization. | | `cms_i64_regular_2x3_matches_golden` | A known i64 regular-path Count-Min matrix serializes to its checked-in golden bytes. | Verifies `CountMinSketch::, RegularPath>::from_storage` over the `2x3` matrix `[[0, 1, 127], [128, 300, 65536]]` serializes to `asapv1_golden/cms_i64_regular_2x3.hex`, and that decoding the fixture reports `rows()` `2`, `cols()` `3`, the six cells in row-major order, and a byte-identical re-serialization. | diff --git a/src/asapv1/kind.rs b/src/asapv1/kind.rs index 2d392865..7c6180ad 100644 --- a/src/asapv1/kind.rs +++ b/src/asapv1/kind.rs @@ -42,7 +42,11 @@ pub(crate) const UNIVMON: &[u8] = &[0x10, 0x00]; pub(crate) const UNIVMON_TWO_TIER: &[u8] = &[0x11, 0x00]; pub(crate) const EXPONENTIAL_HISTOGRAM: &[u8] = &[0x13, 0x00]; pub(crate) const EH_BUCKET_SKETCH: &[u8] = &[0x14, 0x00]; -pub(crate) const BLOOM: &[u8] = &[0x17, 0x00]; +/// Bloom: the packed bit grid (spec §3.4). +pub(crate) const BLOOM: &[u8] = &[0x17, 0x01]; +/// Retired Bloom kind that also carried the `inserted` counter: decoded, with +/// the counter dropped, and never encoded. +pub(crate) const BLOOM_WITH_INSERTED: &[u8] = &[0x17, 0x00]; pub(crate) const SPACE_SAVING: &[u8] = &[0x18, 0x00]; pub(crate) const COUNT_SKETCH_WITH_L2_NORM: &[u8] = &[0x19, 0x00]; pub(crate) const UNIVMON_Q: &[u8] = &[0x1a, 0x00]; diff --git a/src/sketches/bloom.rs b/src/sketches/bloom.rs index 42f7afc5..a4d6332f 100644 --- a/src/sketches/bloom.rs +++ b/src/sketches/bloom.rs @@ -62,7 +62,6 @@ impl BloomMode for FastPath { #[derive(Clone, Debug)] pub struct Bloom { bits: BitMatrix, - inserted: u64, _mode: PhantomData, _hasher: PhantomData, } @@ -85,7 +84,6 @@ impl Bloom { ); Self { bits: BitMatrix::new(rows, cols), - inserted: 0, _mode: PhantomData, _hasher: PhantomData, } @@ -166,12 +164,6 @@ impl Bloom { self.bits.size_in_bytes() } - /// Number of `update` calls this filter has seen, duplicates included. - #[inline(always)] - pub fn inserted(&self) -> u64 { - self.inserted - } - /// Fraction of bits set. pub fn fill_ratio(&self) -> f64 { self.bits.fill_ratio() @@ -182,10 +174,9 @@ impl Bloom { self.bits.count_ones() == 0 } - /// Clears every bit and the insert counter. + /// Clears every bit. pub fn clear(&mut self) { self.bits.clear(); - self.inserted = 0; } /// Read access to the underlying bit grid. @@ -215,7 +206,6 @@ impl Bloom { /// result is exactly the filter the concatenated streams would have built. pub fn merge(&mut self, other: &Self) { self.bits.union_from(&other.bits); - self.inserted = self.inserted.saturating_add(other.inserted); } } // Regular-path membership: one seeded hash per slice. @@ -230,7 +220,6 @@ impl Bloom { let col = ((hashed & LOWER_32_MASK) as usize) % cols; self.bits.set(r, col); } - self.inserted = self.inserted.saturating_add(1); } /// Records every value in `values`. @@ -264,7 +253,6 @@ impl Bloom { let hashed_val = >::hash_for_matrix(&self.bits, value); self.bits .fast_insert(|cell, _, _| *cell = true, (), &hashed_val); - self.inserted = self.inserted.saturating_add(1); } /// Records every value in `values`. @@ -585,68 +573,40 @@ mod tests { } } - /// `inserted` counts calls, not distinct keys — it is the denominator a - /// caller compares against `predicted_fpp`, so silently deduplicating it - /// would make the filter look better sized than it is. `clear` has to take - /// it back down with the bits. + /// `clear` takes the filter back to the empty state: no bit set, a zero + /// fill ratio, and no former member reported present. #[test] - fn the_insert_counter_counts_calls_and_clear_returns_it_to_zero() { + fn clear_returns_the_filter_to_empty() { let mut filter = Bloom::::with_dimensions(4, 256); - assert_eq!(filter.inserted(), 0, "a fresh filter has a non-zero count"); + assert!(filter.is_empty(), "a fresh filter has bits set"); let repeated = DataInput::Str("same"); for _ in 0..5 { filter.update(&repeated); } - assert_eq!( - filter.inserted(), - 5, - "five inserts of one key counted as {}", - filter.inserted() - ); - assert!(!filter.is_empty(), "an inserted-into filter reports empty"); - filter.update_batch(&keys(3)); - assert_eq!( - filter.inserted(), - 8, - "update_batch of three left the count at {}", - filter.inserted() - ); + assert!(!filter.is_empty(), "an inserted-into filter reports empty"); filter.clear(); - assert_eq!( - filter.inserted(), - 0, - "clear left the count at {}", - filter.inserted() - ); assert!(filter.is_empty(), "clear left bits set"); assert_eq!(filter.fill_ratio(), 0.0, "clear left a non-zero fill ratio"); + assert!(!filter.contains(&repeated), "clear left a member present"); } - /// A merge unions the bits, so the merged filter has seen both streams and - /// its count has to say so; a count that stayed put would understate the - /// load the filter is carrying. + /// A merge unions the bits, so the merged filter answers yes for every + /// member of either side and keeps its geometry. #[test] - fn merging_identical_geometries_sums_the_insert_counts() { + fn merging_identical_geometries_unions_the_members() { let mut left = Bloom::::with_dimensions(5, 512); let mut right = Bloom::::with_dimensions(5, 512); - for key in keys(30) { - left.update(&key); + for value in 0..30u64 { + left.update(&DataInput::U64(value)); } for value in 30..70u64 { right.update(&DataInput::U64(value)); } - let expected = left.inserted() + right.inserted(); left.merge(&right); - assert_eq!( - left.inserted(), - expected, - "merged count is {} not {expected}", - left.inserted() - ); for value in 0..70u64 { assert!( left.contains(&DataInput::U64(value)), diff --git a/src/sketches/bloom/wire.rs b/src/sketches/bloom/wire.rs index 5b624655..932ac3d7 100644 --- a/src/sketches/bloom/wire.rs +++ b/src/sketches/bloom/wire.rs @@ -4,14 +4,20 @@ //! ASAPv1 serialization (the metadata/payload DTOs, the kind_id constant, and //! the `to_asapv1` / `from_asapv1` impls) while the //! algorithm lives in the parent module file. Being a descendant module, it -//! reads the sketch's private `bits` / `inserted` fields directly without -//! widening any field visibility. See `docs/asapv1_wire_format.md` §3.4. +//! reads the sketch's private `bits` field directly without widening any field +//! visibility. See `docs/asapv1_wire_format.md` §3.4. //! -//! Bloom is one algorithm — a single kind_id `0x17 0x00`. The structural -//! parameters — the grid dimensions (`rows` / `cols`) and the column-derivation -//! **mode** (fast/regular, the [`BloomMode`] tag the filter already carries) — -//! live in the metadata, so the payload is `[words, inserted]`: the packed bit -//! words row-major, plus the insert counter. +//! The structural parameters — the grid dimensions (`rows` / `cols`) and the +//! column-derivation **mode** (fast/regular, the [`BloomMode`] tag the filter +//! already carries) — live in the metadata, so the payload is `[words]`: the +//! packed bit words row-major. +//! +//! ## Kind ids +//! +//! The filter encodes as `0x17 0x01`: the bit grid alone. The retired +//! `0x17 0x00` also carried an `inserted` counter after the words; it still +//! decodes, with the counter read and dropped, so bytes written before the +//! change stay readable. Nothing encodes it. //! //! ## Wire-eligible geometries //! @@ -27,7 +33,7 @@ use std::marker::PhantomData; use crate::asapv1::envelope; use crate::asapv1::kind; -use crate::asapv1::kind::BLOOM; +use crate::asapv1::kind::{BLOOM, BLOOM_WITH_INSERTED}; use crate::{BitMatrix, HashProfile, SketchHasher}; use super::{BLOOM_MAX_BITS, BLOOM_MAX_SLICES, Bloom, BloomMode}; @@ -76,15 +82,13 @@ fn bloom_metadata(rows: u32, cols: u32, mode: &str) -> BloomMeta } } -/// Bloom payload (ASAPv1 §3.4), a msgpack **array** (`to_vec`, positional): -/// `[words, inserted]`. `words` is the bit grid packed row-major, one row after -/// another, each row padded out to a whole number of `u64`s; the stride is -/// `ceil(cols / 64)`, derived from the metadata dimensions. `inserted` is the -/// insert counter, which no other field determines. +/// Bloom payload (ASAPv1 §3.4, kind `0x17 0x01`), a msgpack **array** +/// (`to_vec`, positional): `[words]`, the bit grid packed row-major, one row +/// after another, each row padded out to a whole number of `u64`s; the stride +/// is `ceil(cols / 64)`, derived from the metadata dimensions. #[derive(Debug, Serialize, Deserialize)] struct BloomPayload { words: Vec, - inserted: u64, } /// Checks a `(rows, cols)` pair against the wire-eligible subset: both non-zero, @@ -117,15 +121,15 @@ fn check_geometry(rows: usize, cols: usize) -> Result<(), String> { } // Wire serialization for the Bloom filter. `wire` is a descendant of the sketch -// module, so these impls read the private `bits` / `inserted` fields and -// construct the struct directly. +// module, so these impls read the private `bits` field and construct the struct +// directly. impl Bloom where Mode: BloomMode, H: SketchHasher + HashProfile, { /// Serializes the filter into an ASAPv1 MessagePack envelope - /// (kind_id `0x17 0x00`). The metadata is derived from the hasher's + /// (kind_id `0x17 0x01`). The metadata is derived from the hasher's /// [`HashProfile`], so it truthfully describes how the filter was hashed. /// /// A geometry outside the wire-eligible subset (see the module docs) is an @@ -141,21 +145,22 @@ where ))?; let payload = rmp_serde::to_vec(&BloomPayload { words: self.bits.words().to_vec(), - inserted: self.inserted, })?; Ok(envelope::encode(kind::BLOOM, &metadata, &payload)) } - /// Deserializes a filter from an ASAPv1 MessagePack envelope. The grid - /// dimensions are read from the (validated) metadata and the word stride is - /// derived from them; the payload carries only the words and the insert - /// counter. Every inconsistency fails closed. + /// Deserializes a filter from an ASAPv1 MessagePack envelope of kind + /// `0x17 0x01`, or of the retired `0x17 0x00`, whose `inserted` counter is + /// read and dropped. The grid dimensions are read from the (validated) + /// metadata and the word stride is derived from them; the payload carries + /// only the words. Every inconsistency fails closed. pub fn from_asapv1(bytes: &[u8]) -> Result { let (kind_id, metadata, payload) = envelope::split(bytes).map_err(RmpDecodeError::Uncategorized)?; - if kind_id != kind::BLOOM { + if kind_id != BLOOM && kind_id != BLOOM_WITH_INSERTED { return Err(RmpDecodeError::Uncategorized(format!( - "Bloom kind_id mismatch: stored {kind_id:?}, expected {BLOOM:?}" + "Bloom kind_id mismatch: stored {kind_id:?}, expected {BLOOM:?} \ + (or the retired {BLOOM_WITH_INSERTED:?})" ))); } let meta: BloomMetadata = envelope::decode_block(metadata)?; @@ -176,7 +181,13 @@ where "Bloom word count overflows: rows={rows}, words_per_row={words_per_row}" )) })?; - let p: BloomPayload = envelope::decode_block(payload)?; + let p: BloomPayload = if kind_id == BLOOM { + envelope::decode_block(payload)? + } else { + // The retired payload `[words, inserted]`: the counter is dropped. + let (words, _inserted): (Vec, u64) = envelope::decode_block(payload)?; + BloomPayload { words } + }; if p.words.len() != expected_words { return Err(RmpDecodeError::Uncategorized(format!( "Bloom words length {} != rows*words_per_row {expected_words}", @@ -202,7 +213,6 @@ where Ok(Bloom { bits: BitMatrix::from_words(rows, cols, p.words) .map_err(RmpDecodeError::Uncategorized)?, - inserted: p.inserted, _mode: PhantomData, _hasher: PhantomData, }) @@ -261,9 +271,9 @@ mod tests { assert_eq!(bytes[7], 2, "kind_id_len"); // Literal, not `kind::BLOOM`: the constant compared against itself pins // nothing, and these bytes are what a Go reader dispatches on. - assert_eq!(&bytes[8..10], &[0x17, 0x00]); + assert_eq!(&bytes[8..10], &[0x17, 0x01]); let (kind_id, metadata, _) = envelope::split(&bytes).expect("split"); - assert_eq!(kind_id, &[0x17, 0x00]); + assert_eq!(kind_id, &[0x17, 0x01]); let meta: BloomMetadata = from_slice(metadata).expect("metadata"); assert_eq!(meta.metadata_version, 1); assert_eq!((meta.rows, meta.cols), (7, 1 << 14)); @@ -275,7 +285,6 @@ mod tests { bytes, "Bloom serialized bytes differed after round trip" ); - assert_eq!(decoded.inserted(), filter.inserted()); assert_eq!(decoded.fill_ratio(), filter.fill_ratio()); assert_answers_preserved(|k| filter.contains(k), |k| decoded.contains(k)); } @@ -290,7 +299,6 @@ mod tests { let decoded = Bloom::::from_asapv1(&bytes).expect("decode"); assert_eq!(decoded.to_asapv1().expect("re-serialize"), bytes); - assert_eq!(decoded.inserted(), filter.inserted()); assert_answers_preserved(|k| filter.contains(k), |k| decoded.contains(k)); } @@ -301,7 +309,6 @@ mod tests { let decoded = Bloom::::from_asapv1(&bytes).expect("decode"); assert!(decoded.is_empty(), "an empty filter decoded with bits set"); - assert_eq!(decoded.inserted(), 0); assert_eq!(decoded.rows(), filter.rows()); assert_eq!(decoded.cols(), filter.cols()); assert_eq!(decoded.to_asapv1().expect("re-serialize"), bytes); @@ -444,13 +451,54 @@ mod tests { /// Builds a real envelope with real metadata around a hand-built payload, /// so the crafted-bytes tests exercise the decoder's own rules rather than /// the envelope's framing. - fn crafted(rows: u32, cols: u32, mode: &str, words: Vec, inserted: u64) -> Vec { + fn crafted(rows: u32, cols: u32, mode: &str, words: Vec) -> Vec { let metadata = rmp_serde::to_vec_named(&bloom_metadata::(rows, cols, mode)).unwrap(); - let payload = rmp_serde::to_vec(&BloomPayload { words, inserted }).unwrap(); + let payload = rmp_serde::to_vec(&BloomPayload { words }).unwrap(); envelope::encode(kind::BLOOM, &metadata, &payload) } + /// Builds an envelope of `kind_id` around the retired two-field payload + /// `[words, inserted]` (2x64, regular path). + fn crafted_with_inserted(kind_id: &[u8], words: Vec, inserted: u64) -> Vec { + let metadata = + rmp_serde::to_vec_named(&bloom_metadata::(2, 64, "regular")).unwrap(); + let payload = rmp_serde::to_vec(&(words, inserted)).unwrap(); + envelope::encode(kind_id, &metadata, &payload) + } + + /// The retired `0x17 0x00` still decodes: the `inserted` counter is read + /// and dropped, the bits are kept, and the filter re-encodes as + /// `0x17 0x01` with the words alone. + #[test] + fn the_retired_kind_decodes_and_re_encodes_without_the_counter() { + let words = vec![0b1011, 1 << 63]; + let retired = crafted_with_inserted(&[0x17, 0x00], words.clone(), 300); + let decoded = Bloom::::from_asapv1(&retired).expect("decode retired"); + assert_eq!(decoded.as_bits().words(), &words[..]); + let re_encoded = decoded.to_asapv1().expect("re-encode"); + assert_eq!(&re_encoded[8..10], &[0x17, 0x01]); + assert_eq!(re_encoded, crafted(2, 64, "regular", words)); + } + + /// The current kind carries the words alone: a `0x17 0x01` payload that + /// still carries a counter is rejected, and so is a retired `0x17 0x00` + /// payload without one. + #[test] + fn each_kind_rejects_the_other_kinds_payload() { + let with_counter = crafted_with_inserted(&[0x17, 0x01], vec![0, 0], 0); + assert!( + Bloom::::from_asapv1(&with_counter).is_err(), + "a 0x17 0x01 payload carrying a counter must be rejected" + ); + let mut without_counter = crafted(2, 64, "regular", vec![0, 0]); + without_counter[9] = 0x00; + assert!( + Bloom::::from_asapv1(&without_counter).is_err(), + "a 0x17 0x00 payload without its counter must be rejected" + ); + } + /// The message a rejected decode fails with, so a test can pin *which* /// rule fired rather than settling for any error at all. fn decode_error(bytes: &[u8]) -> String { @@ -465,11 +513,7 @@ mod tests { fn bloom_rejects_foreign_kind_id() { let metadata = rmp_serde::to_vec_named(&bloom_metadata::(2, 64, "regular")).unwrap(); - let payload = rmp_serde::to_vec(&BloomPayload { - words: vec![0, 0], - inserted: 0, - }) - .unwrap(); + let payload = rmp_serde::to_vec(&BloomPayload { words: vec![0, 0] }).unwrap(); let bytes = envelope::encode(&[0x02, 0x00], &metadata, &payload); assert!( Bloom::::from_asapv1(&bytes).is_err(), @@ -482,7 +526,7 @@ mod tests { #[test] fn bloom_rejects_zero_dimension() { for (rows, cols) in [(0u32, 64u32), (4, 0)] { - let err = decode_error(&crafted(rows, cols, "regular", Vec::new(), 0)); + let err = decode_error(&crafted(rows, cols, "regular", Vec::new())); assert!( err.contains("dimensions must be non-zero"), "rows={rows}, cols={cols} must be rejected by the dimension rule, got {err}" @@ -495,12 +539,12 @@ mod tests { /// rejects it and not the word count. #[test] fn bloom_rejects_non_power_of_two_cols() { - let err = decode_error(&crafted(2, 96, "regular", vec![0; 4], 0)); + let err = decode_error(&crafted(2, 96, "regular", vec![0; 4])); assert!( err.contains("not a power of two"), "a non-power-of-two cols must be rejected as such, got {err}" ); - let ok = crafted(2, 128, "regular", vec![0; 4], 0); + let ok = crafted(2, 128, "regular", vec![0; 4]); assert!( Bloom::::from_asapv1(&ok).is_ok(), "the power-of-two control case must still decode" @@ -513,7 +557,7 @@ mod tests { #[test] fn bloom_rejects_too_many_rows() { let rows = BLOOM_MAX_SLICES + 1; - let err = decode_error(&crafted(rows as u32, 64, "regular", vec![0; rows], 0)); + let err = decode_error(&crafted(rows as u32, 64, "regular", vec![0; rows])); assert!( err.contains("BLOOM_MAX_SLICES"), "rows past BLOOM_MAX_SLICES must be rejected as such, got {err}" @@ -524,7 +568,6 @@ mod tests { 64, "regular", vec![0; BLOOM_MAX_SLICES], - 0, ); assert!(Bloom::::from_asapv1(&ok).is_ok()); } @@ -535,7 +578,7 @@ mod tests { #[test] fn bloom_rejects_oversized_bit_capacity() { let cols = (BLOOM_MAX_BITS / 2) as u32; - let err = decode_error(&crafted(4, cols, "regular", vec![0; 2], 0)); + let err = decode_error(&crafted(4, cols, "regular", vec![0; 2])); assert!( err.contains("BLOOM_MAX_BITS"), "the capacity rule must fire before the word count is even derived, got {err}" @@ -549,13 +592,13 @@ mod tests { // 3 rows x 128 cols needs 3 * 2 = 6 words. for words in [vec![0u64; 5], vec![0u64; 7]] { let len = words.len(); - let err = decode_error(&crafted(3, 128, "regular", words, 0)); + let err = decode_error(&crafted(3, 128, "regular", words)); assert!( err.contains("words length"), "{len} words for a 3x128 grid must be rejected by the stride rule, got {err}" ); } - let ok = crafted(3, 128, "regular", vec![0; 6], 0); + let ok = crafted(3, 128, "regular", vec![0; 6]); assert!(Bloom::::from_asapv1(&ok).is_ok()); } @@ -568,7 +611,7 @@ mod tests { // Bit 8 is the first padding bit and bit 63 the last; both are rejected, // and either row is checked. for words in [vec![0xFF, 1 << 8], vec![1 << 63, 0], vec![0, 1 << 40]] { - let err = decode_error(&crafted(2, 8, "regular", words, 0)); + let err = decode_error(&crafted(2, 8, "regular", words)); assert!( err.contains("row padding"), "bits set in the row padding must be rejected as such, got {err}" @@ -576,12 +619,9 @@ mod tests { } // Same words with the padding clear decode, and report the fill the // eight live bits imply rather than the inflated one. - let clean = crafted(2, 8, "regular", vec![0xFF, 0], 0); + let clean = crafted(2, 8, "regular", vec![0xFF, 0]); let decoded = Bloom::::from_asapv1(&clean).expect("clean decode"); assert_eq!(decoded.fill_ratio(), 0.5); - // Eight bits set against an insert counter of zero: emptiness is a - // property of the bits. - assert_eq!(decoded.inserted(), 0); assert!(!decoded.is_empty(), "a filter with bits set reported empty"); } @@ -627,28 +667,24 @@ mod tests { use crate::asapv1::envelope::hex; use std::marker::PhantomData; - fn bloom( - rows: usize, - cols: usize, - words: &[u64], - inserted: u64, - ) -> Bloom { + fn bloom(rows: usize, cols: usize, words: &[u64]) -> Bloom { Bloom { bits: BitMatrix::from_words(rows, cols, words.to_vec()).expect("valid grid"), - inserted, _mode: PhantomData, _hasher: PhantomData, } } - let regular = bloom::(3, 128, &[1, 1 << 63, 0, 0xff, u64::MAX, 0x10000], 300); + let regular = bloom::(3, 128, &[1, 1 << 63, 0, 0xff, u64::MAX, 0x10000]); let golden = hex(include_str!( - "../../../asapv1_golden/bloom_regular_3x128.hex" + "../../../asapv1_golden/bloom_bits_regular_3x128.hex" )); assert_eq!(regular.to_asapv1().unwrap(), golden); - let fast = bloom::(5, 32, &[1, 0x8000_0000, 0, 0xffff_ffff, 0x10000], 9); - let golden = hex(include_str!("../../../asapv1_golden/bloom_fast_5x32.hex")); + let fast = bloom::(5, 32, &[1, 0x8000_0000, 0, 0xffff_ffff, 0x10000]); + let golden = hex(include_str!( + "../../../asapv1_golden/bloom_bits_fast_5x32.hex" + )); assert_eq!(fast.to_asapv1().unwrap(), golden); } } diff --git a/tests/PBT_COVERAGE.md b/tests/PBT_COVERAGE.md index df856027..00858f4d 100644 --- a/tests/PBT_COVERAGE.md +++ b/tests/PBT_COVERAGE.md @@ -21,15 +21,15 @@ Compared against: the bit grid read via `as_bits`, and the stream's distinct key - `contains(k)` for every inserted `k`; 1-7 slices x 8-255 bits - once `contains(p)`, always `contains(p)`; every probe re-read after every insert - both hash paths, `RegularPath` and `FastPath`, through their own `update` / `contains` - - no member lost; `slices <= bits set <= slices * distinct` (non-empty stream); `fill_ratio() == bits set / bit_capacity()`; `bit_capacity() <= BLOOM_MAX_BITS`; `inserted() == arrivals`. Geometries 1-4 x 1-8 (saturates) and 1-7 x 256-1023 (mostly zero) + - no member lost; `slices <= bits set <= slices * distinct` (non-empty stream); `fill_ratio() == bits set / bit_capacity()`; `bit_capacity() <= BLOOM_MAX_BITS`. Geometries 1-4 x 1-8 (saturates) and 1-7 x 256-1023 (mostly zero) - grid - `grid(S(xs)) == grid(S(perm(xs)))` - - after `clear`: grid, `inserted()`, `is_empty()`, `fill_ratio()` equal a fresh `with_dimensions`; no former member `contains` + - after `clear`: grid, `is_empty()`, `fill_ratio()` equal a fresh `with_dimensions`; no former member `contains` - all bits set: every probe `contains`; `fill_ratio() == 1.0`; `estimated_fpp() == 1.0` - merge - `merge(a, b)` contains every member of `a` and of `b` - `merge(a, b) == merge(b, a)`; `merge(a, a) == a` - - `grid(merge(S(xs), S(ys))) == grid(S(xs ++ ys))`; `inserted()` adds + - `grid(merge(S(xs), S(ys))) == grid(S(xs ++ ys))` - sizing - `dimensions_for(expected, target)`: `1 <= slices <= BLOOM_MAX_SLICES`; width a power of two; `slices * width <= BLOOM_MAX_BITS`. `expected` up to `usize::MAX`; `target` includes `0.0`, `-0.0`, `1.0`, `-1.0`, `MIN_POSITIVE`, `EPSILON`, `MAX`, `MIN` - wire diff --git a/tests/asapv1_golden.rs b/tests/asapv1_golden.rs index 50833521..12fda029 100644 --- a/tests/asapv1_golden.rs +++ b/tests/asapv1_golden.rs @@ -129,8 +129,13 @@ const GOLDEN_CS_ADMISSION: &str = include_str!("../asapv1_golden/cs_i64_regular_ const GOLDEN_DD_ADMISSION: &str = include_str!("../asapv1_golden/ddsketch_signed_a001_admission.hex"); const GOLDEN_KMV: &str = include_str!("../asapv1_golden/kmv_k8.hex"); +// The retired Bloom kind `17 00`, decode-only. const GOLDEN_BLOOM_REGULAR: &str = include_str!("../asapv1_golden/bloom_regular_3x128.hex"); const GOLDEN_BLOOM_FAST: &str = include_str!("../asapv1_golden/bloom_fast_5x32.hex"); +// The current Bloom kind `17 01`. +const GOLDEN_BLOOM_BITS_REGULAR: &str = + include_str!("../asapv1_golden/bloom_bits_regular_3x128.hex"); +const GOLDEN_BLOOM_BITS_FAST: &str = include_str!("../asapv1_golden/bloom_bits_fast_5x32.hex"); const GOLDEN_SPACE_SAVING_I64: &str = include_str!("../asapv1_golden/space_saving_i64_c8.hex"); const GOLDEN_SPACE_SAVING_BYTES: &str = include_str!("../asapv1_golden/space_saving_bytes_c3.hex"); #[cfg(feature = "experimental")] @@ -1782,19 +1787,21 @@ fn kmv_k8_matches_golden() { // --------------------------------------------------------------------------- // Bloom: golden decodes to the known bit grid and round-trips. Encoding the // known grid is checked in `src/sketches/bloom/wire.rs`. Column `c` of row `r` is bit `c % 64` of word -// `r * ceil(cols / 64) + c / 64`. +// `r * ceil(cols / 64) + c / 64`. Each retired `17 00` fixture decodes, its +// `inserted` counter dropped, to the same grid and re-encodes as its +// `bloom_bits_*` twin. // --------------------------------------------------------------------------- +/// Decodes `golden` and checks the decoded grid. A current `17 01` fixture +/// re-serializes to itself; a retired `17 00` fixture re-serializes to `twin`. fn assert_bloom_golden( golden: &str, + twin: &str, (rows, cols): (usize, usize), words: &[u64], - inserted: u64, ) { - let want = decode_hex(golden); - let decoded = Bloom::::from_asapv1(&want).expect("decode"); + let decoded = Bloom::::from_asapv1(&decode_hex(golden)).expect("decode"); assert_eq!((decoded.rows(), decoded.cols()), (rows, cols)); - assert_eq!(decoded.inserted(), inserted); let stride = cols.div_ceil(64); for r in 0..rows { for c in 0..cols { @@ -1802,19 +1809,40 @@ fn assert_bloom_golden( assert_eq!(decoded.as_bits().get(r, c), bit, "bit ({r}, {c})"); } } - assert_eq!(decoded.to_asapv1().expect("re-serialize"), want); + assert_eq!(decoded.to_asapv1().expect("re-serialize"), decode_hex(twin)); +} + +const BLOOM_REGULAR_WORDS: [u64; 6] = [1, 1 << 63, 0, 0xff, u64::MAX, 0x10000]; +const BLOOM_FAST_WORDS: [u64; 5] = [1, 0x8000_0000, 0, 0xffff_ffff, 0x10000]; + +#[test] +fn bloom_bits_regular_3x128_matches_golden() { + let (golden, words) = (GOLDEN_BLOOM_BITS_REGULAR, &BLOOM_REGULAR_WORDS); + assert_bloom_golden::(golden, golden, (3, 128), words); } #[test] -fn bloom_regular_3x128_matches_golden() { - let words = [1, 1 << 63, 0, 0xff, u64::MAX, 0x10000]; - assert_bloom_golden::(GOLDEN_BLOOM_REGULAR, (3, 128), &words, 300); +fn bloom_bits_fast_5x32_matches_golden() { + let (golden, words) = (GOLDEN_BLOOM_BITS_FAST, &BLOOM_FAST_WORDS); + assert_bloom_golden::(golden, golden, (5, 32), words); } +/// Each retired `17 00` fixture decodes to its twin's grid and re-encodes as +/// the twin. #[test] -fn bloom_fast_5x32_matches_golden() { - let words = [1, 0x8000_0000, 0, 0xffff_ffff, 0x10000]; - assert_bloom_golden::(GOLDEN_BLOOM_FAST, (5, 32), &words, 9); +fn bloom_retired_fixtures_decode_to_their_bits_twins() { + assert_bloom_golden::( + GOLDEN_BLOOM_REGULAR, + GOLDEN_BLOOM_BITS_REGULAR, + (3, 128), + &BLOOM_REGULAR_WORDS, + ); + assert_bloom_golden::( + GOLDEN_BLOOM_FAST, + GOLDEN_BLOOM_BITS_FAST, + (5, 32), + &BLOOM_FAST_WORDS, + ); } // --------------------------------------------------------------------------- diff --git a/tests/e2e/membership.rs b/tests/e2e/membership.rs index 634818a3..d64d0856 100644 --- a/tests/e2e/membership.rs +++ b/tests/e2e/membership.rs @@ -99,9 +99,7 @@ fn an_empty_filter_rejects_every_probe() { let regular = Bloom::::with_capacity(MEMBERS as usize, 0.01); let fast = Bloom::::with_capacity(MEMBERS as usize, 0.01); assert!(regular.is_empty()); - assert_eq!(regular.inserted(), 0); assert!(fast.is_empty()); - assert_eq!(fast.inserted(), 0); for key in probes().into_iter().take(10_000) { assert!(!regular.contains(&DataInput::I64(key))); assert!(!fast.contains(&DataInput::I64(key))); @@ -141,8 +139,8 @@ fn the_measured_false_positive_rate_matches_what_the_sizing_predicts() { } } -/// `estimated_fpp` reads the bits actually set rather than the insert count, -/// so it tracks the measured rate on a filter that saw duplicates. +/// `estimated_fpp` reads the bits actually set, so it tracks the measured rate +/// on a filter that saw duplicates. #[test] fn the_fill_based_estimate_tracks_the_measured_rate() { let mut filter = Bloom::::with_capacity(MEMBERS as usize, 0.01); @@ -151,7 +149,6 @@ fn the_fill_based_estimate_tracks_the_measured_rate() { filter.update(&DataInput::I64(key)); } } - assert_eq!(filter.inserted(), 3 * MEMBERS as u64); let measured = false_positive_rate(|k| filter.contains(&DataInput::I64(k))); let estimated = filter.estimated_fpp(); @@ -160,18 +157,17 @@ fn the_fill_based_estimate_tracks_the_measured_rate() { "measured {measured:.5} outside 0.75-1.25x of estimated {estimated:.5}" ); - // The same keys once set the same bits and a third of the inserts, so an - // estimate that read the counter would move and this one does not. + // The same keys once set the same bits from a third of the inserts, so the + // estimate does not move. let mut once = Bloom::::with_capacity(MEMBERS as usize, 0.01); for key in members() { once.update(&DataInput::I64(key)); } assert_eq!(all_bits(once.as_bits()), all_bits(filter.as_bits())); - assert_ne!(once.inserted(), filter.inserted()); assert_eq!( once.estimated_fpp(), estimated, - "estimated_fpp moved with the insert count" + "estimated_fpp moved with the duplicate inserts" ); } @@ -189,8 +185,6 @@ fn repeated_updates_leave_the_bits_unchanged() { assert_eq!(once.as_bits().count_ones(), twice.as_bits().count_ones()); // The same population is not the same bits; compare the grids cell by cell. assert_eq!(all_bits(once.as_bits()), all_bits(twice.as_bits())); - assert_eq!(once.inserted(), 1_000); - assert_eq!(twice.inserted(), 2_000); } /// A union is exact: merging two filters gives the same bits as one filter @@ -212,7 +206,6 @@ fn a_union_equals_the_filter_of_the_concatenated_stream() { left.merge(&right); assert_eq!(left.as_bits().count_ones(), whole.as_bits().count_ones()); - assert_eq!(left.inserted(), whole.inserted()); for key in 0..10_000i64 { assert!(left.contains(&DataInput::I64(key)), "union lost key {key}"); @@ -475,7 +468,6 @@ fn clearing_restores_an_empty_filter() { let (rows, cols) = (filter.rows(), filter.cols()); filter.clear(); assert!(filter.is_empty()); - assert_eq!(filter.inserted(), 0); assert_eq!((filter.rows(), filter.cols()), (rows, cols)); for key in members().into_iter().take(1_000) { assert!(!filter.contains(&DataInput::I64(key))); @@ -491,7 +483,6 @@ fn an_asapv1_round_trip_preserves_every_answer() { assert_eq!(decoded.rows(), filter.rows()); assert_eq!(decoded.cols(), filter.cols()); - assert_eq!(decoded.inserted(), filter.inserted()); assert_eq!( decoded.as_bits().count_ones(), filter.as_bits().count_ones() @@ -627,7 +618,6 @@ fn a_fast_path_asapv1_round_trip_preserves_every_answer() { (decoded.rows(), decoded.cols()), (filter.rows(), filter.cols()) ); - assert_eq!(decoded.inserted(), filter.inserted()); assert_eq!(all_bits(decoded.as_bits()), all_bits(filter.as_bits())); for key in probes().into_iter().take(20_000) { assert_eq!( @@ -654,7 +644,6 @@ fn a_fast_path_union_equals_the_filter_of_the_concatenated_stream() { } left.merge(&right); assert_eq!(all_bits(left.as_bits()), all_bits(whole.as_bits())); - assert_eq!(left.inserted(), whole.inserted()); for key in 0..10_000i64 { assert!(left.contains(&DataInput::I64(key)), "union lost key {key}"); } @@ -688,7 +677,6 @@ fn update_batch_matches_updating_one_at_a_time() { let mut bulk = Bloom::::with_capacity(MEMBERS as usize, 0.01); bulk.update_batch(&batch); assert_eq!(all_bits(bulk.as_bits()), all_bits(one_by_one.as_bits())); - assert_eq!(bulk.inserted(), one_by_one.inserted()); let mut fast_one_by_one = Bloom::::with_capacity(MEMBERS as usize, 0.01); for key in members() { @@ -700,5 +688,4 @@ fn update_batch_matches_updating_one_at_a_time() { all_bits(fast_bulk.as_bits()), all_bits(fast_one_by_one.as_bits()) ); - assert_eq!(fast_bulk.inserted(), fast_one_by_one.inserted()); } diff --git a/tests/e2e/wire.rs b/tests/e2e/wire.rs index e68d4771..08bb32c6 100644 --- a/tests/e2e/wire.rs +++ b/tests/e2e/wire.rs @@ -57,12 +57,12 @@ fn a_bloom_envelope_round_trips_on_both_paths_and_names_its_kind() { } let regular_bytes = regular.to_asapv1().expect("encode regular"); - assert_kind(®ular_bytes, 0x17, 0x00, "Bloom"); + assert_kind(®ular_bytes, 0x17, 0x01, "Bloom"); let decoded_regular = Bloom::::from_asapv1(®ular_bytes).expect("decode regular"); let fast_bytes = fast.to_asapv1().expect("encode fast"); - assert_kind(&fast_bytes, 0x17, 0x00, "Bloom"); + assert_kind(&fast_bytes, 0x17, 0x01, "Bloom"); let decoded_fast = Bloom::::from_asapv1(&fast_bytes).expect("decode fast"); for i in 0..2_000i64 { @@ -83,9 +83,9 @@ fn a_bloom_envelope_round_trips_on_both_paths_and_names_its_kind() { ); } assert_eq!( - decoded_regular.inserted(), - regular.inserted(), - "insert count" + decoded_regular.as_bits().count_ones(), + regular.as_bits().count_ones(), + "bits set" ); } diff --git a/tests/pbt/bloom.rs b/tests/pbt/bloom.rs index 149471e7..04c7e738 100644 --- a/tests/pbt/bloom.rs +++ b/tests/pbt/bloom.rs @@ -99,14 +99,6 @@ macro_rules! mode_laws { "{} path fill ratio is not the bits set over the addressable grid", $path ); - prop_assert_eq!( - filter.inserted(), - $stream.len() as u64, - "{} path counted {} of {} inserts", - $path, - filter.inserted(), - $stream.len() - ); }}; } @@ -277,7 +269,6 @@ proptest! { bloom_bits(&Bloom::with_dimensions(rows, cols)), "clear left a {}x{} filter unlike a fresh one", rows, cols ); - prop_assert_eq!(filter.inserted(), 0, "clear left the insert count standing"); prop_assert!(filter.is_empty(), "a cleared filter reports bits set"); prop_assert_eq!(filter.fill_ratio(), 0.0, "a cleared filter reports a fill"); for v in &stream { @@ -306,12 +297,6 @@ proptest! { "merging {} and {} keys into a {}x{} filter is not the concatenated stream", a.len(), b.len(), rows, cols ); - prop_assert_eq!( - merged.inserted(), - concatenated.len() as u64, - "the merged count is {} over {} inserts", - merged.inserted(), concatenated.len() - ); } // ===== Where the ceilings bind =====