Repository navigation
Conversation
zzylol
force-pushed
the
refactor/1207-pass1
branch
from
November 7, 2026 04:14
d4bde07 to
441f3a9
Compare
zzylol
force-pushed
the
refactor/1207-pass2
branch
from
November 7, 2026 05:10
72a4037 to
24dd00c
Compare
zzylol
force-pushed
the
refactor/1207-pass1
branch
from
November 9, 2026 06:37
2c6d993 to
61f585a
Compare
zzylol
force-pushed
the
refactor/1207-pass2
branch
from
November 9, 2026 07:11
bdf0888 to
bcae9e5
Compare
zzylol
force-pushed
the
refactor/1207-pass1
branch
from
November 9, 2026 16:50
61f585a to
a71e517
Compare
zzylol
force-pushed
the
refactor/1207-pass2
branch
2 times, most recently
from
November 9, 2026 20:52
8107319 to
5d3f6de
Compare
This was referenced Nov 10, 2026
zzylol
force-pushed
the
refactor/1207-pass1
branch
from
November 10, 2026 01:26
a71e517 to
d77278a
Compare
zzylol
force-pushed
the
refactor/1207-pass2
branch
from
November 10, 2026 01:26
5d3f6de to
dea088e
Compare
zzylol
force-pushed
the
refactor/1207-pass1
branch
from
November 10, 2026 13:37
d77278a to
0c11cd7
Compare
zzylol
force-pushed
the
refactor/1207-pass2
branch
from
November 10, 2026 13:37
dea088e to
e98e80b
Compare
zzylol
force-pushed
the
refactor/1207-pass1
branch
from
November 11, 2026 03:35
0c11cd7 to
321a1ae
Compare
…requirements, dead-candidate elimination, chain groups (#1207 PR 4) Pass 2 runs, in order: the general rollup rule with one synthesized finest common group per cluster of equal definitions (replaces scan widening, finer-aggregate reuse and the rollup rules); top-k reuse; reach, caps and readout requirements (one readout candidate per requirement level, written with Memo::set_candidate_input_requirements); dead-candidate elimination with every root cause (PrunedLogicalCandidate); chain groups SummaryTumblingWindowChain[w](X) for the surviving state groups, widths from the gcd closure of the reaching queries' grids, the window and the offsets, then elimination again; and the logical edges between survivors. A merge, and a Filter/TimeRange/TimeShift over states, carries the θ of the states it reads (one candidate per θ, reading only producers with that θ). The metric rule cuts edges from readouts whose metric the consumer's input does not accept (accepted metrics pushed down from the targeted queries), not candidates; a candidate cut off from an input is eliminated too. A whole-result δ with n_q unknown prunes the readouts it reaches (ValuesPerEvaluationUnknown). - stage1-pass2: the entry generate_logical_asap_memo_candidates and the possible edges in lib.rs; rules.rs (Pass2RuleError, add_candidate, add_candidates_per_input_parameters, add_group_computing_itself), rules/{rollup,topk,chains}.rs, requirements.rs, dead_candidates.rs, tests.rs and tests/decided_examples.rs. - types: AccuracyMetric::ALL; RootPruneReason::{HeapSmallerThanTopK, ValuesPerEvaluationUnknown}; MetricRule.accepted_metrics; PruneCause.through_pruned_candidates; Memo::producers; add_candidate checks θ against the group's state (CandidateError::LogicalProperties, candidate::parameters_fit); a candidate's input requirements are part of its identity (hash_consing::same_candidate); LogicalProperties::new accepts any θ on a candidate passing states on; aggregate coverage and selection accessors no longer test-only; origins SummaryInput and WidenedScans deleted. - accuracy: error_propagation(memo, consumer, metric) with the constant contracts (key-only Filter by column lineage, abs / clamp / negation, min / max / avg / count, + - / *), b = 0.1 private; theoretical_guarantee at a failure probability (KLL and KMV evaluated at it, others checked against their own); producer_satisfies_logical_requirements returns the failing root reason; accepted_metrics and readout_metric. Node-tree composition, propagation (with its count-min estimate_has_bound test), budget split, function rules, PropagationStats and the test-only sizing (guarantee/sizing_oracle) deleted; the classic HLL precision tests move into guarantee/hll.rs. - devtools: one pass2 stage; docs follow. - integration-tests: pass1_candidates_sql re-adds a_filter_or_group_key_over_an_estimate_has_no_guarantee.
zzylol
force-pushed
the
refactor/1207-pass2
branch
from
November 11, 2026 18:33
e98e80b to
5ac57fc
Compare
… guarantees no stage reads yet (#1207 PR 4) classic_guarantee, ClassicHllConfidence, classic_failure_probability and the classic statistic helpers, and UnivMon's distinct_guarantee with its incomplete-top-layer statistic, had no caller outside their own tests. PR 8 re-adds what its HLL and UnivMon distinct-count guarantees use.
… error contract (#1207 PR 4) An Aggregate or exact accumulator build gave a count beta = 0 (and min/max/avg beta = 1) without checking where its group-by keys come from. Grouping by an estimate moves rows between groups, so the cap passed down was infinite and no readout was pruned. The keys now have to be copied from a source, as a Filter's predicate columns must.
…dout no query gives a level (#1207 PR 4) A query whose whole-result failure probability has no per-value one marked every readout copy it reached, including the levels other queries wrote, so a readout another query could use was pruned with ValuesPerEvaluationUnknown. Only a readout without any level is pruned now.
…ation grid its width divides (#1207 PR 4) Chain widths are the gcd closure over every query reaching a state group, so a state read every 15 s and every 10 s got 5, 10 and 15 s chains, and the 10 s reader could be given the 15 s chain. The edge from a SummaryTumblingWindowChain into a group whose result depends on the evaluation time now requires the width to divide that group's evaluation grid; the merges over the other widths are pruned.
…er, with the source frontend's comparison semantics (#1207 PR 4) rollup_from added the compensating Filter group before the memo checked the rollup candidate, so a rejected rollup (COUNT(*) WHERE src = 'a' from COUNT(*) GROUP BY src) left a Filter group no group reads. The Filter now stays pending until the rollup over it derives the coarse group's schema (Memo::output_schema_of_pending_expression). The Filter also took PromQL comparison semantics for any group that is not a relation, such as an exact sum state over a SQL table; it now takes the semantics of the frontend that reads the column's source.
… and chain edges as the code does them (#1207 PR 4) The TopKReuse row described a Limit(k_max) over Sort; the rule builds Aggregate[TopK k] over the largest top-k group's ranked rows. A reader without a declared phase has no known evaluation grid, so no chain is built for it, rather than being taken to evaluate at multiples of its interval. The TumblingWindows row now states the chain-width edge rule.
…g: per-series layers and scans listing different labels)
…PromQL scans by lineage A coarse cross-series aggregate over PromQL rows now rolls up from a finer one: the rule writes each member's input out down to the source, each TimeRange, TimeShift and per-series Aggregate an unlifted operator (same range, same per-series function), lifts the scan's label matchers into the selection by source column, and drops the scan's schema when no unlifted operator names a column by position, so scans listing different labels of one metric compare equal. constraint_of is public for the lift.
…chStatistic::for_intent, drop PR numbers Delete CompositionOperator, AccuracyError, the GuaranteeSource variants nothing creates, AccuracyGuarantee::over_scope and approximate_layer_count, and VALUES_PER_RESULT, which only over_scope used. Move sketch_statistic from asap-accuracy's sizing module to SketchStatistic::for_intent and delete sizing.rs. Remove #509 from the Pass 2 crate header and #694 from two comments.
Replace the descriptions of the deleted composition rules, budget split, propagation statistics and sizing module with the error contracts of asap-accuracy and the readout requirement levels Pass 2 sets.
…, remove duplicates (#1207 PR 4) Pass 2: - Pass2RuleError moves to lib.rs; its LogicalProperties variant is gone (Memo::add_candidate is the one θ check). - CandidateEdge and candidate_edges move to edges.rs. - rules::add_candidate is renamed add_candidate_if_output_matches; one distinct_parameters helper and one query_evaluation_grid_ms lookup replace the copies in rules.rs, chains.rs and lib.rs. - requirements.rs becomes readout_requirements.rs and keeps only the memo writes; the caps, query levels and failure-probability split move to asap_accuracy::readout_levels. - Tests: fixtures in test_fixtures.rs on memo_fixtures, one test module per step, without dates or decision codes. Accuracy: - requirements.rs splits into logical_edge.rs (alpha^log) and metric_rule.rs (accepted_metrics, readout_metric, sketch_readout_metric); target_limit and the test-only meets move to guarantee/mod.rs. - One guarantee lookup that takes the failure probability; Hydra reads its subpopulation's at the same probability. The fixed-probability KLL and KMV guarantees are deleted. - readout_caps.rs: readout_levels, caps keyed by metric. Types: - LogicalProperties::new(parameters) is infallible. - CandidateForMemoGroup::new and listed_build_parameters replace the per-crate copies of building a candidate and listing a build's θ. - GroupProperties::sketch_algorithm_of_state moves to group.rs. - MAX_ERROR_LEVELS is one constant beside the parameter lists. - memo_fixtures::state_grouped_by.
…Pass 2 (#1207 PR 4) - end-to-end-accuracy-guarantees.md: the composition rules give way to the error contracts error_propagation has, and where the metric rule, the readout levels and alpha^log use them. - planner-layering.md: the ratio-of-quantiles example's Pass 2 step and the stage table follow the current rules and file names. - accuracy-models.md and asap-aware-mapping-architecture.md: the accuracy crate's new files.
…valuation grid (#1207 PR 4) A chain whose width does not divide the consumer group's evaluation period no longer drops its edges silently: they fail with RootPruneReason::ChainWidthDoesNotDivideEvaluationGrid, so the candidates over that chain trace to it instead of an empty cause list.
…adouts without a guarantee (#1207 PR 4) - A Project's contract judges only the items reading a computed value: copied labels, the time and promql_drop_metric_name of the series identity keep any error; promql_abs/clamp/clamp_min/clamp_max and a negation keep an absolute error; a product with or quotient by a nonzero literal keeps a relative one. abs(count_over_time(m[1m])), -count_over_time(m[1m]) and quantile_over_time(...) / 2 now keep their sketch readouts. - readout_metric no longer needs a finite failure probability (HLL's distinct count is relative). Under a query with a target, an edge into a readout with no metric (UnivMon's distinct count) or whose guarantee's failure probability is unknown (HLL) fails with the new RootPruneReason::NoGuarantee { consumer, input }. - sketch_guarantee is replaced by guarantee_of_parameters (θ's own guarantee) and the δ check in theoretical_guarantee.
…-in, per-query exact-only groups (#1207 PR 4) - Evaluation grids are computed per (query, group) walking down from each root (evaluation_grids.rs): below PromqlSubquery { resolution: Some(r) } the grid is r, with None it is unknown (no chain). Chains and the chain edge check read it, and evaluation_grid_of_group is the corrected grid. - A readout's per-value failure probability is δ / max(n_q, the readout group's values per evaluation), from the new Memo::series_count_scanned_below: max(quantile_over_time(0.9, m[5m])) over 1000 series at δ = 0.01 requires 1e-5 per readout. - A factor-0 contract (count) passes no cap: no infinite requirement. - New LogicalASAPMemoCandidates::exact_only_groups_of_query: per query, the groups it reaches through a consumer with no error contract for its metric, plus the readout groups another query gave a level while its own whole-result δ has no per-value one; closed downward. Stage 3 is to enforce it per query.
… with the frontend's comparison (#1207 PR 4) - A rolled-up Aggregate names each measure as the coarse group's column, so count by (job) rolls up count by (job, instance) as a sum named count. The memo's schema check still rejects a SQL global count over a grouped one (sum over no group is NULL, the count 0). - A compensating filter is built from comparisons with the source frontend's semantics: a PromQL != matcher becomes Compare Ne with PromQL semantics, NOT IN a conjunction of Ne, IN a disjunction of Eq.
…tracts, readout fan-in and exact-only groups (#1207 PR 4)
…ing tests, view coverage in plan (#1207 PR 4)
…rator and plan nodes (#1207 PR 4)
…ers, tests of floor helpers in the test module, updated file paths (#1207 PR 4)
…ted metrics and caps; readout metrics and accepted input metrics computed once per edge pass; readout.rs, binomial.rs, target_limit beside the edge check (#1207 PR 4) - accepted_metrics derives from the readout caps' walk (a reached flag per group); AccuracyTarget::metrics is the one target-metric expansion. - readout_levels keys its result by readout candidate, so Pass 2 no longer detects unrequired readouts a second time. - readout_metric_of_candidate and metrics_accepted_on_inputs are computed once per edge pass / consumer instead of per edge; accepted metrics and evaluation grids once per memo state and passed in. - A readout's summary type is read with GroupProperties::sketch_algorithm_of_state everywhere. - MAX_ERROR_LEVELS moves next to readout_levels; guarantee, hll and univmon are private; kmv_guarantee and meets are inlined. - Docs: a chain's edge fails with ChainWidthDoesNotDivideEvaluationGrid; max_distinct_per_evaluation has no reader yet.
…res in test_support; Pass 2 tests keep what the stage does with the accuracy results (#1207 PR 4)
…d failure probability, the delta grid, top-k and Hydra status, current Pass 2 and accuracy file layout (#1207 PR 4)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
todo!(), so every planning run stops there; the old rules matched shapes pairwise by column name, and θ floors per input mixed ε and δ and could not express a multi-readout allocation.generate_logical_asap_memo_candidates) runs the reuse rules over memo groups (rollup from the finest common group, top-k reuse), gives each readout its required accuracy, eliminates dead candidates with root causes, builds chain groups and the logical edgesquantile_over_time(0.99, m[5m]) / quantile_over_time(0.5, m[5m])at relative 0.05 every 60 s, Pass 2 prunes both KLL readouts (MetricRule, rank error) and the DDSketch builds with α 0.1 and 0.05, gives each DDSketch readout five levels 0.02 … 0.001, and builds one 60 s DDSketch chain with five builds; planning now stops at Stage 2 (PR 5). The snapshot workloadratio-of-quantiles-1h-mvpplanned at 7.481e-06/s with the full stack before this PR's review fixes (D1); it is re-measured when the stack above is restacked.Review guide
main(PR 3, refactor(stage1-pass1): per-computation candidates over memo groups (#1207 PR 3) #1232, is merged); refactor(stage2): physical expansion (decision-tree levels 3–4), physical edges, pruned alternatives; Count-Min weights via column_sign (#1207 PR 5) #1235 (PR 5) builds on it. 25 commits, headc8e67f130.stage1-pass2-reuse-candidates/src/lib.rs(the six steps in order),edges.rs,readout_requirements.rs,dead_candidates.rs,evaluation_grids.rs,exact_only_groups.rs.rules/rollup.rs(add_rollup_candidates),rules/topk.rs,rules/chains.rs.accuracy/src/error_propagation.rs(the contracts),readout_caps.rs(the caps walk,readout_levels),metric_rule.rs(accepted_metrics,metrics_accepted_on_inputs),readout.rs(readout_metric),logical_edge.rs(producer_satisfies_logical_requirements,target_limit).types/src/stages/logical/{candidate,memo,group}/mod.rs(θ on pass-through candidates,Memo::producers,Memo::series_count_scanned_below,same_candidate) andtypes/src/stages/pruning.rs.asap_accuracy::{composition, propagation, propagation_stats, budget, function_rules, requirements, sizing},guarantee/sizing_oracle.rs, the classic HLL and UnivMon distinct-count guarantees, the unused guarantee vocabulary intypes/src/accuracy/guarantee.rs, and the never-setguaranteefield ofOperatorNode,FlatNodeandPhysicalASAPDAGNode(withOperatorNode::with_guarantee; the executor and integration fixtures drop it).requirements.rssplit intological_edge.rs(withtarget_limit),metric_rule.rs,readout.rsandreadout_caps.rs;binomial_upper_tailintoguarantee/binomial.rs;MAX_ERROR_LEVELSfromtypes/src/summary_type/parameter_lists.rsintoreadout_caps.rs; types'group.rs,aggregate_coverage.rsandcolumn_properties.rsintostages/logical/group/,view_coverage.rsintoplan/, inline test modules into siblingtests.rsfiles, hash-consing tests next tohash_consing.rs.tests/{rollup,topk,readout_requirements,dead_candidates,chains,edges,evaluation_grids,exact_only_groups}.rsplustest_fixtures.rs; the accuracy-math tests sit in the accuracy crate next to their files; shared memo builders inasap_types::test_support::memo_builders(pluscompareandarithmeticinmemo_fixtures); devtools drops the two removed origins from its labels.Filter,Projectand reduction contracts that read column lineage (error_propagation); the caps walk and the readout-level grid with D1's extension; the per-value δ split over the readout group's values (series_count_scanned_below);NoGuarantee; two-direction elimination and its traces; θ matching on edges (Memo::producers); the metric rule cutting edges, not candidates, andexact_only_groups_of_query; evaluation grids below subqueries, chain widths (gcd closure) and the chain-edge divisibility check; the rollup pre-check before the compensatingFilterand its frontend semantics.crates/accuracy/src:lib.rs,evidence.rs,error_propagation.rs,readout.rs,metric_rule.rs,readout_caps.rs,logical_edge.rs,guarantee/{mod,binomial,cardinality,cms,count_sketch,ddsketch,hll,kll,univmon}.rs.crates/stage1-pass2-reuse-candidates/src:lib.rs,edges.rs,evaluation_grids.rs,readout_requirements.rs,dead_candidates.rs,exact_only_groups.rs,rules.rs,rules/{rollup,topk,chains}.rs,test_fixtures.rs,tests.rs,tests/{rollup,topk,readout_requirements,dead_candidates,chains,edges,evaluation_grids,exact_only_groups}.rs.Behavior: before → after
Concepts doc "ratio of quantiles" (test
dead_candidates_are_pruned_and_listed_before_chain_groups):quantile_over_time(0.99, m[5m]) / quantile_over_time(0.5, m[5m]), relative error 0.05, every 60 s at phase 0. It serves the design need that Pass 2 enumerates only candidates some query can use, with a reason for every one it removes.todo!(); planning stopsLogicalASAPMemoCandidates(memo, edges, pruned, evaluation grids, exact-only groups per query); planning stops at Stage 2 (todo!of PR 5)MetricRule { producer_metric: Rank, accepted_metrics: [Relative] }; every KLL build through its edge into a KLL readoutBelowRequiredAccuracy; five surviveSummaryTumblingWindowChain[DDSketch, w = 60 s](Scan m)with the 5 surviving builds;SummaryMerge(TimeRange(5m, chain))in the DDSketch state group… > 1over the same ratio would list the ratio group and every group below itInterfaces
Permalinks are at head
c8e67f1308e4dc5acab976a4102b6c64a5523e82.crates/stage1-pass2-reuse-candidates/src/lib.rs/// Stage 1, Pass 2: steps 1–6; `entries[i]` is query i's workload entry (its evaluation grid). /// The planner facade calls it after Pass 1. Signature unchanged; now implemented. pub fn generate_logical_asap_memo_candidates( pass1: Memo, entries: &[QueryWorkloadEntry], evidence: &dyn AccuracyEvidence, ) -> Result<LogicalASAPMemoCandidates, Pass2RuleError>; /// Why Pass 2 failed on a memo: a rule bug or a cyclic memo. Stays; now defined in `lib.rs`. pub enum Pass2RuleError { - /// Never produced. - InvalidGroupReference, - /// Never produced. - TooManyGroups, /// A group, or one of its candidates, reads itself through other groups. CyclicDependencies, + /// Internal bug: a rule built a group expression the node rules reject. + Group(#[from] MemoGroupError), + /// Internal bug: a rule built a candidate expression the node rules reject, or gave it θ its + /// group's output cannot have. + Candidate(CandidateError), }crates/types/src/stages/logical/candidate/mod.rs·memo/mod.rs·group/mod.rs·memo_candidates.rs·crates/types/src/stages/pruning.rsadd_candidatekeeps its signature but now identifies a candidate by expression, θ and input requirements (same_candidate), and rejects a θ that does not fit the group.crates/types/src/ir/operator/node.rs·ir/flat.rs·plan/physical_asap_dag.rs/// A node of the logical operator DAG. Stays. pub struct OperatorNode { pub operator: Operator, pub result_kind: OperatorResultKind, pub schema: Schema, - pub guarantee: Option<AccuracyGuarantee>, } impl OperatorNode { - pub fn with_guarantee(mut self, guarantee: Option<AccuracyGuarantee>) -> Self; } /// An `OperatorNode` whose children are `NodeId`s. Stays. pub struct FlatNode { pub operator: Operator<NodeId>, pub(crate) result_kind: OperatorResultKind, pub(crate) schema: Schema, - pub(crate) guarantee: Option<AccuracyGuarantee>, } /// A node of the installable physical plan. Stays; every other field unchanged. pub struct PhysicalASAPDAGNode { pub id: PhysicalASAPNodeId, pub payload: PhysicalASAPOperatorPayload, pub output_state: ExecutionDataState, pub lifetime: Lifetime, pub parameters: Option<SketchParams>, pub output_schema: Schema, - pub guarantee: Option<AccuracyGuarantee>, pub retention_ms: Option<u64>, pub selected_physical_candidate: Option<PhysicalCandidateId>, pub view_coverage: Option<ViewCoverage>, }The only writer of these fields was the deleted
asap_accuracy::propagation; every remaining site only copied the value, so it was alwaysNone. Wire format: a serialized plan node no longer has theguaranteekey (PhysicalASAPDAGNodeisdeny_unknown_fields, so a plan written before this PR with"guarantee": nullno longer parses); the fixturesplan_wire_version_10.jsonandstored_outputs_on_main.jsondrop it.INSTALLABLE_PLAN_WIRE_VERSIONis not bumped here (see Needs reviewer decision).crates/types/src/ir/properties/selection.rs·stages/logical/group/aggregate_coverage.rs·accuracy/·summary_type/Removed because nothing reads them: composition and budget types,
AccuracyError,over_scopeandapproximate_layer_countand the guarantee variants only the removed composition wrote. The/* N variants */lines stand for whole enums removed with all their variants.crates/accuracy/src/lib.rs·logical_edge.rs·metric_rule.rs·readout.rs·readout_caps.rs·error_propagation.rs·guarantee/mod.rs/// α^log of edge (producer → consumer, input): the readout rule (the producer's own θ, heap ≥ k, /// an evaluable guarantee) or the metric rule; Err names the failing root reason. Was `todo!()`; /// moved from `requirements.rs` into `logical_edge.rs`. The two tables are computed once per edge pass. pub fn producer_satisfies_logical_requirements( memo: &Memo, + readout_metric_of_candidate: &[Vec<Option<AccuracyMetric>>], + metrics_accepted_on_inputs: Option<&[AccuracyMetric]>, producer: CandidateForMemoGroupId, consumer: CandidateForMemoGroupId, input_index_of_consumer: usize, evidence: &dyn AccuracyEvidence, -) -> bool; +) -> Result<(), RootPruneReason>; /// The largest evaluated error (or δ) that meets a required `limit`. Unchanged; moved from /// `requirements.rs` into `logical_edge.rs`. pub fn target_limit(limit: f64) -> f64; +/// Per group, the metrics its output may be off in for the targeted queries reaching it, in +/// `AccuracyMetric::ALL` order; `None` where none reaches (no metric rule). Derived from the +/// readout caps' walk. `order` is `Memo::topological_order`. +pub fn accepted_metrics(memo: &Memo, order: &[MemoGroupId]) -> Vec<Option<Vec<AccuracyMetric>>>; +/// The metrics `consumer` accepts an input's error in: a readout those its output accepts, any +/// other candidate those its contract carries. Pass 2 computes it once per consumer. +pub fn metrics_accepted_on_inputs( + memo: &Memo, + accepted: &[Option<Vec<AccuracyMetric>>], + consumer: CandidateForMemoGroupId, +) -> Option<Vec<AccuracyMetric>>; +/// The metric of reading `statistic` from a state of `summary_type`: its first listed θ's +/// guarantee's, whatever its δ (the estimators' guarantee is the single source). +pub fn readout_metric( + summary_type: &FieldDataType, + statistic: &SketchStatistic, +) -> Option<AccuracyMetric>; +/// Per group, per candidate: its metric when it is a readout of a sketch state, else `None`. +/// Pass 2 computes it once per edge pass. +pub fn readout_metric_of_candidate(memo: &Memo) -> Vec<Vec<Option<AccuracyMetric>>>; +/// The required-accuracy levels of each sketch readout candidate without a requirement yet, with +/// the queries reaching it whose δ has no per-value one there. `order` is `Memo::topological_order`. +pub fn readout_levels( + memo: &Memo, + order: &[MemoGroupId], +) -> BTreeMap<CandidateForMemoGroupId, (Vec<RequiredAccuracy>, Vec<usize>)>; /// g_c's linear bound for errors in `metric` (β and b per input); `None` = no contract (the metric /// rule). A `Filter`, a `Project` or a grouping reads its input group's column lineage in `memo`. Was `todo!()`. pub fn error_propagation( + memo: &Memo, consumer: &Operator<MemoGroupId>, metric: AccuracyMetric, ) -> Option<ErrorPropagation>; +/// The metrics among `metrics` that `consumer`'s contract carries to its inputs. +pub fn metrics_carried( + memo: &Memo, + consumer: &Operator<MemoGroupId>, + metrics: &[AccuracyMetric], +) -> Vec<AccuracyMetric>; -#[derive(Debug, Clone, PartialEq)] +#[derive(Debug)] +#[cfg_attr(test, derive(PartialEq))] pub struct ErrorPropagation { pub factor_per_input: Vec<f64>, pub validity_cap_per_input: Vec<Option<f64>>, } /// The error bound of θ for `statistic` at `failure_probability`; None if θ cannot fail that rarely /// or its δ is unknown. Signature unchanged; was `todo!()`. pub fn theoretical_guarantee( summary_type: &FieldDataType, parameters: &SketchParams, statistic: &SketchStatistic, failure_probability: f64, ) -> Option<AccuracyGuarantee>; -/// The per-algorithm guarantees (`guarantee::hll`, `guarantee::univmon` were public). -pub mod guarantee; +mod guarantee; -/// Node-tree guarantee composition. -pub mod composition; -/// Node-tree guarantee propagation through a DAG. -pub mod propagation; -/// The statistic a summary is read for. -pub mod sizing; -/// The target of one of `approximate_inputs` inputs composing into `target`. -pub fn per_input_target( - target: &AccuracyTarget, - metric: AccuracyMetric, - approximate_inputs: usize, -) -> Option<AccuracyTarget>;Removed: node-tree composition and propagation (
composition,propagation,propagation_stats,budget::per_input_target, privatefunction_rules) are replaced by per-edge error contracts plus readout requirements (agreed 2026-11-06). Also removed: the test-onlyguarantee::sizing_oracle, the classic HLL and UnivMon distinct-count guarantees (a later PR in this stack brings back certified distinct-count guarantees), the fixed-δ KLL and cardinality guarantee lookups (θ's own guarantee plus one δ check intheoretical_guaranteereplace them), the test-onlymeets(its one caller checks the bound withtarget_limit), andPerComputationCandidateError::LogicalProperties(θ is now checked in one place,Memo::add_candidate).ErrorPropagationlostClone(nothing clones it) and keepsPartialEqfor tests only.guaranteeis private because no crate readsguarantee::*beyond the re-exportedtheoretical_guarantee.Usage, end to end (test
dead_candidates_are_pruned_and_listed_before_chain_groupsatc8e67f130):Algorithm changes
Pass 2's steps:
rules/rollup.rsadd_rollup_candidatesSummaryMerge(group_by)for states,sum/min/maxfor finalized sums, counts, minima, maxima, each measure named asTimeRange,TimeShiftand per-series layers (e.g.rate), and scans listing different labels match by column lineagerules/topk.rsreadout_requirements.rs,accuracy::readout_capserror_propagation; each sketch readout gets one candidate per requirement level (single-read query: the cap; composed query: the ε grid × δ grid not looser than the cap and per-value δ); per-value δ = δ / max(dead_candidates.rsLogicalASAPMemoCandidates::prunedlists each with every root cause (one shortest path per failing edge)rules/chains.rsSummaryTumblingWindowChain[w](X)per width (gcd closure ofTimeShift/TimeRangegroups,SummaryMergein the state group;@ Treuses existing chains only;@ end()/@ start()noneedges.rs,evaluation_grids.rs,lib.rsproducer_satisfies_logical_requirements; an edge from a chain into a reader whose evaluation grid (evaluation_grids: gcd of the reaching queries' grids) its width does not divide fails withChainWidthDoesNotDivideEvaluationGridexact_only_groups.rsexact_only_groups_of_query)Rules add only through
find_or_add_group/add_candidateand swallow onlyOutputDiffersFromGroup;add_candidatetreats input requirements as part of a candidate's identity (two readouts with different requirements are different candidates);RELATIVE_OPERAND_VALIDITY_CAP = 0.1is private.theoretical_guaranteeevaluates KLL (rank error scaled below δ = 0.01) and KMV (bisection on ε) at the required δ, and checks every other algorithm's own failure probability against it.Rollup
TimeRange,TimeShiftand per-series layers (Member::see_through_per_series_layers), so equal computations compare equal whatever their keys, label selections and the labels their scans list (scans match by lineage; an operator that names columns by position, such as a measure over a column, still needs equal column lists), and equal intervals; (2) per cluster,Rollup(by keys)(Filter(compensation, G_f))when keys ⊆ and selection ⊆ on keys by lineage. The rollup and its schema are checked over a pendingFilterfirst, so a rollup the memo would reject leaves no compensating group behind.Aggregatenames each measure ascount by (job)rolls upcount by (job, instance)as a sum namedcount(a SQL global count over a grouped one is still rejected by the schema check: a sum over no group is NULL, the count 0). The compensatingFilteris built from comparisons with the source frontend's semantics: a PromQL!=matcher becomesCompare Newith PromQL semantics,NOT INa conjunction ofNe,INa disjunction ofEq.a_coarse_exact_sum_rolls_up_the_finer_sum,a_coarse_count_rolls_up_the_finer_count_as_a_sum,a_filtered_coarse_sketch_state_merges_the_finer_states_through_a_compensating_filter,a_finest_common_group_no_query_asks_for_serves_both,a_narrowing_on_a_column_that_is_no_key_gives_no_rollup,a_compensating_filter_over_a_sql_table_has_sql_semantics,a_not_equal_matcher_compensates_with_promql_semantics,a_rejected_rollup_leaves_no_compensating_filter, and through production PromQL lowering (crates/integration-tests/tests/pass2_rollup_promql.rs):an_instant_sum_rolls_up_from_the_finer_sum_over_a_scan_listing_more_labels,an_instant_count_rolls_up_from_the_finer_count,a_sum_of_rates_rolls_up_from_the_finer_sum_of_the_same_rates,a_label_matcher_becomes_a_compensating_filter_over_the_finer_sum, with the negative casesrates_over_different_ranges_do_not_roll_up,different_per_series_functions_do_not_roll_up,a_standard_deviation_does_not_roll_up,a_finer_sum_narrowed_by_a_matcher_serves_no_wider_sum.withoutgroupings, label filters between per-series layers, regex matchers; no new unionScan; equal intervals. The pre-check covers the schema only. The A filtered query (city = 'NYC') doesn't read its group from a grouped summary another query builds #902 case (city = 'NYC'over the per-city group) is not widened: Rollup rule: a filtered query (city = 'NYC') does not read the per-city group another query builds (#902 regression on the #1207 stack) #1252.Readout requirements, the metric rule and dead candidates
readout_caps::caps: one walk down from the query roots in reverse topological order. A root with a target is reached with its target in each of its metrics (AccuracyTarget::metrics); every candidate of a reached group reaches its inputs and passes each metric its contract carries, with capsreadout_levels: per readout candidate without a requirement, its levels from its group's caps in its metric (single read: the cap at the per-value δ; composed: the ε gridMAX_ERROR_LEVELS× the δ grid 1e-2 … 1e-5 continued in powers of ten down to the first at or below a tenth of the per-value δ, D1).accepted_metrics= the metrics of the same walk per group (None: not reached);metrics_accepted_on_inputsper consumer;readout_metric_of_candidateonce per memo.exact_only_groups_of_query: per query, the metrics every path to a group carries (an intersection, where the metric rule'saccepted_metricsis a union over queries); a group where none is left is exact-only for that query, with every group below it.Projectcontract judges only the items reading a computed value: copied labels, the time andpromql_drop_metric_nameof the series identity keep any error;promql_abs/clamp/clamp_min/clamp_maxand a negation keep an absolute error; a product with or quotient by a nonzero literal keeps a relative one.abs(count_over_time(m[1m])),-count_over_time(m[1m])andquantile_over_time(...) / 2now keep their sketch readouts (key_filters_and_value_preserving_projects_have_unit_contracts,a_function_of_the_value_keeps_the_readouts_below_it).readout_metricno longer needs a finite δ (HLL's distinct count is relative); under a query with a target, an edge into a readout with no metric (UnivMon's distinct count) or whose guarantee's δ is unknown (HLL) fails withNoGuarantee(a_distinct_count_readout_without_an_evaluable_guarantee_serves_no_query_with_a_target).Memo::series_count_scanned_below):max(quantile_over_time(0.9, m[5m]))over 1000 series at δ = 0.01 requires 1e-5 per readout, not 1e-2 (a_max_over_many_readouts_splits_the_failure_probability_over_them).a_count_over_readouts_gives_them_no_level).exact_only_groups_of_query(a_query_comparing_a_shared_ratio_reads_it_only_exactly,a_readout_another_query_gives_a_level_survives_unknown_values_per_evaluation).a_reduction_keyed_by_an_estimate_has_no_contract).ValuesPerEvaluationUnknownprunes only a readout that no query gives a level; a readout another query reaches with a knowna_readout_another_query_gives_a_level_survives_unknown_values_per_evaluation).dead_candidates_are_pruned_and_listed_before_chain_groups,a_single_readout_requires_the_query_target, and in the accuracy cratea_whole_result_failure_probability_is_split_per_value,a_ratio_reads_no_rank_error,a_ratio_whose_per_value_failure_probability_is_below_the_grid_splits_it_over_its_readouts,a_max_over_many_readouts_splits_the_failure_probability_over_them,a_count_over_readouts_gives_them_no_level.exact_only_groups_of_query.Evaluation grids and chain groups
evaluation_grids.rs): belowPromqlSubquery { resolution: Some(r) }the grid is r, withNoneit is unknown (no chain);evaluation_grid_of_groupis the gcd over the reaching queries. Range-free chain groups after the reach; widths include offsets; built in two phases (all chain builds, then theTimeShift/TimeRange/merge groups) so a chain shared by state groups offers each every θ it has; a chain's edges into a reader whose own evaluation grid the chain's width does not divide fail withChainWidthDoesNotDivideEvaluationGrid { consumer, input, tumbling_window_width_ms, evaluation_period_ms }(a 15 s chain never serves a reader every 10 s, which gave wrong answers before); dead-candidate elimination runs again after chain groups.a_chain_whose_width_does_not_divide_the_readers_grid_fails_its_edge,an_offset_enters_the_gcd_of_the_widths,one_range_free_chain_group_serves_every_range_and_offset,chain_groups_run_after_reach_so_a_rolled_up_query_adds_its_grid,an_absolute_anchor_reuses_only_existing_chains,no_chain_without_a_merge_an_anchor_or_a_grid,a_mergeable_state_reads_a_chain_with_one_build_per_surviving_parameters,a_groups_evaluation_grid_is_the_gcd_of_its_queries_grids,below_a_subquery_the_grid_is_its_resolution,below_a_subquery_without_a_resolution_there_is_no_grid,a_subquery_s_resolution_is_the_grid_of_the_groups_below_it.@ Treuses existing chains only;@ end()/@ start()get none.Decisions and open questions
Needs reviewer decision:
PhysicalASAPDAGNode::guaranteeremoves theguaranteekey from serialized plans, anddeny_unknown_fieldsrejects an older plan that has it. BumpINSTALLABLE_PLAN_WIRE_VERSION(14 → 15), or keep 14 (no plan in the field ever had a non-null value)? Not bumped in this PR.Decided (all 2026-11-09 unless noted; concepts doc updated):
SummaryMerge, and aFilter,TimeRangeorTimeShiftover states, carry the θ of the states they read: one candidate per θ (add_candidates_per_input_parameters), input edges only to producers with the same θ (Memo::producers);LogicalProperties::newis infallible andMemo::add_candidatechecks θ against the group. α^log of a readout reads its producer's own θ. Rejected (a): α^log(merge → readout) holds only when every θ the merge may carry meets the requirement (safe, but a chain with weak builds blocks tight levels). Example (a_merge_is_judged_by_the_parameters_it_carries): P =quantile_over_time(0.9, m[5m])at rank 0.005, Q =quantile_over_time(0.9, m[10m])at rank 0.01, one 60 s chain with KLL k ∈ {269, 547, …}:SummaryMerge[k=547]→ P's readout holds;SummaryMerge[k=269]is pruned withBelowRequiredAccuracy { required: rank 0.005 }.accepted_metricspushes down, per group, the metrics its output may be off in (root: the target's metrics; input i of c: the accepted metrics c's contract carries). An edge is cut only when its producer is a sketch readout whose metric the consumer does not accept on that input, or, on a readout's own input, when its metric is not accepted at its output. Every surviving readout a targeted query reaches has a requirement, or fails withNoGuarantee. The metric rule in dead-candidate elimination is a union over queries, so a query can still reach approximate producers through a consumer with no contract for its metric; Pass 2 lists those groups per query inexact_only_groups_of_query, which Stage 3 enforces per query. Example (the_metric_rule_cuts_the_readout_edges_and_keeps_the_exact_division):avg by (job)(q0.5 / q0.5 + q0.9)at absolute 0.05: the readout edges into the division are cut, the exact division is not pruned and its edge into+holds.PruneCause.through_pruned_consumers→through_pruned_candidates.RootPruneReason::ValuesPerEvaluationUnknown { consumer, query }: a whole-result δ witha_whole_result_failure_probability_without_values_per_evaluation_prunes_the_readout). Rejected: δ = 0 (no sketch meets it, with no stated reason).ratio-of-quantiles-1h-mvp:quantile_over_time(0.9, data[1h]) / quantile_over_time(0.5, data[1h]), relative 0.05, δ = 0.01 over 10,000 series = 1e-6 per value: both readouts now get levels 1e-6 and 1e-7 (1e-7 + 1e-7 ≤ 1e-6), and the full stack planned it at 7.481e-06/s before this PR's review fixes (one DDSketch α = 0.02 chain of 60 s windows, refactor: tools and snapshots over the #1207 memo, edges and contexts (#1207 PR 9) #1239); re-measured on restack.Memo::producers,CandidateError::LogicalProperties,accepted_metrics,MetricRule.accepted_metrics(wasconsumer_carries). Example: formerge[k=547]in P's state group,memo.producers(merge, 0)= the k = 547 candidate ofTimeRange(5m, G_w)only; adding aSummaryMergeover KLL states without θ fails withCandidateError::LogicalProperties(SketchBuildWithoutParameters)(a_merge_carries_the_parameters_of_the_states_it_reads).error_propagationtakes the memo, because the key-onlyFiltercontract (E1(b)) reads the input group's column lineage. Example: over R ofquantile by (job)(latency)(columns[job: SourceColumn(latency.job), value: Computed]),Filter(job = "api", R)givesSome { factor_per_input: [1.0], .. }in every metric;Filter(value > 0.5, R)givesNone. Rejected: the doc signature(consumer, metric)with the caller deciding filters (two places decide contracts).producer_satisfies_logical_requirementsreturns the failing reason. Example: KLL k = 26 into the p99 readout requiring relative 0.02 givesErr(BelowRequiredAccuracy { consumer: G4#2, input: 0, required: relative 0.02 δ 1 });LogicalEdge::satisfies_logical_requirementis.is_ok(). Rejected: a bool plus a second function (decides it twice).RootPruneReason::HeapSmallerThanTopK(the doc's α^log includes "a top-k's heap holds k", which fails also without a requirement). Example:topk(50, …)overCmsWithHeap: the edges from the 56 builds with heap 10 or 20 fail, so they are pruned withHeapSmallerThanTopK { k: 50 }(a_heap_smaller_than_k_fails_the_topk_readouts_edge).readout_metricreads the metric off the estimators' guarantee. Example:(Sketch(Kll), Quantile{0.5})→Rank;(Sketch(DDSketch), Quantile{0.5})→Relative;(Sketch(Hll), Cardinality)→Relative;(ExactAggregate(..), _)→None. Used by step 3 and the metric rule; a readout's summary type is read withGroupProperties::sketch_algorithm_of_state, the one helper for it.AccuracyMetric::ALL. P6 (N6):AggregateCoverage::{definition, selection}andSelectionBox::columnspublic (were test-support only);constraint_ofpublic.without_targets_every_edge_is_admissible), matching "(K) has no rows".Scanof the union of label sets, so{job="a"}with{job="b"}gets no $G^$ (scan widening, R7, not folded in). Members groupedwithoutkeys or per entity are not rolled up; interval constraints must be equal within a cluster.every_listed_parameters_has_a_guarantee_at_one_percent,a_guarantee_is_evaluated_at_the_required_failure_probability).readout_levelskeys its result by readout candidate;MAX_ERROR_LEVELSlives withreadout_levels;guaranteeprivate; accuracy-math tests in the accuracy crate; shared memo builders inasap_types::test_support.Verification
Test scopes run at head
c8e67f130withcargo test -p <crate>; the workspace total iscargo test --workspace.Memo::producersinput requirements,add_candidateθ check31df962bc): 1211 / 184 / 9Every workspace failure stops at a later-PR
todo!()(Stage 2 entrystage2-physical-optimization/src/lib.rs, the executor's θ, devtools) or atmaterialized_view.rs:529/materialized_views.rs:67(fixture). I did not compare the failures with main's by test name.Behavior-preserving cleanup check: the last cleanup commit is meant to change no result. A dump program printing Pass 2's full output (groups, candidates, requirements, edges, pruned candidates with causes, exact-only groups, plus an integrity checker) ran over 113 workloads (PromQL and SQL cases, chains, subqueries, rollups, random seeds) before (
31df962bc) and after: every dump is identical except one, where theaccepted_metricslist insideMetricRulereasons is now inAccuracyMetric::ALLorder (the same set; it was the old walk's insertion order); the integrity checker reports 0 problems on all 113.New tests:
a_coarse_exact_sum_rolls_up_the_finer_sum,a_coarse_count_rolls_up_the_finer_count_as_a_sum,a_filtered_coarse_sketch_state_merges_the_finer_states_through_a_compensating_filter,a_finest_common_group_no_query_asks_for_serves_both,a_narrowing_on_a_column_that_is_no_key_gives_no_rollup,a_compensating_filter_over_a_sql_table_has_sql_semantics,a_not_equal_matcher_compensates_with_promql_semantics,a_rejected_rollup_leaves_no_compensating_filter); top-k (a_smaller_topk_reads_the_largest_topk_of_the_same_input); requirements (a_single_readout_requires_the_query_target,a_whole_result_failure_probability_without_values_per_evaluation_prunes_the_readout,a_readout_another_query_gives_a_level_survives_unknown_values_per_evaluation); dead candidates (dead_candidates_are_pruned_and_listed_before_chain_groups); chains (a_mergeable_state_reads_a_chain_with_one_build_per_surviving_parameters,a_chain_whose_width_does_not_divide_the_readers_grid_fails_its_edge,an_absolute_anchor_reuses_only_existing_chains,no_chain_without_a_merge_an_anchor_or_a_grid,chain_groups_run_after_reach_so_a_rolled_up_query_adds_its_grid,an_offset_enters_the_gcd_of_the_widths,one_range_free_chain_group_serves_every_range_and_offset); grids (a_groups_evaluation_grid_is_the_gcd_of_its_queries_grids,below_a_subquery_the_grid_is_its_resolution,below_a_subquery_without_a_resolution_there_is_no_grid); exact-only (a_query_comparing_a_shared_ratio_reads_it_only_exactly); edges (every_candidate_of_an_input_group_is_an_edge_of_matching_parameters,without_targets_every_edge_is_admissible,a_merge_is_judged_by_the_parameters_it_carries).readout.rsa_readouts_metric_is_its_algorithms(KLL quantile Rank, DDSketch and HLL Relative, exact None);readout_caps.rsa_whole_result_failure_probability_is_split_per_value,a_ratio_whose_per_value_failure_probability_is_below_the_grid_splits_it_over_its_readouts,a_max_over_many_readouts_splits_the_failure_probability_over_them,a_count_over_readouts_gives_them_no_level;metric_rule.rsa_ratio_reads_no_rank_error,the_metric_rule_cuts_the_readout_edges_and_keeps_the_exact_division,a_key_only_filter_keeps_the_ddsketch_readout_admissible,avg_of_readouts_under_a_relative_target_reads_exact_only;logical_edge.rstarget_limit_admits_only_floating_point_noise,a_distinct_count_readout_without_an_evaluable_guarantee_serves_no_query_with_a_target,a_heap_smaller_than_k_fails_the_topk_readouts_edge(56 builds with heap 10 or 20 failHeapSmallerThanTopK { k: 50 });error_propagation.rsa_reduction_keyed_by_an_estimate_has_no_contract,key_filters_and_value_preserving_projects_have_unit_contracts,error_contracts_of_a_ratio_a_count_and_a_sum;guarantee/mod.rsa_guarantee_is_evaluated_at_the_required_failure_probability,every_listed_parameters_has_a_guarantee_at_one_percent.pass2_rollup_promql.rs(listed under Rollup);pass2_accuracy_and_grids_promql.rsa_function_of_the_value_keeps_the_readouts_below_it(abs, negation and/ 2over a sketch readout keep it),a_subquery_s_resolution_is_the_grid_of_the_groups_below_it.a_merge_carries_the_parameters_of_the_states_it_reads,kll_builds_with_k_269_and_k_547_share_one_group,set_candidate_input_requirements_replaces_only_the_named_candidate,add_candidate_checks_parameters_against_the_groups_output.End-to-end evidence: none at this head, since planning stops at Stage 2's
todo!.ratio-of-quantiles-1h-mvpplanned at 7.481e-06/s on the #1239 snapshot before this PR's review fixes; it is re-measured on restack. Screenshots: not applicable.Self-review: one
add_candidate_if_output_matcheshelper in Pass 2, onereadout_metric, one sketch-algorithm helper, one target-metric expansion (AccuracyTarget::metrics), contracts only inerror_propagation, θ only inparameter_list(no strongest/weakest/floor tables); no#[allow(dead_code)]added.Follow-ups
city = 'NYC'filter over the per-city group); among the remaining failures.exact_only_groups_of_queryper query: a later PR in this stack.withoutgroupings, no label filters between per-series layers, no regex matchers; the pre-check covers the schema only.PrunedLogicalCandidate.docs/develop_docs/asap-aware-mapping-*.md,end-to-end-accuracy-guarantees.md) and the concepts doc describe the error contracts, the θ lists with the guarantee at the required δ, and the current file layout;planner-layering.md's Pass 2 rows describe the current code. Its proposal sections still citememo::enumerationand "sized for the strictest" designs, which this PR does not rewrite.