Skip to content

refactor(stage1-pass2): rollup from the finest common group, readout requirements, dead-candidate elimination, chain groups (#1207 PR 4) - #1234

Draft
zzylol wants to merge 25 commits into
mainfrom
refactor/1207-pass2
Draft

zzylol wants to merge 25 commits into
mainfrom
refactor/1207-pass2

Conversation

@zzylol

@zzylol zzylol commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Problem: on main Pass 2's rules and edges end in todo!(), so every planning run stops there; the old rules matched shapes pairwise by column name, and θ floors per input mixed ε and δ and could not express a multi-readout allocation.
  • Change: Pass 2 (generate_logical_asap_memo_candidates) runs the reuse rules over memo groups (rollup from the finest common group, top-k reuse), gives each readout its required accuracy, eliminates dead candidates with root causes, builds chain groups and the logical edges $(u,c,i)$ with $\alpha^{\log}$ (concepts doc §3 "Stage 1, Pass 2" and §4; order of 2026-11-07, accuracy design of 2026-11-08). The node-tree guarantee composition and propagation are replaced by per-operator error contracts plus readout requirements; Pass 2 also lists, per query, the groups it may read only exactly.
  • Effect: for quantile_over_time(0.99, m[5m]) / quantile_over_time(0.5, m[5m]) at relative 0.05 every 60 s, Pass 2 prunes both KLL readouts (MetricRule, rank error) and the DDSketch builds with α 0.1 and 0.05, gives each DDSketch readout five levels 0.02 … 0.001, and builds one 60 s DDSketch chain with five builds; planning now stops at Stage 2 (PR 5). The snapshot workload ratio-of-quantiles-1h-mvp planned at 7.481e-06/s with the full stack before this PR's review fixes (D1); it is re-measured when the stack above is restacked.

Review guide

  • Position: PR 4 of the Tracking: implement the optimization formulation (memo candidates, properties, edges y_{u,c,i}, MILP) as named code per stage #1207 stack (row "4. Pass 2" of Tracking: implement the optimization formulation (memo candidates, properties, edges y_{u,c,i}, MILP) as named code per stage #1207's Order); part of the Tracking: implement the optimization formulation (memo candidates, properties, edges y_{u,c,i}, MILP) as named code per stage #1207 merge unit. Base main (PR 3, refactor(stage1-pass1): per-computation candidates over memo groups (#1207 PR 3) #1232, is merged); refactor(stage2): physical expansion (decision-tree levels 3–4), physical edges, pruned alternatives; Count-Min weights via column_sign (#1207 PR 5) #1235 (PR 5) builds on it. 25 commits, head c8e67f130.
  • Reading order:
    1. stage1-pass2-reuse-candidates/src/lib.rs (the six steps in order), edges.rs, readout_requirements.rs, dead_candidates.rs, evaluation_grids.rs, exact_only_groups.rs.
    2. rules/rollup.rs (add_rollup_candidates), rules/topk.rs, rules/chains.rs.
    3. accuracy/src/error_propagation.rs (the contracts), readout_caps.rs (the caps walk, readout_levels), metric_rule.rs (accepted_metrics, metrics_accepted_on_inputs), readout.rs (readout_metric), logical_edge.rs (producer_satisfies_logical_requirements, target_limit).
    4. types/src/stages/logical/{candidate,memo,group}/mod.rs (θ on pass-through candidates, Memo::producers, Memo::series_count_scanned_below, same_candidate) and types/src/stages/pruning.rs.
  • Mechanical:
    • Deletion of asap_accuracy::{composition, propagation, propagation_stats, budget, function_rules, requirements, sizing}, guarantee/sizing_oracle.rs, the classic HLL and UnivMon distinct-count guarantees, the unused guarantee vocabulary in types/src/accuracy/guarantee.rs, and the never-set guarantee field of OperatorNode, FlatNode and PhysicalASAPDAGNode (with OperatorNode::with_guarantee; the executor and integration fixtures drop it).
    • File moves: requirements.rs split into logical_edge.rs (with target_limit), metric_rule.rs, readout.rs and readout_caps.rs; binomial_upper_tail into guarantee/binomial.rs; MAX_ERROR_LEVELS from types/src/summary_type/parameter_lists.rs into readout_caps.rs; types' group.rs, aggregate_coverage.rs and column_properties.rs into stages/logical/group/, view_coverage.rs into plan/, inline test modules into sibling tests.rs files, hash-consing tests next to hash_consing.rs.
    • Pass 2's tests split into tests/{rollup,topk,readout_requirements,dead_candidates,chains,edges,evaluation_grids,exact_only_groups}.rs plus test_fixtures.rs; the accuracy-math tests sit in the accuracy crate next to their files; shared memo builders in asap_types::test_support::memo_builders (plus compare and arithmetic in memo_fixtures); devtools drops the two removed origins from its labels.
  • Careful: the Filter, Project and reduction contracts that read column lineage (error_propagation); the caps walk and the readout-level grid with D1's extension; the per-value δ split over the readout group's values (series_count_scanned_below); NoGuarantee; two-direction elimination and its traces; θ matching on edges (Memo::producers); the metric rule cutting edges, not candidates, and exact_only_groups_of_query; evaluation grids below subqueries, chain widths (gcd closure) and the chain-edge divisibility check; the rollup pre-check before the compensating Filter and its frontend semantics.
  • Final layout:
    • crates/accuracy/src: lib.rs, evidence.rs, error_propagation.rs, readout.rs, metric_rule.rs, readout_caps.rs, logical_edge.rs, guarantee/{mod,binomial,cardinality,cms,count_sketch,ddsketch,hll,kll,univmon}.rs.
    • crates/stage1-pass2-reuse-candidates/src: lib.rs, edges.rs, evaluation_grids.rs, readout_requirements.rs, dead_candidates.rs, exact_only_groups.rs, rules.rs, rules/{rollup,topk,chains}.rs, test_fixtures.rs, tests.rs, tests/{rollup,topk,readout_requirements,dead_candidates,chains,edges,evaluation_grids,exact_only_groups}.rs.

Behavior: before → after

Concepts doc "ratio of quantiles" (test dead_candidates_are_pruned_and_listed_before_chain_groups): quantile_over_time(0.99, m[5m]) / quantile_over_time(0.5, m[5m]), relative error 0.05, every 60 s at phase 0. It serves the design need that Pass 2 enumerates only candidates some query can use, with a reason for every one it removes.

Before (main) After (this PR)
Pass 2 rules and edges todo!(); planning stops LogicalASAPMemoCandidates (memo, edges, pruned, evaluation grids, exact-only groups per query); planning stops at Stage 2 (todo! of PR 5)
KLL readouts — pruned: MetricRule { producer_metric: Rank, accepted_metrics: [Relative] }; every KLL build through its edge into a KLL readout
DDSketch builds — α 0.1 and 0.05 pruned BelowRequiredAccuracy; five survive
DDSketch readout groups — 5 readout candidates requiring relative 0.02, 0.01, 0.005, 0.002, 0.001 (δ 1)
Chains — one SummaryTumblingWindowChain[DDSketch, w = 60 s](Scan m) with the 5 surviving builds; SummaryMerge(TimeRange(5m, chain)) in the DDSketch state group
Exact-only groups — none for this query (the division carries relative error); … > 1 over the same ratio would list the ratio group and every group below it

Interfaces

Permalinks are at head c8e67f1308e4dc5acab976a4102b6c64a5523e82.

crates/stage1-pass2-reuse-candidates/src/lib.rs

 /// Stage 1, Pass 2: steps 1–6; `entries[i]` is query i's workload entry (its evaluation grid).
 /// The planner facade calls it after Pass 1. Signature unchanged; now implemented.
 pub fn generate_logical_asap_memo_candidates(
     pass1: Memo,
     entries: &[QueryWorkloadEntry],
     evidence: &dyn AccuracyEvidence,
 ) -> Result<LogicalASAPMemoCandidates, Pass2RuleError>;
 /// Why Pass 2 failed on a memo: a rule bug or a cyclic memo. Stays; now defined in `lib.rs`.
 pub enum Pass2RuleError {
-    /// Never produced.
-    InvalidGroupReference,
-    /// Never produced.
-    TooManyGroups,
     /// A group, or one of its candidates, reads itself through other groups.
     CyclicDependencies,
+    /// Internal bug: a rule built a group expression the node rules reject.
+    Group(#[from] MemoGroupError),
+    /// Internal bug: a rule built a candidate expression the node rules reject, or gave it θ its
+    /// group's output cannot have.
+    Candidate(CandidateError),
 }

crates/types/src/stages/logical/candidate/mod.rs · memo/mod.rs · group/mod.rs · memo_candidates.rs · crates/types/src/stages/pruning.rs

+impl CandidateForMemoGroup {
+    /// The candidate computing `operator` with θ `parameters`, added by rule `origin`, fusing
+    /// `fused_groups`. Its inputs have no requirement yet: Pass 2 sets each readout's.
+    pub fn new(
+        operator: Operator<MemoGroupId>,
+        parameters: Option<SketchParams>,
+        origin: OriginOfCandidateForMemoGroup,
+        fused_groups: Vec<MemoGroupId>,
+    ) -> Self;
+}
+/// The θ of each build candidate of `build`: every listed θ of a sketch build, `[None]` for any
+/// other operator. A group gets one build candidate per entry.
+pub fn listed_build_parameters(build: &Operator<MemoGroupId>) -> Vec<Option<SketchParams>>;
 /// Which rule added a candidate. Stays; nothing in planning reads it.
 pub enum OriginOfCandidateForMemoGroup {
     /// Pass 1: exact, exact accumulator, or a summary type.
     Pass1,
-    /// Canonical value preprocessing feeding a summary build.
-    SummaryInput,
     /// A tumbling-window chain, or a merge reading one, of a summary state group.
     TumblingWindows,
     /// A coarse group rolled up from a finer one, the compensating `Filter` it reads, or a build of a finest common group.
     Rollup,
     /// A top-k read from another group's ranking.
     TopKReuse,
-    /// Its metric's scans widened to a shared label set (scan widening).
-    WidenedScans,
 }
 /// A candidate `Memo::add_candidate` rejects; the memo is unchanged. Stays.
 pub enum CandidateError {
     /// Internal bug: the expression fails the node rules.
     NotValidOverInputGroups(MemoGroupError),
     /// Internal bug: its output differs from the group's.
     OutputDiffersFromGroup { group: MemoGroupId },
+    /// Internal bug: its θ does not fit the group's output (a sketch state group's candidate needs
+    /// one from the algorithm's list, any other none).
+    LogicalProperties(LogicalPropertiesError),
 }
 /// θ of a candidate. A `SummaryMerge`, and a `Filter`, `TimeRange` or `TimeShift` over states,
 /// carry the θ of the states they read.
 impl LogicalProperties {
-    pub fn new(
-        expression: &CandidateExpression,
-        parameters: Option<SketchParams>,
-    ) -> Result<Self, LogicalPropertiesError>;
+    /// Infallible; `Memo::add_candidate` checks θ against the group.
+    pub fn new(parameters: Option<SketchParams>) -> Self;
 }
 impl Memo {
+    /// The candidates `consumer` may read on input `input`: those of the input group, except that
+    /// a consumer with a θ reads only producers with the same θ or none.
+    pub fn producers(
+        &self,
+        consumer: CandidateForMemoGroupId,
+        input: usize,
+    ) -> Vec<CandidateForMemoGroupId>;
+    /// The most series (PromQL) or groups of rows (SQL) the `Scan` groups below `group` read: the
+    /// sum of their sources' `SourceFacts::series_count`; `None` when one declares none. Bounds the
+    /// values one evaluation of `group` returns (a readout of `quantile_over_time(0.9, m[5m])`
+    /// over 1000 series: 1000), over which a reduction such as `max` splits δ.
+    pub fn series_count_scanned_below(&self, group: MemoGroupId) -> Option<u64>;
 }
 impl GroupProperties {
+    /// The sketch algorithm of the state the group outputs; `None` for rows, values and exact
+    /// states. The accuracy crate reads a readout's summary type through it.
+    pub fn sketch_algorithm_of_state(&self) -> Option<SketchAlgorithm>;
 }
 /// Stage 1, Pass 2's output. Stays.
 pub struct LogicalASAPMemoCandidates {
     pub memo: Memo,
     pub edges: Vec<LogicalEdge>,
     pub pruned: Vec<PrunedLogicalCandidate>,
     /// Per group: the grid its result is evaluated on (now: below a PromQL subquery `[10m:30s]`
     /// its 30 s steps; `None` below a subquery without a resolution).
     pub evaluation_grid_of_group: Vec<Option<EvaluationGrid>>,
+    /// Indexed by query: the groups it reaches through a consumer with no error contract for its
+    /// metric, plus, for a query whose whole-result δ has no per-value one, each readout group
+    /// another query gave a level; closed downward. In these groups the query may use only exact
+    /// candidates. Pass 2 only lists them: Stage 3 enforces them per query in a later PR.
+    pub exact_only_groups_of_query: Vec<BTreeSet<MemoGroupId>>,
 }
 /// One path from a pruned candidate, through candidates that are themselves eliminated, to the
 /// edge that fails. Stays.
 pub struct PruneCause {
-    /// The eliminated consumers on the path, nearest first.
-    pub through_pruned_consumers: Vec<CandidateForMemoGroupId>,
+    /// The eliminated candidates on the path, nearest first: consumers above, or, for a candidate
+    /// cut off from an input, producers below.
+    pub through_pruned_candidates: Vec<CandidateForMemoGroupId>,
     /// Why the edge at the end fails.
     pub root: RootPruneReason,
 }
 /// Why the edge at the end of a `PruneCause`'s path fails. Stays.
 pub enum RootPruneReason {
     /// Unsatisfiable: the producer's guarantee at the required δ exceeds the input's required error.
     BelowRequiredAccuracy {
         consumer: CandidateForMemoGroupId,
         input: usize,
         required: RequiredAccuracy,
     },
     /// Unsatisfiable: the producer is a readout whose metric the consumer does not accept on that
     /// input (on a readout's own input: at its output).
     MetricRule {
         consumer: CandidateForMemoGroupId,
         input: usize,
         producer_metric: AccuracyMetric,
-        consumer_carries: Vec<AccuracyMetric>,
+        /// The metrics accepted on that input (on a readout's own input: at its output).
+        accepted_metrics: Vec<AccuracyMetric>,
     },
+    /// Unsatisfiable: a top-k readout whose `k` exceeds the producer's heap size, whatever accuracy is required.
+    HeapSmallerThanTopK {
+        consumer: CandidateForMemoGroupId,
+        input: usize,
+        k: usize,
+    },
+    /// Unsatisfiable: under a query with an accuracy target, the readout has a metric but no
+    /// guarantee evaluable at the required δ (HLL's distinct count, failure probability unknown) or
+    /// no guarantee at all (UnivMon's distinct count): its error would reach the root unbounded.
+    NoGuarantee {
+        consumer: CandidateForMemoGroupId,
+        input: usize,
+    },
+    /// Unsatisfiable: query `query` sets a whole-result δ but its values per evaluation ($n_q$) are
+    /// unknown, so no per-value δ follows, and no other query reaching the readout gives it a level.
+    ValuesPerEvaluationUnknown {
+        consumer: CandidateForMemoGroupId,
+        query: usize,
+    },
+    /// Unsatisfiable: the producer is over a tumbling-window chain whose width does not divide the
+    /// consumer group's evaluation period (or the group has no known grid): the chain's windows do
+    /// not end at the consumer's evaluation times (a 15 s chain read every 10 s).
+    ChainWidthDoesNotDivideEvaluationGrid {
+        consumer: CandidateForMemoGroupId,
+        input: usize,
+        tumbling_window_width_ms: u64,
+        evaluation_period_ms: Option<u64>,
+    },
     /// The candidate never had a consumer and is not a query output.
     NoConsumer,
 }

add_candidate keeps its signature but now identifies a candidate by expression, θ and input requirements (same_candidate), and rejects a θ that does not fit the group.

crates/types/src/ir/operator/node.rs · ir/flat.rs · plan/physical_asap_dag.rs

 /// A node of the logical operator DAG. Stays.
 pub struct OperatorNode {
     pub operator: Operator,
     pub result_kind: OperatorResultKind,
     pub schema: Schema,
-    pub guarantee: Option<AccuracyGuarantee>,
 }
 impl OperatorNode {
-    pub fn with_guarantee(mut self, guarantee: Option<AccuracyGuarantee>) -> Self;
 }
 /// An `OperatorNode` whose children are `NodeId`s. Stays.
 pub struct FlatNode {
     pub operator: Operator<NodeId>,
     pub(crate) result_kind: OperatorResultKind,
     pub(crate) schema: Schema,
-    pub(crate) guarantee: Option<AccuracyGuarantee>,
 }
 /// A node of the installable physical plan. Stays; every other field unchanged.
 pub struct PhysicalASAPDAGNode {
     pub id: PhysicalASAPNodeId,
     pub payload: PhysicalASAPOperatorPayload,
     pub output_state: ExecutionDataState,
     pub lifetime: Lifetime,
     pub parameters: Option<SketchParams>,
     pub output_schema: Schema,
-    pub guarantee: Option<AccuracyGuarantee>,
     pub retention_ms: Option<u64>,
     pub selected_physical_candidate: Option<PhysicalCandidateId>,
     pub view_coverage: Option<ViewCoverage>,
 }

The only writer of these fields was the deleted asap_accuracy::propagation; every remaining site only copied the value, so it was always None. Wire format: a serialized plan node no longer has the guarantee key (PhysicalASAPDAGNode is deny_unknown_fields, so a plan written before this PR with "guarantee": null no longer parses); the fixtures plan_wire_version_10.json and stored_outputs_on_main.json drop it. INSTALLABLE_PLAN_WIRE_VERSION is not bumped here (see Needs reviewer decision).

crates/types/src/ir/properties/selection.rs · stages/logical/group/aggregate_coverage.rs · accuracy/ · summary_type/

 /// `column = v`, `!=`, `<`, `<=`, `>`, `>=`, `[NOT] IN (...)` and `OR` of equalities on one column.
-pub(crate) fn constraint_of<C>(expr: &ScalarExpr<C>) -> Option<(ColumnId, Constraint)>;
+pub fn constraint_of<C>(expr: &ScalarExpr<C>) -> Option<(ColumnId, Constraint)>;
-#[cfg(any(test, feature = "test-support"))]
 impl AggregateCoverage {
+    /// The coverage definition; public outside tests (the rollup rule reads it).
     pub fn definition(&self) -> &AggregateDefinition;
+    /// Union of boxes over the rows that went in; public outside tests.
     pub fn selection(&self) -> &[SelectionBox];
 }
-#[cfg(any(test, feature = "test-support"))]
 impl SelectionBox {
+    #[cfg(any(test, feature = "test-support"))]
     pub fn new(columns: BTreeMap<SourceColumn, Constraint>) -> Self;
+    /// The per-column constraints; public outside tests.
     pub fn columns(&self) -> &BTreeMap<SourceColumn, Constraint>;
 }
 impl AccuracyMetric {
+    /// Every metric (`ABSOLUTE`, `ABSOLUTE_STREAM_L1`, `ABSOLUTE_STREAM_L2`, `Relative`, `Rank`).
+    pub const ALL: [Self; 5];
 }
 impl AccuracyTarget {
+    /// The metrics this target accepts an error in: its metric, every metric when it names none.
+    /// The caps walk and the exact-only walk seed each query root with it.
+    pub fn metrics(&self) -> Vec<AccuracyMetric>;
 }
 impl SketchStatistic {
+    /// The statistic a summary is read for to answer `intent` (was `asap_accuracy::sizing::sketch_statistic`).
+    pub fn for_intent(intent: &AggIntent) -> Option<Self>;
 }
-/// How a parent operator consumes its inputs' values.
-pub enum CompositionOperator { /* 10 variants */ }
-/// Why an accuracy check rejected a candidate.
-pub enum AccuracyError { /* 4 variants */ }
-/// The statistic an unbound number of values per result is left as.
-pub const VALUES_PER_RESULT: &str = "values_per_result";
 impl AccuracyGuarantee {
-    pub fn over_scope(&self, scope: AccuracyScope, values_per_result: Option<u64>) -> Self;
-    #[cfg(any(test, feature = "test-support"))]
-    pub fn approximate_layer_count(&self) -> usize;
 }
 /// Provenance of a guarantee: now `Exact` and `SketchEvaluation` only.
 pub enum GuaranteeSource {
     Exact { reason: String },
-    AccuracyTarget { target: AccuracyTarget },
     SketchEvaluation { algorithm: String, contract: String, params: serde_json::Value, query: String },
-    ChildGuarantee { /* ... */ },
-    CompositionStep { /* ... */ },
-    BudgetAllocation { /* ... */ },
-    UnavailableStatistic { statistic: String },
-    RuntimeObservation { /* ... */ },
 }

Removed because nothing reads them: composition and budget types, AccuracyError, over_scope and approximate_layer_count and the guarantee variants only the removed composition wrote. The /* N variants */ lines stand for whole enums removed with all their variants.

crates/accuracy/src/lib.rs · logical_edge.rs · metric_rule.rs · readout.rs · readout_caps.rs · error_propagation.rs · guarantee/mod.rs

 /// α^log of edge (producer → consumer, input): the readout rule (the producer's own θ, heap ≥ k,
 /// an evaluable guarantee) or the metric rule; Err names the failing root reason. Was `todo!()`;
 /// moved from `requirements.rs` into `logical_edge.rs`. The two tables are computed once per edge pass.
 pub fn producer_satisfies_logical_requirements(
     memo: &Memo,
+    readout_metric_of_candidate: &[Vec<Option<AccuracyMetric>>],
+    metrics_accepted_on_inputs: Option<&[AccuracyMetric]>,
     producer: CandidateForMemoGroupId,
     consumer: CandidateForMemoGroupId,
     input_index_of_consumer: usize,
     evidence: &dyn AccuracyEvidence,
-) -> bool;
+) -> Result<(), RootPruneReason>;
 /// The largest evaluated error (or δ) that meets a required `limit`. Unchanged; moved from
 /// `requirements.rs` into `logical_edge.rs`.
 pub fn target_limit(limit: f64) -> f64;
+/// Per group, the metrics its output may be off in for the targeted queries reaching it, in
+/// `AccuracyMetric::ALL` order; `None` where none reaches (no metric rule). Derived from the
+/// readout caps' walk. `order` is `Memo::topological_order`.
+pub fn accepted_metrics(memo: &Memo, order: &[MemoGroupId]) -> Vec<Option<Vec<AccuracyMetric>>>;
+/// The metrics `consumer` accepts an input's error in: a readout those its output accepts, any
+/// other candidate those its contract carries. Pass 2 computes it once per consumer.
+pub fn metrics_accepted_on_inputs(
+    memo: &Memo,
+    accepted: &[Option<Vec<AccuracyMetric>>],
+    consumer: CandidateForMemoGroupId,
+) -> Option<Vec<AccuracyMetric>>;
+/// The metric of reading `statistic` from a state of `summary_type`: its first listed θ's
+/// guarantee's, whatever its δ (the estimators' guarantee is the single source).
+pub fn readout_metric(
+    summary_type: &FieldDataType,
+    statistic: &SketchStatistic,
+) -> Option<AccuracyMetric>;
+/// Per group, per candidate: its metric when it is a readout of a sketch state, else `None`.
+/// Pass 2 computes it once per edge pass.
+pub fn readout_metric_of_candidate(memo: &Memo) -> Vec<Vec<Option<AccuracyMetric>>>;
+/// The required-accuracy levels of each sketch readout candidate without a requirement yet, with
+/// the queries reaching it whose δ has no per-value one there. `order` is `Memo::topological_order`.
+pub fn readout_levels(
+    memo: &Memo,
+    order: &[MemoGroupId],
+) -> BTreeMap<CandidateForMemoGroupId, (Vec<RequiredAccuracy>, Vec<usize>)>;
 /// g_c's linear bound for errors in `metric` (β and b per input); `None` = no contract (the metric
 /// rule). A `Filter`, a `Project` or a grouping reads its input group's column lineage in `memo`. Was `todo!()`.
 pub fn error_propagation(
+    memo: &Memo,
     consumer: &Operator<MemoGroupId>,
     metric: AccuracyMetric,
 ) -> Option<ErrorPropagation>;
+/// The metrics among `metrics` that `consumer`'s contract carries to its inputs.
+pub fn metrics_carried(
+    memo: &Memo,
+    consumer: &Operator<MemoGroupId>,
+    metrics: &[AccuracyMetric],
+) -> Vec<AccuracyMetric>;
-#[derive(Debug, Clone, PartialEq)]
+#[derive(Debug)]
+#[cfg_attr(test, derive(PartialEq))]
 pub struct ErrorPropagation {
     pub factor_per_input: Vec<f64>,
     pub validity_cap_per_input: Vec<Option<f64>>,
 }
 /// The error bound of θ for `statistic` at `failure_probability`; None if θ cannot fail that rarely
 /// or its δ is unknown. Signature unchanged; was `todo!()`.
 pub fn theoretical_guarantee(
     summary_type: &FieldDataType,
     parameters: &SketchParams,
     statistic: &SketchStatistic,
     failure_probability: f64,
 ) -> Option<AccuracyGuarantee>;
-/// The per-algorithm guarantees (`guarantee::hll`, `guarantee::univmon` were public).
-pub mod guarantee;
+mod guarantee;
-/// Node-tree guarantee composition.
-pub mod composition;
-/// Node-tree guarantee propagation through a DAG.
-pub mod propagation;
-/// The statistic a summary is read for.
-pub mod sizing;
-/// The target of one of `approximate_inputs` inputs composing into `target`.
-pub fn per_input_target(
-    target: &AccuracyTarget,
-    metric: AccuracyMetric,
-    approximate_inputs: usize,
-) -> Option<AccuracyTarget>;

Removed: node-tree composition and propagation (composition, propagation, propagation_stats, budget::per_input_target, private function_rules) are replaced by per-edge error contracts plus readout requirements (agreed 2026-11-06). Also removed: the test-only guarantee::sizing_oracle, the classic HLL and UnivMon distinct-count guarantees (a later PR in this stack brings back certified distinct-count guarantees), the fixed-δ KLL and cardinality guarantee lookups (θ's own guarantee plus one δ check in theoretical_guarantee replace them), the test-only meets (its one caller checks the bound with target_limit), and PerComputationCandidateError::LogicalProperties (θ is now checked in one place, Memo::add_candidate). ErrorPropagation lost Clone (nothing clones it) and keeps PartialEq for tests only. guarantee is private because no crate reads guarantee::* beyond the re-exported theoretical_guarantee.

Usage, end to end (test dead_candidates_are_pruned_and_listed_before_chain_groups at c8e67f130):

// What the user writes: quantile_over_time(0.99, m[5m]) / quantile_over_time(0.5, m[5m]),
// relative error 0.05, evaluated every 60 s at phase 0 (FixedIntervalAt).
// `memo`: Stage 0 + Pass 1 (built by the test as they build it): G2 = TimeRange(5m, Scan m);
// per quantile an Aggregate group with its exact candidate and readouts of a KLL state
// (7 builds) and a DDSketch state (7 builds); G6 = Divide(p99, p50), the query output.
// `target`: the query's accuracy target; Pass 2 derives each readout's required accuracy from it.
let target = AccuracyTarget::max_error(0.05).accuracy_metric(AccuracyMetric::Relative);
query(&mut memo, ratio, Some(target)); // MemoQuery { root: G6, accuracy_target, values_per_evaluation: 1 }
// `logical`: Pass 2's output, as the planner facade calls it after Pass 1. Inside, step 3 calls
// readout_levels(&memo, &order), whose caps walk calls error_propagation(&memo, Divide, Relative)
// = Some { factor 1/(1-0.1) on both inputs, cap 0.1 on the denominator }; steps 4 and 6 compute
// accepted_metrics(&memo, &order) and readout_metric_of_candidate(&memo) once per edge pass,
// metrics_accepted_on_inputs once per consumer, and producer_satisfies_logical_requirements per edge.
let logical = generate_logical_asap_memo_candidates(memo, &[entry(60_000)], &NoMeasuredGuarantees)?;
// logical.pruned: each KLL readout -> MetricRule { producer_metric: Rank, accepted_metrics: [Relative] };
//   every KLL build through it; DDSketch alpha 0.1 and 0.05 -> BelowRequiredAccuracy.
// Each DDSketch readout group: 5 readout candidates requiring relative 0.02, 0.01, 0.005, 0.002, 0.001 (δ 1).
// One chain group SummaryTumblingWindowChain[DDSketch, w = 60 s](Scan m) with the 5 surviving builds,
// and SummaryMerge(TimeRange(5m, chain)) in the DDSketch state group.
// logical.exact_only_groups_of_query[0]: the scan rows below the builds only (a build has no contract).
// logical.edges: only between survivors; Stage 2 (PR 5) expands them.

Algorithm changes

Pass 2's steps:

Step Code What it adds or decides
1. Rollup from the finest common group rules/rollup.rs add_rollup_candidates Coarse aggregate group $G_c$ from finer $G_f$ of an equal definition (keys ⊆, selection ⊆ with compensation only on $G_f$'s keys by lineage, equal time window): SummaryMerge(group_by) for states, sum/min/max for finalized sums, counts, minima, maxima, each measure named as $G_c$'s column; one synthesized finest common group $G^*$ per cluster. The definition is compared through TimeRange, TimeShift and per-series layers (e.g. rate), and scans listing different labels match by column lineage
2. Top-k reuse rules/topk.rs A smaller exact top-k reads the largest top-k of the same input
3. Reach, caps, readout requirements readout_requirements.rs, accuracy::readout_caps One top-down walk: reach, accepted metrics and caps $\min(b_{c,i}, \max_q \varepsilon_q/\lambda)$ through error_propagation; each sketch readout gets one candidate per requirement level (single-read query: the cap; composed query: the ε grid × δ grid not looser than the cap and per-value δ); per-value δ = δ / max($n_q$, the readout group's values per evaluation)
4. Dead-candidate elimination dead_candidates.rs Removes candidates no query can use, transitively; LogicalASAPMemoCandidates::pruned lists each with every root cause (one shortest path per failing edge)
5. Chain groups rules/chains.rs SummaryTumblingWindowChain[w](X) per width (gcd closure of $\gcd(g_q, W, o_j)$ over reaching queries with a known grid at the state, below a subquery its resolution), one build per surviving θ, re-wrapped TimeShift/TimeRange groups, SummaryMerge in the state group; @ T reuses existing chains only; @ end()/@ start() none
6. Logical edges edges.rs, evaluation_grids.rs, lib.rs Every $(u,c,i)$ between surviving candidates with $\alpha^{\log}$ from producer_satisfies_logical_requirements; an edge from a chain into a reader whose evaluation grid (evaluation_grids: gcd of the reaching queries' grids) its width does not divide fails with ChainWidthDoesNotDivideEvaluationGrid
then exact_only_groups.rs Per query, the groups it may read only exactly (exact_only_groups_of_query)

Rules add only through find_or_add_group/add_candidate and swallow only OutputDiffersFromGroup; add_candidate treats input requirements as part of a candidate's identity (two readouts with different requirements are different candidates); RELATIVE_OPERAND_VALIDITY_CAP = 0.1 is private. theoretical_guarantee evaluates KLL (rank error scaled below δ = 0.01) and KMV (bisection on ε) at the required δ, and checks every other algorithm's own failure probability against it.

Rollup

  • Problem: three rules (scan widening, finer aggregate, rollup) each matched their own shapes by column name; no group finer than every member was ever built.
  • Before: rule 2 widened scans, rule 5 read a finer SQL result, rule 6 merged a finer state, each pairwise.
  • After: (1) cluster aggregate groups by coverage definition without keys, written out down to the source through TimeRange, TimeShift and per-series layers (Member::see_through_per_series_layers), so equal computations compare equal whatever their keys, label selections and the labels their scans list (scans match by lineage; an operator that names columns by position, such as a measure over a column, still needs equal column lists), and equal intervals; (2) per cluster, $G^*$ = union selection, keys ∪ narrowing columns, built over a member's rows; (3) every ordered pair gets Rollup(by keys)(Filter(compensation, G_f)) when keys ⊆ and selection ⊆ on keys by lineage. The rollup and its schema are checked over a pending Filter first, so a rollup the memo would reject leaves no compensating group behind.
  • Review fixes: a rolled-up Aggregate names each measure as $G_c$'s column, so count by (job) rolls up count by (job, instance) as a sum named count (a SQL global count over a grouped one is still rejected by the schema check: a sum over no group is NULL, the count 0). The compensating Filter is built from comparisons with the source frontend's semantics: a PromQL != matcher becomes Compare Ne with PromQL semantics, NOT IN a conjunction of Ne, IN a disjunction of Eq.
  • Correctness: a compensation reads only $G_f$'s keys, whose rows each combine all raw rows of one key. Tests: a_coarse_exact_sum_rolls_up_the_finer_sum, a_coarse_count_rolls_up_the_finer_count_as_a_sum, a_filtered_coarse_sketch_state_merges_the_finer_states_through_a_compensating_filter, a_finest_common_group_no_query_asks_for_serves_both, a_narrowing_on_a_column_that_is_no_key_gives_no_rollup, a_compensating_filter_over_a_sql_table_has_sql_semantics, a_not_equal_matcher_compensates_with_promql_semantics, a_rejected_rollup_leaves_no_compensating_filter, and through production PromQL lowering (crates/integration-tests/tests/pass2_rollup_promql.rs): an_instant_sum_rolls_up_from_the_finer_sum_over_a_scan_listing_more_labels, an_instant_count_rolls_up_from_the_finer_count, a_sum_of_rates_rolls_up_from_the_finer_sum_of_the_same_rates, a_label_matcher_becomes_a_compensating_filter_over_the_finer_sum, with the negative cases rates_over_different_ranges_do_not_roll_up, different_per_series_functions_do_not_roll_up, a_standard_deviation_does_not_roll_up, a_finer_sum_narrowed_by_a_matcher_serves_no_wider_sum.
  • Complexity: O(groups²) per cluster.
  • Assumptions and limits: see Q5. Not covered: without groupings, label filters between per-series layers, regex matchers; no new union Scan; equal intervals. The pre-check covers the schema only. The A filtered query (city = 'NYC') doesn't read its group from a grouped summary another query builds #902 case (city = 'NYC' over the per-city group) is not widened: Rollup rule: a filtered query (city = 'NYC') does not read the per-city group another query builds (#902 regression on the #1207 stack) #1252.

Readout requirements, the metric rule and dead candidates

  • Problem: θ floors per input (Pass 1 heap floor, Pass 2 cap floors) mixed ε and δ and could not express multi-readout allocation.
  • After:
    1. readout_caps::caps: one walk down from the query roots in reverse topological order. A root with a target is reached with its target in each of its metrics (AccuracyTarget::metrics); every candidate of a reached group reaches its inputs and passes each metric its contract carries, with caps $\min(b_{c,i}, \text{cap}/\beta_{c,i})$, keeping the largest per query and metric; an input with $\beta = 0$ (a count's) gets the metric but no cap.
    2. readout_levels: per readout candidate without a requirement, its levels from its group's caps in its metric (single read: the cap at the per-value δ; composed: the ε grid MAX_ERROR_LEVELS × the δ grid 1e-2 … 1e-5 continued in powers of ten down to the first at or below a tenth of the per-value δ, D1).
    3. accepted_metrics = the metrics of the same walk per group (None: not reached); metrics_accepted_on_inputs per consumer; readout_metric_of_candidate once per memo.
    4. $\alpha^{\log}$ per edge, then elimination to a fixpoint in two directions (a candidate whose consumers all died, or one with an input whose every edge is cut or reads a dead producer) and BFS traces.
    5. exact_only_groups_of_query: per query, the metrics every path to a group carries (an intersection, where the metric rule's accepted_metrics is a union over queries); a group where none is left is exact-only for that query, with every group below it.
  • Review fixes:
    • A Project contract judges only the items reading a computed value: copied labels, the time and promql_drop_metric_name of the series identity keep any error; promql_abs/clamp/clamp_min/clamp_max and a negation keep an absolute error; a product with or quotient by a nonzero literal keeps a relative one. abs(count_over_time(m[1m])), -count_over_time(m[1m]) and quantile_over_time(...) / 2 now keep their sketch readouts (key_filters_and_value_preserving_projects_have_unit_contracts, a_function_of_the_value_keeps_the_readouts_below_it).
    • readout_metric no longer needs a finite δ (HLL's distinct count is relative); under a query with a target, an edge into a readout with no metric (UnivMon's distinct count) or whose guarantee's δ is unknown (HLL) fails with NoGuarantee (a_distinct_count_readout_without_an_evaluable_guarantee_serves_no_query_with_a_target).
    • The per-value δ splits over max($n_q$, the readout group's values per evaluation, Memo::series_count_scanned_below): max(quantile_over_time(0.9, m[5m])) over 1000 series at δ = 0.01 requires 1e-5 per readout, not 1e-2 (a_max_over_many_readouts_splits_the_failure_probability_over_them).
    • A factor-0 contract (count) passes no cap, so no readout gets an infinite requirement (a_count_over_readouts_gives_them_no_level).
    • exact_only_groups_of_query (a_query_comparing_a_shared_ratio_reads_it_only_exactly, a_readout_another_query_gives_a_level_survives_unknown_values_per_evaluation).
  • Error contracts: a reduction (count, min, max, avg) keyed by an estimate gets no contract: an error in a key can move a row to another group, so no bound on the output follows (a_reduction_keyed_by_an_estimate_has_no_contract).
  • Unknown values per evaluation: ValuesPerEvaluationUnknown prunes only a readout that no query gives a level; a readout another query reaches with a known $n_q$ keeps its levels, and the query without a per-value δ lists its group as exact-only (a_readout_another_query_gives_a_level_survives_unknown_values_per_evaluation).
  • Correctness: a pruned candidate has no edge to any live consumer, so no plan can use it; up to ten readouts each find a δ level whose sum stays within the per-value δ. Tests: dead_candidates_are_pruned_and_listed_before_chain_groups, a_single_readout_requires_the_query_target, and in the accuracy crate a_whole_result_failure_probability_is_split_per_value, a_ratio_reads_no_rank_error, a_ratio_whose_per_value_failure_probability_is_below_the_grid_splits_it_over_its_readouts, a_max_over_many_readouts_splits_the_failure_probability_over_them, a_count_over_readouts_gives_them_no_level.
  • Complexity: one top-down walk for reach, accepted metrics and caps per memo state (the edge passes run it once each); the readout metric once per candidate and the accepted metrics on inputs once per consumer, not per edge; elimination to a fixpoint over edges.
  • Assumptions and limits: Q2–Q4 below; Stage 3 does not yet enforce exact_only_groups_of_query.

Evaluation grids and chain groups

  • Problem: tumbling layouts were a property; widths came only from queries reaching a state through Stage 0 expressions; offsets were ignored; a chain could serve a reader whose evaluations it does not end on; below a PromQL subquery the outer grid was used.
  • After: evaluation grids per (query, group) walking down from each root (evaluation_grids.rs): below PromqlSubquery { resolution: Some(r) } the grid is r, with None it is unknown (no chain); evaluation_grid_of_group is the gcd over the reaching queries. Range-free chain groups after the reach; widths include offsets; built in two phases (all chain builds, then the TimeShift/TimeRange/merge groups) so a chain shared by state groups offers each every θ it has; a chain's edges into a reader whose own evaluation grid the chain's width does not divide fail with ChainWidthDoesNotDivideEvaluationGrid { consumer, input, tumbling_window_width_ms, evaluation_period_ms } (a 15 s chain never serves a reader every 10 s, which gave wrong answers before); dead-candidate elimination runs again after chain groups.
  • Correctness: tests a_chain_whose_width_does_not_divide_the_readers_grid_fails_its_edge, an_offset_enters_the_gcd_of_the_widths, one_range_free_chain_group_serves_every_range_and_offset, chain_groups_run_after_reach_so_a_rolled_up_query_adds_its_grid, an_absolute_anchor_reuses_only_existing_chains, no_chain_without_a_merge_an_anchor_or_a_grid, a_mergeable_state_reads_a_chain_with_one_build_per_surviving_parameters, a_groups_evaluation_grid_is_the_gcd_of_its_queries_grids, below_a_subquery_the_grid_is_its_resolution, below_a_subquery_without_a_resolution_there_is_no_grid, a_subquery_s_resolution_is_the_grid_of_the_groups_below_it.
  • Assumptions and limits: @ T reuses existing chains only; @ end()/@ start() get none.

Decisions and open questions

Needs reviewer decision:

  • Plan wire version. Deleting PhysicalASAPDAGNode::guarantee removes the guarantee key from serialized plans, and deny_unknown_fields rejects an older plan that has it. Bump INSTALLABLE_PLAN_WIRE_VERSION (14 → 15), or keep 14 (no plan in the field ever had a non-null value)? Not bumped in this PR.

Decided (all 2026-11-09 unless noted; concepts doc updated):

  • A1 (Q1 = (b)): a merge carries θ. SummaryMerge, and a Filter, TimeRange or TimeShift over states, carry the θ of the states they read: one candidate per θ (add_candidates_per_input_parameters), input edges only to producers with the same θ (Memo::producers); LogicalProperties::new is infallible and Memo::add_candidate checks θ against the group. α^log of a readout reads its producer's own θ. Rejected (a): α^log(merge → readout) holds only when every θ the merge may carry meets the requirement (safe, but a chain with weak builds blocks tight levels). Example (a_merge_is_judged_by_the_parameters_it_carries): P = quantile_over_time(0.9, m[5m]) at rank 0.005, Q = quantile_over_time(0.9, m[10m]) at rank 0.01, one 60 s chain with KLL k ∈ {269, 547, …}: SummaryMerge[k=547] → P's readout holds; SummaryMerge[k=269] is pruned with BelowRequiredAccuracy { required: rank 0.005 }.
  • A2 (D4 / bug 6): the metric rule cuts edges, not candidates. accepted_metrics pushes down, per group, the metrics its output may be off in (root: the target's metrics; input i of c: the accepted metrics c's contract carries). An edge is cut only when its producer is a sketch readout whose metric the consumer does not accept on that input, or, on a readout's own input, when its metric is not accepted at its output. Every surviving readout a targeted query reaches has a requirement, or fails with NoGuarantee. The metric rule in dead-candidate elimination is a union over queries, so a query can still reach approximate producers through a consumer with no contract for its metric; Pass 2 lists those groups per query in exact_only_groups_of_query, which Stage 3 enforces per query. Example (the_metric_rule_cuts_the_readout_edges_and_keeps_the_exact_division): avg by (job)(q0.5 / q0.5 + q0.9) at absolute 0.05: the readout edges into the division are cut, the exact division is not pruned and its edge into + holds.
  • Two-direction elimination: a candidate is also dead when every edge on one of its inputs is cut or reads a dead producer; the side it died on is recorded and its trace follows it. PruneCause.through_pruned_consumers → through_pruned_candidates.
  • N10: RootPruneReason::ValuesPerEvaluationUnknown { consumer, query }: a whole-result δ with $n_q$ unknown no longer gives per-value δ 0; a readout no query gives a level is pruned with this reason (a_whole_result_failure_probability_without_values_per_evaluation_prunes_the_readout). Rejected: δ = 0 (no sketch meets it, with no stated reason).
  • D1: δ levels continue below the per-value δ. Before, a per-value δ below 1e-5 gave each readout that δ itself and two readouts summed past it, so (K) had no solution. Example ratio-of-quantiles-1h-mvp: quantile_over_time(0.9, data[1h]) / quantile_over_time(0.5, data[1h]), relative 0.05, δ = 0.01 over 10,000 series = 1e-6 per value: both readouts now get levels 1e-6 and 1e-7 (1e-7 + 1e-7 ≤ 1e-6), and the full stack planned it at 7.481e-06/s before this PR's review fixes (one DDSketch α = 0.02 chain of 60 s windows, refactor: tools and snapshots over the #1207 memo, edges and contexts (#1207 PR 9) #1239); re-measured on restack.
  • D6 (approved): Memo::producers, CandidateError::LogicalProperties, accepted_metrics, MetricRule.accepted_metrics (was consumer_carries). Example: for merge[k=547] in P's state group, memo.producers(merge, 0) = the k = 547 candidate of TimeRange(5m, G_w) only; adding a SummaryMerge over KLL states without θ fails with CandidateError::LogicalProperties(SketchBuildWithoutParameters) (a_merge_carries_the_parameters_of_the_states_it_reads).
  • P1 (N3): error_propagation takes the memo, because the key-only Filter contract (E1(b)) reads the input group's column lineage. Example: over R of quantile by (job)(latency) (columns [job: SourceColumn(latency.job), value: Computed]), Filter(job = "api", R) gives Some { factor_per_input: [1.0], .. } in every metric; Filter(value > 0.5, R) gives None. Rejected: the doc signature (consumer, metric) with the caller deciding filters (two places decide contracts).
  • P2 (N4): producer_satisfies_logical_requirements returns the failing reason. Example: KLL k = 26 into the p99 readout requiring relative 0.02 gives Err(BelowRequiredAccuracy { consumer: G4#2, input: 0, required: relative 0.02 δ 1 }); LogicalEdge::satisfies_logical_requirement is .is_ok(). Rejected: a bool plus a second function (decides it twice).
  • P3 (N5): RootPruneReason::HeapSmallerThanTopK (the doc's α^log includes "a top-k's heap holds k", which fails also without a requirement). Example: topk(50, …) over CmsWithHeap: the edges from the 56 builds with heap 10 or 20 fail, so they are pruned with HeapSmallerThanTopK { k: 50 } (a_heap_smaller_than_k_fails_the_topk_readouts_edge).
  • P4 (N6, on condition it duplicates nothing): readout_metric reads the metric off the estimators' guarantee. Example: (Sketch(Kll), Quantile{0.5}) → Rank; (Sketch(DDSketch), Quantile{0.5}) → Relative; (Sketch(Hll), Cardinality) → Relative; (ExactAggregate(..), _) → None. Used by step 3 and the metric rule; a readout's summary type is read with GroupProperties::sketch_algorithm_of_state, the one helper for it.
  • P5 (N6): AccuracyMetric::ALL. P6 (N6): AggregateCoverage::{definition, selection} and SelectionBox::columns public (were test-support only); constraint_of public.
  • Q2 (R5): the metric rule applies to an edge only when a query with an accuracy target reaches the consumer; with accuracy not enforced every edge passes (without_targets_every_edge_is_admissible), matching "(K) has no rows".
  • Q3 (R5): a query is composed when a candidate it reaches has two or more inputs below which a readout group lies (a ratio); readouts of one value by different summary types are alternatives, not a composition.
  • Q4 (R5; the $n_q$-unknown case changed by N10): when no grid level is within the cap (exact target, cap 0) the readout gets one level at (cap, δ).
  • Q5 (R5): $G^$ is built over the rows of a member whose selection equals the union; no new Scan of the union of label sets, so {job="a"} with {job="b"} gets no $G^$ (scan widening, R7, not folded in). Members grouped without keys or per entity are not rolled up; interval constraints must be equal within a cluster.
  • Q6 (R5): a chain group gets one build per θ of the state group's surviving builds (union over the state groups sharing it).
  • 2026-11-08 (PR 1 Q6): test-only sizing replaced by the per-entry guarantee test (every_listed_parameters_has_a_guarantee_at_one_percent, a_guarantee_is_evaluated_at_the_required_failure_probability).
  • 2026-11-12 (cleanup review): one caps walk gives reach, accepted metrics and caps; readout metrics and accepted metrics on inputs are tables computed once per edge pass; readout_levels keys its result by readout candidate; MAX_ERROR_LEVELS lives with readout_levels; guarantee private; accuracy-math tests in the accuracy crate; shared memo builders in asap_types::test_support.

Verification

Test scopes run at head c8e67f130 with cargo test -p <crate>; the workspace total is cargo test --workspace.

Scope Passed / failed / ignored Notes
asap-accuracy 27 / 0 / 0 error contracts, guarantee tables, readout metric, readout levels, metric rule, edge check (the composition, budget, propagation and sizing tests of main are deleted with their code)
asap-types 235 / 0 / 0 150 unit, 77 integration, 8 doc; memo tests for θ on pass-through candidates, Memo::producers input requirements, add_candidate θ check
asap-stage1-pass2-reuse-candidates 27 / 0 / 0 all new; what the stage does with the accuracy results
asap-stage1-pass1-per-computation-candidates 37 / 0 / 0 shared memo builders
Workspace 1212 / 184 / 9 before this cleanup (31df962bc): 1211 / 184 / 9

Every workspace failure stops at a later-PR todo!() (Stage 2 entry stage2-physical-optimization/src/lib.rs, the executor's θ, devtools) or at materialized_view.rs:529 / materialized_views.rs:67 (fixture). I did not compare the failures with main's by test name.

Behavior-preserving cleanup check: the last cleanup commit is meant to change no result. A dump program printing Pass 2's full output (groups, candidates, requirements, edges, pruned candidates with causes, exact-only groups, plus an integrity checker) ran over 113 workloads (PromQL and SQL cases, chains, subqueries, rollups, random seeds) before (31df962bc) and after: every dump is identical except one, where the accepted_metrics list inside MetricRule reasons is now in AccuracyMetric::ALL order (the same set; it was the old walk's insertion order); the integrity checker reports 0 problems on all 113.

New tests:

  • Pass 2 crate: rollup (a_coarse_exact_sum_rolls_up_the_finer_sum, a_coarse_count_rolls_up_the_finer_count_as_a_sum, a_filtered_coarse_sketch_state_merges_the_finer_states_through_a_compensating_filter, a_finest_common_group_no_query_asks_for_serves_both, a_narrowing_on_a_column_that_is_no_key_gives_no_rollup, a_compensating_filter_over_a_sql_table_has_sql_semantics, a_not_equal_matcher_compensates_with_promql_semantics, a_rejected_rollup_leaves_no_compensating_filter); top-k (a_smaller_topk_reads_the_largest_topk_of_the_same_input); requirements (a_single_readout_requires_the_query_target, a_whole_result_failure_probability_without_values_per_evaluation_prunes_the_readout, a_readout_another_query_gives_a_level_survives_unknown_values_per_evaluation); dead candidates (dead_candidates_are_pruned_and_listed_before_chain_groups); chains (a_mergeable_state_reads_a_chain_with_one_build_per_surviving_parameters, a_chain_whose_width_does_not_divide_the_readers_grid_fails_its_edge, an_absolute_anchor_reuses_only_existing_chains, no_chain_without_a_merge_an_anchor_or_a_grid, chain_groups_run_after_reach_so_a_rolled_up_query_adds_its_grid, an_offset_enters_the_gcd_of_the_widths, one_range_free_chain_group_serves_every_range_and_offset); grids (a_groups_evaluation_grid_is_the_gcd_of_its_queries_grids, below_a_subquery_the_grid_is_its_resolution, below_a_subquery_without_a_resolution_there_is_no_grid); exact-only (a_query_comparing_a_shared_ratio_reads_it_only_exactly); edges (every_candidate_of_an_input_group_is_an_edge_of_matching_parameters, without_targets_every_edge_is_admissible, a_merge_is_judged_by_the_parameters_it_carries).
  • Accuracy crate: readout.rs a_readouts_metric_is_its_algorithms (KLL quantile Rank, DDSketch and HLL Relative, exact None); readout_caps.rs a_whole_result_failure_probability_is_split_per_value, a_ratio_whose_per_value_failure_probability_is_below_the_grid_splits_it_over_its_readouts, a_max_over_many_readouts_splits_the_failure_probability_over_them, a_count_over_readouts_gives_them_no_level; metric_rule.rs a_ratio_reads_no_rank_error, the_metric_rule_cuts_the_readout_edges_and_keeps_the_exact_division, a_key_only_filter_keeps_the_ddsketch_readout_admissible, avg_of_readouts_under_a_relative_target_reads_exact_only; logical_edge.rs target_limit_admits_only_floating_point_noise, a_distinct_count_readout_without_an_evaluable_guarantee_serves_no_query_with_a_target, a_heap_smaller_than_k_fails_the_topk_readouts_edge (56 builds with heap 10 or 20 fail HeapSmallerThanTopK { k: 50 }); error_propagation.rs a_reduction_keyed_by_an_estimate_has_no_contract, key_filters_and_value_preserving_projects_have_unit_contracts, error_contracts_of_a_ratio_a_count_and_a_sum; guarantee/mod.rs a_guarantee_is_evaluated_at_the_required_failure_probability, every_listed_parameters_has_a_guarantee_at_one_percent.
  • Integration: pass2_rollup_promql.rs (listed under Rollup); pass2_accuracy_and_grids_promql.rs a_function_of_the_value_keeps_the_readouts_below_it (abs, negation and / 2 over a sketch readout keep it), a_subquery_s_resolution_is_the_grid_of_the_groups_below_it.
  • Types: a_merge_carries_the_parameters_of_the_states_it_reads, kll_builds_with_k_269_and_k_547_share_one_group, set_candidate_input_requirements_replaces_only_the_named_candidate, add_candidate_checks_parameters_against_the_groups_output.

End-to-end evidence: none at this head, since planning stops at Stage 2's todo!. ratio-of-quantiles-1h-mvp planned at 7.481e-06/s on the #1239 snapshot before this PR's review fixes; it is re-measured on restack. Screenshots: not applicable.

Self-review: one add_candidate_if_output_matches helper in Pass 2, one readout_metric, one sketch-algorithm helper, one target-metric expansion (AccuracyTarget::metrics), contracts only in error_propagation, θ only in parameter_list (no strongest/weakest/floor tables); no #[allow(dead_code)] added.

Follow-ups

@zzylol zzylol added optimizer Workload optimization and plan selection refactor labels Nov 7, 2026
@zzylol
zzylol force-pushed the refactor/1207-pass1 branch from d4bde07 to 441f3a9 Compare November 7, 2026 04:14
@zzylol
zzylol force-pushed the refactor/1207-pass2 branch from 72a4037 to 24dd00c Compare November 7, 2026 05:10
@zzylol
zzylol force-pushed the refactor/1207-pass1 branch from 2c6d993 to 61f585a Compare November 9, 2026 06:37
@zzylol
zzylol force-pushed the refactor/1207-pass2 branch from bdf0888 to bcae9e5 Compare November 9, 2026 07:11
@zzylol zzylol changed the title refactor(stage1-pass2): reuse rules over groups, logical edges, metric rule, validity and pruning caps (#1207 PR 4) refactor(stage1-pass2): rollup from the finest common group, readout requirements, dead-candidate elimination, chain groups (#1207 PR 4) Nov 9, 2026
@zzylol
zzylol force-pushed the refactor/1207-pass1 branch from 61f585a to a71e517 Compare November 9, 2026 16:50
@zzylol
zzylol force-pushed the refactor/1207-pass2 branch 2 times, most recently from 8107319 to 5d3f6de Compare November 9, 2026 20:52
@zzylol
zzylol force-pushed the refactor/1207-pass1 branch from a71e517 to d77278a Compare November 10, 2026 01:26
@zzylol
zzylol force-pushed the refactor/1207-pass2 branch from 5d3f6de to dea088e Compare November 10, 2026 01:26
@zzylol
zzylol force-pushed the refactor/1207-pass1 branch from d77278a to 0c11cd7 Compare November 10, 2026 13:37
@zzylol
zzylol force-pushed the refactor/1207-pass2 branch from dea088e to e98e80b Compare November 10, 2026 13:37
@zzylol
zzylol force-pushed the refactor/1207-pass1 branch from 0c11cd7 to 321a1ae Compare November 11, 2026 03:35
…requirements, dead-candidate elimination, chain groups (#1207 PR 4)

Pass 2 runs, in order: the general rollup rule with one synthesized
finest common group per cluster of equal definitions (replaces scan
widening, finer-aggregate reuse and the rollup rules); top-k reuse; reach,
caps and readout requirements (one readout candidate per requirement level,
written with Memo::set_candidate_input_requirements); dead-candidate
elimination with every root cause (PrunedLogicalCandidate); chain groups
SummaryTumblingWindowChain[w](X) for the surviving state groups, widths
from the gcd closure of the reaching queries' grids, the window and the
offsets, then elimination again; and the logical edges between survivors.
A merge, and a Filter/TimeRange/TimeShift over states, carries the θ of
the states it reads (one candidate per θ, reading only producers with that
θ). The metric rule cuts edges from readouts whose metric the consumer's
input does not accept (accepted metrics pushed down from the targeted
queries), not candidates; a candidate cut off from an input is eliminated
too. A whole-result δ with n_q unknown prunes the readouts it reaches
(ValuesPerEvaluationUnknown).

- stage1-pass2: the entry generate_logical_asap_memo_candidates and the
  possible edges in lib.rs; rules.rs (Pass2RuleError, add_candidate,
  add_candidates_per_input_parameters, add_group_computing_itself),
  rules/{rollup,topk,chains}.rs, requirements.rs, dead_candidates.rs,
  tests.rs and tests/decided_examples.rs.
- types: AccuracyMetric::ALL; RootPruneReason::{HeapSmallerThanTopK,
  ValuesPerEvaluationUnknown}; MetricRule.accepted_metrics;
  PruneCause.through_pruned_candidates; Memo::producers; add_candidate
  checks θ against the group's state (CandidateError::LogicalProperties,
  candidate::parameters_fit); a candidate's input requirements are part of
  its identity (hash_consing::same_candidate); LogicalProperties::new
  accepts any θ on a candidate passing states on; aggregate coverage and
  selection accessors no longer test-only; origins SummaryInput and
  WidenedScans deleted.
- accuracy: error_propagation(memo, consumer, metric) with the constant
  contracts (key-only Filter by column lineage, abs / clamp / negation,
  min / max / avg / count, + - / *), b = 0.1 private;
  theoretical_guarantee at a failure probability (KLL and KMV evaluated at
  it, others checked against their own);
  producer_satisfies_logical_requirements returns the failing root reason;
  accepted_metrics and readout_metric. Node-tree composition, propagation
  (with its count-min estimate_has_bound test), budget split, function
  rules, PropagationStats and the test-only sizing (guarantee/sizing_oracle)
  deleted; the classic HLL precision tests move into guarantee/hll.rs.
- devtools: one pass2 stage; docs follow.
- integration-tests: pass1_candidates_sql re-adds
  a_filter_or_group_key_over_an_estimate_has_no_guarantee.
@zzylol
zzylol force-pushed the refactor/1207-pass2 branch from e98e80b to 5ac57fc Compare November 11, 2026 18:33
@zzylol
zzylol changed the base branch from refactor/1207-pass1 to main November 11, 2026 18:33
… guarantees no stage reads yet (#1207 PR 4)

classic_guarantee, ClassicHllConfidence, classic_failure_probability and
the classic statistic helpers, and UnivMon's distinct_guarantee with its
incomplete-top-layer statistic, had no caller outside their own tests.
PR 8 re-adds what its HLL and UnivMon distinct-count guarantees use.
… error contract (#1207 PR 4)

An Aggregate or exact accumulator build gave a count beta = 0 (and
min/max/avg beta = 1) without checking where its group-by keys come
from. Grouping by an estimate moves rows between groups, so the cap
passed down was infinite and no readout was pruned. The keys now have
to be copied from a source, as a Filter's predicate columns must.
…dout no query gives a level (#1207 PR 4)

A query whose whole-result failure probability has no per-value one
marked every readout copy it reached, including the levels other
queries wrote, so a readout another query could use was pruned with
ValuesPerEvaluationUnknown. Only a readout without any level is pruned
now.
…ation grid its width divides (#1207 PR 4)

Chain widths are the gcd closure over every query reaching a state
group, so a state read every 15 s and every 10 s got 5, 10 and 15 s
chains, and the 10 s reader could be given the 15 s chain. The edge
from a SummaryTumblingWindowChain into a group whose result depends on
the evaluation time now requires the width to divide that group's
evaluation grid; the merges over the other widths are pruned.
…er, with the source frontend's comparison semantics (#1207 PR 4)

rollup_from added the compensating Filter group before the memo checked
the rollup candidate, so a rejected rollup (COUNT(*) WHERE src = 'a'
from COUNT(*) GROUP BY src) left a Filter group no group reads. The
Filter now stays pending until the rollup over it derives the coarse
group's schema (Memo::output_schema_of_pending_expression).

The Filter also took PromQL comparison semantics for any group that is
not a relation, such as an exact sum state over a SQL table; it now
takes the semantics of the frontend that reads the column's source.
… and chain edges as the code does them (#1207 PR 4)

The TopKReuse row described a Limit(k_max) over Sort; the rule builds
Aggregate[TopK k] over the largest top-k group's ranked rows. A reader
without a declared phase has no known evaluation grid, so no chain is
built for it, rather than being taken to evaluate at multiples of its
interval. The TumblingWindows row now states the chain-width edge rule.
…g: per-series layers and scans listing different labels)
…PromQL scans by lineage

A coarse cross-series aggregate over PromQL rows now rolls up from a
finer one: the rule writes each member's input out down to the source,
each TimeRange, TimeShift and per-series Aggregate an unlifted operator
(same range, same per-series function), lifts the scan's label matchers
into the selection by source column, and drops the scan's schema when
no unlifted operator names a column by position, so scans listing
different labels of one metric compare equal. constraint_of is public
for the lift.
…chStatistic::for_intent, drop PR numbers

Delete CompositionOperator, AccuracyError, the GuaranteeSource variants
nothing creates, AccuracyGuarantee::over_scope and approximate_layer_count,
and VALUES_PER_RESULT, which only over_scope used. Move sketch_statistic
from asap-accuracy's sizing module to SketchStatistic::for_intent and delete
sizing.rs. Remove #509 from the Pass 2 crate header and #694 from two
comments.
Replace the descriptions of the deleted composition rules, budget split,
propagation statistics and sizing module with the error contracts of
asap-accuracy and the readout requirement levels Pass 2 sets.
…, remove duplicates (#1207 PR 4)

Pass 2:
- Pass2RuleError moves to lib.rs; its LogicalProperties variant is gone
  (Memo::add_candidate is the one θ check).
- CandidateEdge and candidate_edges move to edges.rs.
- rules::add_candidate is renamed add_candidate_if_output_matches; one
  distinct_parameters helper and one query_evaluation_grid_ms lookup
  replace the copies in rules.rs, chains.rs and lib.rs.
- requirements.rs becomes readout_requirements.rs and keeps only the memo
  writes; the caps, query levels and failure-probability split move to
  asap_accuracy::readout_levels.
- Tests: fixtures in test_fixtures.rs on memo_fixtures, one test module
  per step, without dates or decision codes.

Accuracy:
- requirements.rs splits into logical_edge.rs (alpha^log) and
  metric_rule.rs (accepted_metrics, readout_metric, sketch_readout_metric);
  target_limit and the test-only meets move to guarantee/mod.rs.
- One guarantee lookup that takes the failure probability; Hydra reads its
  subpopulation's at the same probability. The fixed-probability KLL and
  KMV guarantees are deleted.
- readout_caps.rs: readout_levels, caps keyed by metric.

Types:
- LogicalProperties::new(parameters) is infallible.
- CandidateForMemoGroup::new and listed_build_parameters replace the
  per-crate copies of building a candidate and listing a build's θ.
- GroupProperties::sketch_algorithm_of_state moves to group.rs.
- MAX_ERROR_LEVELS is one constant beside the parameter lists.
- memo_fixtures::state_grouped_by.
…Pass 2 (#1207 PR 4)

- end-to-end-accuracy-guarantees.md: the composition rules give way to the
  error contracts error_propagation has, and where the metric rule, the
  readout levels and alpha^log use them.
- planner-layering.md: the ratio-of-quantiles example's Pass 2 step and
  the stage table follow the current rules and file names.
- accuracy-models.md and asap-aware-mapping-architecture.md: the accuracy
  crate's new files.
…valuation grid (#1207 PR 4)

A chain whose width does not divide the consumer group's evaluation period
no longer drops its edges silently: they fail with
RootPruneReason::ChainWidthDoesNotDivideEvaluationGrid, so the candidates
over that chain trace to it instead of an empty cause list.
…adouts without a guarantee (#1207 PR 4)

- A Project's contract judges only the items reading a computed value:
  copied labels, the time and promql_drop_metric_name of the series
  identity keep any error; promql_abs/clamp/clamp_min/clamp_max and a
  negation keep an absolute error; a product with or quotient by a nonzero
  literal keeps a relative one. abs(count_over_time(m[1m])),
  -count_over_time(m[1m]) and quantile_over_time(...) / 2 now keep their
  sketch readouts.
- readout_metric no longer needs a finite failure probability (HLL's
  distinct count is relative). Under a query with a target, an edge into a
  readout with no metric (UnivMon's distinct count) or whose guarantee's
  failure probability is unknown (HLL) fails with the new
  RootPruneReason::NoGuarantee { consumer, input }.
- sketch_guarantee is replaced by guarantee_of_parameters (θ's own
  guarantee) and the δ check in theoretical_guarantee.
…-in, per-query exact-only groups (#1207 PR 4)

- Evaluation grids are computed per (query, group) walking down from each
  root (evaluation_grids.rs): below PromqlSubquery { resolution: Some(r) }
  the grid is r, with None it is unknown (no chain). Chains and the chain
  edge check read it, and evaluation_grid_of_group is the corrected grid.
- A readout's per-value failure probability is δ / max(n_q, the readout
  group's values per evaluation), from the new
  Memo::series_count_scanned_below: max(quantile_over_time(0.9, m[5m]))
  over 1000 series at δ = 0.01 requires 1e-5 per readout.
- A factor-0 contract (count) passes no cap: no infinite requirement.
- New LogicalASAPMemoCandidates::exact_only_groups_of_query: per query,
  the groups it reaches through a consumer with no error contract for its
  metric, plus the readout groups another query gave a level while its own
  whole-result δ has no per-value one; closed downward. Stage 3 is to
  enforce it per query.
… with the frontend's comparison (#1207 PR 4)

- A rolled-up Aggregate names each measure as the coarse group's column,
  so count by (job) rolls up count by (job, instance) as a sum named
  count. The memo's schema check still rejects a SQL global count over a
  grouped one (sum over no group is NULL, the count 0).
- A compensating filter is built from comparisons with the source
  frontend's semantics: a PromQL != matcher becomes Compare Ne with PromQL
  semantics, NOT IN a conjunction of Ne, IN a disjunction of Eq.
…tracts, readout fan-in and exact-only groups (#1207 PR 4)
…ers, tests of floor helpers in the test module, updated file paths (#1207 PR 4)
…ted metrics and caps; readout metrics and accepted input metrics computed once per edge pass; readout.rs, binomial.rs, target_limit beside the edge check (#1207 PR 4)

- accepted_metrics derives from the readout caps' walk (a reached flag
  per group); AccuracyTarget::metrics is the one target-metric expansion.
- readout_levels keys its result by readout candidate, so Pass 2 no
  longer detects unrequired readouts a second time.
- readout_metric_of_candidate and metrics_accepted_on_inputs are computed
  once per edge pass / consumer instead of per edge; accepted metrics and
  evaluation grids once per memo state and passed in.
- A readout's summary type is read with
  GroupProperties::sketch_algorithm_of_state everywhere.
- MAX_ERROR_LEVELS moves next to readout_levels; guarantee, hll and
  univmon are private; kmv_guarantee and meets are inlined.
- Docs: a chain's edge fails with ChainWidthDoesNotDivideEvaluationGrid;
  max_distinct_per_evaluation has no reader yet.
…res in test_support; Pass 2 tests keep what the stage does with the accuracy results (#1207 PR 4)
…d failure probability, the delta grid, top-k and Hydra status, current Pass 2 and accuracy file layout (#1207 PR 4)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

optimizer Workload optimization and plan selection refactor

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant