Skip to content

refactor(stage1-pass1): per-computation candidates over memo groups (#1207 PR 3) - #1232

Draft
zzylol wants to merge 12 commits into
refactor/1207-memofrom
refactor/1207-pass1
Draft

zzylol wants to merge 12 commits into
refactor/1207-memofrom
refactor/1207-pass1

Conversation

@zzylol

@zzylol zzylol commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Merge unit. This is PR 3 of #1207's merge unit (PRs 0–9, with 0b). It is stacked on PR 2 (#1230, refactor/1207-memo), which is stacked on PR 1 (#1227), PR 0b (#1229) and PR 0 (#1226). The PRs merge into main together, after PR 9. This PR fills in Pass 1, which PR 0 left as todo!(). Planning now gets through Pass 1 and stops at the first todo!() of Pass 2 (PR 4); see "Test status".

Why

#1207's Order, row "3. Pass 1": Pass 1 adds each group's initial candidates (exact, exact accumulator, summary with one candidate per θ, fused) to the memo, with every intermediate result its own group (#1203: one operator per candidate, no private nodes). It also re-creates the Pass 1 test coverage that PR 0b deleted, with tests that build their input Memo through the Memo methods, so no stage crate depends on another, even in tests.

PR 0 left Pass 1 stopping at a private stub logical_node_trees(memo): target discovery walked node trees, and candidates came from a whole-workload node-tree composition (compose_logical_candidate) sized per budget share. Neither has a place in the memo model (concepts doc §3: "Everything that was computed on operator-node trees is computed on groups").

What

  • Target discovery over memo groups. computation_targets(memo, supported_summaries) reads memo.groups(): each group's expression and group_properties. It finds one ComputationTarget per measure of each Aggregate group with a realization besides plain exact execution, once however many queries read it. Consumers are the group's parents. The logical_node_trees stub is deleted.
  • Each way to compute a group becomes candidates, through Memo::find_or_add_group and Memo::add_candidate (add_pass1_candidates):
    • exact: every group's own expression is its first candidate (no θ, no floor);
    • exact accumulator: a state group SummaryAgg[ExactAggregate(kind)] (one candidate, no θ) and a FinalizeExactAccumulator over it. avg finalizes its sum and count states and divides them: a PromQL BinaryOp, or for SQL a Join group and a Project;
    • summary: one state group SummaryAgg[family] per supported family, with one build candidate per θ in parameter_list(family), all with the group's own expression (θ only in LogicalProperties::parameters). A separate readout group SummaryEstimate(state) is read by the result group;
    • fused builds: the whole-expression top-k (a heap state group over the rows below the absorbed sums) and the nested-sum merge (an exact Sum build over the rows below the inner sum, added as a candidate of the outer sum's own accumulator state group, Logical memo candidate as (one operator over groups, logical properties θ) with a requirement per input; no private nodes #1203 "fused builds"). Their builds list the absorbed groups in fused_groups;
    • multi-measure aggregates (SQL): one single-measure Aggregate group per measure, each with its own candidates, joined back on the group keys by one candidate of the aggregate's group;
    • HAVING: a key-only consumer Filter group gets the candidates that push its predicate below the aggregate (a Filter group over the input rows, which the states read). A value HAVING stays a Filter above each readout.
  • Every intermediate result is its own group; each candidate is one operator over input groups. A candidate must output its group's columns (add_candidate rejects it otherwise), so a readout whose column names differ from the group's (quantile for quantile_0_9) is its own group, read by a renaming Project candidate. A heap top-k's Float64 estimate is cast to the exact top-k's column type (a SQL count ranks Int64).
  • First LogicalRequirements of each consumer input (input_requirements): a top-k readout of a heap state needs heap_size ≥ k (the family's weakest listed width and depth with heap_size = k). Every other input has no floor. See "Decisions" for validity caps and the metric rule.
  • A way Pass 1 cannot build is not offered (Unsupported, e.g. a SQL count(*) with no non-null column): the group keeps its other candidates. A candidate that breaks the node rules or the group's output is a bug and fails Pass 1 (PerComputationCandidateError::Candidate).
  • pass1_memo(memo, supported_summaries) no longer takes the queries' accuracy targets, the accuracy model, evidence or data. They served only without_summaries_no_query_can_use, which composed whole-workload node trees to prune families no query could use; that pruning is query-sensitive (§4: "query-sensitive pruning follows logical generation and reachability analysis") and moves to PR 4's pruning caps.
  • Deleted (replaced by the memo candidates): compose_logical_candidate, compose, compose_roots, rewrite, substitute, substitute_measure, with_names_of, with_aggregate_qualifiers, whole_expression, nested_sum, Sizing, ChosenComputation, Chosen, PerComputationCandidates (with combinations, absorbed_choice_varies, without_unsupported_summaries, without_summaries_no_query_can_use), output_can_meet_target, sources_read, absorbed_targets, generate_per_computation_candidates, logical_node_trees, ComputationTarget::{choice_count, computation_for}, PerComputationCandidateError::InvalidChoice, the Pass 1 feature test-support and its test_support module; asap_accuracy::test_support::{estimate_has_bound, estimate_bound_over_raw_data_samples, local_guarantee} (their only callers were Pass 1's tests, which now live in asap-accuracy); the executor test helper computation_candidates (no caller) and the executor's dev-dependencies on Stage 0 and Pass 1.

Decisions and pending items

# Item Decision taken here Status
1 Where the parameter floor from the validity cap goes Pass 1 sets no validity-cap floor. θ lives only on build candidates in state groups, and the only consumers of state groups in Pass 1 are readouts, finalizers and merges, whose error propagation is linear (no validity cap). A non-linear consumer (a ratio's denominator) reads a readout group, whose candidates carry no θ, so a floor there would never bind. Translating that cap into a floor on the state-group edge goes through the readout, per edge: PR 4, with error_propagation (still todo!() there). Pass 1 sets only the direct need heap_size ≥ k. question for the user (recommendation: PR 4)
2 Where the metric rule goes PR 4 (edge building). It is $\alpha^{\log}_{u,c,i}$, a property of a (producer, consumer input) pair, and one group can have consumers with different error contracts (G4 is read by B's ratio and could be another query's root). So Pass 1 offers both KLL and DDSketch readouts to every quantile group, and PR 4 drops the KLL → ratio edges. question for the user (recommendation: PR 4)
3 Dropping θ whose error alone exceeds every reaching query's target (§2 level 2) PR 4. It is the pruning cap ($\max_q \varepsilon_q / \lambda_{q,c,i}$), which needs the queries reaching each edge and their factors; §4 says query-sensitive pruning follows logical generation. Pass 1 keeps the whole list. question for the user (recommendation: PR 4)
4 add_candidate's output check It now compares the output columns (name, type, nullability), time index and unique keys, and not each field's table qualifier or the schema's closed flag (same_output_columns, replacing ==). Without this, no candidate but the plain aggregate fits: a grouped SQL aggregate's keys keep their table while its measures have none, and a Project sets one qualifier for all columns; a PromQL per-series aggregate leaves its schema open, and the renaming Project closes it. The old composition patched the node's schema (with_aggregate_qualifiers), which a derived group schema cannot. Plans are unchanged: the old candidates renamed through the same Project. pending (changes PR 2's check)
5 fused_groups of a fused build The absorbed targets' groups, as memo_population.rs set them on main (its implements). Under edges, the candidate does not compute those groups' results (it reads the rows below them); a consumer of an absorbed group still needs one of that group's own producers. Stage 3 (PR 6) must not treat fused_groups as available results. question for the user
6 Plain exact candidate beside the accumulator Every group offers its own expression; an aggregate with an accumulator form offers both (old Pass 1 listed only the accumulator as "the one exact option"). Stage 3 picks the cheaper. pending
7 HAVING push-down A key-only consumer Filter's predicate goes below in the Filter group's own candidates, whatever the aggregate's other consumers read; old Pass 1 pushed it only when every consumer applied it (#1089), because a candidate replaced the aggregate for all of them. pending
8 Nested-sum merge placement A candidate of the outer sum's accumulator state group (#1203 "fused builds": "a fused build with the outer target's state is a candidate of that state group"), reading the raw rows. per #1203
9 pass1_memo arguments Drops accuracy_targets, accuracy, evidence and data (see What). pending
10 Tests outside Pass 1 that composed node trees Helpers that returned composed node trees to execute (planning::explicit_local_candidates, planning::explicit_summary_candidate, avg_as_sum_and_count) now todo!("#1207 PR 6: ..."): realizing one plan per way from the memo is PR 6's realizer. Interim inside the merge unit; PR 6 fills them in. pending

New abstractions not in #1207 or the doc (pending approval)

Item Where Why
pub struct MemoGroupAsInput (was pub(crate)), MemoGroupAsInput::group_id(), Memo::group_as_input(group_id) types/src/stages/{memo,candidates}.rs Pass 1 reads a group's operator through the same OperatorInput rules as a node (ranked_column, the keyed-heap rule, nested-sum detection), and needs the group id back to build over it.
private same_output_columns(left, right) types/src/stages/candidates.rs Decision 4.
raw_data_samples_of_metric<I: OperatorInput>(operator: &Operator<I>) (was (node: &OperatorNode)); any_measure_filtered<C> (was node-typed) accuracy/src/propagation.rs, types/src/ir/operator/non_asap.rs Read a group's expression and a node alike; no copy of either.
HavingEntry::filter: Option<MemoGroupId> (was Option<Rc<OperatorNode>>); HavingEntry::predicate, ComputationTarget::measure_filter typed Predicate<MemoGroupId> types/src/stages/per_computation_logical_memo_candidates.rs Targets name groups, not nodes.
PerComputationCandidateError::{Candidate(CandidateError), LogicalProperties(LogicalPropertiesError)} Pass 1 A candidate the memo rejects (bug in the rule that built it).
private computation_targets, add_pass1_candidates, AggregateWays, SingleMeasure, add_aggregate_candidates, add_fused_candidate, measure_output, accumulator_state, finalized_with, accumulator_build, exact_accumulator_build, add_output_candidate, renaming_project, add_group, add_state_group, add_candidate, input_requirements, heap_floor, unless_unsupported, expression, schema, without_unsupported_summaries (free fn); realization::non_asap, generic PhysicalSummaryInput<I> Pass 1 The implementation; the ported helpers (nested_sum_input, whole_expression_input, realize_keyed_additive_summary_input, summed_per_series_samples, ranked_value, reads_only_time_series, measure_input, counted_column) are generic over OperatorInput instead of taking nodes.
test helpers: metric_schema, scan_operator, group, scan_group, aggregate_group(_with), query (test_fixtures.rs); memo_printout, label, candidates, groups_where, state_group, range, instant, table_group, compare, filter, divide, quantile, sum, count, topk, pushed_below (Pass 1 tests); pass1, pass1_over, having_pushed_below (per_computation_candidates_sql.rs); pass1, root_group, aggregate_groups, candidates_of_aggregates, sketch_builds, avg_candidate_groups, pass1_sql (stage1_logical_asap_optimization.rs) tests Fixtures.

How

  1. computation_targets walks memo.groups(). For each Aggregate expression it records the own having and each parent Filter group whose input is the aggregate (key-only → BelowAggregate). It lists the ways per measure (computation_candidates_for_intent, unchanged), then detects fused computations over groups (add_whole_expression_targets, add_nested_sum_targets, close_absorptions), prunes a PromQL top-k's own heaps, and drops unsupported families.
  2. add_pass1_candidates adds every Stage 0 group's own expression as a candidate, then, per aggregate group (and per key-only Filter group above it), add_aggregate_candidates builds each way bottom-up: add_state_group (θ list or one exact build), add_group for each intermediate (the operator is also its group's candidate), and add_output_candidate for the top operator: added to the result group, or, when its column names differ, added as its own group and read through renaming_project.
  3. Every candidate goes through add_candidate (this crate) → LogicalProperties::new (θ in the list, none on exact operators) → input_requirements → Memo::add_candidate.

Key code interfaces added / modified

stage1-pass1-per-computation-candidates/src/logical_memo_candidates.rs

/// Stage 1, Pass 1's entry: adds each group's initial candidates to Stage 0's memo. Stays.
pub fn pass1_memo(
    memo: Memo,
    supported_summaries: Option<&[SupportedSummary]>,
) -> Result<Memo, Pass1Error>; // was (memo, accuracy_targets, accuracy, evidence, data, supported_summaries)
/// Why Pass 1 failed. Stays.
pub enum Pass1Error {
    /// Bad input (ASAP operators, invalid accuracy) or a bug (a rejected candidate).
    PerComputation(PerComputationCandidateError), // unchanged
}

stage1-pass1-per-computation-candidates/src/per_computation_candidates.rs

/// Why Pass 1 could not add a group's candidates. Stays.
pub enum PerComputationCandidateError {
    /// A group's expression breaks the node rules: bad input. // unchanged
    Structure(SchemaDerivationError),
    /// A Stage 0 group is already an ASAP operator: bad input. // unchanged
    ContainsAsap,
    /// A measure's approximate target has ε or δ outside (0, 1): bad input. // unchanged
    InvalidAccuracy { root: Option<String>, measure: Option<usize> },
    /// `Memo::add_candidate` rejected a candidate Pass 1 built: internal bug. // new
    Candidate(CandidateError),
    /// A candidate's θ or window does not fit its operator: internal bug. // new
    LogicalProperties(LogicalPropertiesError),
    /// A way Pass 1 cannot build; it is not offered (never returned by `pass1_memo`). // unchanged
    Unsupported(&'static str),
    // InvalidChoice — removed (no whole-workload choices)
}
// removed: pub mod test_support { compose_logical_candidate, generate_per_computation_candidates }, feature `test-support`

types/src/stages/memo.rs, types/src/stages/candidates.rs

/// A memo group read as an operator's input. Stays; now `pub` (was `pub(crate)`).
pub struct MemoGroupAsInput<'m> {
    /// The memo. // unchanged, pub(crate)
    pub(crate) memo: &'m Memo,
    /// The group read. // unchanged, pub(crate)
    pub(crate) group_id: MemoGroupId,
}
impl MemoGroupAsInput<'_> {
    /// The group read. // new
    pub fn group_id(&self) -> MemoGroupId;
}
impl Memo {
    /// Group `group_id` read as an operator's input (`OperatorInput`). // new
    pub fn group_as_input(&self, group_id: MemoGroupId) -> MemoGroupAsInput<'_>;
}
/// add_candidate's output check: columns (name, type, nullability), time index, unique keys;
/// not each field's table qualifier nor `closed`. // new (private), was `schema ==`
fn same_output_columns(left: &Schema, right: &Schema) -> bool;

types/src/stages/per_computation_logical_memo_candidates.rs

/// One measure of an aggregate group with a realization besides exact execution. Stays.
pub struct ComputationTarget {
    /// The aggregate's group. // unchanged
    pub aggregate_group: MemoGroupId,
    /// The measure's index. // unchanged
    pub measure_index: usize,
    /// Its ways to compute, the exact option first. // unchanged
    pub ways_to_compute: Vec<HowToComputeAggregateMeasure>,
    /// The measure's own filter. // was Option<Predicate>
    pub measure_filter: Option<Predicate<MemoGroupId>>,
    /// The aggregate's own `having`, then each consumer `Filter` group's. // unchanged
    pub having_filters: Vec<HavingEntry>,
    /// The fused computation with the targets beneath. // unchanged
    pub fused_computation: Option<FusedComputation>,
}
// removed: ComputationTarget::choice_count, ComputationTarget::computation_for
/// One `HAVING` over a target's output. Stays.
pub struct HavingEntry {
    /// The consumer `Filter` group that applies it; `None` for the own `having`. // was Option<Rc<OperatorNode>>
    pub filter: Option<MemoGroupId>,
    /// The predicate over the aggregate's output. // was Predicate
    pub predicate: Predicate<MemoGroupId>,
    /// Below the aggregate (key-only) or above the output. // unchanged
    pub placement: HavingPlacement,
}

accuracy/src/propagation.rs, types/src/ir/operator/non_asap.rs

/// The metric whose raw samples (or plain sums of them) `operator` outputs. Stays.
pub fn raw_data_samples_of_metric<I: OperatorInput>(operator: &Operator<I>) -> Option<String>; // was (node: &OperatorNode)
/// Whether any measure has its own filter. Stays.
pub fn any_measure_filtered<C>(filters: &[Option<Predicate<C>>]) -> bool; // was (&[Option<Predicate>])
// removed from asap_accuracy::test_support: estimate_has_bound, estimate_bound_over_raw_data_samples, local_guarantee

Usage (from running_example_gives_one_build_per_theta_and_separate_readouts, run at this head):

// Stage 0's memo, built as Stage 0 builds it: `latency` (instant selector),
// p95 / p99 / p50 aggregates over it, and B's ratio p99 / p50.
let mut memo = Memo::default();
let g1 = instant(&mut memo, "latency", &[]);            // the input rows (doc G1)
let g3 = aggregate_group(&mut memo, g1, Reduction::by(vec![]), vec![quantile(0.95)]); // query A
let g4 = aggregate_group(&mut memo, g1, Reduction::by(vec![]), vec![quantile(0.99)]);
let g5 = aggregate_group(&mut memo, g1, Reduction::by(vec![]), vec![quantile(0.5)]);
let g6 = divide(&mut memo, g4, g5);                      // query B
memo.add_query(QueryRoot::Operator(g3));
memo.add_query(QueryRoot::Operator(g6));
// Pass 1 (the facade calls it after Stage 0 with the deployment's supported summaries).
let memo = pass1_memo(memo, None)?;
// KLL's state group: 7 build candidates, θ = k ∈ {26, 52, 132, 269, 547, 1404, 2863};
// DDSketch's: 7, α ∈ {0.1 … 0.001}; G3/G4/G5: own aggregate + one readout per family;
// G6: its own BinaryOp. Pass 2 (PR 4) adds reuse candidates and the edges, where the
// metric rule drops KLL → B's ratio; Stage 3 picks one producer per input.

Major algorithm changes

Pass 1 candidate generation.

  • Problem. On PR 2, Pass 1 stops at todo!(). Before PR 0, it built one whole-workload node tree per choice and sized each sketch for an equal budget share; the result was copied into private node chains.
  • Before / after. Before: targets over node trees → compose_logical_candidate(choice) → node tree per choice → sizing per accuracy_budget_parts → private chains. After: targets over groups → per way, groups bottom-up through find_or_add_group, one build candidate per listed θ, one operator per candidate.
  • Correctness. Every candidate passes the node rules and outputs its group's columns (add_candidate); the ported frontend tests (stage1_logical_asap_optimization::{sql,promql}_candidates_output_the_query_columns) run every SQL and PromQL query of the old composition test through Pass 1.
  • Complexity. Linear in the targets × ways; each summary way adds |parameter_list(family)| (4–70) build candidates to one shared state group, instead of one node tree per whole-workload combination.
  • Limits. No accuracy pruning (PR 4). Hydra families have empty lists (Planner: error models and parameter lists for Hydra (and other unsized families), so Pass 1 can offer them #1228) and are not offered.

Before / After

The doc's running example (concepts doc §1): query A = p95 of latency, query B = p99 / p50 of it (running_example_gives_one_build_per_theta_and_separate_readouts). Before (PR 2): pass1_memo panics at todo!("#1207 PR 3: discover targets over the memo groups, not node trees"). After, the memo (printed by the test; _ = PerSubpopulationInstance):

G0: Scan
  [0] Scan inputs=[]
G1: TimeRange
  [0] TimeRange inputs=[G0]
G2: Aggregate[Quantile]
  [0] Aggregate[Quantile] inputs=[G1]
  [1] Project inputs=[G7]
  [2] Project inputs=[G9]
G3: Aggregate[Quantile]
  [0] Aggregate[Quantile] inputs=[G1]
  [1] Project inputs=[G10]
  [2] Project inputs=[G11]
G4: Aggregate[Quantile]
  [0] Aggregate[Quantile] inputs=[G1]
  [1] Project inputs=[G12]
  [2] Project inputs=[G13]
G5: BinaryOp
  [0] BinaryOp inputs=[G3, G4]
G6: SummaryAgg[Sketch(Kll, _)]
  [0] SummaryAgg[Sketch(Kll, _)] inputs=[G1] θ=Kll { k: 26,  }
  [1] SummaryAgg[Sketch(Kll, _)] inputs=[G1] θ=Kll { k: 52,  }
  [2] SummaryAgg[Sketch(Kll, _)] inputs=[G1] θ=Kll { k: 132,  }
  [3] SummaryAgg[Sketch(Kll, _)] inputs=[G1] θ=Kll { k: 269,  }
  [4] SummaryAgg[Sketch(Kll, _)] inputs=[G1] θ=Kll { k: 547,  }
  [5] SummaryAgg[Sketch(Kll, _)] inputs=[G1] θ=Kll { k: 1404,  }
  [6] SummaryAgg[Sketch(Kll, _)] inputs=[G1] θ=Kll { k: 2863,  }
G7: SummaryEstimate[Quantile { q: 0.95 }]
  [0] SummaryEstimate[Quantile { q: 0.95 }] inputs=[G6]
G8: SummaryAgg[Sketch(DDSketch, _)]
  [0] SummaryAgg[Sketch(DDSketch, _)] inputs=[G1] θ=DDSketch { alpha: 0.1 }
  [1] SummaryAgg[Sketch(DDSketch, _)] inputs=[G1] θ=DDSketch { alpha: 0.05 }
  [2] SummaryAgg[Sketch(DDSketch, _)] inputs=[G1] θ=DDSketch { alpha: 0.02 }
  [3] SummaryAgg[Sketch(DDSketch, _)] inputs=[G1] θ=DDSketch { alpha: 0.01 }
  [4] SummaryAgg[Sketch(DDSketch, _)] inputs=[G1] θ=DDSketch { alpha: 0.005 }
  [5] SummaryAgg[Sketch(DDSketch, _)] inputs=[G1] θ=DDSketch { alpha: 0.002 }
  [6] SummaryAgg[Sketch(DDSketch, _)] inputs=[G1] θ=DDSketch { alpha: 0.001 }
G9: SummaryEstimate[Quantile { q: 0.95 }]
  [0] SummaryEstimate[Quantile { q: 0.95 }] inputs=[G8]
G10: SummaryEstimate[Quantile { q: 0.99 }]
  [0] SummaryEstimate[Quantile { q: 0.99 }] inputs=[G6]
G11: SummaryEstimate[Quantile { q: 0.99 }]
  [0] SummaryEstimate[Quantile { q: 0.99 }] inputs=[G8]
G12: SummaryEstimate[Quantile { q: 0.5 }]
  [0] SummaryEstimate[Quantile { q: 0.5 }] inputs=[G6]
G13: SummaryEstimate[Quantile { q: 0.5 }]
  [0] SummaryEstimate[Quantile { q: 0.5 }] inputs=[G8]

Mapping to the doc's table: G1 here is the doc's G1 (latency, via Scan G0 and TimeRange G1); G6 = doc G2 (KLL state, one build per θ of KLL's list; the doc's k = 200/345 are illustrative); G8 = doc G2′ (DDSketch); G2/G3/G4 = doc G3/G4/G5, each with its exact aggregate and a renaming Project over a readout group per family (G7/G9, G10/G11, G12/G13); G5 = doc G6. The doc shows G4/G5 reading only G2′: that is the metric rule, which drops the KLL → ratio edges in PR 4 (decision 2).

Verification

cargo check, cargo clippy --workspace --all-targets --all-features -- -D warnings and cargo fmt --all --check are clean.

Pass 1's tests (stage1-pass1-per-computation-candidates/src/per_computation_candidates_tests.rs, 25 tests). Each builds its memo through Memo::find_or_add_group and Memo::add_query (test_fixtures.rs); no other stage crate is a dependency, even in tests. PR 0b's list (#1229), by category:

Category Test Verifies
running example running_example_gives_one_build_per_theta_and_separate_readouts KLL and DDSketch state groups hold one build per θ of their list, with the group's own expression; p95/p99/p50 each read either state through a readout group; the ratio keeps its own expression
target discovery multi_measure_aggregate_has_one_target_per_measure each measure of a multi-measure aggregate is a target
aggregate_without_realization_is_not_a_target an exact-only measure gets no candidate beyond its own expression
an_aggregate_read_by_several_roots_is_one_target a group read by two roots is one target
asap_input_and_invalid_accuracy_are_rejected ASAP groups and ε outside (0, 1) fail at the boundary
unsupported_summaries_are_not_offered the deployment's unsupported families add no group
candidate composition every_group_offers_its_own_expression every Stage 0 group's first candidate is its expression, no θ, no floor
exact_accumulator_is_a_state_group_and_a_finalize an accumulator is a one-candidate state group read by a finalize
multi_measure_candidate_joins_single_measure_groups a SQL multi-measure aggregate joins single-measure groups, each with its own candidates
only_a_topk_readout_has_a_parameter_floor the only floor is a top-k readout's heap_size ≥ k
HAVING placement key_only_having_is_pushed_below_into_the_filter_group a key-only consumer Filter gets candidates over filtered input rows; the aggregate keeps its own
value_having_stays_above a value predicate pushes nothing below
global_aggregate_having_stays_above a global aggregate's constant HAVING stays above
own_value_having_filters_each_readout the aggregate's own value having is a Filter over each readout
fused heaps / whole-expression top-k / raw-data signs whole_expression_topk_heaps_read_the_raw_samples the heaps read the range's raw samples keyed by job, with RawDataSamplesOfMetric, one build per θ, fused_groups = the absorbed sums
ranked items count_ranked_topk_counts_with_unit_weight a count-ranked top-k uses unit, non-negative weights
ranked_item_is_the_keys_outside_the_partition the top-k's partition keys stay out of the ranked item
nested-sum merges nested_sum_merges_into_the_outer_state_group the merged build is a candidate of the outer sum's state group, over the raw rows, fused_groups = the inner sum
nested_sum_merge_needs_a_sum_of_sums a sum over a max gets no merge
stream norms update_weights_follow_the_measure a Count-Min count adds 1 per row; a sum's state reads the raw value column
intents cardinality_keeps_…, frequency_statistics_keep_universal_choices, topk_keeps_both_specialized_heap_choices, summary_candidates_are_unsized, approximate_count_lists_its_accumulator_and_summaries, a_count_offers_no_count_sketch the ways listed per intent (unchanged tests)

Guarantee composition is no longer Pass 1's: candidates carry θ, and each build's ε follows from θ (PR 1's parameter_list); composition to the query root is error_propagation (PR 4) and constraint (K) (PR 8). The two Count-Min bound tests that lived in Pass 1 but test asap-accuracy (count_min_estimate_has_no_bound_over_an_open_requirement, count_min_estimate_over_raw_data_samples_is_gated_on_the_metric) moved to accuracy/src/propagation.rs.

Ported to the memo (integration-tests): per_computation_candidates_sql.rs (the four HAVING tests, the three heap-update tests, over Pass 1's memo of SQL queries) and stage1_logical_asap_optimization.rs: sql_candidates_output_the_query_columns and promql_candidates_output_the_query_columns (were *_candidates_compose: every query of the old composition test goes through Pass 1 with every candidate's output checked by add_candidate), topk_offers_both_heap_families (was topk_estimates_carry_no_frequency_bound; the guarantee half moves to PR 8), topk_over_an_aggregate_ranks_its_labels, subquery_in_a_measure_filter_is_an_input_group_of_the_state (was …_is_substituted), filtered_sql_avg_is_null_without_matching_rows, promql_avg_divides_sum_and_count_vectors, sql_avg_keeps_its_explicit_output_name, sql_count_star_without_a_non_null_column_is_only_exact (was …_is_unsupported), sql_count_of_a_nullable_column_counts_its_values.

Deleted: devtools/src/candidate_labels.rs connected_labels_describe_selected_states and connected_labels_name_actual_shared_state (they labelled composed node trees; tools are PR 9).

Test status

cargo test -p <crate> --all-features --no-fail-fast, per crate, passed / failed / ignored. PR 2 = #1230 at 4915a82; PR 3 = this PR at 7802d56. No test fails inside Pass 1 any more; planning now stops at Pass 2's first todo!() (PR 4: "rule 1, tumbling layouts as candidates of the state group").

Crate PR 2 PR 3 Where the failures stop now
asap-stage1-pass1-per-computation-candidates 12 / 1 / 0 31 / 0 / 0 — (the one PR-4 failure moved to asap-accuracy with its test)
asap-accuracy 75 / 0 / 0 76 / 1 / 0 +2 tests moved from Pass 1; count_min_estimate_over_raw_data_samples_is_gated_on_the_metric stops at PR 4 (local_guarantee of θ), as it did in Pass 1
asap-types 192 / 0 / 0 192 / 0 / 0 —
asap-stage0-memo 13 / 0 / 0 13 / 0 / 0 —
asap-planner 9 / 19 / 1 9 / 19 / 1 19 at PR 4 (were at PR 3)
asap-devtools 38 / 58 / 3 40 / 54 / 3 46 at PR 4 (38 + 8 in asap_planner subprocesses), 8 at PR 9; −2 tests deleted (composed-node labels)
asap-integration-tests 84 / 87 / 4 101 / 70 / 4 49 at PR 4; 20 at PR 6 (17 realize-from-memo helpers, 3 view coverage); 1 pre-existing (materialized_views, as on PR 2)
asap-executor 210 / 63 / 1 210 / 63 / 1 unchanged (PR 6 θ, one PR 4)
asap-cost 31 / 1 / 0 31 / 1 / 0 unchanged (PR 6)
asap-frontend-promql / -sql / -common / -metricsql, asap-sql-function-catalog, metricsql_common, asap-stage3-plan-selection, Pass 2, Stage 2 218 / 174 / 49 / 15 / 10 / 3 / 3 / 0 / 0, no failures same —
Total 1136 / 229 / 9 1175 / 208 / 9

crate_graph (in asap-integration-tests) passes: no stage crate depends on another, in [dependencies] or [dev-dependencies] (the executor's dev-dependencies on Stage 0 and Pass 1 are gone).

Done means

  • Code pushed (head 7802d56).
  • Body current with the head.
  • Tests re-measured at the final head (table above).
  • Stale-name grep clean over the diff and this body (reader, kept, MILP row, σ/Schedule, EvaluationSchedule, logical_node, child_groups, accuracy_budget_parts, operators_kept_in_definition): no hit in the code; in the body only the names of deleted items (logical_node_trees, accuracy_budget_parts) and SQL/PromQL data rows (row/rows).
  • Rebased onto PR 2's next head (pending: PR 2 is being updated).

zzylol added 11 commits November 6, 2026 21:57
Move-only. Split asap-logical-asap-optimization into asap-stage0-memo,
asap-stage1-pass1-per-computation-candidates and
asap-stage1-pass2-reuse-candidates; rename asap-physical-asap-optimization
to asap-stage2-physical-optimization and asap-plan-selection to
asap-stage3-plan-selection. DeploymentCapabilities moves to asap-types,
and the old test_support splits by caller. The crate-graph test checks
the new crates; no stage crate depends on another.
Pass 1's tests that build Stage 0's memo and Stage 0's tests that lower
PromQL through Pass 1 are deleted, with the dev-dependencies and helpers
only they used. lower_promql moves to Pass 1's test fixtures, so Pass 1
no longer depends on the PromQL frontend outside its own tests. The
crate-graph test now also checks dev-dependencies between stage crates.
…1207 PR 0b)

Frontend tests that compose Pass 1 candidates and Pass 1 tests that
lower PromQL through the frontend are deleted, with the helpers,
dev-dependencies and test targets only they used. MetricsQL's test that
compares with PromQL's lowering goes too, so no frontend tests depend on
another frontend. The crate-graph test checks dev-dependencies between
stage crates with no exceptions.
…R 0b)

The PromQL test that planned through asap-planner is deleted with its
planner_support helpers; the PromQL and SQL frontends drop their
asap-planner dev-dependency, and PromQL its unused asap-accuracy one.
The crate-graph test now also forbids a stage crate's dev-dependency on
the facade.
… PR 0b)

The feature came through asap-logical-asap-optimization's test-support,
which the move deleted, so cargo test -p asap-planner did not build.
@zzylol zzylol added refactor optimizer Workload optimization and plan selection labels Nov 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

optimizer Workload optimization and plan selection refactor

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant