Skip to content

refactor(stage2): physical expansion (decision-tree levels 3–4), physical edges, pruned alternatives; Count-Min weights via column_sign (#1207 PR 5) - #1235

Draft
zzylol wants to merge 30 commits into
refactor/1207-pass2from
refactor/1207-stage2
Draft

zzylol wants to merge 30 commits into
refactor/1207-pass2from
refactor/1207-stage2

Conversation

@zzylol

@zzylol zzylol commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Problem: at PR 4's head Stage 2 was todo!(), so every planning run stopped there; Count-Min weight signs were a WeightDomain proof on each update, set by Pass 1 and checked by the executor.
  • Change: generate_physical_asap_memo_candidates expands each logical candidate Pass 2 kept into its offered $p^{\mathrm{phys}} = (\tau, \ell)$ (decision-tree levels 3–4), each logical edge into physical edges with $\alpha^{\log}$ and $\alpha^{\mathrm{phys}}$, and records every rejected alternative with its PruneReason; a Count-Min build's weight sign is now the state group's column_sign.
  • Effect: for query A of the concepts doc, quantile_over_time(0.95, latency[1m]) every 60 s, Stage 2 offers the chain (QT, None), (QT, OneEvaluation), (QT, AcrossEvaluations), (IT, AcrossEvaluations), prunes its (IT, None) as ReachesNoQueryOutput, and emits 12 physical edges into the chain from the scan's three alternatives; planning now stops at Stage 3 (select_plan, PR 6).

Review guide

Behavior: before → after

Query A of the concepts doc (test offers_each_candidate_its_timings_and_lifetimes): quantile_over_time(0.95, latency[1m]) every 60 s at phase 0, over ingesting data (rows_per_second 1000, max_out_of_order_ms None); the deployment retains raw data and query-time results. It serves the design need of the formulation: Stage 3 chooses among all physical alternatives, Stage 2 only enumerates and records why an alternative is removed.

Before (PR 4) After (this PR)
Stage 2 todo!(); planning stops PhysicalASAPMemoCandidates with physical edges, admissible-edge indexes and the pruned list; planning stops at Stage 3 (select_plan's todo!() of PR 6)
G1, G2[0], G2[1], G4, G5 — (QT, None), (QT, OneEvaluation), (QT, AcrossEvaluations): time-dependent, so no ingestion time; on the 60 s grid, so AcrossEvaluations
G0 Scan — (QT, None), (QT, OneEvaluation), (IT, None): no grid, so no AcrossEvaluations
G3 chain — (QT, None), (QT, OneEvaluation), (QT, AcrossEvaluations), (IT, AcrossEvaluations); (IT, None) pruned ReachesNoQueryOutput
Count-Min weight sign WeightDomain proof on SummaryUpdate, executor refusal the state group's column_sign; unproven → CountMinWeightsNotProvenNonNegative

Interfaces

crates/stage2-physical-optimization/src/catalog/mod.rs

 /// Stage 2: expand every logical candidate Pass 2 kept into its offered (τ, ℓ), every logical
 /// edge into physical edges; record each rejected alternative with its reason. Cannot fail.
 /// Signature unchanged; was `todo!()`.
 pub fn generate_physical_asap_memo_candidates(
     logical: &LogicalASAPMemoCandidates,
     data: &DataWorkload,
     capabilities: &DeploymentCapabilities,
 ) -> PhysicalASAPMemoCandidates;

crates/types/src/stages/physical/memo_candidates.rs

+impl PhysicalEdge {
+    /// Whether a plan may use the edge: α^log = α^phys = 1. Stage 2's pruning, the indexes and Stage 3 call it.
+    pub fn is_logically_and_physically_admissible(&self) -> bool;
+}
 impl PhysicalASAPMemoCandidates {
     /// Stage 2's output, with the admissible-edge indexes built once (signature unchanged; was `todo!()`).
     pub fn new(
         logical_candidates: LogicalASAPMemoCandidates,
         physical_candidates: Vec<PhysicalCandidate>,
         edges: Vec<PhysicalEdge>,
         pruned_candidates: Vec<PrunedCandidate<PhysicalCandidate>>,
     ) -> Self;
     /// P^adm_{c,i}: the admissible edges into input i of `consumer`; empty when none (was `todo!()`).
     pub fn admissible_edges_into(
         &self,
         consumer: PhysicalCandidateId,
         input_index_of_consumer: usize,
     ) -> &[PhysicalEdgeId];
     /// The admissible edges out of `producer` (was `todo!()`).
     pub fn admissible_edges_out_of(&self, producer: PhysicalCandidateId) -> &[PhysicalEdgeId];
 }

crates/types/src/stages/pruning.rs

 /// Why a stage removed a candidate. Stays; variants unchanged, Stage 2's are now produced.
 pub enum PruneReason {
     /// Stage 2: IT, and the data has no ingestion pipeline (at rest or unknown arrival). Input fact.
     NoIngestionPipeline,
-    /// Stage 2: it scans raw data at query time, and the deployment does not retain raw data.
+    /// Stage 2: it scans raw data at query time, or retains the raw rows it scans across
+    /// evaluations, and the deployment does not retain raw data. Input fact.
     RawDataNotRetained,
-    /// Stage 2: a query-time result materialized (`OneEvaluation` or `AcrossEvaluations`), and the
-    /// deployment retains no query-time results.
+    /// Stage 2: a query-time result materialized across evaluations, and the deployment retains
+    /// none (`DeploymentCapabilities::query_time_results_retained`); C1.
     QueryTimeResultsNotRetained,
     /// Stage 2: an input has no buildable admissible producer.
     InputHasNoBuildableProducer,
     /// Stage 2: no query output reads it over admissible edges.
     ReachesNoQueryOutput,
     /// Stage 2: a Count-Min build (`Cms`, `CmsWithHeap`, `HydraCms`) whose state column's
     /// `column_sign` is not `NonNegative`.
     CountMinWeightsNotProvenNonNegative,
     /// Stage 3: dominated by the named candidate.
     DominatedBy(PhysicalCandidateId),
     /// Stage 3: `asap-cost` could not cost it.
     UsageNotCostable,
 }

crates/types/src/ir/schema/sketch.rs · crates/types/src/ir/schema/mod.rs

 /// One summary-state update. Stays; loses `weight_domain`.
 pub struct SummaryUpdate {
     /// The key the row is counted under, for keyed summary types.
     pub item: Option<SummaryInputExpr>,
     /// The value added to the state.
     pub weight: SummaryInputExpr,
-    /// Planner proof for algorithms that need a non-negative stream.
-    #[serde(default)]
-    pub weight_domain: WeightDomain,
 }
-/// The proven or required sign of an update's weights. Removed.
-pub enum WeightDomain {
-    /// No proof (default).
-    #[default]
-    UnknownOrSigned,
-    /// Proven non-negative.
-    NonNegative {
-        proof: NonNegativeWeightProof,
-    },
-    /// Open requirement: raw data samples of `metric`, confirmed from source facts.
-    #[serde(rename = "samples_of_metric")]
-    RawDataSamplesOfMetric {
-        metric: String,
-    },
-    /// Open requirement: reset-corrected `rate`/`increase`/`irate` of `metric`'s samples.
-    CounterDerivativeOfMetric {
-        metric: String,
-    },
-}
-/// Why weights are non-negative. Removed.
-pub enum NonNegativeWeightProof {
-    /// A unit count.
-    UnitCount,
-    /// `rate`/`increase`/`irate` of a counter's samples.
-    ResetAwareCounterDerivative,
-    /// Raw data samples of a metric declared a counter.
-    #[serde(rename = "counter_samples")]
-    CounterRawDataSamples,
-    /// Raw data samples of a metric declared non-negative.
-    #[serde(rename = "declared_non_negative_samples")]
-    DeclaredNonNegativeRawDataSamples,
-    /// A constant weight greater than 0.
-    PositiveConstant,
-}

Removed (decision 2026-11-09): the state group's column_sign (derived from the weight's sign and the declared source signs) replaces them, read by Stage 2's CountMinWeightsNotProvenNonNegative; schema::mod no longer re-exports either.

crates/types/src/test_support.rs (test-support feature only; review D11)

+/// `[ts, value, labels...]`, time index 0, series identity realized (closed, `$promql_series_identity` last).
+pub fn series_schema(labels: &[&str]) -> Schema;
+/// A `TimeSeries(metric)` scan with `schema`, filtered by `predicates`.
+pub fn time_series_scan<C>(
+    metric: &str,
+    schema: Schema,
+    predicates: Vec<Predicate<C>>,
+) -> Operator<C>;
+/// Adds `operator` with `parameters` to `group` as Pass 1 does (no input requirement, no fused group);
+/// a merge or a `TimeRange` over a chain gets the θ of the states it reads (A1).
+pub fn add_candidate(
+    memo: &mut Memo,
+    group: MemoGroupId,
+    operator: Operator<MemoGroupId>,
+    parameters: Option<SketchParams>,
+) -> Result<CandidateForMemoGroupId, CandidateError>;
+/// The group of `operator`, with `operator` as its own candidate.
+pub fn group_with_own_candidate(memo: &mut Memo, operator: Operator<MemoGroupId>) -> MemoGroupId;

Example: group_with_own_candidate(&mut memo, time_series_scan("latency", series_schema(&["job"]), vec![])) is query A's scan group; Pass 2's and Stage 2's tests build their memos with these.

Also removed (crate-private, so no block): Pass 1's CmsWithHeap gate on unproven weights in realize_keyed_additive_summary_input, raw_data_samples_of_metric, with_raw_data_samples_requirement, SingleMeasure::raw_data_samples_of (Stage 2's Count-Min prune replaces them); Pass 1's private reads_only_time_series(memo, input) and Stage 2's recursive copy (the group property GroupProperties::reads_only_time_series, PR 2, N7); the executor's CmsWithHeap "nonnegative weight contract" check (physical_planner), the rate-weight refusal in create_planner_accumulator (keyed off ResetAwareCounterDerivative) and the Hydra "weight's sign is unknown" binding check (the planner proves the sign before offering a Count-Min build; the Hydra build still rejects a negative weight value at run time, invalid_weight_values_fail_the_build).

Not reintroduced from the old PR 5 head (decisions after it): executor_can_run / PerEntityAggregateNotExecutable (G4/H3: no executor-support check), AcrossEvaluationsWithoutWindowLayout (F1/C7: a value-set rule, not a prune reason), PhysicalCandidateKey, PrunedCandidate.group, PhysicalASAPMemoCandidates::query_roots, the private Deployment struct (D1–D4, F2, F3; the PR 0 types already have these shapes).

Usage, end to end (test offers_each_candidate_its_timings_and_lifetimes at 513a1e78e, query A of the concepts doc):

// What the user writes: quantile_over_time(0.95, latency[1m]), every 60 s at phase 0, over
// ingesting data (rows_per_second 1000, max_out_of_order_ms None); the deployment retains raw
// data and query-time results (`data`, `capabilities`, read by Stage 2's offered values and prune rules).
// `logical`: what Stage 0 + Pass 1 + Pass 2 give (built by the test as they build it):
// G0 Scan latency; G1 TimeRange 1m (G0); G2 KLL state with [0] SummaryAgg(G1) and [1] SummaryMerge(G4);
// G3 SummaryTumblingWindowChain[KLL, w = 60 s](G0); G4 TimeRange 1m (G3); G5 p95 readout (root).
// evaluation_grid_of_group: 60 s @ 0 for G1, G2, G4, G5; None for G0 and G3 (time-independent).
// `physical`: Stage 2's output, which the planner facade hands to Stage 3 (PR 6).
let physical = generate_physical_asap_memo_candidates(&logical, &data, &capabilities);
// G1, G2[0], G2[1], G4, G5: (QT, None), (QT, OneEvaluation), (QT, AcrossEvaluations).
// G0 Scan: (QT, None), (QT, OneEvaluation), (IT, None).
// G3 chain: (QT, None), (QT, OneEvaluation), (QT, AcrossEvaluations), (IT, AcrossEvaluations);
//   pruned_candidates: G3 (IT, None) ReachesNoQueryOutput (only an ingestion-time consumer could read it).
// edges: into G3's four alternatives from G0's three: 12 edges (test `edges_carry_both_requirements`);
//   is_logically_and_physically_admissible exactly when both run at the same timing.
// E.g. the edge from the scan's (QT, None) into the chain's (IT, AcrossEvaluations) has
//   satisfies_logical_requirement: true, satisfies_physical_requirement: false, so neither
//   pruning nor admissible_edges_into lists it.

Algorithm changes

Stage 2 expansion and pruning

New; the previous head's version was written against the old memo and is replaced.

  • Problem: at PR 4's head every planning run stopped at Stage 2's todo!().

  • After:

    Step Code What it decides
    1. Offered values catalog/configurations.rs enumerate_candidate_configurations τ = QueryTime always; IngestionTime only for a candidate whose group is time-independent (GroupProperties::depends_on_evaluation_time false) or an input of one (ingestion_time_offered_per_group), and for a counter state (rate/increase/irate) only when its rows are ordered (max_out_of_order_ms declared, or PromQL rows); ℓ = None, OneEvaluation(Memory), and AcrossEvaluations(Memory) only for a chain or a group with a known evaluation_grid_of_group. Values not offered are not recorded
    2. Prune rules (levels 3–4) same CountMinWeightsNotProvenNonNegative, NoIngestionPipeline (IT, arrival AtRest/Unknown), RawDataNotRetained (a Scan at QT, or materialized across evaluations: either retains raw rows), QueryTimeResultsNotRetained (QT AcrossEvaluations)
    3. Physical edges catalog/mod.rs Pass 2's edges in (consumer, input) chunks; every pair of the two candidates' alternatives, $\alpha^{\log}$ copied, $\alpha^{\mathrm{phys}}$ = producer_satisfies_physical_requirements(producer, consumer_timing)
    4. Unusable alternatives catalog/pruning.rs InputHasNoBuildableProducer (least fixed point of "every input has a buildable admissible producer"), ReachesNoQueryOutput (not reached from a query output: an alternative of a root group a query-time consumer may read)
    5. Output PhysicalASAPMemoCandidates::new emitted candidates renumbered; edges grouped by (consumer, input) with a stable sort; admissible-edge indexes (HashMap<(consumer, input), Vec<PhysicalEdgeId>>, Vec<Vec<PhysicalEdgeId>> by producer) built once

    Candidates Pass 2 eliminated (LogicalASAPMemoCandidates::pruned) are not expanded. Stage 2 checks no executor support and costs nothing; alignment with each execution context stays Stage 3's (PhysicalCandidate::delivers_on, PR 1).

  • Correctness: an emitted candidate has, for each input, an admissible edge from an emitted producer, and is reachable from a query output (checked by every test's generate helper, which also checks the edge order and the indexes); a pruned alternative is recorded exactly once, with the first reason that applies (the Count-Min sign is checked first: it holds for every (τ, ℓ)).

  • Complexity: O(|C|·5) alternatives; O(Σ_{(c,i)} |alt(c)|·Σ_u |alt(u)|) ≤ 25·|logical edges| physical edges; the buildable fixed point O(depth·|E|), reach O(|E|).

  • Assumptions and limits: Memory storage only (the capabilities offer no disk, Executor: store Disk-medium materialized views and one-evaluation results on disk #1217); no executor-support check (G4/H3: executor gaps are PR 6's).

Representation: PhysicalCandidate { logical_candidate_id, physical_properties }; PhysicalEdge { …, satisfies_logical_requirement, satisfies_physical_requirement }; PrunedCandidate<PhysicalCandidate> { candidate, reason }. Enumeration: every offered (τ, ℓ) per kept logical candidate; every producer/consumer alternative pair per logical edge. Stage 3 chooses.

Decisions and open questions

Needs reviewer decision: None.

Decided:

  • 2026-11-07: F1–F3 and D1–D13 of the PR 5 review; G2-counter; G4/H3 (no executor-support check anywhere); L8a (Count-Min prune via column_sign); time indexing (any grid-evaluated group may be AcrossEvaluations).
  • 2026-11-09: WeightDomain deleted in PR 5 (rejected: keeping a per-update proof set by Pass 1 and checked by the executor).
  • 2026-11-09 (N7, was N1 here): PhysicalEdge::is_admissible renamed is_logically_and_physically_admissible before landing (the name says both requirements; no alias). Rejected: is_admissible.
  • 2026-11-09 (N7, was N2): reads_only_time_series is the group property GroupProperties::reads_only_time_series, derived once in PR 2; the recursive asap_types::stages::reads_only_time_series is gone. Example: for rate(c_total[5m])'s chain over Scan(TimeSeries c_total) it is true, so the counter chain is offered ingestion time with max_out_of_order_ms: None; over Scan(Table events) false, so only with Some(_).
  • 2026-11-09 (N3, D11): shared memo fixtures under test-support (above).
  • C1 (R6): QueryTimeResultsNotRetained prunes only (QueryTime, AcrossEvaluations). The prune-reason table and PR 0's comment said OneEvaluation too; the decision-tree diagram, DeploymentCapabilities::query_time_results_retained (default false) and the old PR 5 said AcrossEvaluations only. Rejected: OneEvaluation too, which with default capabilities removes every within-evaluation sharing (C3: a None producer has at most one consumer per context). Query A keeps the readout's (QT, OneEvaluation) and prunes its (QT, AcrossEvaluations). The concepts doc's row now reads "QueryTime with ℓ = AcrossEvaluations".
  • C2 (R7): ingestion time is offered through inputs transitively (the closure). Rejected: a one-level rule, which could not build anything below an @ T. Example: sum_over_time(m[1m] @ 1000): TimeShift @1000 is time-independent; its input TimeRange 1m and that one's Scan are offered (IT, None) (an_at_t_subtree_is_offered_ingestion_time). Without @ T the closure adds nothing.
  • C3 (R7): the Count-Min weight sign is read from the build's own state group (Stage 0 derives it from the weight: a constant's sign, or the weight column's column_sign), equal for every candidate of the group. Rejected: a second derivation of the weight's sign. Example: CmsWithHeap keyed by series, weighted by SampleValue of m: pruned in all three alternatives (QT None, QT One, IT None) without a declaration, offered with column_signs: {"value": "non_negative"} (a_count_min_build_needs_weights_proven_non_negative).
  • C4 (R7, review G3): a query reads its output as a query-time consumer: any query-time alternative of a root group and (IngestionTime, AcrossEvaluations); (IngestionTime, None) is not, since its result is gone when the query runs. Example: sum_over_time(m[1m] @ 1000)'s root TimeShift keeps (QT, None) and (QT, OneEvaluation); its (IT, None) is ReachesNoQueryOutput.
  • C5 (R7): the executor no longer refuses rate-weighted updates in create_planner_accumulator (with the proof gone the factory cannot tell a rate weight from a sum weight); typed binding of window-weighted heaps listed for PR 6.
  • A4 (2026-11-09): no summary build (SummaryAgg, chain, MaintainPopulation) is offered (IngestionTime, None): fed at ingestion it keeps its state between batches, so it is materialized; matches the exported plan's validation. Downstream: example 3a and o11y q18/q21 get valid plans (PR 9 snapshots).
  • A7 (2026-11-09, option (a)): under an absolute @ T with no known horizon end, no AcrossEvaluations candidate (its consumer reaches back horizon end − T + range, unbounded); it still runs at query time. Example: x @ 1700000000 [5m]: the scan keeps (QT, None), (QT, OneEvaluation), (IT, None). A producer deeper below (a chain under the TimeRange) is not excluded here: Stage 3 gives it no $h_v$ (PR 7, P4).
  • Fix (2026-11-09): RawDataNotRetained also rejects an ingestion-time Scan materialized across evaluations (it retains raw rows; a no-raw plan held a raw Scan buffer at ingestion). PruneReason doc, planner-layering.md, asap-aware-mapping-contracts.md and the concepts doc prune table say so.

Verification

cargo fmt --all -- --check, cargo clippy --workspace --all-features --all-targets -- -D warnings clean; crate_graph passes. Per crate cargo test -p <crate> --all-features. By test name against the base PR 4 (5d3f6de85): 0 newly failing, 13 newly passing (the new Stage 2 tests), 2 removed (weighted_topk_binding::rate_updates_cannot_enter_integer_heap_factory, hydra_cms_execution::unknown_sign_weight_is_rejected, with the checks they tested).

Scope Passed / failed / ignored Newly failing vs base Notes (where failures stop)
asap-types 206 / 0 / 0 0
asap-accuracy 24 / 0 / 0 0
asap-cost 31 / 1 / 0 0 PR 6: θ on the exported node (1)
asap-frontend-common 49 / 0 / 0 0
asap-sql-function-catalog 10 / 0 / 0 0
metricsql_common 3 / 0 / 0 0
asap-frontend-promql 218 / 0 / 0 0
asap-frontend-metricsql 15 / 0 / 0 0
asap-frontend-sql 175 / 0 / 0 0
asap-executor 209 / 62 / 1 0 PR 6: θ on the node (32), Hydra grid θ (12), retention_ms = $h_v$ (8), weighted-frequency θ (7 + 2), baseline MILP (1)
asap-planner 8 / 18 / 1 0 PR 6: baseline MILP (18)
asap-devtools 37 / 56 / 3 0 PR 6: baseline MILP (41); PR 9: group node from expression (9), group label (2), pruned candidates (2, #1205); output.status.success() (2)
asap-integration-tests 100 / 50 / 4 0 PR 6: baseline MILP (43), retention_ms (3), avg sum-and-count plan (3); old-wire fixture JSON (1)
asap-stage3-plan-selection 3 / 0 / 0 0
asap-stage0-memo 15 / 0 / 0 0
asap-stage2-physical-optimization 13 / 0 / 0 0 all new
asap-stage1-pass2-reuse-candidates 25 / 0 / 0 0
asap-stage1-pass1-per-computation-candidates 32 / 0 / 0 0
Total 1173 / 187 / 9 0

New tests (Stage 2, catalog::tests): offers_each_candidate_its_timings_and_lifetimes (query A's offered values); no_summary_build_is_offered_at_ingestion_time_without_a_lifetime (A4); without_an_ingestion_pipeline_nothing_runs_at_ingestion_time; without_raw_data_only_ingestion_time_builds_read_the_scan (raw data not retained, incl. the materialized ingestion-time scan); query_time_results_across_evaluations_need_the_deployment_to_retain_them (C1); an_at_t_subtree_is_offered_ingestion_time (C2); a_subtree_under_an_absolute_anchor_is_not_materialized_across_evaluations (A7); a_counter_state_runs_at_ingestion_time_over_ordered_rows (SQL vs PromQL, bound declared or not); a_count_min_build_needs_weights_proven_non_negative (C3); edges_carry_both_requirements (12 edges, α^log/α^phys); an_input_without_an_admissible_producer_is_not_buildable; candidates_pass_2_eliminated_are_not_expanded; a_scalar_roots_operand_is_a_query_output.

Self-review: no new abstraction (no Stage 2 input struct (F2), no new prune reason, no executor-support table); two admissible-producer indexes stay (before pruning by provisional index, after by ID; D6 allows both, both call is_logically_and_physically_admissible); duplication removed (reads_only_time_series, one set of memo fixtures, one weight-sign derivation); no dead code added. The concepts doc's D13.2 crate path is already stage2-physical-optimization. Screenshots and measurements: not applicable.

Follow-ups

@zzylol zzylol added optimizer Workload optimization and plan selection refactor labels Nov 7, 2026
@zzylol
zzylol force-pushed the refactor/1207-pass2 branch 2 times, most recently from bdf0888 to bcae9e5 Compare November 9, 2026 07:11
@zzylol
zzylol force-pushed the refactor/1207-stage2 branch from d11f12e to 616f765 Compare November 9, 2026 07:43
@zzylol zzylol changed the title refactor(stage2): physical expansion (decision-tree levels 4–5), physical edges, pruned_candidates (#1207 PR 5) refactor(stage2): physical expansion (decision-tree levels 3–4), physical edges, pruned alternatives; Count-Min weights via column_sign (#1207 PR 5) Nov 9, 2026
@zzylol
zzylol force-pushed the refactor/1207-stage2 branch from 616f765 to f20177e Compare November 9, 2026 11:53
@zzylol
zzylol force-pushed the refactor/1207-pass2 branch from bcae9e5 to 8107319 Compare November 9, 2026 17:11
@zzylol
zzylol force-pushed the refactor/1207-stage2 branch from f20177e to e05016c Compare November 9, 2026 17:44
@zzylol
zzylol force-pushed the refactor/1207-pass2 branch from 8107319 to 5d3f6de Compare November 9, 2026 20:52
@zzylol
zzylol force-pushed the refactor/1207-stage2 branch from e05016c to 513a1e7 Compare November 9, 2026 21:28
@zzylol
zzylol force-pushed the refactor/1207-pass2 branch from 5d3f6de to dea088e Compare November 10, 2026 01:26
@zzylol
zzylol force-pushed the refactor/1207-stage2 branch from 513a1e7 to 4c77ec6 Compare November 10, 2026 01:26
…iming, lifetime }

Deletes Schedule and the σ-only InvalidPhysicalProperties variants; the only
invalid combination left is (IngestionTime, OneEvaluation). satisfies_physical
checks timing x lifetime only; delivers_on is the per-context alignment check
(w | I and w | φ). A OneEvaluation result runs in one evaluation context, so
x_{κ,v} covers it as for ℓ = None.
…physical drops memo

InvalidLogicalProperties gains SketchBuildWithoutParameters and renames
FloorNotInParameterList to ParametersNotInList, returned by
LogicalProperties::new. LogicalRequirement::new no longer checks list
membership (a floor is any value of the family's parameter type, e.g.
heap_size >= k), so it returns Self. satisfies_physical takes only the
producer and consumer.
Context::QueryEvaluation(QueryId) becomes Context::Evaluation(EvaluationSchedule):
queries with the same (interval, phase) are evaluated together and share
their OneEvaluation results. evaluation_schedule returns that schedule.
QueryId stays for z_{q,v}. A non-periodic query has no context yet.
InvalidLogicalProperties -> LogicalPropertiesError, InvalidPhysicalProperties
-> PhysicalPropertiesError, LogicalRequirement -> LogicalRequirements
(parameter_floor -> parameter_floor_for_required_accuracy), satisfies_logical
/ satisfies_physical -> producer_satisfies_logical_requirements /
producer_satisfies_physical_requirements, Medium -> StorageMedium,
Context / ContextId -> ExecutionContext / ExecutionContextId
(x_in_context / y_in_context -> x_in_execution_context /
y_in_execution_context). PhysicalCandidate and PhysicalCandidateKey fields
-> logical_candidate_id, physical_properties; edge fields ->
producer_candidate_id, consumer_candidate_id; PhysicalASAPMemoCandidates
fields -> logical_candidates, physical_candidates.
Deletes EvaluationSchedule and ExecutionContext::evaluation_schedule. An
evaluation context is Evaluation { recurrence, range_evaluation }: queries
with equal ones share it and its OneEvaluation results, one-off queries
included. ExecutionContext, ExecutionContextId and RefreshWindowId move to
asap_types::stages so PhysicalCandidate::delivers_on can take the context.
Contexts are deduplicated by equality (QueryRecurrence holds an f64).
PhysicalASAPDAGValidationError::InvalidPhysicalProperties becomes
PhysicalProperties { node, reason }. In the files this unit edits, "kept"
describing a materialized result or retained data becomes "materialized"
or "retained" (cost, materialized views, realizer, installable plan,
deployment capabilities; locals kept/raw_kept_rows -> retained/
raw_retained_rows), and "reader" becomes "query" where it is a query
(execution demand, ScheduleGroup::queries, cost usage, labels, tests) or
"consumer" where it is a consuming candidate.
find_or_add_group returns the node rules' SchemaDerivationError and
add_candidate a CandidateError (the node rules, or an output that differs
from the group's). Bodies stay todo!() until PR 2.
…mo (#1207 PR 0)

Memo::groups is private, read through groups(); groups are added only
through find_or_add_group and queries through add_query (todo!() until
PR 2). memo_of_queries returns Result and its body is todo!() until PR 2,
which builds Stage 0 on find_or_add_group; Stage 0's node-level CSE (cse.rs),
its test_support export and the dead share_identical helper are deleted.
…OperatorInput; coverage only in the memo (#1207 PR 0)

MemoGroup { expression, group_properties: GroupProperties, parents, queries,
candidates } replaces logical_node and child_groups; GroupProperties {
schema, result_kind, summary_coverage,
shareable_when_equal } is derived once by derive_group_properties, which
applies the existing rules to the expression over MemoGroupAsInputs.

The rules (output_schema, output_kind, validate_inputs,
is_shareable_when_equal, summary builds, window chains, scalar subqueries,
top-k, populations) are generic over OperatorInput, implemented for
Rc<OperatorNode> and MemoGroupAsInput; only how they read an input changed.
BuiltOrInput carries the builds a rule makes while deriving (one build over
a chain's window, the build a rollup or a selection stands for).

SummaryCoverage { definition: SummaryStateDefinition { input_group,
unlifted_operators }, selection } is derived only from memo groups
(derive_from_inputs) and stored in GroupProperties; node-tree coverage,
NodeToSummaryCoverageCache, derive_cached and OperatorNode::coverage are
deleted, so node trees have no merge check. Memo::logical_node_trees is
deleted: Pass 1 target discovery (PR 3), devtools group labels (PR 9) and
the export's view coverage (PR 6) are todo!(). find_or_add_group and
add_candidate derive the properties, then todo!() until PR 2.
…elpers (#1207 PR 0)

One state's θ is no longer sized for several consumers: θ comes from
parameter_list per candidate. size_for_all_readers had no non-test caller;
min_recorded, kll::shared_k and univmon::shared_heap served only it.
…perators; delete merged_window_build_node (#1207 PR 0)

- BuiltOrInput / BuiltInput renamed InputOrTemporaryBuild / TemporaryBuild:
  a build a rule constructs only while it derives, never stored in the memo.
- SummaryStateDefinition::unlifted_operators is Vec<Operator<Option<MemoGroupId>>>:
  None marks an operator's main input, Some(g) another input group it reads
  (a scalar subquery's). States whose definitions read subqueries compare and
  merge; CoverageError::SubqueryInDefinition deleted.
- merged_window_build_node deleted with its test; the accuracy guarantee of a
  read of merged tumbling windows is todo!() until PR 4 rewrites it over groups.
…, TimeBound, Pass 2 pruned list, Stage 2 entry (#1207 PR 0)

- Delete WindowLayout, LogicalProperties::window_layout and its two error
  variants, SummaryMerge::newest_tumbling_windows and
  SummaryTumblingWindowChain::tumbling_window_count, with the count-based
  chain builds and expansion; behaviour that read the count is todo!() for
  PR 1, 2, 6, 7 or 9.
- SelectionBox { columns } only, and TimeBound { At, FromEvaluationTime };
  lifting TimeRange/TimeShift is todo!() for PR 2.
- PrunedLogicalCandidate, PruneCause, RootPruneReason and
  LogicalASAPMemoCandidates::pruned.
- error_propagation(consumer, metric) -> Option<ErrorPropagation>;
  Memo::set_input_requirements.
- PhysicalCandidateKey deleted (PrunedCandidate<PhysicalCandidate>),
  PrunedCandidate::group and PhysicalASAPMemoCandidates::query_roots
  dropped; new(logical, physical, edges, pruned).
- Stage 2 entry takes &DeploymentCapabilities (moved into asap-types) and
  cannot fail; PruneReason follows the doc's Stage 2 table.
- DataArrival::{Ingesting, Mixed} gain max_out_of_order_ms.
- GroupProperties::summary_coverage -> aggregate_coverage.
…l; ScalarValue::Timestamp (#1207 PR 0)

Replace TimeBound by IntervalConstraintLimit { FixedValue(ScalarValue),
EvaluationTimePlusMs(i64) }: Constraint::Interval holds
Bound<IntervalConstraintLimit> on every column, so the timestamp column
and value columns share one interval type. Doc comments carry the
examples (m[5m], offset 1h, @ T, SQL ts >= literal, price > 10).
ScalarValue::Timestamp(ms since the epoch) is the fixed value of an
@ T or a SQL timestamp literal.
…s; delete ParametersOfAnotherFamily (#1207 PR 0)

A range m[r] evaluated at t reads (t - r, t]: IntervalConstraintLimit and
Constraint::Interval examples, the chain window doc and the primitive-schema
table now use Excluded lower and Included upper ends. A parameter list holds
only its own family's parameters, so ParametersNotInList also rejects another
family's parameters.
…acy, the plan is one PhysicalASAPDAG (#1207 PR 0)

- Terminology: ops' `family` is `summary_type`; FieldDataType::Sketch(SketchAlgorithm)
  (SketchKind, GroupingStrategy, HydraKind and the ops' `grouping` deleted);
  FamilyCapability is SummaryTypeCapability; parameter_list(algorithm).
- θ is structure only: SketchParams derives equality; KLL failure_probability,
  UnivMon distinct_failure_probability and HLL/KMV relative_error deleted.
- ValueSign and GroupProperties::column_sign; SourceFacts::column_signs
  replaces values_non_negative.
- LogicalRequirements { required_accuracy: Option<RequiredAccuracy> };
  RootPruneReason::BelowRequiredAccuracy; Memo::add_query takes the target
  and n_q; AggIntent measures carry no accuracy and front ends take none.
- asap-accuracy: AccuracyEvidence::measured_guarantee, theoretical_guarantee
  (was local_guarantee), producer_satisfies_logical_requirements(memo,
  producer, consumer, input_index, evidence); the AccuracyModel trait and
  evidence provider, Pass 1's accuracy pruning and AccuracyCheckInputs deleted.
- Physical: ExecutionContext::Evaluation { grid: EvaluationGrid },
  IngestionPipeline(RefreshWindow); RefreshWindowId deleted;
  producer_satisfies_physical_requirements(producer, consumer_timing);
  PruneReason::CountMinWeightsNotProvenNonNegative.
- Plan: PlanOutput { plan: PhysicalASAPDAG }; PhysicalASAPDAGNode::
  selected_physical_candidate; select_plan returns (PhysicalASAPDAG,
  Selection); Selection { accepted_accuracy_per_query, pruned_candidates,
  milp_* }; RealizedPlan, PhysicalWorkloadPlan, DagOfSelectedCandidates,
  SelectedEdge, SelectedNodeId, AssignmentRealizer, ScanWindow and the
  scan/view window functions, SearchReport, SearchStatus, InexactObjective,
  ObjectiveDiffersFromPlanCost, SelectedPlanFailedCheck and
  PlanCost::cost_unit deleted.
- Plan file wire version 14 (summary_type, Sketch(algorithm), no grouping).
- Cost names: NodeCostSource, cost_source, PlanRanking::Cost; Serialize on
  PruneReason, PhysicalProperties, OriginOfCandidateForMemoGroup and
  RootPruneReason.
…e trait, MemoQuery, evaluation grids of groups (#1207 PR 0)

2026-11-09 decisions:
- theoretical_guarantee and AccuracyEvidence::measured_guarantee take the
  failure probability and return the error bound that holds at it (None
  when θ cannot fail that rarely); bodies stay todo!() for PR 4.
- AccuracyEvidence is a trait; NoMeasuredGuarantees has no measurement.
  PlanningModels::evidence and Pass 2 take &dyn AccuracyEvidence.
- The memo holds its queries in one Vec<MemoQuery> { root,
  accuracy_target, values_per_evaluation }; add_query(query).
- SupportedSummary::family is summary_type.
- satisfies is deleted with metric_meets, scope_meets, native_target and
  AccuracyScope::covers, which only it used; target_limit is the
  comparison limit with the tolerance.
- PhysicalCandidate::delivers_on(&PhysicalASAPMemoCandidates, context).
- GroupProperties::depends_on_evaluation_time (derived in PR 2) and
  LogicalASAPMemoCandidates::evaluation_grid_of_group (filled in PR 4).
… the tests that use it (#1207 PR 0)

pass1/composition.rs composed one whole-workload candidate DAG from one
per-computation candidate per target, for tests only; planning adds every
way to compute to the memo instead. It and the 74 tests and fixture helpers
that compose through it are deleted (no interim code for PR 3).
Move-only. Split asap-logical-asap-optimization into asap-stage0-memo,
asap-stage1-pass1-per-computation-candidates and
asap-stage1-pass2-reuse-candidates; rename asap-physical-asap-optimization
to asap-stage2-physical-optimization and asap-plan-selection to
asap-stage3-plan-selection. The old test_support splits by caller. The
crate-graph test checks the new crates; no stage crate depends on another.
Pass 1's tests that build Stage 0's memo and Stage 0's tests that lower
PromQL through Pass 1 are deleted, with the dev-dependencies and helpers
only they used. lower_promql moves to Pass 1's test fixtures, so Pass 1
no longer depends on the PromQL frontend outside its own tests. The
crate-graph test now also checks dev-dependencies between stage crates.
…1207 PR 0b)

Frontend tests that compose Pass 1 candidates and Pass 1 tests that
lower PromQL through the frontend are deleted, with the helpers,
dev-dependencies and test targets only they used. MetricsQL's test that
compares with PromQL's lowering goes too, so no frontend tests depend on
another frontend. The crate-graph test checks dev-dependencies between
stage crates with no exceptions.
…R 0b)

The PromQL test that planned through asap-planner is deleted with its
planner_support helpers; the PromQL and SQL frontends drop their
asap-planner dev-dependency, and PromQL its unused asap-accuracy one.
The crate-graph test now also forbids a stage crate's dev-dependency on
the facade.
- LogicalProperties::new(expression, parameters): a sketch build (a chain
  included) needs θ from parameter_list(algorithm) of its
  FieldDataType::Sketch(algorithm); an exact operator has none.
- parameter_list(algorithm): structures only. Count-Min lists each width
  with depths 5, 7, 10 and 12, Count Sketch with 47, 81, 117 and 155 (its
  median-of-rows δ levels 1e-2 … 1e-5); heap variants each of those
  with the 1-2-5 heap sizes; UnivMon 40 entries; KLL, DDSketch, HLL, KMV
  one entry per ε level. A test keeps the lists equal to today's sizing,
  which is test-only.
- PhysicalProperties::new rejects (IngestionTime, OneEvaluation);
  producer_satisfies_physical_requirements(producer, consumer_timing).
- PhysicalCandidate::delivers_on(candidates, context): None in every
  context, OneEvaluation only in Evaluation contexts, a chain with
  AcrossEvaluations when its width divides the grid's period and phase
  (or the refresh window's width and phase), any other AcrossEvaluations
  result everywhere when its group does not depend on the evaluation
  time, else when its group's evaluation grid holds the context's times;
  Materialized(v) contexts stay todo!() for PR 7.
- Chains: one schema (window_end at column 0, keys, state), TimeRange and
  TimeShift over a chain select windows (an instant TimeRange is
  rejected), a SummaryMerge over them finalizes and rolls up as the
  chain's build over that range (the two PR 0 todos).
- summary_type_of_operator; QueryRecurrence::evaluation_grid_ms is None
  for a fixed interval without a phase.
…, depends_on_evaluation_time, aggregate_coverage), @ end()/@ start() lifted above their subtree (#1207 PR 2)

- Memo: hash-consing (find_or_add_group by expression hash), candidate
  storage (add_candidate), add_query(MemoQuery); Memo::new(declared column
  signs), private declared_column_signs read by a Scan's column_sign.
- Stage 0: memo_of_queries(roots, accuracy_targets, data) adds each query
  with its accuracy target and n_q (values_per_result over series_read),
  and collects the declared signs of the scanned sources (a counter's
  value is non-negative without a declaration).
- GroupProperties::column_sign derived per operator (concepts doc, "Value
  sign"); depends_on_evaluation_time: a TimeRange or time()/now() makes a
  result depend on the evaluation time, @ T pins it;
  reads_only_time_series: a Scan of a time series, or every input group
  reading only time series (OperatorInput::reads_only_time_series computes
  the same rule over node trees; the recursive walks in topk.rs and Pass 1
  are deleted).
- aggregate_coverage for summary states, chains, TimeRange/TimeShift over
  a chain, merges and exact Aggregates; timestamp constraints are
  IntervalConstraintLimit intervals (t - o - W, t - o]; SQL timestamp
  literals and identity casts fold so absolute filters lift.
- PromQL: @ end() / @ start() are lowered as TimeShift { at } above the
  selector or subquery they anchor, which is lowered without them, so it is
  the group of the same expression without @; coverage stops at that
  TimeShift (it is the definition's input group).
…1207 PR 2b)

GroupProperties gains column_lineage (one ColumnLineage per output column,
derived once from the input groups'). Lineage and column_sign are derived
together by one per-operator column layout (stages/column_properties.rs,
which replaces column_sign.rs): a copied column keeps its lineage and sign,
a computed one is Computed with the operator's sign rule. Chains compute
window_end and their state and keep their keys' lineage; a TimeRange or
TimeShift over a chain, and TimeShift { at: End | Start } above a selector,
pass lineage through; a merge of a chain's selected windows has the chain
build's layout.

Summary coverage names a selection's column by its SourceColumn lineage
(the one SourceColumn type, beside Source); ColumnIdentity is deleted. A
predicate on a Computed column, or on one of two columns of one lineage,
is not lifted; the regrouping key check compares lineage. Schema fields
and the IR Project lose their table qualifier; the front ends bind
qualified names with their own scope while lowering. The requalification
patches (with_aggregate_qualifiers, node.rs) are deleted.
…1207 PR 3)

Pass 1 fills Stage 0's memo with each group's initial candidates, every
intermediate result its own group and every candidate one operator over
input groups:

- computation_targets walks memo.groups(): one target per measure of each
  Aggregate group with a realization besides plain exact execution.
- add_pass1_candidates adds every group's own expression, exact
  accumulators (state group + FinalizeExactAccumulator; avg from sum and
  count), one SummaryAgg state group per sketch algorithm with one build
  candidate per configuration of parameter_list(algorithm) and separate
  SummaryEstimate readout groups, fused builds (whole-expression top-k
  heaps; nested sum/min/max merges) listing the replaced groups in
  fused_groups, multi-measure joins, and key-only HAVING pushed below in
  the consumer Filter group's candidates.
- Pass 1 sets no input requirement and no accuracy floor: every
  candidate's LogicalRequirements are empty; Pass 2 gives each readout its
  required accuracy (a top-k readout's heap_size >= k included).
- ComputationTarget and FusedComputation move into Pass 1 (crate-private),
  FusedComputation::absorbed_groups replaces target indices, HavingEntry,
  HavingPlacement and Pass1Error are deleted; pass1_memo returns
  PerComputationCandidateError.
- types: topk::ranked_column_of_group (ranked_column becomes pub(crate));
  one unique-key rule for a top-k's ranked rows, exact or a heap readout;
  any_measure_filtered generic over the predicate's input.
- Tests build their memo through the Memo methods; the integration tests
  stage1_logical_asap_optimization and per_computation_candidates_sql run
  SQL and PromQL queries through Stage 0 and Pass 1.
…requirements, dead-candidate elimination, chain groups (#1207 PR 4)

Pass 2 runs, in order: the general rollup rule with one synthesized
finest common group per cluster of equal definitions (replaces scan
widening, finer-aggregate reuse and the rollup rules); top-k reuse; reach,
caps and readout requirements (one readout candidate per requirement level,
written with Memo::set_input_requirements); dead-candidate elimination with
every root cause (PrunedLogicalCandidate); chain groups
SummaryTumblingWindowChain[w](X) for the surviving state groups, widths
from the gcd closure of the reaching queries' grids, the window and the
offsets, then elimination again; and the logical edges between survivors.
A merge, and a Filter/TimeRange/TimeShift over states, carries the θ of
the states it reads (one candidate per θ, reading only producers with that
θ). The metric rule cuts edges from readouts whose metric the consumer's
input does not accept (accepted metrics pushed down from the targeted
queries), not candidates; a candidate cut off from an input is eliminated
too. A whole-result δ with n_q unknown prunes the readouts it reaches
(ValuesPerEvaluationUnknown). The summary-input and
window-rollup rules and the local-window-composition step are deleted.

- types: LogicalRequirements::error_propagation(memo, consumer, metric)
  with the constant contracts (key-only Filter by column lineage, abs /
  clamp / negation, min / max / avg / count, + - / *), b = 0.1 private;
  AccuracyMetric::ALL; RootPruneReason::{HeapSmallerThanTopK,
  ValuesPerEvaluationUnknown}; MetricRule.accepted_metrics;
  PruneCause.through_pruned_candidates; Memo::producers; add_candidate
  checks θ against the group's state (CandidateError::LogicalProperties);
  a candidate's
  input requirements are part of its identity in add_candidate; summary
  coverage accessors no longer test-only; origins SummaryInput and
  WidenedScans deleted.
- accuracy: theoretical_guarantee at a failure probability (KLL and KMV
  evaluated at it, others checked against their own),
  producer_satisfies_logical_requirements returns the failing root reason,
  readout_metric; node-tree composition, propagation, budget split,
  function rules, PropagationStats, test_support and the test-only sizing
  deleted.
- devtools: one pass2 stage.
…natives; Count-Min weights via column_sign (#1207 PR 5)

Stage 2 expands each logical candidate Pass 2 kept into its offered
(timing, lifetime): ingestion time for time-independent groups and their
inputs (counter states only over ordered rows), AcrossEvaluations for
chains and grid-evaluated groups. Prune reasons: NoIngestionPipeline,
RawDataNotRetained, QueryTimeResultsNotRetained,
CountMinWeightsNotProvenNonNegative, InputHasNoBuildableProducer,
ReachesNoQueryOutput. Physical edges carry alpha^log and alpha^phys;
PhysicalASAPMemoCandidates indexes the admissible ones.

Deletes WeightDomain, NonNegativeWeightProof and SummaryUpdate::weight_domain
(the state group's column_sign replaces them), Pass 1's copy of
reads_only_time_series, and per-crate memo test fixtures
(asap_types::test_support).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

optimizer Workload optimization and plan selection refactor

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant