Repository navigation
refactor(stage2): physical expansion (decision-tree levels 3–4), physical edges, pruned alternatives; Count-Min weights via column_sign (#1207 PR 5) - #1235
Draft
zzylol wants to merge 30 commits into
Conversation
zzylol
force-pushed
the
refactor/1207-pass2
branch
2 times, most recently
from
November 9, 2026 07:11
bdf0888 to
bcae9e5
Compare
zzylol
force-pushed
the
refactor/1207-stage2
branch
from
November 9, 2026 07:43
d11f12e to
616f765
Compare
zzylol
force-pushed
the
refactor/1207-stage2
branch
from
November 9, 2026 11:53
616f765 to
f20177e
Compare
zzylol
force-pushed
the
refactor/1207-pass2
branch
from
November 9, 2026 17:11
bcae9e5 to
8107319
Compare
zzylol
force-pushed
the
refactor/1207-stage2
branch
from
November 9, 2026 17:44
f20177e to
e05016c
Compare
zzylol
force-pushed
the
refactor/1207-pass2
branch
from
November 9, 2026 20:52
8107319 to
5d3f6de
Compare
zzylol
force-pushed
the
refactor/1207-stage2
branch
from
November 9, 2026 21:28
e05016c to
513a1e7
Compare
This was referenced Nov 10, 2026
zzylol
force-pushed
the
refactor/1207-pass2
branch
from
November 10, 2026 01:26
5d3f6de to
dea088e
Compare
zzylol
force-pushed
the
refactor/1207-stage2
branch
from
November 10, 2026 01:26
513a1e7 to
4c77ec6
Compare
…nes they replace (#1207 PR 0)
…iming, lifetime }
Deletes Schedule and the σ-only InvalidPhysicalProperties variants; the only
invalid combination left is (IngestionTime, OneEvaluation). satisfies_physical
checks timing x lifetime only; delivers_on is the per-context alignment check
(w | I and w | φ). A OneEvaluation result runs in one evaluation context, so
x_{κ,v} covers it as for ℓ = None.
…physical drops memo InvalidLogicalProperties gains SketchBuildWithoutParameters and renames FloorNotInParameterList to ParametersNotInList, returned by LogicalProperties::new. LogicalRequirement::new no longer checks list membership (a floor is any value of the family's parameter type, e.g. heap_size >= k), so it returns Self. satisfies_physical takes only the producer and consumer.
Context::QueryEvaluation(QueryId) becomes Context::Evaluation(EvaluationSchedule):
queries with the same (interval, phase) are evaluated together and share
their OneEvaluation results. evaluation_schedule returns that schedule.
QueryId stays for z_{q,v}. A non-periodic query has no context yet.
InvalidLogicalProperties -> LogicalPropertiesError, InvalidPhysicalProperties -> PhysicalPropertiesError, LogicalRequirement -> LogicalRequirements (parameter_floor -> parameter_floor_for_required_accuracy), satisfies_logical / satisfies_physical -> producer_satisfies_logical_requirements / producer_satisfies_physical_requirements, Medium -> StorageMedium, Context / ContextId -> ExecutionContext / ExecutionContextId (x_in_context / y_in_context -> x_in_execution_context / y_in_execution_context). PhysicalCandidate and PhysicalCandidateKey fields -> logical_candidate_id, physical_properties; edge fields -> producer_candidate_id, consumer_candidate_id; PhysicalASAPMemoCandidates fields -> logical_candidates, physical_candidates.
Deletes EvaluationSchedule and ExecutionContext::evaluation_schedule. An
evaluation context is Evaluation { recurrence, range_evaluation }: queries
with equal ones share it and its OneEvaluation results, one-off queries
included. ExecutionContext, ExecutionContextId and RefreshWindowId move to
asap_types::stages so PhysicalCandidate::delivers_on can take the context.
Contexts are deduplicated by equality (QueryRecurrence holds an f64).
PhysicalASAPDAGValidationError::InvalidPhysicalProperties becomes
PhysicalProperties { node, reason }. In the files this unit edits, "kept"
describing a materialized result or retained data becomes "materialized"
or "retained" (cost, materialized views, realizer, installable plan,
deployment capabilities; locals kept/raw_kept_rows -> retained/
raw_retained_rows), and "reader" becomes "query" where it is a query
(execution demand, ScheduleGroup::queries, cost usage, labels, tests) or
"consumer" where it is a consuming candidate.
find_or_add_group returns the node rules' SchemaDerivationError and add_candidate a CandidateError (the node rules, or an output that differs from the group's). Bodies stay todo!() until PR 2.
…mo (#1207 PR 0) Memo::groups is private, read through groups(); groups are added only through find_or_add_group and queries through add_query (todo!() until PR 2). memo_of_queries returns Result and its body is todo!() until PR 2, which builds Stage 0 on find_or_add_group; Stage 0's node-level CSE (cse.rs), its test_support export and the dead share_identical helper are deleted.
…OperatorInput; coverage only in the memo (#1207 PR 0) MemoGroup { expression, group_properties: GroupProperties, parents, queries, candidates } replaces logical_node and child_groups; GroupProperties { schema, result_kind, summary_coverage, shareable_when_equal } is derived once by derive_group_properties, which applies the existing rules to the expression over MemoGroupAsInputs. The rules (output_schema, output_kind, validate_inputs, is_shareable_when_equal, summary builds, window chains, scalar subqueries, top-k, populations) are generic over OperatorInput, implemented for Rc<OperatorNode> and MemoGroupAsInput; only how they read an input changed. BuiltOrInput carries the builds a rule makes while deriving (one build over a chain's window, the build a rollup or a selection stands for). SummaryCoverage { definition: SummaryStateDefinition { input_group, unlifted_operators }, selection } is derived only from memo groups (derive_from_inputs) and stored in GroupProperties; node-tree coverage, NodeToSummaryCoverageCache, derive_cached and OperatorNode::coverage are deleted, so node trees have no merge check. Memo::logical_node_trees is deleted: Pass 1 target discovery (PR 3), devtools group labels (PR 9) and the export's view coverage (PR 6) are todo!(). find_or_add_group and add_candidate derive the properties, then todo!() until PR 2.
…elpers (#1207 PR 0) One state's θ is no longer sized for several consumers: θ comes from parameter_list per candidate. size_for_all_readers had no non-test caller; min_recorded, kll::shared_k and univmon::shared_heap served only it.
…perators; delete merged_window_build_node (#1207 PR 0) - BuiltOrInput / BuiltInput renamed InputOrTemporaryBuild / TemporaryBuild: a build a rule constructs only while it derives, never stored in the memo. - SummaryStateDefinition::unlifted_operators is Vec<Operator<Option<MemoGroupId>>>: None marks an operator's main input, Some(g) another input group it reads (a scalar subquery's). States whose definitions read subqueries compare and merge; CoverageError::SubqueryInDefinition deleted. - merged_window_build_node deleted with its test; the accuracy guarantee of a read of merged tumbling windows is todo!() until PR 4 rewrites it over groups.
…, TimeBound, Pass 2 pruned list, Stage 2 entry (#1207 PR 0) - Delete WindowLayout, LogicalProperties::window_layout and its two error variants, SummaryMerge::newest_tumbling_windows and SummaryTumblingWindowChain::tumbling_window_count, with the count-based chain builds and expansion; behaviour that read the count is todo!() for PR 1, 2, 6, 7 or 9. - SelectionBox { columns } only, and TimeBound { At, FromEvaluationTime }; lifting TimeRange/TimeShift is todo!() for PR 2. - PrunedLogicalCandidate, PruneCause, RootPruneReason and LogicalASAPMemoCandidates::pruned. - error_propagation(consumer, metric) -> Option<ErrorPropagation>; Memo::set_input_requirements. - PhysicalCandidateKey deleted (PrunedCandidate<PhysicalCandidate>), PrunedCandidate::group and PhysicalASAPMemoCandidates::query_roots dropped; new(logical, physical, edges, pruned). - Stage 2 entry takes &DeploymentCapabilities (moved into asap-types) and cannot fail; PruneReason follows the doc's Stage 2 table. - DataArrival::{Ingesting, Mixed} gain max_out_of_order_ms. - GroupProperties::summary_coverage -> aggregate_coverage.
…l; ScalarValue::Timestamp (#1207 PR 0) Replace TimeBound by IntervalConstraintLimit { FixedValue(ScalarValue), EvaluationTimePlusMs(i64) }: Constraint::Interval holds Bound<IntervalConstraintLimit> on every column, so the timestamp column and value columns share one interval type. Doc comments carry the examples (m[5m], offset 1h, @ T, SQL ts >= literal, price > 10). ScalarValue::Timestamp(ms since the epoch) is the fixed value of an @ T or a SQL timestamp literal.
…s; delete ParametersOfAnotherFamily (#1207 PR 0) A range m[r] evaluated at t reads (t - r, t]: IntervalConstraintLimit and Constraint::Interval examples, the chain window doc and the primitive-schema table now use Excluded lower and Included upper ends. A parameter list holds only its own family's parameters, so ParametersNotInList also rejects another family's parameters.
…acy, the plan is one PhysicalASAPDAG (#1207 PR 0) - Terminology: ops' `family` is `summary_type`; FieldDataType::Sketch(SketchAlgorithm) (SketchKind, GroupingStrategy, HydraKind and the ops' `grouping` deleted); FamilyCapability is SummaryTypeCapability; parameter_list(algorithm). - θ is structure only: SketchParams derives equality; KLL failure_probability, UnivMon distinct_failure_probability and HLL/KMV relative_error deleted. - ValueSign and GroupProperties::column_sign; SourceFacts::column_signs replaces values_non_negative. - LogicalRequirements { required_accuracy: Option<RequiredAccuracy> }; RootPruneReason::BelowRequiredAccuracy; Memo::add_query takes the target and n_q; AggIntent measures carry no accuracy and front ends take none. - asap-accuracy: AccuracyEvidence::measured_guarantee, theoretical_guarantee (was local_guarantee), producer_satisfies_logical_requirements(memo, producer, consumer, input_index, evidence); the AccuracyModel trait and evidence provider, Pass 1's accuracy pruning and AccuracyCheckInputs deleted. - Physical: ExecutionContext::Evaluation { grid: EvaluationGrid }, IngestionPipeline(RefreshWindow); RefreshWindowId deleted; producer_satisfies_physical_requirements(producer, consumer_timing); PruneReason::CountMinWeightsNotProvenNonNegative. - Plan: PlanOutput { plan: PhysicalASAPDAG }; PhysicalASAPDAGNode:: selected_physical_candidate; select_plan returns (PhysicalASAPDAG, Selection); Selection { accepted_accuracy_per_query, pruned_candidates, milp_* }; RealizedPlan, PhysicalWorkloadPlan, DagOfSelectedCandidates, SelectedEdge, SelectedNodeId, AssignmentRealizer, ScanWindow and the scan/view window functions, SearchReport, SearchStatus, InexactObjective, ObjectiveDiffersFromPlanCost, SelectedPlanFailedCheck and PlanCost::cost_unit deleted. - Plan file wire version 14 (summary_type, Sketch(algorithm), no grouping). - Cost names: NodeCostSource, cost_source, PlanRanking::Cost; Serialize on PruneReason, PhysicalProperties, OriginOfCandidateForMemoGroup and RootPruneReason.
…e trait, MemoQuery, evaluation grids of groups (#1207 PR 0) 2026-11-09 decisions: - theoretical_guarantee and AccuracyEvidence::measured_guarantee take the failure probability and return the error bound that holds at it (None when θ cannot fail that rarely); bodies stay todo!() for PR 4. - AccuracyEvidence is a trait; NoMeasuredGuarantees has no measurement. PlanningModels::evidence and Pass 2 take &dyn AccuracyEvidence. - The memo holds its queries in one Vec<MemoQuery> { root, accuracy_target, values_per_evaluation }; add_query(query). - SupportedSummary::family is summary_type. - satisfies is deleted with metric_meets, scope_meets, native_target and AccuracyScope::covers, which only it used; target_limit is the comparison limit with the tolerance. - PhysicalCandidate::delivers_on(&PhysicalASAPMemoCandidates, context). - GroupProperties::depends_on_evaluation_time (derived in PR 2) and LogicalASAPMemoCandidates::evaluation_grid_of_group (filled in PR 4).
… the tests that use it (#1207 PR 0) pass1/composition.rs composed one whole-workload candidate DAG from one per-computation candidate per target, for tests only; planning adds every way to compute to the memo instead. It and the 74 tests and fixture helpers that compose through it are deleted (no interim code for PR 3).
…racy argument (#1207 PR 0)
Move-only. Split asap-logical-asap-optimization into asap-stage0-memo, asap-stage1-pass1-per-computation-candidates and asap-stage1-pass2-reuse-candidates; rename asap-physical-asap-optimization to asap-stage2-physical-optimization and asap-plan-selection to asap-stage3-plan-selection. The old test_support splits by caller. The crate-graph test checks the new crates; no stage crate depends on another.
Pass 1's tests that build Stage 0's memo and Stage 0's tests that lower PromQL through Pass 1 are deleted, with the dev-dependencies and helpers only they used. lower_promql moves to Pass 1's test fixtures, so Pass 1 no longer depends on the PromQL frontend outside its own tests. The crate-graph test now also checks dev-dependencies between stage crates.
…1207 PR 0b) Frontend tests that compose Pass 1 candidates and Pass 1 tests that lower PromQL through the frontend are deleted, with the helpers, dev-dependencies and test targets only they used. MetricsQL's test that compares with PromQL's lowering goes too, so no frontend tests depend on another frontend. The crate-graph test checks dev-dependencies between stage crates with no exceptions.
…R 0b) The PromQL test that planned through asap-planner is deleted with its planner_support helpers; the PromQL and SQL frontends drop their asap-planner dev-dependency, and PromQL its unused asap-accuracy one. The crate-graph test now also forbids a stage crate's dev-dependency on the facade.
- LogicalProperties::new(expression, parameters): a sketch build (a chain included) needs θ from parameter_list(algorithm) of its FieldDataType::Sketch(algorithm); an exact operator has none. - parameter_list(algorithm): structures only. Count-Min lists each width with depths 5, 7, 10 and 12, Count Sketch with 47, 81, 117 and 155 (its median-of-rows δ levels 1e-2 … 1e-5); heap variants each of those with the 1-2-5 heap sizes; UnivMon 40 entries; KLL, DDSketch, HLL, KMV one entry per ε level. A test keeps the lists equal to today's sizing, which is test-only. - PhysicalProperties::new rejects (IngestionTime, OneEvaluation); producer_satisfies_physical_requirements(producer, consumer_timing). - PhysicalCandidate::delivers_on(candidates, context): None in every context, OneEvaluation only in Evaluation contexts, a chain with AcrossEvaluations when its width divides the grid's period and phase (or the refresh window's width and phase), any other AcrossEvaluations result everywhere when its group does not depend on the evaluation time, else when its group's evaluation grid holds the context's times; Materialized(v) contexts stay todo!() for PR 7. - Chains: one schema (window_end at column 0, keys, state), TimeRange and TimeShift over a chain select windows (an instant TimeRange is rejected), a SummaryMerge over them finalizes and rolls up as the chain's build over that range (the two PR 0 todos). - summary_type_of_operator; QueryRecurrence::evaluation_grid_ms is None for a fixed interval without a phase.
…, depends_on_evaluation_time, aggregate_coverage), @ end()/@ start() lifted above their subtree (#1207 PR 2) - Memo: hash-consing (find_or_add_group by expression hash), candidate storage (add_candidate), add_query(MemoQuery); Memo::new(declared column signs), private declared_column_signs read by a Scan's column_sign. - Stage 0: memo_of_queries(roots, accuracy_targets, data) adds each query with its accuracy target and n_q (values_per_result over series_read), and collects the declared signs of the scanned sources (a counter's value is non-negative without a declaration). - GroupProperties::column_sign derived per operator (concepts doc, "Value sign"); depends_on_evaluation_time: a TimeRange or time()/now() makes a result depend on the evaluation time, @ T pins it; reads_only_time_series: a Scan of a time series, or every input group reading only time series (OperatorInput::reads_only_time_series computes the same rule over node trees; the recursive walks in topk.rs and Pass 1 are deleted). - aggregate_coverage for summary states, chains, TimeRange/TimeShift over a chain, merges and exact Aggregates; timestamp constraints are IntervalConstraintLimit intervals (t - o - W, t - o]; SQL timestamp literals and identity casts fold so absolute filters lift. - PromQL: @ end() / @ start() are lowered as TimeShift { at } above the selector or subquery they anchor, which is lowered without them, so it is the group of the same expression without @; coverage stops at that TimeShift (it is the definition's input group).
…1207 PR 2b) GroupProperties gains column_lineage (one ColumnLineage per output column, derived once from the input groups'). Lineage and column_sign are derived together by one per-operator column layout (stages/column_properties.rs, which replaces column_sign.rs): a copied column keeps its lineage and sign, a computed one is Computed with the operator's sign rule. Chains compute window_end and their state and keep their keys' lineage; a TimeRange or TimeShift over a chain, and TimeShift { at: End | Start } above a selector, pass lineage through; a merge of a chain's selected windows has the chain build's layout. Summary coverage names a selection's column by its SourceColumn lineage (the one SourceColumn type, beside Source); ColumnIdentity is deleted. A predicate on a Computed column, or on one of two columns of one lineage, is not lifted; the regrouping key check compares lineage. Schema fields and the IR Project lose their table qualifier; the front ends bind qualified names with their own scope while lowering. The requalification patches (with_aggregate_qualifiers, node.rs) are deleted.
…1207 PR 3) Pass 1 fills Stage 0's memo with each group's initial candidates, every intermediate result its own group and every candidate one operator over input groups: - computation_targets walks memo.groups(): one target per measure of each Aggregate group with a realization besides plain exact execution. - add_pass1_candidates adds every group's own expression, exact accumulators (state group + FinalizeExactAccumulator; avg from sum and count), one SummaryAgg state group per sketch algorithm with one build candidate per configuration of parameter_list(algorithm) and separate SummaryEstimate readout groups, fused builds (whole-expression top-k heaps; nested sum/min/max merges) listing the replaced groups in fused_groups, multi-measure joins, and key-only HAVING pushed below in the consumer Filter group's candidates. - Pass 1 sets no input requirement and no accuracy floor: every candidate's LogicalRequirements are empty; Pass 2 gives each readout its required accuracy (a top-k readout's heap_size >= k included). - ComputationTarget and FusedComputation move into Pass 1 (crate-private), FusedComputation::absorbed_groups replaces target indices, HavingEntry, HavingPlacement and Pass1Error are deleted; pass1_memo returns PerComputationCandidateError. - types: topk::ranked_column_of_group (ranked_column becomes pub(crate)); one unique-key rule for a top-k's ranked rows, exact or a heap readout; any_measure_filtered generic over the predicate's input. - Tests build their memo through the Memo methods; the integration tests stage1_logical_asap_optimization and per_computation_candidates_sql run SQL and PromQL queries through Stage 0 and Pass 1.
…requirements, dead-candidate elimination, chain groups (#1207 PR 4) Pass 2 runs, in order: the general rollup rule with one synthesized finest common group per cluster of equal definitions (replaces scan widening, finer-aggregate reuse and the rollup rules); top-k reuse; reach, caps and readout requirements (one readout candidate per requirement level, written with Memo::set_input_requirements); dead-candidate elimination with every root cause (PrunedLogicalCandidate); chain groups SummaryTumblingWindowChain[w](X) for the surviving state groups, widths from the gcd closure of the reaching queries' grids, the window and the offsets, then elimination again; and the logical edges between survivors. A merge, and a Filter/TimeRange/TimeShift over states, carries the θ of the states it reads (one candidate per θ, reading only producers with that θ). The metric rule cuts edges from readouts whose metric the consumer's input does not accept (accepted metrics pushed down from the targeted queries), not candidates; a candidate cut off from an input is eliminated too. A whole-result δ with n_q unknown prunes the readouts it reaches (ValuesPerEvaluationUnknown). The summary-input and window-rollup rules and the local-window-composition step are deleted. - types: LogicalRequirements::error_propagation(memo, consumer, metric) with the constant contracts (key-only Filter by column lineage, abs / clamp / negation, min / max / avg / count, + - / *), b = 0.1 private; AccuracyMetric::ALL; RootPruneReason::{HeapSmallerThanTopK, ValuesPerEvaluationUnknown}; MetricRule.accepted_metrics; PruneCause.through_pruned_candidates; Memo::producers; add_candidate checks θ against the group's state (CandidateError::LogicalProperties); a candidate's input requirements are part of its identity in add_candidate; summary coverage accessors no longer test-only; origins SummaryInput and WidenedScans deleted. - accuracy: theoretical_guarantee at a failure probability (KLL and KMV evaluated at it, others checked against their own), producer_satisfies_logical_requirements returns the failing root reason, readout_metric; node-tree composition, propagation, budget split, function rules, PropagationStats, test_support and the test-only sizing deleted. - devtools: one pass2 stage.
…natives; Count-Min weights via column_sign (#1207 PR 5) Stage 2 expands each logical candidate Pass 2 kept into its offered (timing, lifetime): ingestion time for time-independent groups and their inputs (counter states only over ordered rows), AcrossEvaluations for chains and grid-evaluated groups. Prune reasons: NoIngestionPipeline, RawDataNotRetained, QueryTimeResultsNotRetained, CountMinWeightsNotProvenNonNegative, InputHasNoBuildableProducer, ReachesNoQueryOutput. Physical edges carry alpha^log and alpha^phys; PhysicalASAPMemoCandidates indexes the admissible ones. Deletes WeightDomain, NonNegativeWeightProof and SummaryUpdate::weight_domain (the state group's column_sign replaces them), Pass 1's copy of reads_only_time_series, and per-crate memo test fixtures (asap_types::test_support).
zzylol
force-pushed
the
refactor/1207-pass2
branch
from
November 10, 2026 13:37
dea088e to
e98e80b
Compare
zzylol
force-pushed
the
refactor/1207-stage2
branch
from
November 10, 2026 13:37
4c77ec6 to
8876072
Compare
zzylol
force-pushed
the
refactor/1207-pass2
branch
from
November 11, 2026 18:33
e98e80b to
5ac57fc
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
todo!(), so every planning run stopped there; Count-Min weight signs were aWeightDomainproof on each update, set by Pass 1 and checked by the executor.generate_physical_asap_memo_candidatesexpands each logical candidate Pass 2 kept into its offeredPruneReason; a Count-Min build's weight sign is now the state group'scolumn_sign.quantile_over_time(0.95, latency[1m])every 60 s, Stage 2 offers the chain (QT, None), (QT, OneEvaluation), (QT, AcrossEvaluations), (IT, AcrossEvaluations), prunes its (IT, None) asReachesNoQueryOutput, and emits 12 physical edges into the chain from the scan's three alternatives; planning now stops at Stage 3 (select_plan, PR 6).Review guide
refactor/1207-pass2, PR 4); refactor(stage3): baseline selection MILP and its PhysicalASAPDAG; executor θ, native chains and a time-indexed store (#1207 PR 6) #1236 (PR 6) builds on it. One commit, head4c77ec62d.stage2-physical-optimization/src/catalog/configurations.rs(enumerate_candidate_configurations,read_at_absolute_time,counter_state_rows_ordered,count_min_weights_not_proven_non_negative): offered values and prune rules.catalog/mod.rs(generate_physical_asap_memo_candidates,ingestion_time_offered_per_group): physical edges.catalog/pruning.rs(prune_reasons): buildability fixed point and reach.types/src/stages/physical/memo_candidates.rs:PhysicalASAPMemoCandidates::newand the admissible-edge indexes.WeightDomain/NonNegativeWeightProofandweight_domain:lines in every test; shared memo fixtures replacing per-crate copies (Pass 2'stry_candidate,candidate,group,series_schema,scan_operator; Pass 1'scfg(test)scan_operator);thiserrordropped from Stage 2; doc lines inplanner-layering.md,asap-primitive-schema.md,glossary.md.configurations.rs; executor checks removed inphysical_planner/mod.rs,summary_kernels/factory.rsandcapability.rs(C5).Behavior: before → after
Query A of the concepts doc (test
offers_each_candidate_its_timings_and_lifetimes):quantile_over_time(0.95, latency[1m])every 60 s at phase 0, over ingesting data (rows_per_second 1000, max_out_of_order_ms None); the deployment retains raw data and query-time results. It serves the design need of the formulation: Stage 3 chooses among all physical alternatives, Stage 2 only enumerates and records why an alternative is removed.todo!(); planning stopsPhysicalASAPMemoCandidateswith physical edges, admissible-edge indexes and the pruned list; planning stops at Stage 3 (select_plan'stodo!()of PR 6)ReachesNoQueryOutputWeightDomainproof onSummaryUpdate, executor refusalcolumn_sign; unproven →CountMinWeightsNotProvenNonNegativeInterfaces
crates/stage2-physical-optimization/src/catalog/mod.rs/// Stage 2: expand every logical candidate Pass 2 kept into its offered (τ, ℓ), every logical /// edge into physical edges; record each rejected alternative with its reason. Cannot fail. /// Signature unchanged; was `todo!()`. pub fn generate_physical_asap_memo_candidates( logical: &LogicalASAPMemoCandidates, data: &DataWorkload, capabilities: &DeploymentCapabilities, ) -> PhysicalASAPMemoCandidates;crates/types/src/stages/physical/memo_candidates.rscrates/types/src/stages/pruning.rs/// Why a stage removed a candidate. Stays; variants unchanged, Stage 2's are now produced. pub enum PruneReason { /// Stage 2: IT, and the data has no ingestion pipeline (at rest or unknown arrival). Input fact. NoIngestionPipeline, - /// Stage 2: it scans raw data at query time, and the deployment does not retain raw data. + /// Stage 2: it scans raw data at query time, or retains the raw rows it scans across + /// evaluations, and the deployment does not retain raw data. Input fact. RawDataNotRetained, - /// Stage 2: a query-time result materialized (`OneEvaluation` or `AcrossEvaluations`), and the - /// deployment retains no query-time results. + /// Stage 2: a query-time result materialized across evaluations, and the deployment retains + /// none (`DeploymentCapabilities::query_time_results_retained`); C1. QueryTimeResultsNotRetained, /// Stage 2: an input has no buildable admissible producer. InputHasNoBuildableProducer, /// Stage 2: no query output reads it over admissible edges. ReachesNoQueryOutput, /// Stage 2: a Count-Min build (`Cms`, `CmsWithHeap`, `HydraCms`) whose state column's /// `column_sign` is not `NonNegative`. CountMinWeightsNotProvenNonNegative, /// Stage 3: dominated by the named candidate. DominatedBy(PhysicalCandidateId), /// Stage 3: `asap-cost` could not cost it. UsageNotCostable, }crates/types/src/ir/schema/sketch.rs·crates/types/src/ir/schema/mod.rs/// One summary-state update. Stays; loses `weight_domain`. pub struct SummaryUpdate { /// The key the row is counted under, for keyed summary types. pub item: Option<SummaryInputExpr>, /// The value added to the state. pub weight: SummaryInputExpr, - /// Planner proof for algorithms that need a non-negative stream. - #[serde(default)] - pub weight_domain: WeightDomain, } -/// The proven or required sign of an update's weights. Removed. -pub enum WeightDomain { - /// No proof (default). - #[default] - UnknownOrSigned, - /// Proven non-negative. - NonNegative { - proof: NonNegativeWeightProof, - }, - /// Open requirement: raw data samples of `metric`, confirmed from source facts. - #[serde(rename = "samples_of_metric")] - RawDataSamplesOfMetric { - metric: String, - }, - /// Open requirement: reset-corrected `rate`/`increase`/`irate` of `metric`'s samples. - CounterDerivativeOfMetric { - metric: String, - }, -} -/// Why weights are non-negative. Removed. -pub enum NonNegativeWeightProof { - /// A unit count. - UnitCount, - /// `rate`/`increase`/`irate` of a counter's samples. - ResetAwareCounterDerivative, - /// Raw data samples of a metric declared a counter. - #[serde(rename = "counter_samples")] - CounterRawDataSamples, - /// Raw data samples of a metric declared non-negative. - #[serde(rename = "declared_non_negative_samples")] - DeclaredNonNegativeRawDataSamples, - /// A constant weight greater than 0. - PositiveConstant, -}Removed (decision 2026-11-09): the state group's
column_sign(derived from the weight's sign and the declared source signs) replaces them, read by Stage 2'sCountMinWeightsNotProvenNonNegative;schema::modno longer re-exports either.crates/types/src/test_support.rs(test-supportfeature only; review D11)Example:
group_with_own_candidate(&mut memo, time_series_scan("latency", series_schema(&["job"]), vec![]))is query A's scan group; Pass 2's and Stage 2's tests build their memos with these.Also removed (crate-private, so no block): Pass 1's
CmsWithHeapgate on unproven weights inrealize_keyed_additive_summary_input,raw_data_samples_of_metric,with_raw_data_samples_requirement,SingleMeasure::raw_data_samples_of(Stage 2's Count-Min prune replaces them); Pass 1's privatereads_only_time_series(memo, input)and Stage 2's recursive copy (the group propertyGroupProperties::reads_only_time_series, PR 2, N7); the executor'sCmsWithHeap"nonnegative weight contract" check (physical_planner), the rate-weight refusal increate_planner_accumulator(keyed offResetAwareCounterDerivative) and the Hydra "weight's sign is unknown" binding check (the planner proves the sign before offering a Count-Min build; the Hydra build still rejects a negative weight value at run time,invalid_weight_values_fail_the_build).Not reintroduced from the old PR 5 head (decisions after it):
executor_can_run/PerEntityAggregateNotExecutable(G4/H3: no executor-support check),AcrossEvaluationsWithoutWindowLayout(F1/C7: a value-set rule, not a prune reason),PhysicalCandidateKey,PrunedCandidate.group,PhysicalASAPMemoCandidates::query_roots, the privateDeploymentstruct (D1–D4, F2, F3; the PR 0 types already have these shapes).Usage, end to end (test
offers_each_candidate_its_timings_and_lifetimesat513a1e78e, query A of the concepts doc):Algorithm changes
Stage 2 expansion and pruning
New; the previous head's version was written against the old memo and is replaced.
Problem: at PR 4's head every planning run stopped at Stage 2's
todo!().After:
catalog/configurations.rsenumerate_candidate_configurationsGroupProperties::depends_on_evaluation_timefalse) or an input of one (ingestion_time_offered_per_group), and for a counter state (rate/increase/irate) only when its rows are ordered (max_out_of_order_msdeclared, or PromQL rows); ℓ = None, OneEvaluation(Memory), and AcrossEvaluations(Memory) only for a chain or a group with a knownevaluation_grid_of_group. Values not offered are not recordedCountMinWeightsNotProvenNonNegative,NoIngestionPipeline(IT, arrivalAtRest/Unknown),RawDataNotRetained(aScanat QT, or materialized across evaluations: either retains raw rows),QueryTimeResultsNotRetained(QT AcrossEvaluations)catalog/mod.rsproducer_satisfies_physical_requirements(producer, consumer_timing)catalog/pruning.rsInputHasNoBuildableProducer(least fixed point of "every input has a buildable admissible producer"),ReachesNoQueryOutput(not reached from a query output: an alternative of a root group a query-time consumer may read)PhysicalASAPMemoCandidates::newHashMap<(consumer, input), Vec<PhysicalEdgeId>>,Vec<Vec<PhysicalEdgeId>>by producer) built onceCandidates Pass 2 eliminated (
LogicalASAPMemoCandidates::pruned) are not expanded. Stage 2 checks no executor support and costs nothing; alignment with each execution context stays Stage 3's (PhysicalCandidate::delivers_on, PR 1).Correctness: an emitted candidate has, for each input, an admissible edge from an emitted producer, and is reachable from a query output (checked by every test's
generatehelper, which also checks the edge order and the indexes); a pruned alternative is recorded exactly once, with the first reason that applies (the Count-Min sign is checked first: it holds for every (τ, ℓ)).Complexity: O(|C|·5) alternatives; O(Σ_{(c,i)} |alt(c)|·Σ_u |alt(u)|) ≤ 25·|logical edges| physical edges; the buildable fixed point O(depth·|E|), reach O(|E|).
Assumptions and limits: Memory storage only (the capabilities offer no disk, Executor: store Disk-medium materialized views and one-evaluation results on disk #1217); no executor-support check (G4/H3: executor gaps are PR 6's).
Representation:
PhysicalCandidate { logical_candidate_id, physical_properties };PhysicalEdge { …, satisfies_logical_requirement, satisfies_physical_requirement };PrunedCandidate<PhysicalCandidate> { candidate, reason }. Enumeration: every offered (τ, ℓ) per kept logical candidate; every producer/consumer alternative pair per logical edge. Stage 3 chooses.Decisions and open questions
Needs reviewer decision: None.
Decided:
column_sign); time indexing (any grid-evaluated group may beAcrossEvaluations).WeightDomaindeleted in PR 5 (rejected: keeping a per-update proof set by Pass 1 and checked by the executor).PhysicalEdge::is_admissiblerenamedis_logically_and_physically_admissiblebefore landing (the name says both requirements; no alias). Rejected:is_admissible.reads_only_time_seriesis the group propertyGroupProperties::reads_only_time_series, derived once in PR 2; the recursiveasap_types::stages::reads_only_time_seriesis gone. Example: forrate(c_total[5m])'s chain overScan(TimeSeries c_total)it is true, so the counter chain is offered ingestion time withmax_out_of_order_ms: None; overScan(Table events)false, so only withSome(_).test-support(above).QueryTimeResultsNotRetainedprunes only (QueryTime, AcrossEvaluations). The prune-reason table and PR 0's comment said OneEvaluation too; the decision-tree diagram,DeploymentCapabilities::query_time_results_retained(default false) and the old PR 5 said AcrossEvaluations only. Rejected: OneEvaluation too, which with default capabilities removes every within-evaluation sharing (C3: aNoneproducer has at most one consumer per context). Query A keeps the readout's (QT, OneEvaluation) and prunes its (QT, AcrossEvaluations). The concepts doc's row now reads "QueryTime with ℓ = AcrossEvaluations".@ T. Example:sum_over_time(m[1m] @ 1000):TimeShift @1000is time-independent; its inputTimeRange 1mand that one'sScanare offered (IT, None) (an_at_t_subtree_is_offered_ingestion_time). Without@ Tthe closure adds nothing.column_sign), equal for every candidate of the group. Rejected: a second derivation of the weight's sign. Example:CmsWithHeapkeyed by series, weighted bySampleValueofm: pruned in all three alternatives (QT None,QT One,IT None) without a declaration, offered withcolumn_signs: {"value": "non_negative"}(a_count_min_build_needs_weights_proven_non_negative).sum_over_time(m[1m] @ 1000)'s rootTimeShiftkeeps (QT, None) and (QT, OneEvaluation); its (IT, None) isReachesNoQueryOutput.create_planner_accumulator(with the proof gone the factory cannot tell a rate weight from a sum weight); typed binding of window-weighted heaps listed for PR 6.SummaryAgg, chain,MaintainPopulation) is offered (IngestionTime, None): fed at ingestion it keeps its state between batches, so it is materialized; matches the exported plan's validation. Downstream: example 3a and o11y q18/q21 get valid plans (PR 9 snapshots).@ Twith no known horizon end, noAcrossEvaluationscandidate (its consumer reaches back horizon end − T + range, unbounded); it still runs at query time. Example:x @ 1700000000 [5m]: the scan keeps (QT, None), (QT, OneEvaluation), (IT, None). A producer deeper below (a chain under theTimeRange) is not excluded here: Stage 3 gives it noRawDataNotRetainedalso rejects an ingestion-timeScanmaterialized across evaluations (it retains raw rows; a no-raw plan held a rawScanbuffer at ingestion).PruneReasondoc,planner-layering.md,asap-aware-mapping-contracts.mdand the concepts doc prune table say so.Verification
cargo fmt --all -- --check,cargo clippy --workspace --all-features --all-targets -- -D warningsclean;crate_graphpasses. Per cratecargo test -p <crate> --all-features. By test name against the base PR 4 (5d3f6de85): 0 newly failing, 13 newly passing (the new Stage 2 tests), 2 removed (weighted_topk_binding::rate_updates_cannot_enter_integer_heap_factory,hydra_cms_execution::unknown_sign_weight_is_rejected, with the checks they tested).retention_ms=output.status.success()(2)retention_ms(3),avgsum-and-count plan (3); old-wire fixture JSON (1)New tests (Stage 2,
catalog::tests):offers_each_candidate_its_timings_and_lifetimes(query A's offered values);no_summary_build_is_offered_at_ingestion_time_without_a_lifetime(A4);without_an_ingestion_pipeline_nothing_runs_at_ingestion_time;without_raw_data_only_ingestion_time_builds_read_the_scan(raw data not retained, incl. the materialized ingestion-time scan);query_time_results_across_evaluations_need_the_deployment_to_retain_them(C1);an_at_t_subtree_is_offered_ingestion_time(C2);a_subtree_under_an_absolute_anchor_is_not_materialized_across_evaluations(A7);a_counter_state_runs_at_ingestion_time_over_ordered_rows(SQL vs PromQL, bound declared or not);a_count_min_build_needs_weights_proven_non_negative(C3);edges_carry_both_requirements(12 edges, α^log/α^phys);an_input_without_an_admissible_producer_is_not_buildable;candidates_pass_2_eliminated_are_not_expanded;a_scalar_roots_operand_is_a_query_output.Self-review: no new abstraction (no Stage 2 input struct (F2), no new prune reason, no executor-support table); two admissible-producer indexes stay (before pruning by provisional index, after by ID; D6 allows both, both call
is_logically_and_physically_admissible); duplication removed (reads_only_time_series, one set of memo fixtures, one weight-sign derivation); no dead code added. The concepts doc's D13.2 crate path is alreadystage2-physical-optimization. Screenshots and measurements: not applicable.Follow-ups
@ Tproducers deeper below: Stage 3 gives them no