Repository navigation
Conversation
Move-only. Split asap-logical-asap-optimization into asap-stage0-memo, asap-stage1-pass1-per-computation-candidates and asap-stage1-pass2-reuse-candidates; rename asap-physical-asap-optimization to asap-stage2-physical-optimization and asap-plan-selection to asap-stage3-plan-selection. DeploymentCapabilities moves to asap-types, and the old test_support splits by caller. The crate-graph test checks the new crates; no stage crate depends on another.
Pass 1's tests that build Stage 0's memo and Stage 0's tests that lower PromQL through Pass 1 are deleted, with the dev-dependencies and helpers only they used. lower_promql moves to Pass 1's test fixtures, so Pass 1 no longer depends on the PromQL frontend outside its own tests. The crate-graph test now also checks dev-dependencies between stage crates.
…1207 PR 0b) Frontend tests that compose Pass 1 candidates and Pass 1 tests that lower PromQL through the frontend are deleted, with the helpers, dev-dependencies and test targets only they used. MetricsQL's test that compares with PromQL's lowering goes too, so no frontend tests depend on another frontend. The crate-graph test checks dev-dependencies between stage crates with no exceptions.
…R 0b) The PromQL test that planned through asap-planner is deleted with its planner_support helpers; the PromQL and SQL frontends drop their asap-planner dev-dependency, and PromQL its unused asap-accuracy one. The crate-graph test now also forbids a stage crate's dev-dependency on the facade.
… PR 0b) The feature came through asap-logical-asap-optimization's test-support, which the move deleted, so cargo test -p asap-planner did not build.
… memo groups (#1207 PR 2)
…ries identity (#1207 PR 3)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
#1207's Order, row "3. Pass 1": Pass 1 adds each group's initial candidates (exact, exact accumulator, summary with one candidate per θ, fused) to the memo, with every intermediate result its own group (#1203: one operator per candidate, no private nodes). It also re-creates the Pass 1 test coverage that PR 0b deleted, with tests that build their input
Memothrough theMemomethods, so no stage crate depends on another, even in tests.PR 0 left Pass 1 stopping at a private stub
logical_node_trees(memo): target discovery walked node trees, and candidates came from a whole-workload node-tree composition (compose_logical_candidate) sized per budget share. Neither has a place in the memo model (concepts doc §3: "Everything that was computed on operator-node trees is computed on groups").What
computation_targets(memo, supported_summaries)readsmemo.groups(): each group'sexpressionandgroup_properties. It finds oneComputationTargetper measure of eachAggregategroup with a realization besides plain exact execution, once however many queries read it. Consumers are the group'sparents. Thelogical_node_treesstub is deleted.Memo::find_or_add_groupandMemo::add_candidate(add_pass1_candidates):SummaryAgg[ExactAggregate(kind)](one candidate, no θ) and aFinalizeExactAccumulatorover it.avgfinalizes itssumandcountstates and divides them: a PromQLBinaryOp, or for SQL aJoingroup and aProject;SummaryAgg[family]per supported family, with one build candidate per θ inparameter_list(family), all with the group's own expression (θ only inLogicalProperties::parameters). A separate readout groupSummaryEstimate(state)is read by the result group;Sumbuild over the rows below the inner sum, added as a candidate of the outer sum's own accumulator state group, Logical memo candidate as (one operator over groups, logical properties θ) with a requirement per input; no private nodes #1203 "fused builds"). Their builds list the absorbed groups infused_groups;Aggregategroup per measure, each with its own candidates, joined back on the group keys by one candidate of the aggregate's group;HAVING: a key-only consumerFiltergroup gets the candidates that push its predicate below the aggregate (aFiltergroup over the input rows, which the states read). A valueHAVINGstays aFilterabove each readout.add_candidaterejects it otherwise), so a readout whose column names differ from the group's (quantileforquantile_0_9) is its own group, read by a renamingProjectcandidate. A heap top-k's Float64 estimate is cast to the exact top-k's column type (a SQL count ranks Int64).LogicalRequirementsof each consumer input (input_requirements): a top-k readout of a heap state needsheap_size ≥ k(the family's weakest listed width and depth withheap_size = k). Every other input has no floor. See "Decisions" for validity caps and the metric rule.Unsupported, e.g. a SQLcount(*)with no non-null column): the group keeps its other candidates. A candidate that breaks the node rules or the group's output is a bug and fails Pass 1 (PerComputationCandidateError::Candidate).pass1_memo(memo, supported_summaries)no longer takes the queries' accuracy targets, the accuracy model, evidence or data. They served onlywithout_summaries_no_query_can_use, which composed whole-workload node trees to prune families no query could use; that pruning is query-sensitive (§4: "query-sensitive pruning follows logical generation and reachability analysis") and moves to PR 4's pruning caps.compose_logical_candidate,compose,compose_roots,rewrite,substitute,substitute_measure,with_names_of,with_aggregate_qualifiers,whole_expression,nested_sum,Sizing,ChosenComputation,Chosen,PerComputationCandidates(withcombinations,absorbed_choice_varies,without_unsupported_summaries,without_summaries_no_query_can_use),output_can_meet_target,sources_read,absorbed_targets,generate_per_computation_candidates,logical_node_trees,ComputationTarget::{choice_count, computation_for},PerComputationCandidateError::InvalidChoice, the Pass 1 featuretest-supportand itstest_supportmodule;asap_accuracy::test_support::{estimate_has_bound, estimate_bound_over_raw_data_samples, local_guarantee}(their only callers were Pass 1's tests, which now live inasap-accuracy); the executor test helpercomputation_candidates(no caller) and the executor's dev-dependencies on Stage 0 and Pass 1.Decisions and pending items
error_propagation(stilltodo!()there). Pass 1 sets only the direct needheap_size ≥ k.add_candidate's output checktablequalifier or the schema'sclosedflag (same_output_columns, replacing==). Without this, no candidate but the plain aggregate fits: a grouped SQL aggregate's keys keep their table while its measures have none, and aProjectsets one qualifier for all columns; a PromQL per-series aggregate leaves its schema open, and the renamingProjectcloses it. The old composition patched the node's schema (with_aggregate_qualifiers), which a derived group schema cannot. Plans are unchanged: the old candidates renamed through the sameProject.fused_groupsof a fused buildmemo_population.rsset them on main (itsimplements). Under edges, the candidate does not compute those groups' results (it reads the rows below them); a consumer of an absorbed group still needs one of that group's own producers. Stage 3 (PR 6) must not treatfused_groupsas available results.HAVINGpush-downFilter's predicate goes below in theFiltergroup's own candidates, whatever the aggregate's other consumers read; old Pass 1 pushed it only when every consumer applied it (#1089), because a candidate replaced the aggregate for all of them.pass1_memoargumentsaccuracy_targets,accuracy,evidenceanddata(see What).planning::explicit_local_candidates,planning::explicit_summary_candidate,avg_as_sum_and_count) nowtodo!("#1207 PR 6: ..."): realizing one plan per way from the memo is PR 6's realizer. Interim inside the merge unit; PR 6 fills them in.New abstractions not in #1207 or the doc (pending approval)
pub struct MemoGroupAsInput(waspub(crate)),MemoGroupAsInput::group_id(),Memo::group_as_input(group_id)types/src/stages/{memo,candidates}.rsOperatorInputrules as a node (ranked_column, the keyed-heap rule, nested-sum detection), and needs the group id back to build over it.same_output_columns(left, right)types/src/stages/candidates.rsraw_data_samples_of_metric<I: OperatorInput>(operator: &Operator<I>)(was(node: &OperatorNode));any_measure_filtered<C>(was node-typed)accuracy/src/propagation.rs,types/src/ir/operator/non_asap.rsHavingEntry::filter: Option<MemoGroupId>(wasOption<Rc<OperatorNode>>);HavingEntry::predicate,ComputationTarget::measure_filtertypedPredicate<MemoGroupId>types/src/stages/per_computation_logical_memo_candidates.rsPerComputationCandidateError::{Candidate(CandidateError), LogicalProperties(LogicalPropertiesError)}computation_targets,add_pass1_candidates,AggregateWays,SingleMeasure,add_aggregate_candidates,add_fused_candidate,measure_output,accumulator_state,finalized_with,accumulator_build,exact_accumulator_build,add_output_candidate,renaming_project,add_group,add_state_group,add_candidate,input_requirements,heap_floor,unless_unsupported,expression,schema,without_unsupported_summaries(free fn);realization::non_asap, genericPhysicalSummaryInput<I>nested_sum_input,whole_expression_input,realize_keyed_additive_summary_input,summed_per_series_samples,ranked_value,reads_only_time_series,measure_input,counted_column) are generic overOperatorInputinstead of taking nodes.metric_schema,scan_operator,group,scan_group,aggregate_group(_with),query(test_fixtures.rs);memo_printout,label,candidates,groups_where,state_group,range,instant,table_group,compare,filter,divide,quantile,sum,count,topk,pushed_below(Pass 1 tests);pass1,pass1_over,having_pushed_below(per_computation_candidates_sql.rs);pass1,root_group,aggregate_groups,candidates_of_aggregates,sketch_builds,avg_candidate_groups,pass1_sql(stage1_logical_asap_optimization.rs)How
computation_targetswalksmemo.groups(). For eachAggregateexpression it records the ownhavingand each parentFiltergroup whose input is the aggregate (key-only →BelowAggregate). It lists the ways per measure (computation_candidates_for_intent, unchanged), then detects fused computations over groups (add_whole_expression_targets,add_nested_sum_targets,close_absorptions), prunes a PromQL top-k's own heaps, and drops unsupported families.add_pass1_candidatesadds every Stage 0 group's own expression as a candidate, then, per aggregate group (and per key-onlyFiltergroup above it),add_aggregate_candidatesbuilds each way bottom-up:add_state_group(θ list or one exact build),add_groupfor each intermediate (the operator is also its group's candidate), andadd_output_candidatefor the top operator: added to the result group, or, when its column names differ, added as its own group and read throughrenaming_project.add_candidate(this crate) →LogicalProperties::new(θ in the list, none on exact operators) →input_requirements→Memo::add_candidate.Key code interfaces added / modified
stage1-pass1-per-computation-candidates/src/logical_memo_candidates.rsstage1-pass1-per-computation-candidates/src/per_computation_candidates.rstypes/src/stages/memo.rs,types/src/stages/candidates.rstypes/src/stages/per_computation_logical_memo_candidates.rsaccuracy/src/propagation.rs,types/src/ir/operator/non_asap.rsUsage (from
running_example_gives_one_build_per_theta_and_separate_readouts, run at this head):Major algorithm changes
Pass 1 candidate generation.
todo!(). Before PR 0, it built one whole-workload node tree per choice and sized each sketch for an equal budget share; the result was copied into private node chains.compose_logical_candidate(choice)→ node tree per choice → sizing peraccuracy_budget_parts→ private chains. After: targets over groups → per way, groups bottom-up throughfind_or_add_group, one build candidate per listed θ, one operator per candidate.add_candidate); the ported frontend tests (stage1_logical_asap_optimization::{sql,promql}_candidates_output_the_query_columns) run every SQL and PromQL query of the old composition test through Pass 1.|parameter_list(family)|(4–70) build candidates to one shared state group, instead of one node tree per whole-workload combination.Before / After
The doc's running example (concepts doc §1): query A = p95 of
latency, query B = p99 / p50 of it (running_example_gives_one_build_per_theta_and_separate_readouts). Before (PR 2):pass1_memopanics attodo!("#1207 PR 3: discover targets over the memo groups, not node trees"). After, the memo (printed by the test;_=PerSubpopulationInstance):Mapping to the doc's table: G1 here is the doc's G1 (
latency, viaScanG0 andTimeRangeG1); G6 = doc G2 (KLL state, one build per θ of KLL's list; the doc'sk = 200/345are illustrative); G8 = doc G2′ (DDSketch); G2/G3/G4 = doc G3/G4/G5, each with its exact aggregate and a renamingProjectover a readout group per family (G7/G9, G10/G11, G12/G13); G5 = doc G6. The doc shows G4/G5 reading only G2′: that is the metric rule, which drops the KLL → ratio edges in PR 4 (decision 2).Verification
cargo check,cargo clippy --workspace --all-targets --all-features -- -D warningsandcargo fmt --all --checkare clean.Pass 1's tests (
stage1-pass1-per-computation-candidates/src/per_computation_candidates_tests.rs, 25 tests). Each builds its memo throughMemo::find_or_add_groupandMemo::add_query(test_fixtures.rs); no other stage crate is a dependency, even in tests. PR 0b's list (#1229), by category:running_example_gives_one_build_per_theta_and_separate_readoutsmulti_measure_aggregate_has_one_target_per_measureaggregate_without_realization_is_not_a_targetan_aggregate_read_by_several_roots_is_one_targetasap_input_and_invalid_accuracy_are_rejectedunsupported_summaries_are_not_offeredevery_group_offers_its_own_expressionexact_accumulator_is_a_state_group_and_a_finalizemulti_measure_candidate_joins_single_measure_groupsonly_a_topk_readout_has_a_parameter_floorheap_size ≥ kkey_only_having_is_pushed_below_into_the_filter_groupFiltergets candidates over filtered input rows; the aggregate keeps its ownvalue_having_stays_aboveglobal_aggregate_having_stays_aboveHAVINGstays aboveown_value_having_filters_each_readouthavingis aFilterover each readoutwhole_expression_topk_heaps_read_the_raw_samplesjob, withRawDataSamplesOfMetric, one build per θ,fused_groups= the absorbed sumscount_ranked_topk_counts_with_unit_weightranked_item_is_the_keys_outside_the_partitionnested_sum_merges_into_the_outer_state_groupfused_groups= the inner sumnested_sum_merge_needs_a_sum_of_sumsmaxgets no mergeupdate_weights_follow_the_measurecardinality_keeps_…,frequency_statistics_keep_universal_choices,topk_keeps_both_specialized_heap_choices,summary_candidates_are_unsized,approximate_count_lists_its_accumulator_and_summaries,a_count_offers_no_count_sketchGuarantee composition is no longer Pass 1's: candidates carry θ, and each build's ε follows from θ (PR 1's
parameter_list); composition to the query root iserror_propagation(PR 4) and constraint (K) (PR 8). The two Count-Min bound tests that lived in Pass 1 but testasap-accuracy(count_min_estimate_has_no_bound_over_an_open_requirement,count_min_estimate_over_raw_data_samples_is_gated_on_the_metric) moved toaccuracy/src/propagation.rs.Ported to the memo (
integration-tests):per_computation_candidates_sql.rs(the fourHAVINGtests, the three heap-update tests, over Pass 1's memo of SQL queries) andstage1_logical_asap_optimization.rs:sql_candidates_output_the_query_columnsandpromql_candidates_output_the_query_columns(were*_candidates_compose: every query of the old composition test goes through Pass 1 with every candidate's output checked byadd_candidate),topk_offers_both_heap_families(wastopk_estimates_carry_no_frequency_bound; the guarantee half moves to PR 8),topk_over_an_aggregate_ranks_its_labels,subquery_in_a_measure_filter_is_an_input_group_of_the_state(was…_is_substituted),filtered_sql_avg_is_null_without_matching_rows,promql_avg_divides_sum_and_count_vectors,sql_avg_keeps_its_explicit_output_name,sql_count_star_without_a_non_null_column_is_only_exact(was…_is_unsupported),sql_count_of_a_nullable_column_counts_its_values.Deleted:
devtools/src/candidate_labels.rsconnected_labels_describe_selected_statesandconnected_labels_name_actual_shared_state(they labelled composed node trees; tools are PR 9).Test status
cargo test -p <crate> --all-features --no-fail-fast, per crate, passed / failed / ignored. PR 2 = #1230 at 4915a82; PR 3 = this PR at 7802d56. No test fails inside Pass 1 any more; planning now stops at Pass 2's firsttodo!()(PR 4: "rule 1, tumbling layouts as candidates of the state group").asap-accuracywith its test)count_min_estimate_over_raw_data_samples_is_gated_on_the_metricstops at PR 4 (local_guaranteeof θ), as it did in Pass 1asap_plannersubprocesses), 8 at PR 9; −2 tests deleted (composed-node labels)materialized_views, as on PR 2)crate_graph(inasap-integration-tests) passes: no stage crate depends on another, in[dependencies]or[dev-dependencies](the executor's dev-dependencies on Stage 0 and Pass 1 are gone).Done means
reader,kept, MILProw,σ/Schedule,EvaluationSchedule,logical_node,child_groups,accuracy_budget_parts,operators_kept_in_definition): no hit in the code; in the body only the names of deleted items (logical_node_trees,accuracy_budget_parts) and SQL/PromQL data rows (row/rows).