Repository navigation
Conversation
…ailure probability at the source's distinct bound (#1207 PR 8) distinct_count_guarantee_over_source reads classic HLL's and a distinct-count UnivMon's guarantees (kept dead since PR 4 for this) and binds their failure probability at SourceFacts::max_distinct_per_evaluation, for Stage 3's (K).
CandidateErrorModel (milp/error_bounds.rs): each physical candidate's own guarantee (a sketch state's: the largest error per metric and failure probability of the statistics read from its group; none for a read whose bound needs values not proven non-negative, a windowed or merged state that does not keep the guarantee, or a family without an error model), its error contract per metric (error_propagation), and ceilings of E and D used as M. AccuracyVariables E/F per execution and metric, D/H, U (no guarantee); accuracy_constraints: E >= eps_loc + sum beta F, F >= E_u - M(1-y), the metric rule and validity caps on selected edges, D by a union bound, and the root checks E_v <= eps_q, D_v <= delta_q / n, U_v = 0 when z = 1, for every query the accuracy model does not accept regardless of its guarantee. After the solve, each query's composed guarantee is checked with the model and evidence; an output none accepts is SelectedPlanFailedCheck. AccuracyCheckInputs carries the front ends' query roots (G_q and the accept-regardless policy). Tests: #1207's problem example, the DDSketch sign condition, failure probabilities adding, big-M only on selected edges, and the reference enumeration under (K).
…ee per metric; an exact target's limits are exact (#1207 PR 8)
…o some query (#1207 PR 8)
…ignore mode; a whole-result failure probability over 1,000 series (#1207 PR 8)
…its edge; (K) applies unless the model or evidence may accept outputs regardless; one G per query for (K) and the check; no ceiling cap; tighter HiGHS tolerances under (K) (#1207 PR 8) - The error of reading a state is a constant per edge (u, c) for the consumer's statistic(s) on the producer's theta (E_c >= sum eps_uc y + ...), so one query's read no longer changes another's (a DDSketch read at the maximum keeps its guarantee when a p50 read of the same state has none). - root_accuracy_limits follows the AccuracyModel/AccuracyEvidenceProvider contract (may_accept_regardless_of_guarantee, may_provide_evidence): (K) applies to every query or none; the per-query pre-ASAP accept-regardless call is gone. - values_per_result is computed once from the lowered roots and used by both (K) and the check after the solve; a short query_roots slice is an EntryCount error. - Ceilings stop after one round per execution (a valid M for every acyclic selection) with no cap, which could make unrelated plans infeasible. - OwnGuarantee renamed ErrorBounds (read errors and composed errors). - Tests: the regression test above; the DDSketch sign test checks the declared case too; the big-M test's description matches it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #1237 (PR 7, execution contexts). Part of #1207 (PR 8 of the order), #1204 item 3, #1203. Draft; not to be merged on its own (one merge unit, PRs 0–9).
Why
Until this PR nothing checked a query's composed accuracy: Stage 3 selected the cheapest sizes the edges' parameter floors allowed, and a floor is only a pruning cap (Pass 2 takes the largest cap over the queries, and a target whose metric the family does not state leaves the input unbounded). PR 6 showed the result (handoff H1):$E_c$ , $D_c$ composed over the selected edges with big-M, checked at each query root.
quantile_over_time(0.99, latency_ms[60m])with a rank target of 0.01 selected six DDSketch windows atalpha0.1, a relative error of 10 % that says nothing about rank. The concepts doc §4 and #1204 item 3 define constraint (K): per-candidate error boundsWhat
asap_accuracy::estimators::local_guarantee), a constant per edgeErrorSelectingPlan::SelectedPlanFailedCheck(the PR 0 variant, now constructed).may_accept_regardless_of_guarantee,may_provide_evidence; e.g.IgnoreAccuracyGuarantees): the traits' contract is that plan selection then keeps every candidate, and the check after the solve decides. A whole-result failure probability is divided overAccuracyCheckInputs::query_roots, new).todo!("#1207 PR 8 …")is implemented (accuracy_constraints); thedead_codeallowances for PR 8 are gone (AccuracyVariables,accuracy_constraints, HLLclassic_guarantee, UnivMondistinct_guarantee,incomplete_top_layer_statistic).Not here: tools and snapshots (PR 9); latency (L) and resource budgets (B) are out of scope (#1204 items 5, 6).
Representation, rules, enumeration (#1207 (a)/(b)/(c))
CandidateErrorModel::composed_errors) and agrees with the MILP on Tracking: implement the optimization formulation (memo candidates, properties, edges y_{u,c,i}, MILP) as named code per stage #1207's problem example under 12 cost/target sets (targets that admit a plan and targets that admit none).Notation → code
CandidateErrorModel::read_error_on_edge(u, c):ErrorBounds::{error_bound, failure_probability, without_guarantee}per metric, fromread_bounds/read_guaranteeCandidateErrorModel::{factor, validity_cap}(fromLogicalRequirements::error_propagation, per metric)CandidateErrorModel::{error_bound_ceiling, failure_probability_ceiling}(the producer's / output's ceiling)AccuracyVariables::{error_bound, error_bound_of_input, failure_probability, failure_probability_of_input}(keys:Execution, input, metric index)accuracy_constraints(model.rs)RootAccuracyLimits(root_accuracy_limits, per query;Nonewhen (K) does not apply)values_per_result_per_queryoverAccuracyCheckInputs::query_rootsAccuracyVariables::without_guaranteeaccuracy_constraints:RootAccuracyLimits::error_bound_limit[m] = 0for a metric the target does not acceptHow
select_planbuildsCandidateErrorModel::new(physical, data, executions): per physical candidate its error contracts in the five metrics; per admissible edge on which a consumer reads a state a build builds, the read error of the consumer's statistic(s) on the producer's θ; and ceilings (rounds ofroot_accuracy_limits(entries, G, model, evidence): per querymetric_can_meetof the built-in rule),max_values_covered(scope, G_q); allNonewhen the model or evidence may accept outputs regardless. A scalar root that combines outputs (notscalar(v)) requires exact references.solve_selection: when (K) applies,accuracy_variablesandaccuracy_constraintsadd it per execution andaccepted_accuracycomposes each query's error over the selected DAG (composed_errors, no big-M), states it over the target's scope with the sameKey code interfaces added / modified
models.rs, pass.rs
estimators/mod.rs
report.rs
error_bounds.rs, model.rs (crate-private, pending approval)
Usage (end to end;
crates/planner/tests/accuracy_at_query_roots.rsat 55a1384):Major algorithm changes
Stage 3: constraint (K)
quantile_over_time(0.99, latency_ms[60m]), rank 0.01, selected DDSketchalpha0.1 windows (relative error, not rank); the ratio p99/p50 at relative 0.03 could takealpha0.02 for both operands (each within its floor, composedthe_problem_example_meets_each_query_target_at_its_root,a_ddsketch_interior_quantile_needs_non_negative_values,a_read_without_a_guarantee_does_not_affect_another_read_of_the_same_state,failure_probabilities_add_along_the_selected_edges,a_producer_s_error_counts_only_on_its_selected_edge,the_milp_agrees_with_the_reference_enumeration_under_accuracy, end to endeach_query_meets_its_target_at_its_root.metric_can_meet, no cross-metric conversion). A ceiling grows with the rounds on a cycle that carries error (bounded by the round count; none in the tests). Latency (L) and budgets (B) out of scope.Deleted (replaced in this PR)
todo!("#1207 PR 8: constraint (K)")and thedead_codeallowances onAccuracyVariables,accuracy_constraints#[allow(dead_code)]on HLLclassic_guarantee, UnivMondistinct_guarantee,incomplete_top_layer_statisticdistinct_count_guarantee_over_sourceaccepted_accuracythat accepted every sketch output in its target's metric without a checkaccepted_accuracyover the composed guarantee (model, evidence, accept-regardless),composed_guaranteesAccuracyVariableskeyed byPhysicalCandidateId(Vec)HashMap)accuracy_constraints(physical, selection, accuracy)accuracy_constraints(inputs, selection, variables, accuracy)ErrorSelectingPlan::SelectedPlanFailedCheck(PR 0) is now constructed; its doc no longer mentions budget shares.New abstractions not in #1207 or the doc (pending approval)
AccuracyCheckInputs::query_roots: &[QueryRoot](pub;PlanningModels::accuracy_check_inputsnow takes the roots): Stage 3 readsvalues_per_result); a length other than the entries' isErrorSelectingPlan::EntryCount.asap_accuracy::estimators::distinct_count_guarantee_over_source(parameters, source, max_distinct)(pub): classic HLL / distinct-count UnivMon guarantee with δ bound at the source's declared distinct bound.milp/error_bounds.rs):CandidateErrorModel(contracts per metric,read_error_on_edge, ceilings,composed_errors),ErrorBounds(+EXACT),RootAccuracyLimits(+EXACT),root_accuracy_limits,values_per_result_per_query,returns_its_output_unchanged,METRICS/METRIC_COUNT; private helpersread_bounds,statistics_read,read_guarantee,proven_non_negative,raw_data_samples_of_metric,source_distinct_bound.AccuracyVariablesbeyond the doc'swithout_guarantee, part of the output without a guarantee), plus per-execution keys (doc §4:accuracy_variables(creates the family; PR 0 had only the struct andaccuracy_constraints).may_…contract); a scalar root other thanscalar(v)/ a scalar subquery requires exact references; unknown δ counts as 1; read errors sit on edges (spec question 2).Decisions taken conservatively (please confirm)
IgnoreAccuracyGuarantees(PR 4 question). Ignore mode (and any model or evidence provider whosemay_accept_regardless_of_guarantee/may_provide_evidenceis true, including a custom model that keeps the trait defaults) turns off only (K) for every query; the check after the solve then decides, and an output nothing accepts isSelectedPlanFailedCheck(no re-solve). Pass 2'saccepts_output_regardless_of_guaranteeneeds the realized output, so it is asked only after the solve; before it, only the model-widemay_…methods decide.values_non_negative); otherwise no guarantee (Spec questions
with_failure_probability). Options: (a) add δ variants to the lists (KLLfailure_probability, CMS depth); (b) keep as is (such queries are exact or infeasible). 4 planner tests now fail for this (below); I did not change their workloads.Concat) can carry a rank error from one producer and a relative error from another, somay_…defaults aretrue): a deployment that supplies evidence for one query loses (K) for all, and the post-solve check may then reject the cheapest plan with no re-solve. Keep, or should (K) stay on and evidence only be read after the solve?k1404 and Tracking: implement the optimization formulation (memo candidates, properties, edges y_{u,c,i}, MILP) as named code per stage #1207's example DDSketchalpha0.002 wherek269 /alpha0.01 meet the targets at the same objective.Lower-PR items (not changed)
RejectionCause::CountMinWeightsNotProvenNonNegativeis never constructed: PR 6 deleted the executability check (nodes_with_proven_non_negative_weights); a Count-Min over unproven weights is now only "no guarantee" under (K), and in ignore mode it can be selected and the executor refuses it (PR 5/6).RejectionCause::{LatencyBoundExceeded, LatencyOverRepeatInterval, OverClusterCapacity}never constructed ((L), (B) out of scope; PR 6).raw_data_samples_of_metricis duplicated: Pass 1's (realization.rs) is crate-private; Stage 3 needs the same walk and cannot depend on Pass 1. Moving it toasap-typeschanges PR 3.docs/design_docs/proposals/planner-layering.mdlines ~277, 313, 372–421 (memo concepts table and the ratio-of-quantiles example) still describeroot_node,input_groups,accuracy_budget_partsand pre-Tracking: implement the optimization formulation (memo candidates, properties, edges y_{u,c,i}, MILP) as named code per stage #1207 counts (PR 0/PR 3 doc; numbers come from PR 9's re-baselinedstage_pipeline).crates/devtools/src/bin/asap_planner.rs:90comment namesaccuracy_budget_parts(PR 9 tool).local_guaranteegives a Count-MinTopKread a frequency bound (old code had none for top-k membership); unchanged.Before this PR
No query's composed accuracy was checked: sizes were bounded only by the per-edge parameter floors (pruning caps), a target whose metric a family does not state left its inputs unbounded, and every sketch output was reported as accepted in its target's metric.
p99 over 60mat rank 0.01 selected DDSketchalpha0.1 windows; #1207's ratio could compose to 0.044 against a 0.03 target.After this PR
(K) holds for every selected plan unless the deployment's model or evidence may accept outputs regardless; the composed guarantee is rechecked after the solve. Representative plans (
asap-planner --output textstops at PR 9todo!()s, so these come from the sameplan_installablecall through a scratch test; cost model 1 defaults;fixed_interval_at10 min; 1,000 series every 1 s,values_non_negative; "without (K)" is the same build withignore_accuracy_guarantees, which selects exactly as PR 7):alpha0.1 windows, 3.352/salpha0.005, 3.844e-2/sk269 windows, 7.825/s; MILP 96 var / 136 constr, 21 msalpha0.1, 3.352/s; 78 / 110, 24 msk2863), 7.235e-3/s; 488 / 617, 64 msalpha0.005, 6.249e-3/s; 452 / 565, 182 msk2863); B: one DDSketch (alpha0.001) read by both operands; 1.265e-3/s; 498 / 633, 140 msalpha0.005 for A and B, 8.403e-4/s; 462 / 579, 235 msSizes above the floor tie in the objective (memory held during an evaluation is not in it, PR 7 spec question 2), so
k2863 /alpha0.001 are as good ask269 /alpha0.01 to the MILP;plan_costdiffers.Verification
cargo fmt --all;cargo clippy --workspace --all-features --all-targets -- -D warningsclean;crate_graphpasses.the_problem_example_meets_each_query_target_at_its_root(Tracking: implement the optimization formulation (memo candidates, properties, edges y_{u,c,i}, MILP) as named code per stage #1207 Problem: A via KLL, B via one shared DDSketch atalpha0.01 where 0.02 composes to 0.044, metric rule at the root; without (K) the weakest sizes),a_ddsketch_interior_quantile_needs_non_negative_values(infeasible undeclared, feasible declared),failure_probabilities_add_along_the_selected_edges(δ sums, a producer reached twice counted twice: 1.5δ infeasible, 2δ met),a_producer_s_error_counts_only_on_its_selected_edge(big-M only on selected edges),a_read_without_a_guarantee_does_not_affect_another_read_of_the_same_state(regression for review finding 1),the_milp_agrees_with_the_reference_enumeration_under_accuracy(reference over DAGs filtered by composed errors; 12 cost/target sets incl. infeasible ones). stage3 (error_bounds):root_limits_accept_the_target_s_metric_and_divide_delta_over_its_scope,a_read_s_error_is_its_statistic_s_on_the_producer_s_parameters. accuracy:a_distinct_count_over_a_source_binds_its_failure_probability. planner (end to end):each_query_meets_its_target_at_its_root,ignoring_accuracy_guarantees_drops_k,a_whole_result_failure_probability_is_divided_over_the_values.may_…contract (per-query exemptions left per-edge rows on exempt paths, and evidence could never apply); onequery_rootsis an error; no ceiling cap (it could make unrelated plans infeasible); ceilings bounded by executions, not candidates; HiGHS feasibility tolerancesOwnGuarantee→ErrorBounds; stronger tests. Not taken: ceilings over the SCC condensation (no error-carrying cycle in any tested workload: instrumented run, all settle within 8 rounds); a per-(merge, statistic) variable family (merges take the largest over their group's statistics; rollup merges are offered only for reads whose merged guarantee is kept); validity-cap and scalar-root tests (no listed size reaches the capTest table (
cargo test -p <crate> --all-features --no-fail-fast, at 55a1384; PR 7: 1341/94/9):Compared with PR 7: fixed
joint_search_equivalence::custom_accuracy_policy_is_respected(the policy hook:satisfiesis called on the composed guarantee) andproduction_rollup_workloads::production_sql_distinct_region_service_rollup. New failures, all stopping at spec question 1 (whole-result δ 0.01 over 1,000 series, lists fail 0.01 per value; workloads unchanged): plannercolumnar_batches::a_tumbling_window_is_built_from_arrow_columns_with_one_label_entry_per_series,runtime_install::{the_plan_carries_stage3s_predicted_usage, the_plan_file_carries_theta_and_coverage_and_reads_back, tumbling_windows_of_an_hour_share_one_definition_with_distinct_times}; andsharing_issue_regressions::grouped_filtered_quantile_plans(δ 0.001 targets → exact plans; the exact plans tie in the objective, 0.196/s, andplan_costof the one picked is 0.508/s vs 0.216/s: PR 7's inexact objective; it failed in PR 6 and passed in PR 7 by the same tie). Other failures stop where PR 7's did: devtools 37 at PR 9todo!()s, 13 fuzz inconclusive (witness with ASAP operators through Pass 1, PR 9), 2 CLI JSON (PR 9); planner 5 as PR 7 (Pass 2 metric rule / raw data, PR 4; Stage 2 executor-support gap, PR 5;ignoring_accuracy_guarantees_plans_the_undeclared_p99_queryis infeasible with (K) off, Stage 2/PR 5); integration 35 as PR 7 (Stage 2 infeasible outputs, PR 5/Pass 1; pinned plan shapes and θ-less snapshots, PR 9;joint_search_equivalence::rootless_request_overhead_respects_capacity_and_latencyneeds latency (L), out of scope;different_windows_and_raw_retention_policies_produce_valid_plansStage 2 infeasible with raw data off, PR 5; avg-over-quantile /sumcontracts stop at the PR 4 decision). No failure stops at a PR 8todo!(); none isSelectedPlanFailedCheck.