Skip to content

refactor: one crate per stage (#1207 PR 0b) - #1229

Draft
zzylol wants to merge 5 commits into
refactor/1207-interfacesfrom
refactor/1207-crates-per-stage
Draft

zzylol wants to merge 5 commits into
refactor/1207-interfacesfrom
refactor/1207-crates-per-stage

Conversation

@zzylol

@zzylol zzylol commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Merge unit. This is PR 0b of #1207's merge unit (PRs 0–9, with 0b). It is stacked on PR 0 (#1226, refactor/1207-interfaces); PR 1 (#1227) is stacked on it. The PRs merge into main together, after PR 9. This PR only moves code between crates: no logic changes. It also deletes the 82 tests that crossed stage crates (decided 2026-11-06, see "Deleted tests"); every other test passes or fails exactly as on PR 0 (see "Verification").

Why

#1054's crate rules say:

  • stage crates never depend on each other; each reads the previous stage's output through asap-types;
  • shared code lives in asap-types, asap-accuracy or asap-cost;
  • each crate exposes only what other crates use;
  • the planner facade calls the stages in order, and the crate-graph test checks the edges.

Before this PR, Stage 0's hash-consing and both passes of Stage 1 were one crate, asap-logical-asap-optimization, so nothing stopped Pass 2 from calling into Pass 1 (only a source-scanning test, pass_layering.rs). #1207 (Order, decided 2026-11-06) makes each stage and pass its own crate before any stage logic is filled in, so PRs 1–8 land in their final crates.

What

Crates

Folder names are the crate names without the asap- prefix, like the existing crates (asap-types in crates/types).

Stage Crate (folder) Content
Stage 0 asap-stage0-memo (crates/stage0-memo) hash-consing into the Memo (memo_of_queries, whose body PR 2 fills in; PR 0 deleted the old cse.rs), from asap-logical-asap-optimization
Stage 1, Pass 1 asap-stage1-pass1-per-computation-candidates (crates/stage1-pass1-per-computation-candidates) initial candidates (pass1_memo), from asap-logical-asap-optimization/src/pass1/
Stage 1, Pass 2 asap-stage1-pass2-reuse-candidates (crates/stage1-pass2-reuse-candidates) reuse rules and logical edges (generate_logical_asap_memo_candidates), from asap-logical-asap-optimization/src/pass2/
Stage 2 asap-stage2-physical-optimization (crates/stage2-physical-optimization) renamed from asap-physical-asap-optimization; content unchanged
Stage 3 asap-stage3-plan-selection (crates/stage3-plan-selection) renamed from asap-plan-selection; content unchanged

asap-logical-asap-optimization is deleted.

File → crate map

Every file of the deleted crate, and every file of the renamed crates:

Before After
logical-asap-optimization/Cargo.toml split into stage0-memo/Cargo.toml, stage1-pass1-per-computation-candidates/Cargo.toml, stage1-pass2-reuse-candidates/Cargo.toml (each lists only its own dependencies; asap_sketchlib, serde and rmp-serde of the old crate were unused)
logical-asap-optimization/src/lib.rs deleted; each new crate root re-exports its own entry points, and its crate docs moved to them (the accuracy paragraph to Pass 1)
logical-asap-optimization/src/stage0.rs stage0-memo/src/lib.rs
logical-asap-optimization/src/stage0/cse.rs deleted in PR 0 (Stage 0 builds groups through Memo::find_or_add_group, PR 2)
logical-asap-optimization/src/pass1.rs stage1-pass1-per-computation-candidates/src/lib.rs
logical-asap-optimization/src/pass1/logical_memo_candidates.rs stage1-pass1-per-computation-candidates/src/logical_memo_candidates.rs
logical-asap-optimization/src/pass1/per_computation_candidates.rs stage1-pass1-per-computation-candidates/src/per_computation_candidates.rs
logical-asap-optimization/src/pass1/per_computation_candidates_tests.rs stage1-pass1-per-computation-candidates/src/per_computation_candidates_tests.rs
logical-asap-optimization/src/pass1/realization.rs stage1-pass1-per-computation-candidates/src/realization.rs (only Pass 1 calls it)
logical-asap-optimization/src/test_support.rs split, see below
logical-asap-optimization/src/pass2.rs stage1-pass2-reuse-candidates/src/lib.rs
logical-asap-optimization/src/pass2/logical_memo_candidates.rs stage1-pass2-reuse-candidates/src/logical_memo_candidates.rs
logical-asap-optimization/src/pass2/memo.rs stage1-pass2-reuse-candidates/src/memo.rs
logical-asap-optimization/src/pass2/memo/rules.rs stage1-pass2-reuse-candidates/src/memo/rules.rs
logical-asap-optimization/src/pass2/memo/rules/{finer_aggregate,rollup,scan_labels,summary_inputs,topk}.rs stage1-pass2-reuse-candidates/src/memo/rules/{finer_aggregate,rollup,scan_labels,summary_inputs,topk}.rs
logical-asap-optimization/src/pass2/memo/window_candidates.rs stage1-pass2-reuse-candidates/src/memo/window_candidates.rs
logical-asap-optimization/src/deployment_capabilities.rs types/src/deployment_capabilities.rs (shared: Stages 1–3 and the facade read it); summary_supported is now pub, since Pass 1 calls it from another crate
logical-asap-optimization/tests/pass_layering.rs deleted: it checked by scanning source that pass1/ and pass2/ do not import each other; the crates now cannot, and crate_graph.rs checks it
physical-asap-optimization/** (Cargo.toml, src/lib.rs, src/catalog/mod.rs, src/catalog/realize.rs) stage2-physical-optimization/**, same paths
plan-selection/** (Cargo.toml, src/{lib,models,report,select}.rs, src/milp/{mod,model}.rs, tests/cost_inputs_json.rs) stage3-plan-selection/**, same paths

test_support.rs (the old crate's test helpers) is split by who calls each item:

Item Called by After
compose_logical_candidate, generate_per_computation_candidates (re-exports) tests of executor, frontends, devtools, integration-tests asap_stage1_pass1_per_computation_candidates::test_support (feature test-support)
lower_promql only deleted tests (Stage 0's and Pass 1's tests that lowered PromQL) deleted; Pass 1 no longer depends on the PromQL frontend at all
share_common_sub_dags (re-export) — deleted in PR 0 with cse.rs; asap-stage0-memo has no test-support feature
population_over_selector executor tests asap_types::test_support (an IR fixture; no stage involved)
metric_scan, metric_scan_with_keys, scan, scan_from Pass 1's own tests only stage1-pass1-per-computation-candidates/src/test_fixtures.rs (#[cfg(test)], not exported)
cms_summary, weighted, estimate_over only the deleted cross-stage tests deleted
PerComputationCandidates (re-export) nobody outside the crate dropped from the re-exports
hll_classic_failure_statistic, aggregate, agg, without_agg, filter, dedup, time_range, entry, entry_with nobody (dead since PR 0) deleted

Crate graph

Before ([dependencies] edges between workspace crates; frontend helpers, devtools and integration-tests omitted):

graph LR
  planner[asap-planner] --> logical[asap-logical-asap-optimization]
  planner --> physical[asap-physical-asap-optimization]
  planner --> ps[asap-plan-selection]
  planner --> fe[frontends]
  planner --> executor[asap-executor]
  logical --> types[asap-types]
  logical --> accuracy[asap-accuracy]
  logical -. optional, test-support .-> fepromql[asap-frontend-promql]
  physical --> types
  ps --> types
  ps --> accuracy
  ps --> cost[asap-cost]
  accuracy --> types
  cost --> types
  executor --> types
  fe --> types
Loading

After:

graph LR
  planner[asap-planner] --> s0[asap-stage0-memo]
  planner --> p1[asap-stage1-pass1-per-computation-candidates]
  planner --> p2[asap-stage1-pass2-reuse-candidates]
  planner --> s2[asap-stage2-physical-optimization]
  planner --> s3[asap-stage3-plan-selection]
  planner --> fe[frontends]
  planner --> executor[asap-executor]
  s0 --> types[asap-types]
  p1 --> types
  p1 --> accuracy[asap-accuracy]
  p2 --> types
  s2 --> types
  s3 --> types
  s3 --> accuracy
  s3 --> cost[asap-cost]
  accuracy --> types
  cost --> types
  executor --> types
  fe --> types
Loading

No stage crate depends on another, in [dependencies] or in [dev-dependencies], and no stage crate's tests depend on the facade (asap-planner): no test of a stage crate calls another stage crate, directly or through the facade. The stage crates here are Stage 0, Pass 1, Pass 2, Stage 2, Stage 3 and the three query-language frontends (crate_graph.rs gives the frontends the Stage role). The old optional edge Pass 1 → asap-frontend-promql is gone with lower_promql.

crate_graph.rs now also checks [dev-dependencies], with no exceptions. The rule added: a [dev-dependencies] edge from a Stage crate (the five planning-stage crates and the three frontends) to another Stage crate or to the Facade (asap-planner) is forbidden ("a stage crate's tests never call another stage crate, directly or through the facade"). Dev-dependencies on the shared crates (asap-types, asap-accuracy, asap-cost) and the frontend helpers stay allowed; a new forbidden edge fails the test.

MetricsQL → PromQL (test-only). Deleted, not kept: keeping it would have needed an exception or a new role for the frontends in crate_graph.rs. Its only users were external_corpus.rs (via PromQL's tests/support.rs, deleted for calling Pass 1) and lowering_semantics.rs's vector_negation_lowers_as_in_promql (deleted, see "Deleted tests").

Other changes

  • Facade. asap-planner depends on the five stage crates and calls them in order (run.rs: memo_of_queries → pass1_memo → generate_logical_asap_memo_candidates → Stage 2 → select_plan), as before; only the paths changed. asap_planner::{DeploymentCapabilities, SupportedSummary} re-export from asap_types::deployment_capabilities.
  • Crate-graph test (integration-tests/tests/crate_graph.rs). The three new crates get the Stage role; the two renamed crates keep it under their new names. The optional-edge exception (logical → asap-frontend-promql under test-support) is deleted with its only_if_optional field, since no optional edge is left. Dev-dependencies from a stage crate to another stage crate or to the facade are now checked, with no exceptions. Synthetic tests: optional_only_exception_rejects_the_edge_made_non_optional deleted, checker_ignores_dev_dependencies_and_external_crates → checker_ignores_external_crates_and_non_stage_dev_dependencies, checker_rejects_a_dev_dependency_between_stage_crates and checker_rejects_a_frontend_dev_dependency_on_the_facade added.
  • Cargo.toml. Every dependent crate names the new crates. Dead dev-dependencies removed: asap-cost on the logical and physical crates (no test of asap-cost uses them since PR 0 deleted tests/held_memory.rs), asap-planner on the logical crate with test-support (unused); after the test deletions: Pass 1 on asap-stage0-memo and on asap-frontend-promql; Stage 0 on Pass 1; asap-frontend-promql and asap-frontend-sql on asap-stage0-memo and Pass 1; asap-frontend-metricsql on asap-stage0-memo, Pass 1 and asap-frontend-promql. Group 3: asap-frontend-promql and asap-frontend-sql on asap-planner (SQL's had no user), and asap-frontend-promql on asap-accuracy (no user, already on PR 0). The metrics_observability test target is removed from frontend-promql/Cargo.toml (its file is deleted), and docs/develop_docs/metrics-observability-corpora.md says so (its fixtures stay).
  • Docs. Crate names and paths updated in docs/develop_docs/ (architecture, contracts, extension guide, library API), docs/design_docs/architecture/, and the design proposals and decisions that name a file. Historical records are left as written (joint-candidate-workload-results.json, joint-candidate-workload-evaluation.md). The stage-document JSON keys (stage1_logical_asap_optimization, stage2_physical_asap_optimization, stage3_plan_selection) are a document format, not crate names, and are unchanged.

Not changed

  • No function body, type or signature changes, except summary_supported becoming pub (it is now called across crates) and module visibility inside the new crate roots (pub(crate) mod → mod).
  • The JSON stage-document format, the viewer and the snapshots.

How

git mv for every file, so history follows (git log --follow). Inside the moved files only paths change: crate::pass1::X → crate::X, crate::pass2::memo → crate::memo, crate::stage0::memo_of_queries → asap_stage0_memo::memo_of_queries (Pass 1 tests only), crate::deployment_capabilities → asap_types::deployment_capabilities.

Key code interfaces added / modified

Only crate paths change. Each crate's public surface, at [4445d45]:

stage0-memo/src/lib.rs

/// Stage 0: the memo groups of the workload's query roots (no candidates). // was asap_logical_asap_optimization::memo_of_queries
pub fn memo_of_queries(roots: &[QueryRoot]) -> Result<Memo, SchemaDerivationError>;

stage1-pass1-per-computation-candidates/src/lib.rs

/// Pass 1's entry. // was asap_logical_asap_optimization::pass1_memo
pub fn pass1_memo(
    memo: Memo,
    accuracy_targets: &[AccuracyTarget],
    accuracy: &dyn AccuracyModel,
    evidence: &dyn AccuracyEvidenceProvider,
    data: &DataWorkload,
    supported_summaries: Option<&[SupportedSummary]>,
) -> Result<Memo, Pass1Error>;
/// Why `pass1_memo` failed. // was asap_logical_asap_optimization::Pass1Error
pub enum Pass1Error { /* unchanged */ }
/// A query's DAG Pass 1 cannot plan. // was asap_logical_asap_optimization::PerComputationCandidateError
pub enum PerComputationCandidateError { /* unchanged */ }
/// Test-only (feature `test-support`). // was asap_logical_asap_optimization::test_support
pub mod test_support {
    pub use crate::per_computation_candidates::{compose_logical_candidate, generate_per_computation_candidates};
}

stage1-pass2-reuse-candidates/src/lib.rs

// All four were at asap_logical_asap_optimization::*; signatures unchanged.
pub fn generate_logical_asap_memo_candidates(pass1: Memo, entries: &[QueryWorkloadEntry]) -> Result<LogicalASAPMemoCandidates, Pass2RuleError>;
pub fn pass2_local_window_composition(pass1: Memo, entries: &[QueryWorkloadEntry]) -> Result<Memo, Pass2RuleError>;
pub fn pass2_after_local_window_composition(memo: Memo) -> Result<LogicalASAPMemoCandidates, Pass2RuleError>;
pub enum Pass2RuleError { /* unchanged */ }

types/src/deployment_capabilities.rs, types/src/test_support.rs

// asap_types::deployment_capabilities — was asap_logical_asap_optimization::*; bodies unchanged.
pub struct DeploymentCapabilities { /* fields unchanged */ }
pub enum DeploymentCapabilitiesError { /* unchanged */ }
pub use crate::ir::schema::SupportedSummary;
/// Whether `supported_summaries` covers `family` and `readout`. // was pub(crate)
pub fn summary_supported(supported_summaries: Option<&[SupportedSummary]>, family: &SketchAlgorithm, readout: StatisticKind) -> bool;

// asap_types::test_support (feature `test-support`)
/// The maintained-population sub-DAG for executor tests. // was asap_logical_asap_optimization::test_support
pub fn population_over_selector(root: &Rc<OperatorNode>, statistic: PopulationStatistic, grouping: &[&str]) -> Rc<OperatorNode>;

asap-stage2-physical-optimization and asap-stage3-plan-selection: same items as asap-physical-asap-optimization and asap-plan-selection.

Usage. The facade's flow is unchanged; only the paths it calls changed (crates/planner/src/run.rs):

// The workload's lowered query roots, one per query (Stage 0 input from the frontends).
let memo = asap_stage0_memo::memo_of_queries(&roots)?;
// Pass 1: the initial candidates of each group.
let pass1 = asap_stage1_pass1_per_computation_candidates::pass1_memo(memo, &targets, accuracy, evidence, data, supported)?;
// Pass 2: reuse rules and logical edges, which Stage 2 expands.
let logical = asap_stage1_pass2_reuse_candidates::generate_logical_asap_memo_candidates(pass1, &entries)?;
// Stage 2 and Stage 3 as before, under their new crate names.

Planning still reaches PR 0's first todo!() (Stage 0's memo_of_queries, filled in by PR 2), as on PR 0.

Major algorithm changes

None.

New abstractions (pending approval)

None. One module was added: stage1-pass1-per-computation-candidates/src/test_fixtures.rs (#[cfg(test)]), holding the old test_support fixtures that only Pass 1's own tests call, so the crate does not export them.

Design notes for review

  • No stage needs another stage's code today. realization.rs (summary families per intent, summary inputs) was documented as shared by Pass 1 and Pass 2, but only Pass 1 calls it now; it stays in Pass 1. If Pass 2's rules (PR 4) need it, that is a crate-boundary question: it would move to asap-types (IR helpers) rather than Pass 2 depending on Pass 1.

Before this PR

Stage 0 and both Stage 1 passes were modules of asap-logical-asap-optimization; a source-scanning test kept the passes from importing each other. Stage 2 and Stage 3 were asap-physical-asap-optimization and asap-plan-selection. DeploymentCapabilities lived in the Stage 1 crate.

After this PR

One crate per stage and pass; the crate-graph test forbids any edge between them, including test-only edges. DeploymentCapabilities is in asap-types. PRs 1–8 change code in its final crate.

Deleted tests

Decided 2026-11-06: tests that cross stage crates are deleted, not rewritten; PR 3 re-creates Pass 1 coverage against the new interfaces. 82 tests are deleted (63 passed, 18 failed and 1 ignored on PR 0), plus pass_layering.rs's 2. Three groups follow: (1) Stage 0 ↔ Pass 1 (54 tests), (2) frontends → Stage 0 / Pass 1, Pass 1 → PromQL frontend, and MetricsQL → PromQL (27 tests), (3) frontends → the facade (1 test).

(1) Stage 0 ↔ Pass 1

Pass 1's generate_per_computation_candidates takes Stage 0's Memo, so every Pass 1 test that built its input with memo_of_queries (directly or through a helper) goes. Their helpers with no caller left are deleted too (listed after the table).

File Tests PR 0 result What they covered
stage0-memo/src/lib.rs (was stage0.rs), mod tests repeated_results_share_groups_but_quantile_levels_do_not, equal_keyless_deterministic_results_share_one_group, scalar_and_literal_roots_keep_workload_positions (3) 3 passed memo_of_queries over PromQL lowered through Pass 1's lower_promql: equal results share a group, different quantile levels do not; G2 / G2 is one group read twice; scalar and literal roots keep their query positions. cse.rs and its 24 hash-consing tests were deleted in PR 0.
stage1-pass1-per-computation-candidates/src/per_computation_candidates_tests.rs scalar_root_producers_are_discovered_once, example_1_exact_q1_keeps_exact_accumulators_only, asap_input_and_invalid_accuracy_are_rejected, multi_measure_aggregate_has_one_target_per_measure, aggregate_without_realization_is_not_a_target, filtered_measure_keeps_candidates_and_records_its_filter (target discovery and input checks, 6); key_only_having_is_pushed_below_and_keeps_candidates, value_having_stays_above_output_and_keeps_candidates, having_on_one_consumer_of_a_shared_aggregate_stays_above, a_scalar_reference_consumer_keeps_a_key_only_having_above, only_the_adjacent_of_stacked_filters_is_having, global_aggregate_having_stays_above, scalar_reference_filter_is_not_having, promql_scalar_operand_is_not_having, having_shared_with_a_project_consumer_stays_above, without_and_per_entity_having_stays_above, memo_cached_rules_preserve_occurrence_specific_having (HAVING placement, 11); composed_candidate_replaces_chosen_target_with_summary_evaluation, composed_filters_and_having, composed_avg_divides_sum_and_count_states, composed_multi_measure_aggregate_joins_its_measures, composed_mixed_exact_and_sketch_measures_keep_output_names, composed_guarantees, own_having_subquery_is_substituted, per_series_avg_with_series_identity_composes_from_sum_and_count, per_series_rate_keeps_labels_and_retypes_its_state, grouped_avg_composition_preserves_the_whole_schema (composing a chosen candidate, 10) — 27 21 passed, 6 failed (composed_candidate_replaces_chosen_target_with_summary_evaluation, composed_filters_and_having, composed_guarantees, composed_mixed_exact_and_sketch_measures_keep_output_names, composed_multi_measure_aggregate_joins_its_measures, own_having_subquery_is_substituted) Frontend-to-Pass-1 acceptance: which computation targets a query has, where a HAVING goes, and the DAG a chosen candidate composes to. The file's 4 tests of computation_candidates_for_intent stay.
stage1-pass1-per-computation-candidates/src/per_computation_candidates.rs, mod tests whole_expression_topk_replaces_the_heap_over_the_exact_sum, whole_expression_topk_partitions_index_the_heap_input, whole_expression_topk_over_a_grouped_sum_of_range_sums_reads_the_raw_data_samples, heap_weights_over_raw_data_samples_record_an_open_requirement, grouped_range_sums_a_raw_data_samples_heap_cannot_rank_get_no_fused_heap, a_fused_computation_absorbs_what_its_absorbed_targets_merges_absorb, combinations_skip_choices_a_whole_expression_candidate_absorbs, count_ranked_topk_fuses_to_one_global_heap_sketch (fused heaps and whole-expression top-k, 8); nested_sum_merges_into_one_exact_accumulator, nested_sum_merge_needs_a_sum_of_sums, nested_sum_merge_never_groups_by_the_dropped_metric_name, nested_sum_merges_do_not_stack (nested-sum merges, 4); avg_states_out_of_order_are_refused, temporal_average_divides_with_a_finite_division_guard (2); multidimensional_ranked_item_is_a_structured_tuple, subpopulation_columns_are_not_part_of_the_ranked_item_tuple, temporal_per_entity_topk_uses_series_identity_and_sample_value, topk_keeps_known_labels_beside_series_identity (ranked items, 4); a_quantile_estimate_shares_its_raw_data_samples_sign_only_inside_the_unit_interval, a_quantile_of_merged_tumbling_windows_shares_its_raw_data_samples_sign, values_estimating_raw_data_samples_name_their_sign_by_the_metric (raw-data-samples signs, 3); guarantee_composition::normalized_additive_errors_compose_as_absolute_errors (1); stream_norms::a_norm_follows_the_update_weight, stream_norms::the_absolute_bound_names_the_weighted_norm (2) — 24 19 passed, 5 failed (the two quantile-sign tests, normalized_additive_errors_compose_as_absolute_errors, both stream_norms tests) Pass 1's fused and whole-expression candidates, merges, ranked items, and the guarantees and signs of its estimates. Of the module's other 18 tests, 14 are deleted in group (2) for lowering PromQL; 4 stay, with realization.rs's 5.

Helpers deleted with them (no caller left): in per_computation_candidates_tests.rs composed, scan, aggregate_over, aggregate, always, exact_promql, choices_of, assert_sql_count_choices, quantile, grouped_quantile, grouped_count, having_filter, equals, having_of, above_scalar, columns, without_output_aliases, summary_agg; in per_computation_candidates.rs's tests inventory, inventory_root, inventory_at, open_inventory, heap_weight_domains, composed_with, heap_states, topk_target, enumerated_choices, nested_sum_target, fused_heap, heap_update, and the emptied mod stream_norms; in test_fixtures.rs cms_summary, weighted, estimate_over. Dev-dependencies removed: Pass 1 → asap-stage0-memo, Stage 0 → asap-stage1-pass1-per-computation-candidates.

(2) Frontends → Stage 0 / Pass 1, Pass 1 → PromQL frontend, MetricsQL → PromQL

File Tests PR 0 result What they covered
frontend-promql/tests/count_planning.rs grouped_count_offers_hydra_but_selection_keeps_per_group_state, frequency_count_candidates_use_unit_weights, cms_count_updates_total_ten_for_zero_positive_and_negative_samples, logical_candidates_fails_on_a_candidate_that_does_not_compose, aggregate_over_topk_candidates_compose (5) 2 passed (logical_candidates_fails_on_a_candidate_that_does_not_compose, aggregate_over_topk_candidates_compose), 2 failed (frequency_count_candidates_use_unit_weights, cms_count_updates_total_ten_for_zero_positive_and_negative_samples), 1 ignored (grouped_count_offers_hydra_but_selection_keeps_per_group_state) Pass 1 candidates of PromQL count queries: Hydra offered for grouped counts, unit weights for frequency counts, CMS update totals, composition failures surfacing, aggregate-over-topk composing. The file's 4 tests that lower or plan through the facade stay.
frontend-promql/tests/univmon_candidates.rs (file deleted) uncalibrated_frequency_evaluations_do_not_bypass_accuracy_targets (1) failed UnivMon candidates generated by Pass 1 are rejected by selection without calibration.
frontend-promql/tests/observability/promql_corpus.rs binding_is_total_over_the_entire_corpus (1) failed Pass 1 binding (logical_plan) never errors over the Prometheus docs and testdata corpora. lowering_is_total_over_the_entire_corpus stays.
frontend-promql/tests/observability/metrics_observability.rs (file and test target deleted) benchmark_corpora_are_total_and_keep_their_coverage, kubernetes_mixin_aggregates_over_topk_compose (2) 1 failed, 1 passed Coverage ratchets of ASAP candidates per metrics-observability corpus; Kubernetes-mixin aggregates over topk compose through Pass 1.
frontend-sql/tests/frequency_l2.rs stage1_offers_exact_and_summary_l2_candidates (1) passed Pass 1 offers exact and summary candidates for the SQL L2 intent. The file's lowering tests stay.
frontend-metricsql/tests/external_corpus.rs (file deleted) corpus_lowers_and_binds, floor_line (2) 2 passed MetricsQL external corpus lowers and binds through Pass 1 (skipped without the corpus directory); floor_line tested only that test's parsing helpers.
frontend-metricsql/tests/lowering_semantics.rs vector_negation_lowers_as_in_promql (1) passed MetricsQL vector negation lowers to the same IR as PromQL's (the only MetricsQL → PromQL use left).
stage1-pass1-per-computation-candidates/src/per_computation_candidates.rs, mod tests aggregate_replacement_preserves_sql_field_identity, raw_data_samples_of_metric_ends_at_any_other_operator, classic_hll_confidence_needs_one_instance_over_a_sources_own_rows; guarantee_composition::{absolute_errors_compose_through_arithmetic, products_and_quotients_of_estimates_have_closed_or_symbolic_bounds, relative_and_rank_errors_compose_only_where_sound, clamp_and_abs_keep_the_largest_argument_error, comparisons_over_estimates_have_no_guarantee, a_scalar_subquery_inherits_its_guarantee, failure_probability_counts_once_per_distinct_estimate, join_beside_an_exact_input_on_exact_columns_keeps_the_guarantee, join_on_an_approximate_key_has_no_guarantee, case_on_exact_columns_keeps_the_branch_guarantee, projection_over_a_join_reads_each_columns_guarantee} (14) 13 passed, 1 failed (classic_hll_confidence_needs_one_instance_over_a_sources_own_rows) Pass 1's guarantee composition over PromQL-lowered estimates (arithmetic, products and quotients, metrics, clamp and abs, comparisons, scalar subqueries, δ counting, joins, CASE, projections), raw-data-samples requirements, HLL confidence, SQL field identity through aggregate replacement. The emptied mod guarantee_composition (its helpers with_bound, abs, rel, lit, arith, call, of, bound, delta, close, join, column, equals, case_over, project) is deleted.

Helpers deleted with them: lower_promql (Pass 1's test_fixtures.rs); node, node_over, scalar, approximate_and_exact (Pass 1's tests); logical_candidates, local_computation_candidates, LogicalCandidate, require_composed_operator_candidates, logical_plan (PromQL's tests/support.rs); reads_hydra (count_planning.rs); bind_tally, BindTally (promql_corpus.rs). MetricsQL's tests no longer include PromQL's support.rs.

(3) Frontends → the facade (asap-planner)

File Tests PR 0 result What they covered
frontend-promql/tests/count_planning.rs exact_counts_select_count_accumulators (1) failed (Pass 1's todo!()) count(up), count by(job)(up) and count_over_time(up[5m]) at an exact target select a count accumulator when planned through asap_planner::run_planning. The file's 2 fixture tests of lowered count aggregates stay.

Helpers deleted with it: frontend-promql/tests/planner_support.rs (run_all_stages, selected_dag; file deleted). Helpers that had no caller left: physical_asap_dag (its callers were deleted in group 2) and declared_data (no caller on PR 0 either) in PromQL's tests/support.rs. No SQL or MetricsQL test used asap-planner; SQL's dev-dependency on it was unused and is removed.

Verification

  • cargo check --workspace --all-targets, cargo clippy --workspace --all-targets -- -D warnings and cargo fmt --all -- --check: clean.
  • crate_graph.rs: passes with the new crates and edges, including the new dev-dependency checks (6 tests).
  • Rebased 2026-11-06 (three times) onto PR 0's memo changes (Result signatures, private groups, MemoGroup { expression, group_properties }, coverage only in the memo, size_for_all_readers deleted; last, InputOrTemporaryBuild, unlifted_operators: Vec<Operator<Option<MemoGroupId>>> and no node-tree wrapper of merged_window_build, which touch only asap-types and asap-accuracy and applied without conflicts): the move commit carries them to the new paths. One commit is added: the planner's tests now take asap-types' test-support feature directly; it came through asap-logical-asap-optimization's test-support, which the move deleted, so cargo test -p asap-planner did not build (the workspace build hid it by feature unification). cargo check, cargo clippy --workspace --all-targets -- -D warnings and cargo fmt --check are clean, and crate_graph passes (6).
  • Test status at the head (4445d45), cargo test -p <crate> --all-features --no-fail-fast. Every failure panics at a todo!(), except the stale fixture (PR 9):
Crate Passed Failed Ignored Fails at
asap-types 166 2 0 reject the invalid combination of section 2 (PR 1): 2
asap-accuracy 73 0 0 —
asap-cost 31 1 0 check θ, which the exported node carries (PR 6): 1
asap-frontend-common 49 0 0 —
asap-sql-function-catalog 10 0 0 —
metricsql_common 3 0 0 —
asap-frontend-promql 218 0 0 —
asap-frontend-metricsql 15 0 0 —
asap-frontend-sql 174 0 0 —
asap-stage0-memo 0 0 0 —
asap-stage1-pass1-per-computation-candidates 12 1 0 the guarantee of θ from the candidate's LogicalProperties::parameters (PR 4): 1
asap-stage1-pass2-reuse-candidates 0 0 0 —
asap-stage2-physical-optimization 0 0 0 —
asap-stage3-plan-selection 3 0 0 —
asap-executor 210 63 1 check θ, which the exported node carries (PR 6): 32; the Hydra grid's sizes are θ, which the exported node carries (PR 6): 13; the view's coverage from its memo group's GroupProperties::summary_coverage (PR 6): 8; the weighted-frequency θ, which the exported node carries (PR 6): 7; check the weighted-frequency θ, which the exported node carries (PR 6): 2; Stage 0 over Memo::find_or_add_group (PR 2): 1
asap-planner 9 19 1 Stage 0 over Memo::find_or_add_group (PR 2): 19
asap-devtools 38 58 3 Stage 0 over Memo::find_or_add_group (PR 2): 56; an asap_planner subprocess that stops at a todo!(): 2
asap-integration-tests 84 87 4 Stage 0 over Memo::find_or_add_group (PR 2): 83; the view's coverage from its memo group's GroupProperties::summary_coverage (PR 6): 3; the stale fixture (old wire format; PR 9): 1
Total 1095 231 9
  • History. Before the rebases, the move was checked test by test: cargo test --workspace --no-fail-fast on PR 0 (dab3d064f) and on this PR's first commit (80b5b09b8, the move): every test has the same result, compared by name, except pass_layering.rs's 2 deleted tests. On the deletion commits (dfc78f40d, group 1; f6e2daa49, group 2; 4d5537310, group 3) cargo test was re-run for the crates they change (asap-stage0-memo, asap-stage1-pass1-per-computation-candidates, asap-stage1-pass2-reuse-candidates, the three frontends, and crate_graph.rs in asap-integration-tests); each remaining test has its PR 0 result. Every other crate's code is unchanged, and cargo clippy --workspace --all-targets builds all tests. Per crate:
Crate PR 0 (dab3d064f) passed / failed / ignored This PR passed / failed / ignored
asap-types 200 / 2 / 0 201 / 2 / 0
asap-accuracy 77 / 0 / 0 77 / 0 / 0
asap-cost 31 / 1 / 0 31 / 1 / 0
asap-frontend-common 49 / 0 / 0 49 / 0 / 0
asap-sql-function-catalog 10 / 0 / 0 10 / 0 / 0
metricsql_common 3 / 0 / 0 3 / 0 / 0
asap-frontend-promql 221 / 6 / 1 218 / 0 / 0
asap-frontend-metricsql 18 / 0 / 0 15 / 0 / 0
asap-frontend-sql 175 / 0 / 0 174 / 0 / 0
asap-logical-asap-optimization 95 / 13 / 0 —
asap-stage0-memo — 24 / 0 / 0
asap-stage1-pass1-per-computation-candidates — 12 / 1 / 0
asap-stage1-pass2-reuse-candidates — 0 / 0 / 0
asap-stage2-physical-optimization (was asap-physical-asap-optimization) 0 / 0 / 0 0 / 0 / 0
asap-stage3-plan-selection (was asap-plan-selection) 3 / 0 / 0 3 / 0 / 0
asap-executor 217 / 56 / 1 217 / 56 / 1
asap-planner 9 / 19 / 1 9 / 19 / 1
asap-devtools 49 / 48 / 3 49 / 48 / 3
asap-integration-tests 102 / 76 / 4 103 / 76 / 4
Total 1259 / 221 / 10 1195 / 203 / 9

(Measured on PR 0 as it was then, dab3d064f; the head's numbers are in the table above.)

The old crate's 95 / 13 / 0: 24 / 0 / 0 stay in Stage 0 and 12 / 1 / 0 in Pass 1; 1 / 0 / 0 moved to asap-types (deployment_capabilities.rs's default_supports_everything_and_json_fills_the_rest: 200 → 201); 56 / 12 / 0 deleted as cross-stage tests (groups 1 and 2); 2 / 0 / 0 deleted with pass_layering.rs. The frontends lost 7 / 5 / 1 (group 2) and 0 / 1 / 0 (group 3). In asap-integration-tests, crate_graph.rs has 6 tests (one deleted, two added). Every other test has the same result as on PR 0. The remaining failures are PR 0's todo!() panics.

@zzylol zzylol added optimizer Workload optimization and plan selection refactor labels Nov 6, 2026
@zzylol
zzylol force-pushed the refactor/1207-crates-per-stage branch 2 times, most recently from ae4843e to 8d020b0 Compare November 6, 2026 19:10
@zzylol
zzylol force-pushed the refactor/1207-interfaces branch from 200ec5c to 341c98b Compare November 6, 2026 21:28
@zzylol
zzylol force-pushed the refactor/1207-crates-per-stage branch from 8d020b0 to d1feaad Compare November 6, 2026 21:29
@zzylol
zzylol force-pushed the refactor/1207-interfaces branch from 341c98b to 3a75e1e Compare November 6, 2026 21:37
@zzylol
zzylol force-pushed the refactor/1207-crates-per-stage branch 2 times, most recently from 9a3bb19 to 345b5df Compare November 6, 2026 21:57
Move-only. Split asap-logical-asap-optimization into asap-stage0-memo,
asap-stage1-pass1-per-computation-candidates and
asap-stage1-pass2-reuse-candidates; rename asap-physical-asap-optimization
to asap-stage2-physical-optimization and asap-plan-selection to
asap-stage3-plan-selection. DeploymentCapabilities moves to asap-types,
and the old test_support splits by caller. The crate-graph test checks
the new crates; no stage crate depends on another.
Pass 1's tests that build Stage 0's memo and Stage 0's tests that lower
PromQL through Pass 1 are deleted, with the dev-dependencies and helpers
only they used. lower_promql moves to Pass 1's test fixtures, so Pass 1
no longer depends on the PromQL frontend outside its own tests. The
crate-graph test now also checks dev-dependencies between stage crates.
…1207 PR 0b)

Frontend tests that compose Pass 1 candidates and Pass 1 tests that
lower PromQL through the frontend are deleted, with the helpers,
dev-dependencies and test targets only they used. MetricsQL's test that
compares with PromQL's lowering goes too, so no frontend tests depend on
another frontend. The crate-graph test checks dev-dependencies between
stage crates with no exceptions.
…R 0b)

The PromQL test that planned through asap-planner is deleted with its
planner_support helpers; the PromQL and SQL frontends drop their
asap-planner dev-dependency, and PromQL its unused asap-accuracy one.
The crate-graph test now also forbids a stage crate's dev-dependency on
the facade.
… PR 0b)

The feature came through asap-logical-asap-optimization's test-support,
which the move deleted, so cargo test -p asap-planner did not build.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

optimizer Workload optimization and plan selection refactor

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant