Repository navigation
Description
Example
Query A: quantile_over_time(0.95, latency[1m]) with rank error 0.01. Query B: quantile_over_time(0.99, latency[1m]) / quantile_over_time(0.50, latency[1m]) with relative error 0.01. A reads a KLL quantile state over latency[1m]. B's two operands read a DDSketch quantile state over the same input, because KLL bounds rank error, which does not bound a ratio's relative error (metric rule below).
-
Formulation:
- one state group holds candidates
$v = (e_v, p_v)$ , e.g.$e = \texttt{SummaryAgg(KLL)}(G_1)$ with$\theta$ from the predefined list,$\theta \in {k = 200, k = 345, \dots}$ ; - each
$\theta$ has a precomputed error bound$\varepsilon_v$ and failure probability$\delta_v$ ; - accuracy is not a per-input requirement: each query's composed error is bounded at its root (constraint (K), Stage 3 MILP: implement the formulation variables x, y, z, t and rows (A)–(I) as named code over explicit edges #1204, decided 2026-11-04);
- A meets its target with
k=200ork=345; B needs a DDSketchalphasmall enough for both operands; - so B's two operands can read one DDSketch build, and A reads a KLL build.
- one state group holds candidates
-
Code today:
kis a field inside the candidate'sroot_node(theSummaryAgg's family parameters), and each candidate is sized for a fixed share of its query's budget (accuracy_budget_parts). Each consumer is tied to one sized build through its own private node chain, so two queries can only share a build through the extraSharedBuildand per-size state groups that Pass 2 adds.
Representation, rules, enumeration
-
(a) Representation.
- A candidate is one operator over child groups plus
$\theta$ , and each input carries a requirement$r^{\log}$ (a parameter floor$\boldsymbol\theta^{\min}$ ). - Every intermediate result is its own group, so there are no private nodes.
- A mergeable state is one group, and its sizes are candidates of that group.
- A candidate is one operator over child groups plus
-
(b) Rules.
- Pass 1 offers the
$\theta$ list as candidates of each state group. - Pass 2 rules match over groups, not over private operator chains. They add
$(e, \theta)$ candidates to groups and never copy consumers. - The
SharedBuildandSummaryResultrules are deleted: hash-consing already gives the shared group.
- Pass 1 offers the
-
(c) Enumeration.
- The reference enumerator chooses one producer per consumer input, not one candidate per group.
- So two consumers of one state group may pick different sizes;
$\alpha^{\log}$ limits which ones they may pick. - In the MILP this is
$y$ (Producer–consumer edges (u, c, i): make the edge set explicit after Pass 2 and Stage 2, with α^log and α^phys, as the domain of y_{u,c,i} #1201, Stage 3 MILP: implement the formulation variables x, y, z, t and rows (A)–(I) as named code over explicit edges #1204).
Formulation
A memo group
- $\operatorname{op}v$ includes its operator arguments; $G{v,i}$ is its
$i$ -th input group. -
$\boldsymbol\theta_v$ holds the summary algorithm parameters, e.g. KLLk, Count-Min width × depth. They are taken from a predefined list per summary family; they are not solved for. -
Error bounds. Each
$\theta_v$ has a precomputed error bound$\varepsilon_v$ and failure probability$\delta_v$ , constants. Stage 3 bounds each query's composed error with them (constraint (K), Stage 3 MILP: implement the formulation variables x, y, z, t and rows (A)–(I) as named code over explicit edges #1204). So the accuracy budget is not part of$r^{\log}$ . -
Logical requirement (decided 2026-11-05). A field is kept only when it is a real choice: anything the expression,
$G_{c,i}$ 's identity or the consumer's own properties already fix is removed. So$r_{c,i}^{\log}$ =LogicalRequirements { parameter_floor_for_required_accuracy }and nothing else: $\boldsymbol\theta^{\min}{c,i}$ is the smallest $\theta$ in the family's predefined list whose $\varepsilon$, for the statistic the consumer reads, is at most $\text{cap}{c,i}$. The family and the query window are fixed by$G_{c,i}$ 's identity; "width divides step and phase" is the physical schedule rule; the statistic is the consumer's own operator argument, used once to compute the floor. Then
$$\alpha^{\log}{u,c,i} = 1 \iff \boldsymbol\theta_u \succeq \boldsymbol\theta^{\min}{c,i},$$
compared per family on the parameters that set$\varepsilon$ . With no floor, or an exact producer, it always holds. It is 0 whenever the producer's error metric cannot be converted to the consumer's (metric rule below). -
No accuracy budget in
$r^{\log}$ (decided 2026-11-04). A per-input error bound would fix each input's share of the query's budget: the rejected "budget shares" option. It cannot express uneven splits, or an exact operand leaving the whole budget to the other. Constraint (K) bounds each query's composed error instead (Stage 3 MILP: implement the formulation variables x, y, z, t and rows (A)–(I) as named code over explicit edges #1204). -
One accuracy-related requirement stays: the validity cap, folded into the floor
$\boldsymbol\theta^{\min}$ .- Constraint (K) adds a non-linear consumer's (ratio, product) input errors through a linear upper bound. That bound holds only while each input's
$\varepsilon$ is at most a cap. - The cap depends only on the consumer and is fixed before optimization. It says where the linearization is valid, not how the budget is split, so it is a per-edge requirement (stored as the floor):
$$\alpha^{\log}{u,c,i} = 0 \text{ when } \varepsilon_u > \text{cap}{c,i}.$$
- Constraint (K) adds a non-linear consumer's (ratio, product) input errors through a linear upper bound. That bound holds only while each input's
-
Error propagation and the error cap (decided 2026-11-05). Each consumer
$c$ has an error propagation function$g_c$ with$\varepsilon_c \le g_c(\varepsilon_{c,1}, \dots, \varepsilon_{c,n})$ ;$g_c = \sum_i \beta_{c,i}\varepsilon_{c,i}$ with constant local factors$\beta_{c,i}$ read from the consumer's operator and arguments (e.g.sum byover$n$ series:$n$ ).$\beta_{c,i}$ is a local factor of one consumer;$\lambda$ is only for factors composed to a query root.$\lambda_{q,c,i}$ multiplies the$\beta$ factors along$q$ 's expression from input$i$ of$c$ to$q$ 's root (summed over paths), so it comes from the query expression itself, and it is used only for the pruning caps. The per-input cap (not stored; it sets the floor $\boldsymbol\theta^{\min}{c,i}$) is
$$\text{cap}{c,i} = \min\Bigl(\text{validity cap of } g_c,\ \max_{q \text{ that may reach } c} \frac{\varepsilon_q}{\lambda_{q,c,i}}\Bigr), \qquad \alpha^{\log}{u,c,i} = 0 \text{ when } \varepsilon_u > \text{cap}{c,i}.$$-
Validity cap (correctness, non-linear
$g_c$ only): the region where the linear form bounds$g_c$ , e.g.$\varepsilon_Y \le$ cap for$X / Y$ , giving$\beta = 1/(1-\text{cap})$ for both inputs. A linear$g_c$ has none. A ratio caps only its denominator. - Pruning cap (efficiency only, every consumer): an input whose error alone breaks every target it could serve is in no feasible plan, so dropping its edges removes no feasible plan.
- The validity cap is set when the consumer is created; the pruning cap is filled in when Pass 2 builds the logical edges (Producer–consumer edges (u, c, i): make the edge set explicit after Pass 2 and Stage 2, with α^log and α^phys, as the domain of y_{u,c,i} #1201).
- Interface (approved 2026-11-05, revised 2026-11-06):
pub struct ErrorPropagation { pub factor_per_input: Vec<f64>, pub validity_cap_per_input: Vec<Option<f64>> }andLogicalRequirements::error_propagation(consumer: &Operator<MemoGroupId>) -> ErrorPropagation.validity_cap_per_input[i]isNonewhen the bound holds everywhere for input$i$ (a ratio caps only its denominator).
-
Validity cap (correctness, non-linear
-
Metric rule (decided 2026-11-05). Every guarantee states its metric: absolute, relative or rank.
$\beta_{c,i}$ is defined per input metric and output metric. When the consumer's error contract cannot convert the producer's metric, there is no edge ($\alpha^{\log} = 0$ ). Rank error bounds neither value error nor the relative error of a ratio, so a relative-error ratio such as p99/p50 cannot read KLL quantiles unless a declared distribution assumption covers it; it needs, for example, DDSketch operands. -
No
Exactrequirement. It is covered twice:- a query with an exact target has
$\varepsilon_q = 0$ , so constraint (K) forces every input it depends on to$\varepsilon = 0$ ; - a structural need for exact state (e.g. a Finalize reading exact-accumulator state) is the family or the group itself.
- a query with an exact target has
This issue covers
Data structures and interfaces per stage
Types (crates/types/src/stages/candidates.rs):
/// One `Memo` type for Stage 0 and Stage 1 (`DagOfLogicalMemoGroups` is gone).
/// Stage 0 returns it with every group's `candidates` empty; Stage 1 fills them.
pub struct MemoGroup {
pub expression: Operator<MemoGroupId>, // what it computes; each id in it is an input group, never its own
pub group_properties: GroupProperties, // derived once from the input groups' expression and group_properties
pub parents: Vec<MemoGroupId>,
pub queries: Vec<usize>,
pub candidates: Vec<CandidateForMemoGroup>,
}
pub struct GroupProperties {
pub schema: Schema,
pub result_kind: OperatorResultKind,
pub summary_coverage: Option<SummaryCoverage>, // derived from groups, never node trees
pub shareable_when_equal: bool,
}
/// What a summary state computes, with every lifted selection removed.
pub struct SummaryStateDefinition {
/// The first input under the SummaryAgg that is not Filter / range TimeRange /
/// TimeShift without @ / direct-column Project.
pub input_group: MemoGroupId,
/// The operators not lifted, top down from the SummaryAgg, with residual arguments;
/// `None` marks an operator's main input, `Some(g)` another input group it reads
/// (a scalar subquery's). Two states may merge only if both fields are equal.
pub unlifted_operators: Vec<Operator<Option<MemoGroupId>>>,
}
/// Coverage exists only in the memo (no type parameter); the executor reads `definition_id`.
pub struct SummaryCoverage { pub(crate) definition: SummaryStateDefinition, pub(crate) selection: Vec<SelectionBox> }
/// One set of rules: the existing derivation rules read inputs through this trait.
pub(crate) trait OperatorInput: Sized {
fn schema(&self) -> &Schema;
fn result_kind(&self) -> OperatorResultKind;
fn operator(&self) -> Operator<Self>; // the input's own operator, its children as inputs again
fn summary_coverage(&self) -> Option<&SummaryCoverage>;
}
// impl for Rc<OperatorNode> (front-end trees; summary_coverage None) and for
// MemoGroupAsInput<'m> { memo: &'m Memo, group_id: MemoGroupId } (reads group_properties / expression).
// find_or_add_group and add_candidate derive GroupProperties with
// derive_group_properties(memo, &expression) -> Result<GroupProperties, SchemaDerivationError>.
/// Groups are private: added only through `find_or_add_group`, read through `groups()`.
pub struct Memo { groups: Vec<MemoGroup>, pub query_roots: Vec<QueryRoot<MemoGroupId>>, groups_by_expression_hash: HashMap<u64, Vec<MemoGroupId>> }
pub struct CandidateForMemoGroupId { pub group: MemoGroupId, pub index_in_group: usize }
impl Memo {
pub fn groups(&self) -> &[MemoGroup];
/// Hash-consing on the expression (operator, arguments, input groups): the group
/// with this expression, added if new. Fails when the expression breaks the node
/// rules (schema derivation, input validation, merge check).
pub fn find_or_add_group(&mut self, expression: Operator<MemoGroupId>) -> Result<MemoGroupId, SchemaDerivationError>;
/// Checked by the same node rules; its output must equal the group's. A
/// duplicate (same expression and logical properties) returns the existing id.
pub fn add_candidate(&mut self, group: MemoGroupId, candidate: CandidateForMemoGroup) -> Result<CandidateForMemoGroupId, CandidateError>;
/// Adds a query with this root; its groups and those below list it.
pub fn add_query(&mut self, root: QueryRoot<MemoGroupId>) -> usize;
pub fn candidate(&self, id: CandidateForMemoGroupId) -> &CandidateForMemoGroup;
}
/// A candidate `add_candidate` rejects; the memo is unchanged.
pub enum CandidateError {
/// The candidate's expression fails the node rules (schema derivation, input validation, merge check).
Schema(SchemaDerivationError),
/// Its output schema or output kind differs from the group's: it does not compute the group's result.
OutputDiffersFromGroup { group: MemoGroupId },
}
/// v = (e_v, p_v^log): one candidate of a memo group.
pub struct CandidateForMemoGroup {
pub expression: CandidateExpression, // e_v
pub logical_properties: LogicalProperties, // θ_v
/// r^log_{c,i}, one per child group of `expression.operator`.
pub input_requirements: Vec<LogicalRequirements>,
pub origin: OriginOfCandidateForMemoGroup,
pub fused_groups: Vec<MemoGroupId>,
}
/// e_v = op_v(G_{v,1}, …, G_{v,n}): one operator, with its arguments,
/// whose children are groups. No private nodes (decided, see below).
pub struct CandidateExpression {
pub operator: Operator<MemoGroupId>, // op_v; children are G_{v,i}
}
/// p_v^log = (θ_v, ω_v): one field per dimension (decided 2026-11-05: properties are
/// products of dimensions, never one flat enum of their combinations). The family is
/// not a property: it is the operator's argument and part of the group identity.
/// `new` rejects parameters of another family (or on an exact operator) and a
/// window on an operator that cannot be windowed.
pub struct LogicalProperties {
parameters: Option<SketchParams>, // θ_v, from the family's predefined list; None for exact
window_layout: Option<WindowLayout>, // ω_v: tumbling width (#1160); sliding and subtractive in #1206
}
impl LogicalProperties {
pub fn new(expression: &CandidateExpression, parameters: Option<SketchParams>,
window_layout: Option<WindowLayout>) -> Result<Self, LogicalPropertiesError>;
pub fn parameters(&self) -> Option<&SketchParams>;
pub fn window_layout(&self) -> Option<&WindowLayout>;
}
/// The window count is the group's query range ÷ width; retention is h_v (Stage 3).
pub enum WindowLayout { Tumbling { width_ms: u64 } }
/// The family's predefined θ list (#1207 PR 3), each with its error bound
/// derived through `estimators::local_guarantee` for a given statistic.
pub fn parameter_list(family: SketchAlgorithm) -> &'static [SketchParams];
/// r^log_{c,i}: only what G_{c,i}'s identity does not fix. `new` rejects a floor
/// that is not in the family's predefined list.
pub struct LogicalRequirements {
/// θ^min_{c,i}: the smallest θ in the list whose ε, for the statistic the
/// consumer reads, is at most cap_{c,i} = min(validity cap of g_c, pruning cap).
parameter_floor_for_required_accuracy: Option<SketchParams>,
}
impl LogicalRequirements {
pub fn new(parameter_floor_for_required_accuracy: Option<SketchParams>) -> Result<Self, LogicalPropertiesError>;
pub fn parameter_floor_for_required_accuracy(&self) -> Option<&SketchParams>;
/// g_c: β_{c,i} per input and a validity cap per input (None when the bound holds everywhere).
pub fn error_propagation(consumer: &Operator<MemoGroupId>) -> ErrorPropagation;
}
pub struct ErrorPropagation { pub factor_per_input: Vec<f64>, pub validity_cap_per_input: Vec<Option<f64>> }
/// α^log_{u,c,i}: θ_u ⪰ θ^min_{c,i}, compared per family on the parameters that set ε.
pub fn producer_satisfies_logical_requirements(producer: &LogicalProperties, requirement: &LogicalRequirements) -> bool;These are the interfaces of the concepts doc (docs/design_docs/concepts/memo-groups-properties-requirements.md, section 3, #1224). A PR that needs an abstraction not defined here or in the doc lists it and asks first. accuracy_budget_parts is deleted: no candidate is sized for a share of a budget.
Stage 0 (frontends, hash-consing): groups stay one per logical expression: Stage 0 builds them bottom-up from the query nodes through find_or_add_group. It returns a Memo whose groups have empty candidate lists; candidates live in MemoGroup::candidates, not in a Memo-level list.
Stage 1, Pass 1 (logical-asap-optimization/src/pass1):
- Every intermediate result of a way to compute becomes a group, hash-consed. Each candidate is one operator over groups.
- Example:
Project ← SummaryEstimate ← SummaryAggbecomes three groups.
- Example:
- Sizing becomes enumeration: for each summary family, Pass 1 offers the predefined
$\theta$ list as candidates of the state group (Pass 2: tumbling-window builds as candidates of the state group, and merge-then-aggregate over raw tumbling windows (window-specific; memo representation is #1203) #1160) and writes$\theta_v$ intological_properties;$\varepsilon_v$ ,$\delta_v$ are derived from it for the statistic a consumer reads, once, when the consumer's floor is computed (parameter_listandestimators::local_guarantee). -
List bounds. Candidates no query could use are dropped: those whose
$\varepsilon_v$ alone exceeds every reading query's target (Pass 1'sNoErrorBoundForQueryTarget). -
Requirements. A consumer's input records only its parameter floor
$\boldsymbol\theta^{\min}$ : from the validity cap when the consumer is created, tightened by the pruning cap in Pass 2.
Stage 1, Pass 2 (logical-asap-optimization/src/pass2):
- Rules that match a candidate's operator chain match over groups instead, e.g.
summary_results,topk,summary_inputs.-
SummaryResult, which today factors a readout out of a composite candidate, is no longer needed: the readout is already a group.
-
- Rules add candidates
$(e, \theta)$ to existing groups. They do not copy consumers. - Every place that treats a group's candidates as interchangeable checks
producer_satisfies_logical_requirements.
Stage 2 (physical-asap-optimization): reads logical_properties and input_requirements to compute root_node.
- No private nodes. Every operator node is an implementation root of its group, so
is_implementation_rootis always true. - No factoring of identical consumers. Stage 2 no longer factors identical terminal consumers of different candidates (
candidates_for_memo_group): hash-consing already makes them one group.
Stage 3 (plan-selection): ResultSlot::PrivateNode (groups.rs) goes away. Every slot is a group.
Memo choice (enumerate, ChosenCandidatePerMemoGroup): chooses a producer per consumer input (a choice of k=200 and another k=345.
Current vs proposed
| Notation | Today | Proposed |
|---|---|---|
MemoGroup (memo.rs:14): one per logical node, plus Pass 2's per-size state groups |
MemoGroup { expression, group_properties, … }, one per logical result or state family; no per-size groups (#1160) |
|
CandidateForMemoGroup (candidates.rs:25) |
same type, restructured as above | |
root_node plus input_groups; root_node may hold a chain of private nodes down to input_groups
|
expression: CandidateExpression { operator: Operator<MemoGroupId> }: one operator over groups |
|
| private node chain | e.g. G4[1] = Project ← SummaryEstimate TopK(k=10) over Gcms; each private node is a Stage 2 physical candidate whose single input has one fixed producer |
one group per intermediate result: Gest = SummaryEstimate TopK(k=10)(Gcms), G4[1] = Project(Gest)
|
|
|
Stage 2 gives each private node its own timing and materialization, so one logical candidate has several | one |
inside root_node's operator arguments (sketch family parameters), sized per query for a budget share (accuracy_budget_parts) |
logical_properties: LogicalProperties, from the predefined list |
|
|
|
computed by Pass 1 sizing for one budget share | precomputed per |
| not represented: each consumer's private chain fixes its producer's size |
input_requirements[i]: the parameter floor |
|
| not computed: there is no requirement to test | producer_satisfies_logical_requirements(&u.logical_properties, &c.input_requirements[i]) |
|
| sharing one build |
SharedBuild candidates and per-size state groups (Pass 2, add_shared_summary_state_groups) |
one candidate in the state group that several consumers select; constraint (K) checks each query's accuracy |
Mergeable states as their own groups (moved from #1160, decided 2026-11-04)
With one operator per candidate and
SummaryAgg(k=200)(G1)andSummaryAgg(k=345)(G1)are one groupSummaryAgg[KLL](G1)with two candidates;- the aggregate's group keeps only the consumer (
FinalizeExactAccumulatoror the estimate) over it.
This PR also takes from #1160:
- fused builds (Pass 2: tumbling-window builds as candidates of the state group, and merge-then-aggregate over raw tumbling windows (window-specific; memo representation is #1203) #1160 rules 1 and 2): a fused build with the outer target's state is a candidate of that state group; one with another state kind is its own state group;
- shared builds as plain group identity:
SharedBuildcandidates and per-size state groups are deleted; - memo choice per consumer input, since two consumers may use different sizes of one group.
#1160 keeps what hash-consing cannot find: tumbling-window builds as candidates of the state group, and sub-window order (b). The
Decision: one operator per candidate, no private nodes (2026-11-04)
The formulation writes $e_v = \operatorname{op}v(G{v,1}, \dots)$ with a single operator. Today a candidate's root_node can hold a chain of private operators.
Example from #1160, topk(10, sum by (job) (sum_over_time(http_requests_total[1m]))):
today (after #1160) decided
G4: Aggregate top_k(k=10) Gest: SummaryEstimate TopK(k=10)
[1] Project ← SummaryEstimate TopK(k=10) [0] SummaryEstimate TopK(k=10) inputs=[Gcms]
inputs=[Gcms] G4: Aggregate top_k(k=10)
[1] Project inputs=[Gest]
Decided: every operator is a candidate of its own group (option b), as in Cascades. In Cascades, each memo multi-expression is one operator over input groups, and each intermediate result is a group. Reasons:
-
Physical properties are per operator node. Stage 2 already gives
SummaryEstimateandProjecttheir own timing and materialization, so a chain as one candidate would carry several$p^{\mathrm{phys}}$ . Cascades folds several logical operators into one only for a physical operator whose inside cannot be materialized, which is not the case here. -
One mechanism.
- Every edge is in
$\mathcal E$ (Producer–consumer edges (u, c, i): make the edge set explicit after Pass 2 and Stage 2, with α^log and α^phys, as the domain of y_{u,c,i} #1201); a former private edge is an ordinary edge with$|P_{c,i}| = 1$ . -
ResultSlot::PrivateNode,is_implementation_rootand Stage 2's factoring of identical consumers go away.
- Every edge is in
- Same MILP size. Stage 2 already makes one physical candidate per private node.
-
Same direction as existing code.
SummaryResultand Pass 2: tumbling-window builds as candidates of the state group, and merge-then-aggregate over raw tumbling windows (window-specific; memo representation is #1203) #1160's state groups already split parts of chains into groups.
Rejected: a chain as one
Cost:
- more groups, mostly with one candidate each;
- the Pass 2 rules that match operator chains must match over groups.
Plan decisions (2026-11-04)
Order: see #1207, "Order (rewritten 2026-11-06: interfaces first, baseline before extensions)". This issue's parts are PRs 0 to 4 and 8.
| # | Question | Decision |
|---|---|---|
| 1 | What leaves a group's identity (interning key) (PR 3) | Taken out of the key, as candidate properties:
|
| 2 | "Declared by measure" equivalence (a build is a candidate of a state group though it is a different logical node) | One mechanism, built with the fused builds in PR 3; #1160 reuses it. |
| 3 | Rule 9 (shared_configuration_readers) |
Deleted in PR 0, with no replacement. Plans that get costlier before the |
| 4 | Overlap with #1200 | None: #1207's PR 0 deletes private nodes, ResultSlot::PrivateNode and the old physical types together. |
| 5 | Viewer (viewer_contract) |
Ignored for now; each PR states in its body that the viewer tests are not updated. |
| 6 | Plan cost / planning time | Each PR lists the snapshots whose selected cost changes, and the change in planning time; plans are checked by hand with asap-planner --output text. |
Scope and acceptance
In scope:
- the
CandidateForMemoGrouprestructure; - PRs 2 to 3 (Tracking: implement the optimization formulation (memo candidates, properties, edges y_{u,c,i}, MILP) as named code per stage #1207): mergeable states as their own groups, fused builds, shared builds as group identity, memo choice per consumer input,
$\theta$ asLogicalProperties(today's sizes, unchanged),CandidateExpression=Operator<MemoGroupId>; - PRs 1 and 3 (Tracking: implement the optimization formulation (memo candidates, properties, edges y_{u,c,i}, MILP) as named code per stage #1207): the
$\theta$ list with$\varepsilon$ ,$\delta$ ,parameter_list, the parameter floor, deletingaccuracy_budget_parts; - one operator per candidate: Pass 1 groups for intermediate results, Pass 2 rules over groups, and removing private nodes from Stage 2 and Stage 3;
- Pass 1 enumerating the
$\theta$ list with$\varepsilon$ ,$\delta$ , and writing$r^{\log}$ ; - deleting
accuracy_budget_parts; - Pass 2 rules checking
producer_satisfies_logical_requirements; - Stage 2 reading
$\theta$ from the new field.
Out of scope:
- physical properties (Physical properties: replace Materialization with timing × lifetime (None / OneEvaluation / AcrossEvaluations) and a schedule for shareable results, with r^phys and α^phys #1200);
- edges (Producer–consumer edges (u, c, i): make the edge set explicit after Pass 2 and Stage 2, with α^log and α^phys, as the domain of y_{u,c,i} #1201);
- Stage 3 (Stage 3 MILP: implement the formulation variables x, y, z, t and rows (A)–(I) as named code over explicit edges #1204).
Acceptance:
- Deletes what it replaces in the same PR, with no dead code, fallback, flag, shim or reuse of an old abstraction; the PR body lists what it deleted (Tracking: implement the optimization formulation (memo candidates, properties, edges y_{u,c,i}, MILP) as named code per stage #1207, Policy).
- No code reads sketch sizes out of
root_nodeto decide compatibility. - No candidate has a private node. For the top-k example,
stage_pipeline --textshowsGestandG4[1] = Projectover it. - Stage 2's physical candidate count per snapshot is the same or lower. It can be lower because hash-consing merges identical chains.
- For the example, the state group lists the KLL candidates from the list, each with its
$\varepsilon$ and$\delta$ , and no consumer input carries a size requirement, as shown bystage_pipeline --text. -
corpus_snapshotsandsketch_plan_snapshotskeep every selected plan's cost, or the PR lists each difference.
Related
- Pass 2: tumbling-window builds as candidates of the state group, and merge-then-aggregate over raw tumbling windows (window-specific; memo representation is #1203) #1160: state groups. Its size-requirement proposal is replaced by constraint (K) (Stage 3 MILP: implement the formulation variables x, y, z, t and rows (A)–(I) as named code over explicit edges #1204).
- Producer–consumer edges (u, c, i): make the edge set explicit after Pass 2 and Stage 2, with α^log and α^phys, as the domain of y_{u,c,i} #1201: edges and
$\alpha$ ; Physical properties: replace Materialization with timing × lifetime (None / OneEvaluation / AcrossEvaluations) and a schedule for shareable results, with r^phys and α^phys #1200:$p^{\mathrm{phys}}$ .
Activity
[-]Logical memo candidate as (expression, logical properties θ) with a requirement per input[/-][+]Logical memo candidate as (one operator over groups, logical properties θ) with a requirement per input; no private nodes[/+]refactor(stage1): one node equality for memo groups; review fixes for #…
zzylol commentedon Nov 5, 2026
Decision (2026-11-05), from 2a-6 (tumbling-window chain and merge become one group shared by every reader of the same state and width): Stage 3 currently gives each candidate one accuracy weight, so a state shared by a ratio's numerator and denominator takes the stricter reader's weight and can no longer serve the numerator alone. Decided: accuracy weights move from the candidate to the (query, reader position) pair, as in row (K)'s$\lambda_{q,u} a_{q,u}$ . This lands as its own stacked PR before 2a-6.
refactor(plan-selection): row (K) terms per path of edges, not a weig…
refactor(plan-selection): row (K) terms per path of edges, not a weig…