Repository navigation
Description
Example
Take one KLL state, quantile_over_time(0.99, m[1m]), kept at ingestion time on disk: physical properties KeptAcrossEvaluationsAtIngestion(Disk), plan-file node field "lifetime": {"AcrossEvaluations": "Disk"}.
| Planner (Stage 3, cost model 1) | Execution | |
|---|---|---|
| Today | Prices the kept bytes as disk: cost_per_disk_byte_second × bytes, nothing added to average memory. Tested by state_on_disk_is_counted_as_disk_not_memory. |
The state runs in memory, exactly as AcrossEvaluations(Memory). The executor's physical planner makes it a materialized-view slot (every_valid_state.rs::physical_planner_over_every_valid_state). The MaterializedView a deployment installs (asap_types::ir::materialized_view::MaterializedView) has no medium field, so the deployment's DataStore cannot tell disk from memory. The only store in this repository, InMemoryDataStore, keeps every view in a BTreeMap in memory. |
| Wanted | Same price. | The view is written to disk and read back from it. Its retention is evicted on disk. |
KeptForOneEvaluationAtQueryTime(Disk) has the same gap. It is a query-time result shared within one evaluation: the planner prices it as disk, and the executor computes it in memory, once per query run.
For now no plan reaches either state. Stage 2 offers only the Memory medium, by rule; the rule comment at enumerate_candidate_configurations cites this issue. Every consumer still handles Disk when given it (#1216's table tests).
Why it isn't done today
- No medium on the install contract.
MaterializedViewcarriesnode_id,refresh,coverage,inputsandretention_ms. The medium exists only on the plan node'slifetime. - No disk-backed store.
asap_executor::sources::DataStoreis the deployment's view and raw-data store. The only implementation here,InMemoryDataStore, is memory only. - One-evaluation results are not stored at all. They are recomputed inside the query run, so a
OneEvaluation(Disk)result has nowhere to spill.
Proposal
- Medium on the install contract.
MaterializedViewgainsmedium: Medium, filled from the node's lifetime byviews_with_node_retention. This bumps the plan-file wire version. - A disk-backed view store. A
DataStoreimplementation (or a mode of the deployment's store) writes aDiskview's states to files on refresh, reads them back for the view's slot, and evicts states older thanretention_mson disk. - One-evaluation results. Either spill a
OneEvaluation(Disk)result to disk within the query run, or decide to keep it in memory and price it so. The latter changes the cost model, not the executor. - Runtime install.
InstallablePlaninstallation creates each view on its medium.
Alternatives. Drop Medium::Disk from the IR entirely until a store exists. That would delete the cost model's disk terms and the disk tests.
Scope and acceptance
- In scope: the medium on
MaterializedView, a disk-backed store, install and retention eviction on disk, and executor tests. - Acceptance:
- A plan with an
AcrossEvaluations(Disk)view installs it on disk. - Refresh writes it, the query reads it back with the same answer as
Memory, and eviction removes states pastretention_ms. - Executor tests cover both
AcrossEvaluations(Disk)timings andOneEvaluation(Disk). - Once this lands, Stage 2 may offer
Disk, which is a separate decision.
- A plan with an
- Out of scope: choosing between memory and disk in Stage 3. That already happens through cost once Stage 2 offers both.
Status and related work
- Decided 2026-11-05: Stage 2 keeps not offering
Disk; this issue tracks executor disk storage. Until it lands, aDiskstate executes in memory. This is stated in the rule comment and in refactor(types, stages): every valid timing × lifetime state explicit; PhysicalRequirement, satisfies_physical, no named states #1216's state table. - Related: Physical properties: replace Materialization with timing × lifetime (None / OneEvaluation / AcrossEvaluations) and a schedule for shareable results, with r^phys and α^phys #1200 (physical properties), Tracking: implement the optimization formulation (memo candidates, properties, edges y_{u,c,i}, MILP) as named code per stage #1207 (tracking), refactor(types, stages): replace Materialization with PhysicalProperties (timing, lifetime) #1214 and refactor(types, stages): every valid timing × lifetime state explicit; PhysicalRequirement, satisfies_physical, no named states #1216 (timing × lifetime and the state table).
Activity
refactor: PhysicalProperties = NotKept(timing) | Kept(KeptProperties)