Skip to content

Executor: store Disk-medium materialized views and one-evaluation results on disk #1217

Open

Description

@zzylol

Example

Take one KLL state, quantile_over_time(0.99, m[1m]), kept at ingestion time on disk: physical properties KeptAcrossEvaluationsAtIngestion(Disk), plan-file node field "lifetime": {"AcrossEvaluations": "Disk"}.

Planner (Stage 3, cost model 1) Execution
Today Prices the kept bytes as disk: cost_per_disk_byte_second × bytes, nothing added to average memory. Tested by state_on_disk_is_counted_as_disk_not_memory. The state runs in memory, exactly as AcrossEvaluations(Memory). The executor's physical planner makes it a materialized-view slot (every_valid_state.rs::physical_planner_over_every_valid_state). The MaterializedView a deployment installs (asap_types::ir::materialized_view::MaterializedView) has no medium field, so the deployment's DataStore cannot tell disk from memory. The only store in this repository, InMemoryDataStore, keeps every view in a BTreeMap in memory.
Wanted Same price. The view is written to disk and read back from it. Its retention is evicted on disk.

KeptForOneEvaluationAtQueryTime(Disk) has the same gap. It is a query-time result shared within one evaluation: the planner prices it as disk, and the executor computes it in memory, once per query run.

For now no plan reaches either state. Stage 2 offers only the Memory medium, by rule; the rule comment at enumerate_candidate_configurations cites this issue. Every consumer still handles Disk when given it (#1216's table tests).

Why it isn't done today

  • No medium on the install contract. MaterializedView carries node_id, refresh, coverage, inputs and retention_ms. The medium exists only on the plan node's lifetime.
  • No disk-backed store. asap_executor::sources::DataStore is the deployment's view and raw-data store. The only implementation here, InMemoryDataStore, is memory only.
  • One-evaluation results are not stored at all. They are recomputed inside the query run, so a OneEvaluation(Disk) result has nowhere to spill.

Proposal

  1. Medium on the install contract. MaterializedView gains medium: Medium, filled from the node's lifetime by views_with_node_retention. This bumps the plan-file wire version.
  2. A disk-backed view store. A DataStore implementation (or a mode of the deployment's store) writes a Disk view's states to files on refresh, reads them back for the view's slot, and evicts states older than retention_ms on disk.
  3. One-evaluation results. Either spill a OneEvaluation(Disk) result to disk within the query run, or decide to keep it in memory and price it so. The latter changes the cost model, not the executor.
  4. Runtime install. InstallablePlan installation creates each view on its medium.

Alternatives. Drop Medium::Disk from the IR entirely until a store exists. That would delete the cost model's disk terms and the disk tests.

Scope and acceptance

  • In scope: the medium on MaterializedView, a disk-backed store, install and retention eviction on disk, and executor tests.
  • Acceptance:
    • A plan with an AcrossEvaluations(Disk) view installs it on disk.
    • Refresh writes it, the query reads it back with the same answer as Memory, and eviction removes states past retention_ms.
    • Executor tests cover both AcrossEvaluations(Disk) timings and OneEvaluation(Disk).
    • Once this lands, Stage 2 may offer Disk, which is a separate decision.
  • Out of scope: choosing between memory and disk in Stage 3. That already happens through cost once Stage 2 offers both.

Status and related work

Metadata

Metadata

Assignees

No one assigned

    Labels

    cost-modelCost formulas, statistics, ranking, and selectionenhancementNew feature or requestirPre-ASAP or post-ASAP intermediate representation

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions