[milindsrivastava1997]

Summary

• Problem: the viewer's memo view was hard to read.
  • Large groups were unusable: example1 G9 has 280 CmsWithHeap candidates.
  • Every candidate drew its own arrow, even when they all read the same group.
  • Stage 2's physical variants were only a text dump.
  • Inputs and outputs were mixed on the page, and the deployment line printed undefined (Total), ….
• Change:
  • Large groups split into counts by any varying dimension (origin, sketch, each θ field, window width), and you can drill into a count.
  • One arrow from the group when all its candidates read the same groups.
  • Pass 2's pruned candidates are hidden behind a checkbox.
  • A "selected plan only" filter.
  • A timing × lifetime grid of each candidate's Stage 2 variants.
  • The page is split into Planner inputs and Planner output.
  • Zoom buttons, a light theme by default, and the tool renamed to Plan Viewer.
• Effect: for stage_dump/q3_10m5m_5m5m_3m3m.json (three quantile queries):
  • The Pass 2 memo goes from ~100 boxes to 53 with 29 arrows, because pruned candidates are hidden.
  • With "selected plan only" on, it shows the 14 groups the plan uses.
  • Each candidate's grid shows that the plan builds the shared 5-minute KLL chain at query time (C53) instead of at ingest (C54). That choice is why the materialized-views panel says "no installable plan: … a tumbling-window chain is built in ingestion runs".

Closes #1262.

Screenshots

These are in the light theme, taken in headless Chrome on stage_dump/q3_10m5m_5m5m_3m3m.json (1–5) and example1 (6–7). The images live on the pr-assets/1262 branch, which is not for merging.

1. Planner inputs: the queries, the queries' DAG labelled Input, the data workload, and deployment and planner settings.

2. Planner output: candidate counts, the outcome line, and the selected plan.

3. Memo, Pass 2. Pruned candidates are hidden. Each candidate has its Stage 2 grid (rows ingest/query; columns not kept / one run / across runs; filled, crossed, or dark outline = used by the plan).

4. Memo with "selected plan only".

5. A candidate's details: the Stage 2 grid as a table, then each variant's physical edges. G19#0 is the shared 5-minute KLL chain, and the plan uses its query-time variant C53.

6. Drilling into example1's G9 (280 CmsWithHeap candidates, with "show pruned candidates" on): depth=5 (70) split into 10 heap_size counts.

7. G9's details: "split by" picks the dimension the counts split on.

Review guide

• Position: stacked on feat/1261-readable-stage-output (feat(devtools): asap_planner text output is terse by default; --verbose prints the details (#1261) #1276), which is stacked on refactor/1207-tools (refactor: tools and snapshots over the #1207 memo, edges and contexts (#1207 PR 9) #1239).
• Reading order:
  1. tools/dag-viewer/memo.js: memoCandidateDims, memoGroupDrill, memoPhysicalGrid and memoGridImage, then the changes to memoStepElements and the controller (draw, tap, showCandidate).
  2. tools/dag-viewer/index.html: the inputs/outputs layout.
  3. tools/dag-viewer/app.js: outcome line, settings, zoom, theme.
  4. tools/dag-viewer/stages.js: settingsRows, fitGraph.
• Mechanical:
  • The Stage Viewer → Plan Viewer rename across README.md, docs/** (docs/user_guide_docs/stage-viewer.md → plan-viewer.md), tools/dag-viewer/* and one comment in crates/devtools/src/bin/asap_planner.rs.
  • Test updates in test_viewer.py for removed or renamed elements.
• Careful review:
  • memoGroupDrill: buckets, and which dimension to split by next.
  • The edge dedup in memoStepElements.
  • memoPhysicalGrid's rule for "used by the plan": a physical edge into or out of the variant has selected: true.

Behavior: before → after

The user runs asap_planner --workload stage_dump/q3_10m5m_5m5m_3m3m.json --run-until stage3 --format json --out out/q3.json and opens ?doc=out/q3.json.

	Before	After
Page layout	Queries with data workload beside them; a Stage 3 sidebar on the left; deployment summaries: undefined (Total), …	Planner inputs (query workload, the queries' DAG labelled Input, data workload, Deployment and planner settings with summaries Cms (Total), …, accuracy guarantees, cost model analytical-cost-v2 (illustrative statistics)) above Planner output (outcome line plan selected · candidate-connection search · MILP 340 variables, 514 constraints · …, plan, views, memo)
G9 in example1 (280 candidates)	Collapsed by origin only: one box "280 Pass1"	Counts 70 · depth=5 … ; click to drill in (e.g. 10 heap_size buckets → 7 candidates θ width=28 …); the group's details offer split by width / heap_size
Arrows	One per candidate (4 parallel arrows G9 → G1)	One from the group when all candidates read the same groups (g9>g1)
Pruned candidates	Always drawn	Hidden unless show pruned candidates is ticked
Stage 2	Text list per candidate	A 2×3 grid per candidate (ingest/query × not kept/one run/across runs): filled, crossed (pruned), dark outline (used by the plan); the same grid as a table with C<id>s in the details
Memo tabs	Stage 0, the frontends' DAG hash-consed, Stage 1, Pass 1, …	Stage 0, Stage 1 · Pass 1, Stage 1 · Pass 2
Accuracy without a metric	(metric not set)	(error metric not set)

Need served: debugging and explaining the planner's memo, i.e. which candidates exist, why they were pruned, and how the plan was chosen (issue #1262).

Interfaces

Viewer JavaScript only; there are no Rust interface changes.

tools/dag-viewer/memo.js

+// A candidate's dimensions, in split order: [['origin', 'Pass1'], ['sketch', 'Kll'], ['k', '1404'], ['window_ms', '300000']].
+function memoCandidateDims(c)
+// Which of `candidates` a group shows under its drill `state` ({path: [[dim, value]], next}):
+// {path, shown, choices (dims that still vary), dim (split by, or null), buckets ([[value, candidates]], numeric, 'none' last)}.
+function memoGroupDrill(candidates, state, collapseAbove)
+// Stage 2's variants of candidate (group, index) as rows MEMO_TIMINGS × columns MEMO_LIFETIMES,
+// each {ids, pruned, selected}; null without a Stage 2 section.
+function memoPhysicalGrid(doc, group, index)
+// That grid as an SVG data URI drawn under the candidate's label.
+function memoGridImage(grid)
+const MEMO_TIMINGS = [['ingestion_time', 'ingest'], ['query_time', 'query']];
+const MEMO_LIFETIMES = [['None', 'not kept'], ['OneEvaluation', 'one run'], ['AcrossEvaluations', 'across runs']];
+const MEMO_STEP_NAMES = { stage0: 'Stage 0', pass1: 'Stage 1 · Pass 1', pass2: 'Stage 1 · Pass 2' };
+const memoStepName = (step) => MEMO_STEP_NAMES[step.step] || step.title;
 // Step s as cytoscape elements; options:
-  const { expanded = [], collapseAbove = MEMO_COLLAPSE_ABOVE, queryIds = [] } = options || {};
+  const { drill = {}, collapseAbove = MEMO_COLLAPSE_ABOVE, queryIds = [], hidePruned = false, physical = null, selectedOnly = false } = options || {};
 function memoStepElements(memo, s, options)
-  return { show, highlightQuery, restyle, selectStep };
+  return { show, highlightQuery, restyle, selectStep, graph: () => cy };

expanded (a list of expanded groups) is replaced by drill (per group, its drill path), because a group now splits on more than one dimension.

tools/dag-viewer/stages.js

-function selectedPlanRow(doc)
-function accuracyGuaranteesText(doc)
-function deploymentText(capabilities)
+// The planner's inputs besides the workload as [name, value] rows: raw data, query-time results,
+// summaries, accuracy guarantees, cost model (stage3.cost.source).
+function settingsRows(doc)
+// Fit a graph, at most at 1.3× (first drawing and the ⌂ button).
+function fitGraph(cy)

These three are removed because their only caller, the Stage 3 sidebar and the one-line deployment text, is gone. settingsRows replaces the last two.

tools/dag-viewer/app.js

-  window.StageViewer = { planWorkload };
+  window.PlanViewer = { planWorkload };

Usage, checked in memo.test.mjs on test-fixtures/p90-kll.json:

const doc = fixtureDoc();                         // asap_planner's document for one p90 query (KLL only)
memoPhysicalGrid(doc, 3, 5);                      // G3#5, KLL k=1404:
// [[none, none, none],                           //   ingest: no variant
//  [{ids:[12], selected:true}, {ids:[13]}, {pruned:true}]]  // query: C12 not kept (the plan's), C13 one run, across runs pruned
memoStepElements(doc.memo, 2, { selectedOnly: true });  // nodes g0,g0c0,g1,g1c0,g2,g2c1,g3,g3c5,g4,g4c0 only

Algorithm changes

None in the planner. The viewer only changes how it draws:

1. Drill-down (memoGroupDrill):
   • Keep the candidates matching the path.
   • Of the dimensions that still vary among them, pick the requested next one, or else the first in memoCandidateDims order.
   • Bucket the candidates by that dimension's value, numeric order with "none" last.
   • Stop when at most 12 remain or nothing varies.
   • Cost is O(n · dims) per draw.
2. Edge dedup: when every drawn item of a group reads the same set of groups, the group draws one edge per child.
3. Plan use of a variant: a variant counts as used if any physical edge with selected: true has it as producer or consumer. This covers leaves (Scan) and roots.

Decisions and open questions

• Needs reviewer decision:
  • What Pass 2's inputs[].edges in the JSON means (stored edges vs. recomputed pairs) is filed as asap_planner and viewer: Pass 2 edges are recomputed instead of read from the stored list, and are labeled with α symbols #1275 and is not changed here. The details panel still shows the recomputed list.
  • The selected q3 plan builds a tumbling-window chain at query time, which the executor rejects ("a tumbling-window chain is built in ingestion runs"). Stage 2 offers that variant; whether it should is a planner question, not addressed here.
• Decided (2026-11-10):
  • Stage 2 is shown as a grid inside each candidate, not as a separate graph of 76 physical nodes.
  • The theme is light by default with a toggle.
  • The Stage 3 sidebar is removed: its content is the outcome line and the plan lane's header.
  • The tool is renamed to Plan Viewer; the document format keeps the name asap-stage-pipeline/v2.

Verification

Scope	Passed / failed / ignored	Newly failing vs base	Notes
node --test *.test.mjs (tools/dag-viewer)	33 / 0 / 0	none
python3 -m unittest test_viewer test_server (py-mini-racer 0.6.0)	81 / 0 / 0	none
cargo test -p asap-devtools --test devtools	70 / 2 / 1	none	Two tests fail on the base (7b65e693d) without this PR's commits: the_nine_query_workload_lowers_and_installs, workload_examples_select_valid_plans_with_linked_materialized_views. The only Rust change here is one comment (committed with --no-verify for that reason).

New tests:

• Dimensions: a candidate's dimensions are its origin, sketch, θ fields and window width.
• Splitting and drilling: a large group splits by its first varying dimension, and a count drills into it.
• Next dimension: the dimension chosen next is used, and a missing one counts as "none", sorted last.
• Edge dedup: a group whose candidates share inputs draws one edge from the group (G5 → G0).
• Hidden pruned candidates: pruned candidates, groups left empty, and edges into them are left out.
• Selected plan only: only the plan's candidates and groups remain, and only on the last step.
• Stage 2 grid: the grid shows which variants exist, which Stage 2 pruned, and which the plan uses.
• Page: pruned candidates stay hidden until the checkbox is ticked, and tapping counts drills down and back.

Evidence: see Screenshots above, all taken from this branch in headless Chrome. To reproduce, run python3 tools/dag-viewer/server.py and open ?doc=out/<doc>.json.

Follow-ups

• asap_planner and viewer: Pass 2 edges are recomputed instead of read from the stored list, and are labeled with α symbols #1275: the JSON's Pass 2 edges, and plain-English replacements for the α^log / α^phys labels.
• Clicking a grid cell to draw just that variant's physical edges.
• Shorter operator labels in memo boxes.
• A layered layout (e.g. ELK) that routes arrows around boxes.
• Planner-side:
  • record the cost-inputs file in the document;
  • record the error metric used when a query sets none;
  • check DDSketch α=0.001 being pruned at "Relative ≤ 0.01";
  • query-time tumbling-window chains vs. the executor.

Human review — do not complete with an agent

• The MVP boundary is correct.
• New conceptual layers or public interfaces are necessary.
• The before/after description matches the intended product behavior.
• Human reviewer:
• Decision and rationale:

🤖 Generated with Claude Code

https://claude.ai/code/session_014qhwq4QkRexgK7yPwbksve