Repository navigation
planeval: run the use cases through ASAPPlanner's planner and executor, replacing the single-query planeval - #233
Merged
Merged
Conversation
… fidelity in Grafana aqpbm-inprocess plans each query of a use-case spec with ASAPPlanner (bb981e9b) and runs the installed plan with its executor at every evaluation time, refreshing materialized views as the planner's own runtime test does. Use case 1 (SYN flood: entropy and distinct counts of source IPs) runs an exact arm and seven approximate arms. scripts/ turns the answers into query_result rows, derives the threshold alert and its fidelity metrics, and writes them for the Grafana dashboard in tools/grafana. docs/approx-bench-use-cases.md holds the five use cases. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…dashboard Pin ASAPPlanner to main 41c51546. Specs now describe many labeled series per metric, phased changes, and request streams whose values become labels; use case 1 draws the same samples as before. Add the comparison alert (use case 4), autoscaler decision (use case 2) and top-k set (use case 5) applications, a single run_use_case.py entry point, and a dashboard generator with decision, topk and overview dashboards. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each application now writes exact, approx and annotation strips per evaluation (onset delay, gap, end delay, false positive, decision delay) and reports true/false positives and negatives. - Use case 1: end delay, measured as a mean absolute value. - Use case 2: scaling actions once per crossing, matched within two evaluations; down rule at 0.4 cores and 0.75 GB. - Use case 4: rollout controller over fixed 10-check rollouts replaces the comparison alert; canary at 5% of samples; new rollout dashboard. - Use case 5: exact and approx capped strips per client. - Per-container and per-client data files so the decision and top clients dashboards load quickly. - docs/approx-bench-use-cases.md updated to the current use case doc. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Use case 2 matches the closest same-kind exact and approx actions first, so the result no longer depends on processing order. - Grouped stat queries return nothing on empty input instead of an Infinity error. - Per-container and per-client panels read files through hidden variables that always name an entity in the selected run. - Split run data rejects entity names that collide when case is ignored, and reads only JSON files. - The overview shows use case 5 agreement as mean overlap, with identical lists in the fidelity text. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nd analytics for every planned sketch Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…case runner Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…os in plain short sentences Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes
aqpbm-planevalnow runs recurring PromQL use-case queries through ASAPPlanner's own planner and executor, in process, and measures how faithfully each approximate plan preserves the application's decisions. It replaces the single-query planeval on main. The old one is pinned to ASAPPlanner 3c31338, depends onasap-aware-mapping(gone from planner main), and executes the DAG with its own hand-written executor.aqpbm-planeval(binaryplaneval), pinned to ASAPPlanner main b822553. It runs the specs inaqpbm-planeval/workloads/for use cases 1, 2, 4 and 5 at every evaluation time.scripts/run_use_case.pywiththreshold_alert.py,decision.py,rollout.pyandtopk_set.py.tools/grafana/, including an overview table that puts sketchlib's bound and memory next to the measured error and the planner's predicted memory.Removed
aqpbm-planevalsources.configs/datagen/planeval_promql_leaf.yamlandconfigs/datagen/planeval_cluster_metrics.yaml, used only by it.Verification
cargo build --release --workspacecargo test -p aqpbm-planeval --release: 4/4 pass; clippy clean.python3 -m unittest discover -s scripts: 170 pass.python3 scripts/run_use_case.py 4 --arm noraw-eps0.01🤖 Generated with Claude Code