feat(optimize/analyze): preview applicable optimizations and check their produced-operator support

[xieofxie]

Summary

Two related, opt-in features for reasoning about optimizations before applying them, sharing one consistent flag name --check-optim:

1. winml optimize --check-optim — answers "which optimizations actually apply to this model, and on which nodes?" without writing any output.
2. winml analyze --check-optim — for each optimization that would change the model, checks whether the operators it introduces are supported on the resolved EP/device.

optimize --check-optim

Probes every registered optimization capability independently against the given model and reports only the ones that would genuinely change it, along with the specific nodes and constants affected.

$ winml optimize -m model.onnx --check-optim
2 applicable optimization(s):

--enable-matmul-add-fusion  (matmul)
  Fuse MatMul+Add operations into single kernel
  nodes: -1 ~1
    removed: MatMul (1)
    modified: Gemm (1)

--enable-clamp-constant-values  (surgery)
  constants: ~1

Add -v to list every affected node/constant instead of a sample. --check-optim requires --model and writes nothing to disk.

analyze --check-optim

Reuses the optimize probe engine to obtain the model each optimization would produce, then feeds the produced (added/modified) nodes into analyze's RuntimeChecker and classifies them against the target EP/device rule data. Answers "if I apply this optimization, will its output run on my target?".

$ winml analyze --model model.onnx --ep qnn --device NPU --check-optim

Renders a per-target OPTIMIZATION OUTPUT SUPPORT section listing each applicable optimization, its --enable-* flag, and the support level (supported / partial / unsupported / unknown) of every operator it would introduce. The probe is target-independent, so it is materialized once and reused across all resolved targets; only the support lookup runs per EP/device. Disabled by default; console-only.

How it works (universal, no hardcoding)

The probe mirrors Optimizer.optimize(): run mandatory shape inference, then walk the pipes in order. For each pipe it computes a baseline output, then re-runs the pipe on a clone of the upstream model with only one default-off capability enabled (plus auto-enabled dependencies) and diffs the result:

• Node diff is keyed by output tensor names (unique and stable within a valid ONNX graph across input rewiring / attribute changes), yielding removed / added / modified buckets; recurses into subgraphs.
• Initializer diff catches constant-only surgery ops (e.g. clamp-constant-values) that modify no nodes.
• Cloning uses CopyFrom (not a serialize round-trip) because saving can convert tensors to external data in place, and to avoid the 2 GB protobuf limit.
• Each probe is isolated in try/except so a single failing capability is logged and skipped rather than aborting the run.

The support check correlates produced nodes back to op_support results by output-tensor name, and RuntimeChecker.op_support(node_output_filter=...) restricts the check to just the produced nodes. No op/tensor/architecture names are hardcoded (CLAUDE.md Cardinal Rule #1).

Changes

• New src/winml/modelkit/optim/analysis.py — analyze_model, iter_optimization_outputs, CapabilityFinding, NodeRef and diff helpers (shared _iter_findings generator)
• src/winml/modelkit/optim/__init__.py — export the new public API
• src/winml/modelkit/commands/optimize.py — --check-optim option + report renderer (with rich-markup escaping of dynamic labels)
• New src/winml/modelkit/analyze/optim_output.py — bridge that checks produced-operator EP/device support
• src/winml/modelkit/analyze/core/runtime_checker.py — op_support(node_output_filter=...) to restrict the check to a node subset
• src/winml/modelkit/commands/analyze.py — --check-optim flag + per-target renderer
• New/updated tests — tests/unit/optim/test_analysis.py, tests/unit/analyze/test_optim_output.py, and CLI coverage in tests/unit/commands/test_optimize_cli.py and tests/unit/analyze/test_static_analyzer_cli.py
• docs/commands/optimize.md and docs/commands/analyze.md — flag-table rows + worked examples

Validation

• ruff clean; mypy -p winml.modelkit clean; doc/code flag-alignment check ✅
• Targeted optim + analyze suites pass (hardware/fixture-gated skips and one pre-existing xfail only)

[xieofxie]

should move to analyze command to take advantage of analyzer data

[DingmaomaoBJTU]

test

[DingmaomaoBJTU]

Review Summary

PR scope: Adds --check-optim to both winml optimize and winml analyze, with a clean dry-run analysis engine, produced-operator support checking, and good test/doc coverage. Well-structured — new modules (optim/analysis.py, analyze/optim_output.py) are cleanly separated. Inline comments below.

[DingmaomaoBJTU]

Second-pass deeper review — additional findings on correctness and performance.

[DingmaomaoBJTU]

Third-pass review — error handling, edge cases, and correctness deep-dive.

Verified OK (no issues):

• auto_enable_dependencies returns a new dict, no mutation risk between probe iterations
• PIPES ordering is shared between _iter_findings and Optimizer.optimize() — consistent
• node_output_filter.intersection(node.output) handles protobuf repeated fields and empty outputs correctly
• --check-optim properly requires --model (validated before handler)
• Optimization probe runs once, support check runs per EP/device target — good design for --ep all

[DingmaomaoBJTU]

LGTM — approving after three review passes. Well-architected feature with clean separation (analysis.py for dry-run, optim_output.py for EP support checking).

Non-blocking follow-ups (can be addressed in later PRs):

• Add try/except around baseline _run_pipe call (analysis.py:280) for robustness — low risk since default configs rarely fail, but would match the probe's error handling
• _collect_initializers may produce false "modified" findings for external-data models (serialization includes file metadata)
• For very large models with many applicable optimizations, list(iter_optimization_outputs(...)) materializes all ModelProto clones in memory — consider a warning or lazy approach

All other areas verified: dependency management, pipeline ordering, filter logic, multi-target reuse, --model validation — all clean.

[DingmaomaoBJTU]

LGTM. The --check-optim feature is well-designed: the dry-run analysis walks the pipeline pipe-by-pipe with per-capability isolation, the graph diff via output-tensor identity is robust, and the produced-model handoff to the analyze bridge (--check-optim on analyze) cleanly separates target-independent probing from EP-specific support checks. Test coverage is thorough (diff helpers, end-to-end analysis, CLI wiring, rendering). A couple minor observations (non-blocking):$([char]10)$([char]10)1. _matmul_add_model() is duplicated across test_analysis.py, test_optim_output.py, and test_static_analyzer_cli.py -- consider a shared conftest fixture.$([char]10)2. In analyze.py the optim_outputs list is typed as list[tuple[Any, Any]] -- using the concrete types list[tuple[CapabilityFinding, ModelProto]] would be clearer and catch misuse.