e2e_eval: record the precision the build actually applied - #1243
Merged
Conversation
run_eval wrote EvalJob.precision into eval_result.json, which is None for any winml-config fallback job that pins no precision. But that build is not precision-less: winml config resolves auto to the device default, so a --device npu / --device auto run still produces a w8a16 QDQ model. The downstream coverage report maps a missing precision to the 'unquantized' label, so those quantized runs were published as unquantized baselines. _run_build now reports the effective precision back to the caller: the pinned flag when there is one, otherwise read from the generated build config's quant section (uint8/uint16 -> w8a16, no quant stage -> fp32). _run_recipe_build reports its variant precision. The job's declared precision still drives the result-dir slug and the log label, so folder naming and --continue resume are unchanged.
A build config with no quant stage was reported as fp32. That is an inference about the graph, not something the config states: winml only skipped quantization, and the exported dtype depends on the source weights. Report only what the config carries, so an unquantized build records no precision at all (as before) while a quantized one is still named.
Falling back to job.precision when the build reported none could stamp a value the build deliberately ignored: for a skip-quant EP (VitisAI) _resolve_precision drops even an explicit per-model precision and the model is built with --no-quant, yet the result would still claim that precision. The build-reported value is now the sole source of truth -- pinned builds already report their own flag, so nothing is lost. _run_build reports the caller's resolved precision for pre-exported composite ONNX, where no build happens.
DingmaomaoBJTU
approved these changes
Jul 29, 2026
DingmaomaoBJTU
left a comment
Collaborator
There was a problem hiding this comment.
LGTM. Fixes precision reporting so auto-resolved quantization (e.g. NPU -> w8a16) is faithfully recorded instead of appearing as 'no precision'. Clean extraction of _merge_backfill_result, proper stale-precision cleanup, and comprehensive test coverage.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
run_evalwroteEvalJob.precision— the declared precision — intoeval_result.json. That value isNonefor anywinml configfallback job that pins no precision, but such a build is not precision-less:winml configresolvesautoto the device default, which on NPU isw8a16.So a
--device npu(pre-#1163) or--device auto --ep <npu ep>run produced a w8a16 QDQ model while the result recorded no precision at all, and the result dir carried no__<precision>suffix. The downstream coverage report maps a missing precision to theunquantizedlabel, so those quantized runs were published as unquantized baselines — models that never had an unquantized NPU run showed a PASS "unquantized" row.Verified against the published data:
openai__clip-vit-base-patch32__zero-shot-image-classification(no suffix, noprecisionfield) hasonnx_size_bytes= 178,466,841 ≈ 2 × the model's w8a16 artifact (89.6 MB), vs 253.9 MB for its fp16 build — i.e. the "unquantized" cell is a quantized run.Change
_run_buildreports the effective precision back to the caller: the pinned--precisionflag when there is one, otherwise read from the generated build config'squantsection (weight_type/activation_typebit widths ->w{x}a{y},mode="fp16"->fp16,mode="rtn"->int{bits}). Reading the config back is whatwinml configactually decided, so the harness never has to re-implement the CLI's precision policy._run_recipe_buildreports its authored variant precision.main()writes that effective precision intoeval_result.json(including the accuracy-backfill merge, which rebuilds the model).The job's declared precision still drives the result-dir slug and the log label, so folder naming,
--continueresume and the existing data layout are unchanged.Verification
Ran the harness on
prajjwal1/bert-tiny(perf only):precisiononnx_size_bytes--device cpu…__feature-extraction--device auto --ep openvino…__feature-extractionw8a16--device npu --ep openvino…__feature-extraction__w8a8w8a8(pinned)--device npu --ep openvino…__feature-extraction__w8a16w8a16(pinned)The second row is exactly the previously-broken case: same untagged folder, but the generated config is
weight_type=uint8, activation_type=uint16and the artifact is half the size. Before this change that file carried noprecisionat all.Recipe path (
microsoft/resnet-18, which hasfp16+w8a16recipes) recordsfp16/w8a16from the variant, as it already did.Note the real config uses
quant.mode = "static", not"qdq", which is why the mapping keys off the weight/activation bit widths rather than the mode.uv run --no-sync pytest tests/unit/eval-> 540 passed.