Skip to content

Fix QNN E2E evaluation failures - #1247

Open
DingmaomaoBJTU wants to merge 7 commits into
mainfrom
dingmaomaobjtu-run-e2e-tests
Open

Fix QNN E2E evaluation failures#1247
DingmaomaoBJTU wants to merge 7 commits into
mainfrom
dingmaomaobjtu-run-e2e-tests

Conversation

@DingmaomaoBJTU

@DingmaomaoBJTU DingmaomaoBJTU commented Jul 28, 2026

Copy link
Copy Markdown
Collaborator

QNN E2E started failing after transformers 5 changed CLIP export to produce per-layer SDPA attention mask guard graphs that include IsNaN and Where. Those graphs are rejected by the QNN backend during validate/finalize, so this fixes the export regression instead of disabling the affected eval targets.

Summary

  • Add a reusable export context manager in transformers_compat.py that temporarily switches HF-style configs exposing _attn_implementation to eager attention during torch.onnx.export, then restores the original values afterward.
  • Wrap the HTP ONNX export path with that context so transformers 5 exports avoid the QNN-incompatible SDPA guard pattern without model ID, architecture, node, or op-name hardcoding.
  • Stabilize QNN E2E handling by classifying explicit QNN backend validation/finalize rejections as unsupported failures, tightening registry target filtering, and covering the related subprocess/template behavior with tests.
  • Treat Windows native access violation exits from winml build as non-fatal only when the completed ONNX artifact can be resolved, so post-build teardown crashes do not hide usable artifacts.

Validation

  • Targeted exporter, transformers compat, and run_eval unit pytest coverage passed.
  • winml sys confirms local QNNExecutionProvider registration for NPU/GPU/CPU.
  • openai/clip-vit-base-patch32 QNN GPU feature-extraction and zero-shot-image-classification E2E perf passed.
  • google-bert/bert-base-multilingual-cased QNN GPU masked-lm and fill-mask E2E perf passed.
  • google-bert/bert-base-multilingual-cased QNN NPU fill-mask E2E perf passed for fp16 and w8a16.
  • google/flan-t5-base QNN GPU E2E perf passed.

@DingmaomaoBJTU
DingmaomaoBJTU requested a review from a team as a code owner July 28, 2026 11:54
github-actions Bot and others added 2 commits July 28, 2026 20:19
Parse wrapped winml build artifact paths so successful builds are not misreported as export failures, and classify QNN backend graph-finalization rejections as unsupported perf skips instead of relying on registry target blacklists.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Temporarily switch HF-style attention configs to eager only while torch.onnx.export runs, then restore the original attention implementation. This keeps transformers 5 exports compatible with QNN by avoiding SDPA mask guard graphs.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@DingmaomaoBJTU DingmaomaoBJTU changed the title Fix QNN E2E CI failures Fix QNN E2E evaluation failures Jul 29, 2026
github-actions Bot and others added 2 commits July 29, 2026 09:10
Move the eager-attention export helper into transformers_compat so transformers 5 export compatibility behavior stays in one module while preserving lazy import behavior.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Treat Windows native access violation exits from winml build as non-fatal only when the completed ONNX artifact can be resolved from the build output. This preserves real build failures while allowing QNN eval jobs to continue after interpreter teardown crashes.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant