Skip to content

[AIMIGRAPHX-1100] Add no-rebuild callback for verify#5067

Draft
eddieliao wants to merge 8 commits into
developfrom
verify_callback
Draft

[AIMIGRAPHX-1100] Add no-rebuild callback for verify#5067
eddieliao wants to merge 8 commits into
developfrom
verify_callback

Conversation

@eddieliao

Copy link
Copy Markdown
Contributor

Motivation

Technical Details

Changelog Category

Add a CHANGELOG.md entry for any option other than Not Applicable

    • Added: New functionality.
    • Changed: Changes to existing functionality.
    • Removed: Functionality or support that has been removed. (Compared to a previous release)
    • Optimized: Component performance that has been optimized or improved.
    • Resolved Issues: Known issues from a previous version that have been resolved.
    • Not Applicable: This PR is not to be included in the changelog.

Follow the LLVM AI Tool Use Policy for contributions using AI.

@eddieliao eddieliao self-assigned this Jul 15, 2026
@eddieliao eddieliao added enhancement New feature or request Tooling Touches tools used for or that aid in MIGraphX development and debugging labels Jul 15, 2026
@codecov

codecov Bot commented Jul 15, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #5067      +/-   ##
===========================================
+ Coverage    92.89%   92.90%   +0.01%     
===========================================
  Files          603      603              
  Lines        32448    32526      +78     
===========================================
+ Hits         30140    30217      +77     
- Misses        2308     2309       +1     
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@gh-app-migraphx-bot-pr-write

gh-app-migraphx-bot-pr-write Bot commented Jul 15, 2026

Copy link
Copy Markdown
Test Batch New Rate (867d18) Old Rate (939a41)* Diff Status
torchvision-resnet50 64 3,030.26 3,137.65 -3.42%
torchvision-resnet50_fp16 64 4,648.36 6,664.17 -30.25% 🔴
torchvision-densenet121 32 2,483.62 2,696.53 -7.90% 🔴
torchvision-densenet121_fp16 32 4,993.56 4,546.91 9.82% 🔆
torchvision-inceptionv3 32 2,057.98 1,789.44 15.01% 🔆
torchvision-inceptionv3_fp16 32 2,324.04 2,837.26 -18.09% 🔴
cadene-inceptionv4 16 821.72 820.89 0.10%
cadene-resnext64x4 16 428.50 584.90 -26.74% 🔴
slim-mobilenet 64 8,392.42 7,391.43 13.54% 🔆
slim-nasnetalarge 64 206.92 208.95 -0.97%
slim-resnet50v2 64 2,880.55 2,401.77 19.93% 🔆
bert-mrpc-onnx 8 1,169.02 888.28 31.61% 🔆
bert-mrpc-tf 1 503.11 369.82 36.04% 🔆
pytorch-examples-wlang-gru 1 476.71 482.91 -1.28%
pytorch-examples-wlang-lstm 1 377.25 436.18 -13.51% 🔴
torchvision-resnet50_1 1 1,038.39 699.41 48.47% 🔆
cadene-dpn92_1 1 451.50 442.31 2.08%
cadene-resnext101_1 1 363.83 364.07 -0.07%
onnx-taau-downsample 1 845.96 401.79 110.55% 🔆
dlrm-criteoterabyte 1 nan 31.86 nan
dlrm-criteoterabyte_fp16 1 51.75 51.73 0.04%
agentmodel 1 9,707.32 12,642.67 -23.22% 🔴
unet_fp16 2 58.78 57.08 2.98%
resnet50v1_fp16 1 1,316.11 943.42 39.50% 🔆
resnet50v1_int8 1 1,648.77 935.85 76.18% 🔆
bert_base_cased_fp16 64 1,097.85 1,072.35 2.38%
bert_large_uncased_fp16 32 344.98 342.15 0.83%
bert_large_fp16 1 204.49 80.24 154.84% 🔆
distilgpt2_fp16 16 2,089.62 859.88 143.01% 🔆
yolov5s 1 557.71 526.66 5.90% 🔆
tinyllama 1 45.79 45.81 -0.05%
vicuna-fastchat 1 44.12 44.09 0.07%
whisper-tiny-encoder 1 412.67 412.40 0.07%
whisper-tiny-decoder 1 408.88 410.21 -0.32%
llama2_7b 1 20.89 20.70 0.92%
qwen1.5-7b 1 23.54 23.58 -0.17%
phi3-3.8b 1 26.68 26.47 0.78%
llama3-8b 1 21.65 21.78 -0.63%
whisper-large-encoder 1 10.18 10.16 0.23%
whisper-large-decoder 1 105.35 105.84 -0.46%
mistral-7b 1 23.62 23.72 -0.39%
FLUX.1-schnell 1 772.03 771.10 0.12%

Regressions detected 🔴

* No develop baseline was found for this PR's branch point; compared against the latest available develop run instead.

@gh-app-migraphx-bot-pr-write

gh-app-migraphx-bot-pr-write Bot commented Jul 15, 2026

Copy link
Copy Markdown
Test Status Result
bert-mrpc-onnx PASSED: MIGraphX meets tolerance
bert-mrpc-tf PASSED: MIGraphX meets tolerance
pytorch-examples-wlang-gru PASSED: MIGraphX meets tolerance
pytorch-examples-wlang-lstm PASSED: MIGraphX meets tolerance
dlrm-criteoterabyte PASSED: MIGraphX meets tolerance
agentmodel PASSED: MIGraphX meets tolerance
unet PASSED: MIGraphX meets tolerance
resnet50v1 PASSED: MIGraphX meets tolerance
bert_base_cased_fp16 PASSED: MIGraphX meets tolerance
bert_large_uncased_fp16 🔴 FAILED: MIGraphX is not within tolerance - check verbose output
bert_large PASSED: MIGraphX meets tolerance
yolov5s PASSED: MIGraphX meets tolerance
tinyllama PASSED: MIGraphX meets tolerance
vicuna-fastchat PASSED: MIGraphX meets tolerance
whisper-tiny-encoder PASSED: MIGraphX meets tolerance
whisper-tiny-decoder PASSED: MIGraphX meets tolerance
distilgpt2_fp16 PASSED: MIGraphX meets tolerance
llama2_7b PASSED: MIGraphX meets tolerance
qwen1.5-7b PASSED: MIGraphX meets tolerance
phi3-3.8b PASSED: MIGraphX meets tolerance
llama3-8b PASSED: MIGraphX meets tolerance
whisper-large-encoder PASSED: MIGraphX meets tolerance
whisper-large-decoder PASSED: MIGraphX meets tolerance
mistral-7b PASSED: MIGraphX meets tolerance
FLUX.1-schnell PASSED: MIGraphX meets tolerance

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds a --no-rebuild verification mode to the MIGraphX driver that performs a single reference run and a single target run while tracing instruction outputs by debug symbol, enabling layer-wise divergence reporting without recompiling for --reduce/--bisect workflows.

Changes:

  • Adds verify_options::no_rebuild and wires --no-rebuild into the driver CLI.
  • Introduces a trace-based verify_callback that captures reference outputs by debug symbol and compares them against target outputs.
  • Updates verify flows (verify_program, reduced, bisected) to use the new layer-wise compare path when --no-rebuild is enabled.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.

File Description
src/driver/verify.hpp Declares verify_callback and exposes layer-wise compare support from the driver verify interface.
src/driver/verify.cpp Implements trace-based capture/compare, baseline error attribution, and integrates --no-rebuild into verify/reduce/bisect modes.
src/driver/verify_options.hpp Adds no_rebuild option to the verify options struct.
src/driver/main.cpp Adds --no-rebuild CLI flag and help text for verify.

Comment thread src/driver/verify.cpp
Comment thread src/driver/verify.cpp Outdated
Comment on lines +296 to +298
// Last-writer-wins on the symbol keeps the fused op's final value, not an interior one.
target_outputs[symbol] = {output.copy(), ins->name(), order, std::move(inputs)};
};
Comment thread src/driver/verify.cpp Outdated
Comment on lines +451 to +462
bool any_failed = false;
for(const auto& [symbol, lr] : vcb->results)
{
if(not lr.passed)
{
any_failed = true;
log::error() << "FAILED at " << lr.symbol << " (" << lr.op << ")";
}
}
if(not any_failed)
log::info() << "MIGraphX verification passed successfully.";
return;
Comment thread src/driver/verify.hpp Outdated
Comment on lines +49 to +53
// Captures ref outputs by debug symbol and compares each target op at its terminal symbol.
struct verify_callback
{
using trace_function = std::function<void(instruction_ref, const argument&)>;

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request Tooling Touches tools used for or that aid in MIGraphX development and debugging

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants