[zaneni6]

What

A new workflow, D3 Test (.github/workflows/d3-test.yml). It asks d3:lite 60 MMMU-Pro vision questions, drawn with a fixed seed (0), through POST /v1/systemone. It runs on the linux, linux-hrx, windows and windows-hrx portable builds that "Build Linux Portable" and "Build Windows Packages" produce for the commit. It follows asr-test.yml: the job that waits for the builds is the same, and on a pull request main's latest build runs first on the same machine as the baseline.

• build_d3_mmmu_eval.py runs on the runner. It downloads MMMU/MMMU_Pro (config vision, split test) into the Hugging Face cache (/scratch/huggingface on Linux, the profile's default cache on Windows). It keeps the questions d3 can be asked: a PNG/JPEG/WebP image of at most 8 MB, 2–26 options, and an answer among them. It sorts them by id, draws 60 with random.Random(seed), and records the dataset commit in manifest.json.
• d3_mmmu_test.py sends one request per question. The screenshot is the only image, and the question is a choice over the option letters with no option text, as in MMMU-Pro's vision setting.
• d3_mmmu_summary.py writes one job summary with the platforms side by side.

When a run fails

Check	When
Every question gets a 200 reply with a choice over exactly its letters, and probabilities that sum to 1	always
Every question's input is longer than the same question without the image, so the image reached the prompt	always
At least 90% of the questions get the same answer as main's build	PRs
No more than 3 fewer questions are right than on main's build	PRs

d3-lite is close to chance on this set (the benchmark page has 15.0% for the NPU), so accuracy alone says little. Agreement with main's build is the check that catches a broken image path. Pushes to main and the weekly run (Tuesdays) have no baseline, so they only require every question to be answered and report the accuracy.

The trigger paths are the decision model and its contract, the d3_npu libraries and D3-Lite-NPU2 kernels, the image reader and processor, FlmImageCodecs.cmake, the server files that host /v1/systemone, and the test scripts. This PR changes the workflow itself, so it runs here.

Not yet verified

• Not yet run on NPU hardware. I tested the scripts locally: the question draw against synthetic Parquet files, and the test against a mock /v1/systemone. A clean run and a run against a baseline both passed. A server that dropped images and a build whose answers drifted both failed. Drawing twice gave the same questions.
• Dataset layout is unconfirmed. The script assumes vision/test-*.parquet with id, options, answer and image columns, and stops with a clear error if they aren't there.
• The dataset version isn't pinned. D3_MMMU_REVISION is main. Once this has run, it should be pinned to the commit printed in the log. A PR and its baseline always get the same questions, because one job draws them for both.
• The limits are starting values. The 90% agreement and the 3-question allowance should be adjusted after the first runs, especially on HRX.
• Accuracy won't match the benchmark page. The prompt wording and the seed-0 sample are not the Decision Index suite's.

🤖 Generated with Claude Code