Skip to content

Verify Llama and Qwen3 inference evidence - #34

Merged
ToluClassics merged 1 commit into
mainfrom
agent/verify-llama-qwen
Jul 29, 2026
Merged

ToluClassics merged 1 commit into
mainfrom
agent/verify-llama-qwen

Conversation

@ToluClassics

Copy link
Copy Markdown
Owner

What changed

  • pin the exact Llama 3.2 1B and Qwen3 0.6B revisions behind the verified support claims
  • record successful public-CLI runtime affine 4-bit/group-64 generation for both checkpoints
  • add both models to the reproducibility log for the short-decode and prefill scenarios
  • refresh the Phi-3 rows on the merged benchmark implementation after its BOS-token fix
  • record the sanitized local environment and exact implementation revision

No package source code or raw benchmark result files are added.

Why

The benchmark runner was verified initially with a quantized Phi-3 checkpoint. This follow-up demonstrates that the same deterministic protocol and public inference APIs work for two actively maintained model families without expanding architecture scope.

Local Apple silicon evidence

All model resolution was strict offline against already-cached immutable snapshots.

Model Scenario Median TTFT Median prefill Median decode Peak MLX memory
Qwen3 0.6B short-decode-128 24.1 ms 2656.7 tok/s 72.4 tok/s 2.95 GiB
Qwen3 0.6B prefill-512-decode-64 98.6 ms 5192.9 tok/s 65.1 tok/s 4.10 GiB
Llama 3.2 1B short-decode-128 28.6 ms 2236.3 tok/s 51.9 tok/s 4.77 GiB
Llama 3.2 1B prefill-512-decode-64 155.9 ms 3284.2 tok/s 50.2 tok/s 5.66 GiB

Each scenario used one warmup and five measured runs. Generated-token checksums matched across every measured run.

Additional inference checks

  • Llama 3.2 1B full-precision bounded generation: passed
  • Llama 3.2 1B runtime affine 4-bit/group-64 bounded generation: passed
  • Qwen3 0.6B full-precision bounded generation: passed
  • Qwen3 0.6B runtime affine 4-bit/group-64 bounded generation: passed

Validation

  • exact PR Add reproducible MLX benchmark protocol #33 merge CI: passed
  • current environment offline suite: 125 tests, 103 passed, 22 skipped
  • minimum compatibility environment offline suite: 125 tests, 103 passed, 22 skipped
  • ruff check .
  • ruff format --check .
  • six protocol v1 result validations
  • wheel and source distribution build
  • twine check on both artifacts

@ToluClassics
ToluClassics marked this pull request as ready for review July 29, 2026 21:41
@ToluClassics
ToluClassics merged commit 5626d27 into main Jul 29, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant