[bri-prism]

What

Follow-up to #160. The PTQ1_0 MMQ tile path landed behind MMQ_PTQ1_0_MAX_BATCH_SIZE = 64, so the default ubatch (512) still took the fp16 dequantize + cuBLAS fallback. The cap is now effectively off (MMQ at every batch, like the other low-bit types), and GGML_CUDA_PTQ1_0_MMQ_MAX_BATCH=<n> restores a threshold as the A/B knob. HIP stays excluded by the existing guard.

Why

The fallback is where PTQ1_0's extra error on CUDA comes from. From the #160 review measurements on an L40S (CUDA 12.8), cuBLAS path vs MMQ at full prefill:

PTQ1_0	mean KLD, cuBLAS	mean KLD, MMQ	top-1, cuBLAS	top-1, MMQ
largest band	9.2e-4	5.6e-4	97.78%	98.39%
9B	7.7e-4	5.0e-4	98.26%	98.60%

With MMQ the largest PTQ1_0 band clears the top-1 certification gate it was the only shipped band to fail on CUDA, and lands at PQ2_0's level.

Cost: MMQ is about 7% slower than cuBLAS at pp512 on Ada (largest band 1689 -> 1578, 9B 5721 -> 4786, 4B 9249 -> 8361), and faster below roughly ubatch 400 (my own run on a second L40S: 9B ubatch 64 1,332 -> 3,392, 128 1,883 -> 4,232, 256 2,956 -> 4,761, 512 5,206 -> 4,739). Decode is unchanged either way.

Verification

Host-side selection change only; the kernel it enables is the one #160 gated (MUL_MAT 1283/1283 incl. 45 PTQ1_0, MUL_MAT_ID 1019/1019 incl. 75). The cap-lifted build was measured for KLD and throughput in the #160 review as quoted above.

[bri-prism]

One small thing on the env parse, not a blocker.

atoll returns 0 for anything it cannot read, so a typo like GGML_CUDA_PTQ1_0_MMQ_MAX_BATCH=64x or =256 with a stray character lands on 0. That routes every batch back to the fp16 dequantize plus cuBLAS path, which is the exact fallback this PR exists to remove. It also fails silently: no warning at load, and the accuracy regression only shows up in a certification run, so whoever set the variable would read the result as "MMQ at every batch is no better" and reach the opposite conclusion from the truth.

Falling back to the default on an unparseable value keeps that failure mode loud:

static const int64_t max_batch = [] {
    const char * s = getenv("GGML_CUDA_PTQ1_0_MMQ_MAX_BATCH");
    if (!s || !*s) {
        return (int64_t) MMQ_PTQ1_0_MAX_BATCH_SIZE;
    }
    char * end = nullptr;
    const long long v = std::strtoll(s, &end, 10);
    if (end == s || *end != '\0') {
        return (int64_t) MMQ_PTQ1_0_MAX_BATCH_SIZE;
    }
    return (int64_t) v;
}();

Cheap insurance on a knob whose wrong value is invisible.

[Copilot]

🟢 Approval recommended

The selection change is narrowly scoped and preserves existing architecture and HIP guards, with only a minor comment-style issue.

Pull request overview

Enables PTQ1_0 CUDA MMQ for effectively all batch sizes while providing an environment-variable cap.

Changes:

• Raises the default PTQ1_0 MMQ batch limit.
• Adds GGML_CUDA_PTQ1_0_MMQ_MAX_BATCH runtime configuration.

File summaries

File	Description
ggml/src/ggml-cuda/mmq.cuh	Raises the default MMQ batch threshold.
ggml/src/ggml-cuda/mmq.cu	Reads and applies the optional environment cap.

Review details

• Files reviewed: 2/2 changed files
• Comments generated: 1
• Review effort level: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.