ggml : speed up batch-1 CPU decode, align large allocations - #27478
ggml : speed up batch-1 CPU decode, align large allocations#27478matevz-kovacic wants to merge 2 commits into
Conversation
|
Hi @matevz-kovacic, thanks for your contribution! Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:
Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below. |
ggml-gh-bot
Bot
added
the
draft
github-actions
Bot
added
the
ggml
github-actions
Bot
removed
the
draft
Two changes to the same workload. ggml_compute_forward_flash_attn_ext_f16_one_chunk block the loop over KV. Therefore the softmax accumulator is rescaled at most once per block instead of on every new maximum. Consume Q directly in F32 where a vectorized mixed F16xF32 dot exists and K is F16. Compute four K.Q dots from one Q load, and accumulate an F16 V in F32 where a vectorized mixed multiply add exists. For Linux only: In ggml_aligned_malloc we align allocations of at least 4 MiB to 2 MiB and hint them with madvise(MADV_HUGEPAGE). If that fails we retry so it is never worse than before. Result for Qwen3-30B-A3B Q4_K_M: single sequence token generation at an 8192 token context and transparent huge pages in madvise mode: +15.29% on a Ryzen 7 9700X and +9.22% on a Neoverse-N1. If we measure the attention change alone the result is +10.67% and +7.43%. Under THP always those four figures are +12.00%, +6.61%, +7.52% and +5.57%. The two changes were measured separately as a 2x2 with THP mode as a third factor. Assisted-by: Claude Opus 5
4cf3846 to
69be7ac
Compare
| int i = 0; | ||
| float f0 = 0.0f, f1 = 0.0f, f2 = 0.0f, f3 = 0.0f; | ||
| #if defined(__AVX512F__) | ||
| __m512 a0 = _mm512_setzero_ps(), a1 = _mm512_setzero_ps(); |
There was a problem hiding this comment.
use macroses that simplify repetable code)
There was a problem hiding this comment.
Good point.
Loop body now uses the existing GGML_F32_VEC / GGML_F16_VEC macros instead of hand-written intrinsics. The AVX2 reduce that was written out four times is now one local macro, GGML_FA_HSUM.
The existing reduce isn't used because GGML_F32_VEC_REDUCE folds an array of accumulators into a single scalar, but this function needs four independent outputs.
The generated code is unchanged: gcc emits byte-identical .text on AVX-512, AVX2 and SSE2; clang differs only by one instruction's scheduling on AVX2.
2 participants
Overview
Bbatch-1 decode attention process KV one position at a time. It uses scalar expf per position, rescales accumulator when new max is fund and converts Q to F16 then widens back (x86). The loop-carried max/rescale is serialized back.
Tiled path can't be uilized. because it requires GGML_FA_TILE_Q query rows, but decode only has one.
This PR: vlocks the KV loop, rescaling once every 32 elements. Keeps Q in F32 where K is F16, and mixed F16xF32 dor exists.
Computes four K.Q dot products per Q load and accumulates V in F32 where a mixed multiply-add is used.
For Linux only: aligns allocations >=r 4MiB to 2MiB + madvise(MADV_HUGEPAGE), retrying at ordinary alignment on failure.
Also the fallback is implemented when tiling is not feasible (quantized K/V, mismatched K/V types, DV not a multiple of the vector width, 2..63 query rows); this is not evaluated in the measurements.
Additional information
Qwen3-30B-A3B Q4_K_M, tg at 8192-token context, THP madvise:
Correctness
Scope limits:
Related work
Commit 9b62573 advises the 2 MiB-aligned interior without changing alignment; it came in PR #22378, a draft closed unmerged
PR #22022 is open and complementary: it hints the mmap path for weights, which this change does not cover
Requirements
I used an AI coding assistant throughout this work, driven by Active Model, a private research harness of my own design (public result archive), with human review at every step. Specifically:
I directed the work, made the design and reporting decisions, reviewed each change before it was kept, and take full responsibility for everything submitted here. No part of this was submitted by an autonomous agent acting without my oversight.