ggml-cpu(s390x): add repack support for q4_0

[taronaeo]

Overview

This PR introduces weights repack support for Q4_0 on s390x. We are seeing an approximate 1.60x performance improvement for Prompt Processing and 1.10x for Token Generation.

Additional information

$ build/bin/llama-bench -hf taronaeo/Granite-3.0-1B-A400M-Instruct-BE-GGUF:Q4_0 -lm mlock

CPU	Model	Test	t/s 91f6a6c	t/s feat/s390x-q4_0-repack	Speedup
CPU	granitemoe 1B.A400M Q4_0	pp512	38.16	60.23	1.58
CPU	granitemoe 1B.A400M Q4_0	tg128	16.44	18.03	1.10

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES, Claude Code + Opus 5

[bernardladenthin]

Heads-up: this PR broke the build for s390x targets without VXE. It is still broken on master (a2878d30d).

What happens

The three helpers vxe_dot_acc / vxe_splat_granule / vxe_fold sit at file scope in
ggml/src/ggml-cpu/arch/s390/repack.cpp (master lines 73, 77, 83) — i.e. between the guarded
blocks at 28–70 and 100–155, but outside both. Their signatures name int16x8_t / int8x16_t /
int32x4_t, which ggml-cpu-impl.h only typedefs inside #if defined(__VXE__) || defined(__VXE2__).
So without VXE the file fails before reaching any of the code it is meant to skip:

repack.cpp:73:15: error: 'int16x8_t' does not name a type; did you mean 'int16_t'?
repack.cpp:77:15: error: 'int8x16_t' does not name a type; did you mean 'int16_t'?
repack.cpp:83:15: error: 'int32x4_t' does not name a type; did you mean 'int32_t'?

ggml-cpu/CMakeLists.txt adds the file to GGML_CPU_SOURCES for every s390x build, so this is a
hard compile error, not a missing optimisation.

It hits a documented configuration

docs/build-s390x.md documents -DGGML_VXE=OFF with a copy-pasteable command (the "By default,
VXE/VXE2 is enabled. To disable it (not recommended)" block), and further down states:

No hardware acceleration is possible with llama.cpp with older systems, such as IBM z14/arch12.
In such systems, the APIs can still run but will use a scalar implementation.

That documented build does not compile any more. Reproduced with s390x-linux-gnu-g++ using
exactly the flags a native GGML_VXE=OFF build produces — -march=z15 and no -mvx -mzvector,
since only -mzvector defines __VEC__, which is what gates the __VXE__/__VXE2__ self-define
in ggml-cpu-impl.h. Same three errors. The sibling file arch/s390/quants.c compiles clean under
the identical flags, which is what makes this look like an oversight rather than a decision.

Why CI did not catch it

Both s390x jobs build with VXE2, so neither reaches the #else path:

• build-ibm.yml → ubuntu-24-s390x runs on a native ubuntu-24.04-s390x runner and passes no
  GGML_NATIVE=OFF, so GGML_NATIVE stays on → GGML_VXE follows it on, and the if (GGML_NATIVE)
  branch reads /proc/cpuinfo and sets -march=z15/z16.
• release.yml passes -DGGML_NATIVE=OFF -DGGML_CPU_ALL_VARIANTS=ON, and the only s390x variants
  registered are ggml_add_cpu_backend_variant(z15 Z15 VXE2) and (z16 Z16 VXE2 NNPA).

There is no scalar s390x configuration anywhere in CI.

Worth noting that this PR clearly intended the non-VXE path to work: each function has an #else
branch calling ..._generic(...), and the PR removes exactly those three _generic aliases from
arch-fallback.h. Only the helper definitions slipped out of the guard.

Fix

Wrapping the three definitions in the same guard is sufficient — all 21 call sites are already
inside __VXE__ blocks:

--- a/ggml/src/ggml-cpu/arch/s390/repack.cpp
+++ b/ggml/src/ggml-cpu/arch/s390/repack.cpp
@@ -70,6 +70,7 @@ void ggml_quantize_mat_q8_0_4x4(const float * GGML_RESTRICT x, void * GGML_RESTR
 #endif
 }
 
+#if defined(__VXE__) || defined(__VXE2__)
 static inline int16x8_t vxe_dot_acc(const int8x16_t v_x, const int8x16_t v_y, const int16x8_t v_acc) {
     return vec_meadd(v_x, v_y, vec_moadd(v_x, v_y, v_acc));
 }
@@ -84,6 +85,7 @@ static inline int32x4_t vxe_fold(const int16x8_t v_sumi) {
     const int16x8_t v_ones = vec_splats((int16_t)1);
     return vec_add(vec_mule(v_sumi, v_ones), vec_mulo(v_sumi, v_ones));
 }
+#endif  // __VXE__ || __VXE2__
 
 void ggml_gemv_q4_0_4x4_q8_0(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, const void * GGML_RESTRICT vy, int nr, int nc) {
     const int qk = QK8_0;

Verified against the pristine source: unpatched + the flags above reproduces the three errors;
patched + the same flags compiles clean; patched + -mvx -mzvector -march=z15 also compiles clean,
so the VXE path is unaffected.

We carry this downstream for now — the patch file and the full write-up (including the measured
comparison against -DGGML_VXE=ON and -DGGML_VXE=ON -march=z15) are here:

• patch: https://github.com/bernardladenthin/java-llama.cpp/blob/894fe541cc28d82e38758d5c6d66aa579760d2d9/llama/patches/0013-s390x-repack-guard-vxe-only-helpers.patch
• context: fix(s390x): guard the VXE-only helpers upstream left at file scope bernardladenthin/java-llama.cpp#432

Found while cross-compiling s390x from x86 for a downstream JNI binding, which passes
-DGGML_NATIVE=OFF without GGML_CPU_ALL_VARIANTS — a third path where neither -march branch in
the s390x block fires, so no -march is set at all and GGML_VXE defaults off with GGML_NATIVE.

Happy to open a PR with the above if that is useful.

---

AI usage disclosure: Claude Code + Opus 5 (investigation, reproduction and this write-up).

[taronaeo]

It is still broken on master (a2878d3).

@bernardladenthin By 'broken' you mean that the build is failing, and not that the logits computed are wrong right? And yes, I missed out those guards, so please feel free to open a PR :)

Edit: I had also intended to refactor those helper functions into ggml/src/ggml-cpu/ggml-cpu-impl.h in the near future but for now please go ahead with the PR.

[bernardladenthin]

PR is open: #28775. And agreed, moving them into ggml-cpu-impl.h would be better. thanks!