bernardladenthin · bernardladenthin · Sep 5, 2026 · Sep 5, 2026
@@ -1410,7 +1410,7 @@ jobs:
         shell: bash
         run: |
           mvn --no-transfer-progress -f llama/pom.xml compile
-          .github/build.sh -DLLAMA_METAL=OFF -DGGML_NATIVE=OFF -DBUILD_TESTING=ON
+          .github/build.sh -DGGML_METAL=OFF -DGGML_NATIVE=OFF -DBUILD_TESTING=ON
       - name: Run C++ unit tests
         run: ctest --test-dir llama/build --output-on-failure
       - name: Upload artifacts

@@ -691,6 +691,7 @@ Used during `llama.cpp` version bumps: when upgrading, scan this file from the r
 | b10792–b10797 | `ggml/src/ggml-cpu/arch/s390/quants.c` (**#28332: `q5_1` used an uninitialized `v_acc`**), `ggml/src/ggml-sycl/**` (#27610 rms_norm+mul+add / add+add fusion, #26863 `GGML_SYCL_ENABLE_MKL_FA` made a global), `src/llama-hparams.{h,cpp}` + `src/llama-model.cpp` + `src/llama-model-loader.cpp` (**#28323: new internal `n_expert_used_max()`**), `src/CMakeLists.txt` + new `src/llama-version.h.in` (**#28278: stop rebuilding the whole source tree on every commit**), `src/llama.cpp`, `tests/test-backend-ops.cpp`, `docs/backend/SYCL.md` | **No project-source change, and this time the review surface is literally empty**: the range touches **zero** files under `common/`, `include/`, `tools/server/`, `tools/mtmd/`, `ggml/include/` or the top-level `CMakeLists.txt`. Server contract re-checked anyway and byte-identical (request fields, response keys). Two entries are still worth recording. **(1)** #28332 is a **big-endian correctness fix on a platform this project ships in the default JAR**: `ggml_vec_dot_q5_1_q8_1`'s s390x path accumulated into an uninitialized `v_acc`, so `Q5_1` inference on IBM Z could return garbage. The `build-linux-s390x` job cross-builds and runs the C++ suite under `qemu-user`, but that suite covers *our* endian-sensitive layer (`tts_wav.hpp`, the JSON/token transforms), not ggml's kernels — so this is precisely the class of bug the job cannot see and upstream has to fix. **(2)** #28278 moves `LLAMA_VERSION`/`LLAMA_COMMIT` out of `target_compile_definitions(llama PRIVATE ...)` into a `configure_file`-generated `src/llama-version.h`. That is the one change in the range that could plausibly have reached the project, since `NativeLibraryLoadSmokeTest.nativeBuildInfoMatchesPinnedVersionConstant` asserts the pin against the linked binary — but `jllama.cpp` reads `llama_build_info()` from `common/build-info.cpp`, a different mechanism, and the test passes at b10797. Verified rather than reasoned about. |
 | b10792–b10797 | patches + upstream verification | **Zero intersection with the patch set, established mechanically.** The 42 files the eight patches touch were intersected against the range's 18 changed files: empty, so no patch context can have moved. Confirmed by running the applier for real anyway: fresh `rm -rf build && cmake -B build -DBUILD_TESTING=ON`, configure clean, stamp at `head 832fd6f1741ad9f66fb2a785002396700666595a`, all eight hashes recorded. **This step needed no chunking decision**: the full `b10792...b10797` diff is 46 KB across 5 commits, comfortably inside the runbook's 100 KiB threshold, so it is a straight bump by the rule rather than in spite of it. |
 | b10797–b10817 | `common/build-info.{h,cpp.in}` (**#28322: `llama_print_build_info` gains a `FILE *` stream**), `ggml/include/ggml-backend.h` (**REMOVES `ggml_backend_op_alloc_size_may_expand`**, via the #28379 ggml sync), `tools/mtmd/mtmd.cpp` + `clip.cpp` (**#28335: gemma4 vision handling; image-token limit 40/280 → 70/1120**), `CMakeLists.txt` (`LLAMA_VERSION_MINOR` 3 → 4, #28386), `vendor/cpp-httplib/CMakeLists.txt` (BoringSSL pin, #28354). Everything else in the range is GPU-backend work (`ggml-opencl` +940 and a new 871-line Adreno SDPA kernel, Metal fa-vec tunings for M3/M3 Max, SYCL memtrace), a new model architecture (`src/models/hy-v4.cpp`, #28127), upstream CI and the WebUI. | **No project-source change.** The whole review surface is **7 files, +12 / −13**. Two entries needed checking rather than skimming. **(1)** `llama_print_build_info(const char *)` became `llama_print_build_info(const char *, FILE * = stderr)` — a **default argument**, so any existing one-argument call still compiles; the project does not call it at all and reads `llama_build_info()`, which is untouched. Worth noting anyway because `build-info` is what `NativeLibraryLoadSmokeTest` cross-checks against the pin, and this is the second consecutive range to move that file (b10792–b10797 carried #28278). **(2)** the **removal** of `ggml_backend_op_alloc_size_may_expand` from a public ggml header — a deletion is the one shape that breaks a build with no diff hunk to notice, so the project source was grepped for it: zero references, in `src/main/cpp` and `src/test/cpp` alike. The server contract was re-checked and is byte-identical (request fields, `set_hard_limits` bounds, response keys in both emit forms); `tools/server/` has no `.cpp`/`.h` change in the range at all. |
-| b10797–b10817 | patches + upstream verification | **Zero intersection with the patch set.** The 42 files the eight patches touch were intersected against the range's changed-file list: empty, so no patch context can have moved. The applier was run for real anyway — fresh `rm -rf build && cmake -B build -DBUILD_TESTING=ON`, configure clean, stamp at `head cd8cdf397dc21d986e69069ce2180651d693fdff`, all eight hashes recorded. **Chunking:** the full diff is 307 KB, over the runbook's 100 KiB threshold, but excluding `ggml/src/**`, `tools/ui/**`, `docs/**`, `.github/**` and `conversion/**` — every one of them outside the review surface the rule exists to bound — it is **68 KB, under the threshold**. Same call as the previous bump, and this time the excluded-diff figure alone justifies it. **This range does NOT address the macOS-15 regression introduced at b10797** (see the two rows above): `src/llama-model-loader.cpp` is untouched here and `hparams.n_expert_used_max()` — the prime suspect from #28323 — is still called at both b10797 sites. Expect `Java Tests macOS 15 arm64` to stay red until that is bisected. |
-| b10817–b10819 | `ggml/src/ggml-metal/ggml-metal-context.m` (**#28399: one-line memory-leak fix on an early-return path**), `ggml/src/ggml-sycl/fwht.cpp` (#28254: restore Kronecker-product FWHT support and unbreak `test-backend-ops` on SYCL), `tests/test-backend-ops.cpp`. | **No project-source change, and the review surface is empty**: all three changed files are ggml backends or upstream tests, none of which this project links against or compiles into `jllama`. The server contract was re-checked and is byte-identical (request-field set, response keys in both emit forms); `tools/server/`, `common/`, `include/`, `tools/mtmd/` and `ggml/include/` have no change at all in the range. The Metal leak fix is worth a note next to the **macOS-15 regression** tracked from b10797: that job logs `ggml_metal_log_allocated_size: warning: current allocated size is greater than the recommended max working set size`, so a leaked Metal object on an early return is at least adjacent to the symptom. It is **not** a fix for it — the observed failure is a libc++ `"vector"` exception thrown ~30 ms into `llama_model_load` of the draft model, before any allocation of that kind, and the no-Metal macOS job fails identically. Treat it as one variable removed, not as the answer. |
+| b10797–b10817 | patches + upstream verification | **Zero intersection with the patch set.** The 42 files the eight patches touch were intersected against the range's changed-file list: empty, so no patch context can have moved. The applier was run for real anyway — fresh `rm -rf build && cmake -B build -DBUILD_TESTING=ON`, configure clean, stamp at `head cd8cdf397dc21d986e69069ce2180651d693fdff`, all eight hashes recorded. **Chunking:** the full diff is 307 KB, over the runbook's 100 KiB threshold, but excluding `ggml/src/**`, `tools/ui/**`, `docs/**`, `.github/**` and `conversion/**` — every one of them outside the review surface the rule exists to bound — it is **68 KB, under the threshold**. Same call as the previous bump, and this time the excluded-diff figure alone justifies it. This range does not address the macOS-15 failure; see the investigation row at the end of this table, which **refutes** the #28323 suspicion recorded here and shows the b10792–b10797 window itself was never established. |
+| b10817–b10819 | `ggml/src/ggml-metal/ggml-metal-context.m` (**#28399: one-line memory-leak fix on an early-return path**), `ggml/src/ggml-sycl/fwht.cpp` (#28254: restore Kronecker-product FWHT support and unbreak `test-backend-ops` on SYCL), `tests/test-backend-ops.cpp`. | **No project-source change, and the review surface is empty**: all three changed files are ggml backends or upstream tests, none of which this project links against or compiles into `jllama`. The server contract was re-checked and is byte-identical (request-field set, response keys in both emit forms); `tools/server/`, `common/`, `include/`, `tools/mtmd/` and `ggml/include/` have no change at all in the range. The Metal leak fix sits next to the **macOS-15 failure** and is not a fix for it; the investigation row below supersedes the reasoning recorded here, including the claim that "the no-Metal macOS job fails identically" (there was no no-Metal job). |
 | b10817–b10819 | patches + upstream verification | **Zero intersection with the patch set**: the 42 files the eight patches touch versus the range's 3 changed files — empty, so no patch context can have moved. The applier was run for real regardless: fresh `rm -rf build && cmake -B build -DBUILD_TESTING=ON`, configure clean, stamp at `head 6a1a922d269908a29cbd4b49c27e6a8e7fd10fae`, all eight hashes recorded. **No chunking question arises**: the full diff is 17 KB across 2 commits, far inside the runbook's 100 KiB threshold, so this is a straight bump *by* the rule rather than an exception to it. |
+| **macOS-15 failure — investigation, 2026-09-05** | `Java Tests macOS 15 arm64 (Metal)` + `(no Metal)`, run **33861339600** (#897, dispatch, pin b10797) | **The #28323 lead is refuted, and the b10792–b10797 bisect window it rested on was never established.** Read this row before spending time on the earlier suspicion recorded two rows up. **What actually fails.** Two tests, `MemoryManagementTest#testPromptCacheCompleteMissAfterWarmup` and `LlamaModelTest#testSpeculativeDecoding`, both with `LlamaException: could not load model from given file path` over a native `llama_model_load: error loading model: vector`. The recorded symptom ("the draft model") is **half wrong**: the first failure is **codellama-7b.Q2_K.gguf**, the second the AMD-Llama-135m draft — so it is not size-specific. Both are *repeat* loads: the same 7B model had already loaded and generated successfully dozens of times in the same JVM over the preceding ten minutes. `what() == "vector"` is libc++'s message for `std::out_of_range` from `vector::at()` and for `vector`'s `length_error`; the 33 ms elapsed (10.06.078 → 10.06.111) and the position right after the vocab warnings put it in `load_hparams`/`load_tensors`, not in the big allocation phase. **Why #28323 cannot be it — three independent reasons.** `n_expert_used_arr` is a `std::array<uint32_t, LLAMA_MAX_LAYERS>` read through `operator[]` behind an `il < n_layer_all` guard, so it cannot raise `out_of_range("vector")` (and its failure mode would be `GGML_ABORT`, which aborts rather than throws). Both failing models are dense (`n_expert == 0`), so the two changed call sites in `weight_buft_supported` (`GGML_OP_MUL_MAT_ID`, `GGML_OP_ADD_ID`) are unreachable and the `load_tensors` guard short-circuits on `n_expert > 0` before ever calling it. And the change is a **widening** — `max(il)` ≥ `[0]` — so it cannot narrow a value into an out-of-range one. **The "no Metal" job has never been a no-Metal job**, which invalidates the "both macOS jobs fail identically, so Metal is not involved" reasoning. It passed `-DLLAMA_METAL=OFF`, and upstream's `llama_option_depr(WARNING LLAMA_METAL GGML_METAL)` forwards only `if (${OLD})` — i.e. it acts on `ON` and **silently ignores `OFF`**, leaving `GGML_METAL` at its `APPLE` default of `ON`. CMake emits no "unused variable" warning either, because the variable *is* read; it just has no effect. Both macOS-15 jobs were therefore the same Metal build, which is why their logs are identical (44 `MTL0`/`ggml_metal` lines in the supposedly Metal-free one) and why "identical failure" carried no information. Fixed in this change set by passing `-DGGML_METAL=OFF`; the two `-DLLAMA_METAL_EMBED_LIBRARY=ON` sites are unaffected, since the shim does forward a truthy value. **The bisect window is wrong, and much wider than 5 commits.** The last CI observation of the macOS *Java test* jobs passing is run **33275073456** (#875, the v5.1.0 release dispatch, 2026-08-29) at pin **b10618**. b10731 was never observed: run #887 (dispatch, b10731) failed at `Code style (spotless) + package graph`, which skipped everything downstream — 30 jobs, no `Java Tests macOS` among them — and every other run between #875 and #897 was cancelled. So the regression window is **b10618 → b10797**, ~180 upstream builds, not b10792 → b10797. **Memory pressure is not the discriminator**, checked rather than assumed: the *green* run #875 logged **8** `ggml_metal_log_allocated_size: current allocated size is greater than the recommended max working set size` warnings and more free pages than the red run, which logged **6**. The 7 GB runner is tight in both. **Not resolved.** No bisect was run: it needs Apple-silicon hardware or one `workflow_dispatch` per step, neither available in the sandbox this was investigated from. The decisive next steps, in order: (1) dispatch one run on the current `main` (b10819) — nothing on it has been validated by CI at all, so even the premise "still red" is unverified; (2) with the flag fixed, compare the now-genuinely-Metal-free job against the Metal one, which is the experiment the old flag silently prevented; (3) only then bisect, over **b10618…b10797**. |