diff --git a/CHANGELOG.md b/CHANGELOG.md index cb6fda20..40d5dbe8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,44 @@ The release procedure (prompt template and step-by-step instructions) lives in [ --- +## [Unreleased] + +### Added +- **`flashAttn` works.** The knob was documented and settable since it was introduced, but could not + be forwarded: `--flash-attn` takes a mandatory `on|off|auto` and `net.ladenthin:llama` offered only + a bare-flag setter, which emitted the key alone and made llama.cpp's parser consume the following + argv token — the load then died naming a flag the user never set. 1.2.0 shipped a refusal at plan + time rather than that diagnostic. + + `net.ladenthin:llama` **5.2.0** adds `ModelParameters.setFlashAttn(FlashAttn)`, so the provider now + emits `--flash-attn on` when the knob is set. `false` still emits nothing, which leaves llama.cpp's + own `auto` default in force — that is the correct behaviour for an unconfigured run, and it is what + the knob's javadoc now says. + + **The knob is exercised against a real model for the first time.** `flashAttn` moves out of + `LlamaCppJniKnobSweepTest`'s `NOT_SWEPT` list into a real sweep case; until now only the refusal had + ever been executed. + +### Removed +- The plan-time and provider-side refusals (`EngineSupport.validateFlashAttnIsNotRequested`, + `LlamaCppJniAiGenerationProvider.FLASH_ATTN_UNSUPPORTED_MESSAGE`) and the three tests that pinned + them. + +### Changed +- **`lazyMode` replaces `tensorReadLazy` — breaking.** llama.cpp b10731 renamed `--tensor-read-lazy` + to `-lzm` / `--lazy-mode` with no alias, and `net.ladenthin:llama` followed the rename rather than + papering over it, so the model-definition field, its `srcmorph.llama.*` surface, the + `LlamaCppJniConfig` accessor and the sweep case move with it. Carrying the old name would leave the + configuration describing a flag that no longer exists. + + Worth knowing why this is a real hazard and not cosmetics: a **new Java name paired with an old + native** produces `Failed to parse model parameters` at load time and nothing earlier catches it. + The knob sweep did — it is what surfaced the mismatch here. + +- **`net.ladenthin:llama` 5.1.0 → 5.2.0-SNAPSHOT.** Deliberately a snapshot: the binding change this + release depends on is not yet published. Building srcmorph therefore requires that snapshot to be + resolvable, and CI stays red until it is — recorded here so nobody mistakes it for a regression. + ## [1.2.0] - 2026-09-01 ### Added diff --git a/TODO.md b/TODO.md index 7c813f52..347d20f3 100644 --- a/TODO.md +++ b/TODO.md @@ -47,7 +47,7 @@ recorded in git history and `crossrepostatus.md`, not here. its fix: the class generates no mutants, so neither the bug nor the tests that now cover it move the number. Do not read that stability as reassurance; PIT structurally could not have caught this. It is now worth adding, because the class finally has model-free coverage of the parts that matter - (`buildInferenceParameters`, `warnOnTruncatedAnswer`, `logPromptCacheReuse`, `tensorReadLazyMode`, + (`buildInferenceParameters`, `warnOnTruncatedAnswer`, `logPromptCacheReuse`, `lazyMode`, `cacheType`). **Measure before committing to it**: the real-model tests take ~4-16 s each, and PIT re-runs every test covering a mutated line, so mutants in `model()` — the long `ModelParameters` chain — could make the run far slower than the current few minutes. If it does, the answer is @@ -55,33 +55,6 @@ recorded in git history and `crossrepostatus.md`, not here. not dropping the idea. Deliberately out of scope for 1.2.0: it is a build-time question, not a correctness one. -- **Wire `flashAttn` up for real after `net.ladenthin:llama` 5.2.0 ships.** 1.2.0 *refuses* the knob - rather than forwarding it, in two places (`LlamaCppJniAiGenerationProvider.model()` for the direct - API path, `EngineSupport.validateFlashAttnIsNotRequested` for the plan phase), because the binding - cannot currently spell it: `-fa` takes a mandatory `[on|off|auto]`, `enableFlashAttn()` routes - through `setFlag` which stores `null`, and the argv renderer then emits the key with no token after - it — so llama.cpp swallows the *next* argv token and the load dies naming a flag the user never set - (`error: unknown value for --flash-attn: '--reasoning-format'`, reproduced against the shipped fat - jar). No workaround exists downstream: `putScalar` is `protected`. - - When 5.2.0 lands with a value-taking setter: bump `llama.version`, replace both guards with the real - call, delete `FLASH_ATTN_UNSUPPORTED_MESSAGE` and the four tests that pin the refusal (two in - `EngineSupportTest`, one in `LlamaCppJniAiGenerationProviderTest`, and - `LlamaCppJniKnobSweepTest#flashAttn_isRefusedRatherThanSilentlyDropped`), and restore the - `flashAttn`/`cacheTypeV` guidance in `srcmorph-maven-plugin/README.md`. Then move `flashAttn` out of - `LlamaCppJniKnobSweepTest.NOT_SWEPT` into a real sweep case — that is the step that finally sets the - knob against a real model, which the refusal has never done. The sweep's completeness check keeps - the bookkeeping honest: it fails if the knob is neither swept nor excluded, so it cannot be left - half-migrated. - - **The trap to avoid: bumping `llama.version` alone changes nothing visible.** The guards keep - throwing, the knob keeps looking broken, and every gate stays green — so this has to be an explicit - line on the bump checklist rather than something a version bump surfaces on its own. The upstream - fix is tracked in java-llama.cpp (`ModelParameters` needs `setFlashAttn(on|off|auto)` mirroring the - `CacheType` enum pattern, `enableFlashAttn()` deprecated, and - `ModelParametersExtendedTest.testToArrayComplexCombination` corrected — it currently pins the broken - 9-token argv shape as correct). - - **`enable_thinking` is sent unconditionally, including at its own default.** `LlamaCppJniAiGenerationProvider.model()` always puts `enable_thinking` into `chatTemplateKwargs`, at whatever `chatTemplateEnableThinking` says — and its default is `true`. diff --git a/srcmorph-maven-plugin/README.md b/srcmorph-maven-plugin/README.md index 8b2e6daf..6a17cee2 100644 --- a/srcmorph-maven-plugin/README.md +++ b/srcmorph-maven-plugin/README.md @@ -439,7 +439,7 @@ below are the shipped values (`AiGenerationConfig.DEFAULT_*`). | `batchSize` | `-1` | Logical maximum batch size (`--batch-size`); `-1` = leave default. A prefill-throughput knob, and prefill is what dominates an indexing run: every file is one large prompt with a short answer. Must be positive when set | | `ubatchSize` | `-1` | Physical maximum batch size (`--ubatch-size`); `-1` = leave default. Bounds how much of a logical batch reaches the backend at once, so it trades prefill speed against peak memory. Must be positive when set | | `threadsBatch` | `-1` | Threads for batch/prompt processing (`--threads-batch`); `-1` = reuse `threads`. Prefill parallelises differently from decode, so the optimum often is not the decode thread count — on a machine with efficiency cores, more threads help prompt processing while hurting generation. Must be positive when set | -| `tensorReadLazy` | *(empty)* | On-demand reading of lazy-loadable tensors (`--tensor-read-lazy`): `off`, `auto` or `on`; empty leaves the default. Trades resident memory for disk reads and shortens model load time; **requires mmap** | +| `lazyMode` | *(empty)* | On-demand reading of lazy-loadable tensors (`--tensor-read-lazy`): `off`, `auto` or `on`; empty leaves the default. Trades resident memory for disk reads and shortens model load time; **requires mmap** | | `mainGpu` | `-1` | Primary GPU index (`--main-gpu`); `-1` = leave default. Matters on multi-GPU hosts (e.g. a Vulkan build enumerates every GPU) | | `devices` | *(empty)* | Explicit device selection (`--device`), comma-separated backend device names (e.g. `Vulkan1`); takes precedence over `mainGpu` | | `chatTemplateEnableThinking` | `true` | Enable the chat template's thinking mode | diff --git a/srcmorph/pom.xml b/srcmorph/pom.xml index c0d176cd..686a400c 100644 --- a/srcmorph/pom.xml +++ b/srcmorph/pom.xml @@ -49,8 +49,20 @@ SPDX-License-Identifier: Apache-2.0 never had an effect, and upstream llama.cpp defaults use_jinja to true, so chat templating is unchanged. The rest of the provider's API surface (LlamaModel, InferenceParameters, ModelParameters, ChatResponse/Timings/Pair, ChatResponseParser, - ReasoningFormat) is untouched by 5.1.0. --> - 5.1.0 + ReasoningFormat) is untouched by 5.1.0. + 5.2.0 is a BREAKING bump this module needs, not an optional one: the binding removed + ModelParameters.enableFlashAttn()/ModelFlag.FLASH_ATTN (--flash-attn takes a mandatory + on|off|auto value since llama.cpp b10273, so the valueless flag emitted a broken argv) + and renamed setTensorReadLazy/TensorReadLazyMode to setLazyMode/LazyMode (llama.cpp + b10731 renamed --tensor-read-lazy to --lazy-mode with no alias). The provider calls + both, so it cannot compile against 5.1.0 any more. + This deliberately names the RELEASE, not 5.2.0-SNAPSHOT: a snapshot would need a + entry for the Central snapshot channel (this repo declares none -- the + distributionManagement/snapshotRepository below is deploy-only) AND would block every + srcmorph release, since Central rejects a release with a -SNAPSHOT dependency. The + build therefore stays red until net.ladenthin:llama:5.2.0 is published, and goes green + with no further edit the moment it is. --> + 5.2.0