diff --git a/CHANGELOG.md b/CHANGELOG.md index ad8ae1e..3d19890 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,79 @@ # Changelog +## v0.8.71 + +llama.cpp bumped to `9581154a2` (b12447+16), 95 commits from `9daa0aa22` +(b12361+7). #32306 deprecates the `llama_batch` API (`llama_batch_init`, +`llama_batch_get_one`, `llama_batch_free`, `llama_decode`, `llama_encode`) +in favour of `llama_batch_ext` and `llama_process`; the NIF now uses only +the new API and builds without deprecation warnings. `llama.h` otherwise +adds `llama_model_dflash_variant`; `common.h` adds the +`draft-dflash2`/`draft-lilicorr` speculative types (#32016). +`llama_model_default_params` and `llama_context_default_params` are +unchanged. `src/llama-vocab.cpp` only adds the `zonos3` and `inkling2` +pre-tokenizers (`tokenizer_st_partition` is untouched), and +`tools/server/server-decision.*` did not change, so `input_tokenize.cpp` +needs nothing and `decision.cpp` is "Ported at" `9581154a2` with no hunk to +port. ggml's headers and the RPC backend did not change: the RPC protocol +stays 10.0.0. + +Checked on an M1 Max (Metal): the default suite on a default build and on an +`LLAMA_RPC=1` build, plus `:rpc_live` against a local `RPC.Server` worker; +`:smoke`, `:embeddings` (Qwen3-Embedding), `:slow`, `:tools`, `:decision`, +`:decision_clef`, `:mtp` (Qwen3.6-27B), `:mtp_sidecar` (Qwen3.8-27B) and +`test/vm_exit_test.exs`; and `scripts/decision_compare.exs` — ALL MATCH on +9 models (tinylaya, tinyopenjev, Clef Flash, d1-3B, d1-omni-600M, d1-2B, +vllm-decision2 Kai, vllm-d3-edge, h2o-lightning-4b) against llama-server at +the same commit. `:dflash` not run (no DFlash drafter on disk); not run on a +DGX Spark. + +### Changed + +- Every decode goes through `llama_batch_ext` (`TokenBatch` in + `llama_nif.h`, one per context, sized to `n_batch`) and `llama_process`. + The MTP path feeds `common_speculative_process` the same entries as its + target decode, as before. +- A batch NIF handed more entries than the context's `n_batch` + (`batch_eval`, `batch_eval_sample`, `decode_batch`, `embed_decode`, + `embed_batch_decode`, an MTP verify batch) now returns + `{:error, "... batch exceeds n_batch (N tokens)"}`. It used to abort the VM + in `llama_context::decode` (`GGML_ASSERT(n_tokens_all <= n_batch)`). +- A token id outside the vocabulary or an out-of-range seq id inside a batch + is now reported by name (`invalid token id N`, `invalid seq_id N ...`) + instead of `... failed with code: -1`. Decode errors read + `" failed with code: N"` or `" failed: "`. +- The `generate` and `generate_tokens` NIFs request logits for the last + prompt token only, not for the last token of every `n_batch` chunk + (`llama_batch_get_one`'s behaviour). +- `LlamaCppEx.MTP` docs: `:dflash` runs DFlash, DFlash 2 and LiliCorr + drafters (upstream picks the head from the drafter's metadata); `:dspark` + refuses any drafter that is not DSpark. + +### Upstream + +- From the 95 commits: + - The `llama_batch` API is deprecated (#32306) and upstream's own callers + moved to `llama_batch_ext` (#32356). + - DFlash speculative decoding split into one driver per head, with + `draft-dflash2` and `draft-lilicorr` types (#32016); DFLASH2 for + Recursive Ripple-65B (#31904). + - New models: Inkling 2 (text + vision, #32217, parser #32381), MiniMax-M3.1 + (#32301); mtmd gains YuE2-3B, fish audio s2, higgs v3 STT, vibevoice-ASR + and zonos3-mini TTS. + - KV cells no longer scan every cell per sequence removal (#32380); the + M-RoPE non-consecutive position warning is fixed (#32287). + - The tensor staging buffer comes from the CPU buffer type (#31438); the + backend scheduler starts a new split at an incompatible weight (#31775). + - Chat: tool calls inside reasoning for Gemma 4 (#32232) and Ministral 3 + (#32271), undeclared tool calls in the LFM2 grammar (#32164), the + Mistral workaround removed (#32230); jinja gains scope-chain lookup, + `sort_keys` and four missing features, and renders faster (#32023). + - `common_sampler_init` throws on an unknown grammar trigger type instead + of aborting (#31478). + - Metal: `mul_mv` destination offset overflow fixed (#32221), fusion + workaround for LFM3 (#32296). CUDA: `MUL_MAT_ID` abort on AMD (#32385). + - Vulkan, OpenCL, Hexagon, WebGPU, SYCL and s390x work. + ## v0.8.70 llama.cpp stays at `9daa0aa22`. The `generate_mtp_tokens` NIF changed, so diff --git a/Makefile b/Makefile index 686e257..08b57f5 100644 --- a/Makefile +++ b/Makefile @@ -36,7 +36,7 @@ endif # Pinned llama.cpp commit, used when vendor/llama.cpp has to be cloned. MUST # match the vendor/llama.cpp submodule; bump both together, see # docs/release-guide.md. Override to build the NIF against another revision. -LLAMA_COMMIT ?= 9daa0aa22b8aa0143344d7549dee1f0e32f048bb +LLAMA_COMMIT ?= 9581154a2250cc9e6948e4c2a2fc4b4e19d1a57e # The commit actually on disk. A submodule can be bumped without LLAMA_COMMIT # following it, and the build has to key off what is really there. diff --git a/c_src/llama_cpp_ex/decision.cpp b/c_src/llama_cpp_ex/decision.cpp index 103a987..1ab116c 100644 --- a/c_src/llama_cpp_ex/decision.cpp +++ b/c_src/llama_cpp_ex/decision.cpp @@ -2,7 +2,7 @@ // of tools/server/server-context.cpp (send_decision, the batch layout rules and // the /v1/systemone handler). See decision.h for scope. // -// Ported at 9daa0aa22. Functions that are upstream's carry its name in a +// Ported at 9581154a2. Functions that are upstream's carry its name in a // trailing comment so a bump can diff them one by one; everything below the // "evaluation" banner replaces server slots and is ours. diff --git a/c_src/llama_cpp_ex/llama_nif.cpp b/c_src/llama_cpp_ex/llama_nif.cpp index d166ce0..a79eaf3 100644 --- a/c_src/llama_cpp_ex/llama_nif.cpp +++ b/c_src/llama_cpp_ex/llama_nif.cpp @@ -278,15 +278,17 @@ LLAMA_NIF(log_set_level, 0); // DIRECTLY are bounded here. Those functions assert on their arguments, // and nothing upstream sees the value first. // -// 2. Values carried INSIDE a `llama_batch` are bounded by upstream's -// `llama_batch_allocr::init` (vendor/llama.cpp/src/llama-batch.cpp:61-64), -// which range-checks batch seq ids and positions and returns false rather -// than asserting — so `llama_decode` returns non-zero and the NIF returns -// `{:error, _}`. That check is load-bearing and invisible from this file, -// which is part of why the three unguarded sites in category 1 -// (embed_decode, embed_batch_decode, batch_eval_sample's purge list) -// looked closed: their neighbours really were safe, for a reason nobody -// had written down. +// 2. Values carried INSIDE a batch are bounded by upstream's +// `llama_batch_ext_add_token` (vendor/llama.cpp/src/llama-batch.cpp), +// which refuses a seq id outside [0, n_seq_max), a token id outside the +// vocabulary and an entry past n_batch with an error code rather than an +// assert; `TokenBatch::add` turns that into `{:error, _}`. Positions are +// checked by `llama_batch_allocr::init` at decode time, which returns +// false, so `llama_process` returns non-zero. Those checks are +// load-bearing and invisible from this file, which is part of why the +// three unguarded sites in category 1 (embed_decode, embed_batch_decode, +// batch_eval_sample's purge list) looked closed: their neighbours really +// were safe, for a reason nobody had written down. // // `test/nif_guards_test.exs` enumerates both categories and is checked against // LlamaCppEx.NIF's own source, so a new seq_id-taking NIF cannot be added @@ -319,52 +321,73 @@ static std::string invalid_seq_id_message(llama_context* ctx, int64_t seq_id) { " (must be 0 <= seq_id < " + std::to_string(llama_n_seq_max(ctx)) + ")"; } -// Single choke point for every llama_decode in this file, so the recorded batch -// shape cannot drift from reality as decode paths are added. Only a successful -// decode publishes a new shape; a failed one leaves the previous logits (and -// therefore the previous shape) in place. +// Single choke point for every target decode in this file, so the recorded +// batch shape cannot drift from reality as decode paths are added. Only a +// successful decode publishes a new shape; a failed one leaves the previous +// logits (and therefore the previous shape) in place. // // This claim was false for seven direct `llama_decode` calls (the two streaming // generators, the blocking `generate`, and the MTP prefill/verify/rollback), and // false in the *permissive* direction: `sampler_sample_at`'s bound was computed // from a stale batch, so it accepted indices upstream would abort on. A // permissively-stale bound is worse than no bound because it reads as safe. -// Every `llama_decode` in this translation unit now goes through here; the only -// decodes that do not are inside upstream's `common_speculative_*`, which run on -// the draft context and are never sampled through this boundary. -static int decode_tracked(LlamaContext& c, const llama_batch& batch) { - int ret = llama_decode(c.ctx, batch); +// Every decode in this translation unit now goes through here; the only ones +// that do not are inside upstream's `common_speculative_*`, which run on the +// draft context and are never sampled through this boundary, and the decision +// engine's, which forgets the shape (decision_decide). +static int decode_tracked(LlamaContext& c, TokenBatch& batch) { + int ret = llama_process(c.ctx, LLAMA_PROCESS_TYPE_DECODE, batch.get()); if (ret == 0) { c.record_batch(batch); } return ret; } -// Append one single-sequence entry to a batch from llama_batch_init(n, 0, 1). -// Stands in for common_batch_add(llama_batch&, ...), which upstream removed in -// 60e9cf7a7 along with the rest of the legacy-batch helpers in common/. -static void batch_push(llama_batch& b, llama_token id, llama_pos pos, - llama_seq_id seq_id, bool logits) { - const int32_t i = b.n_tokens++; - b.token[i] = id; - b.pos[i] = pos; - b.n_seq_id[i] = 1; - b.seq_id[i][0] = seq_id; - b.logits[i] = logits; +// Appends `tokens` to sequence 0 after the last position it holds, in n_batch +// chunks, with logits for the final token only: llama_batch_get_one's append +// semantics without its logit row per chunk. Returns llama_process's code; a +// token the batch refuses returns -1 with `err` set. +static int decode_append(LlamaContext& c, const llama_token* tokens, size_t n_tokens, + std::string& err) { + const llama_pos start = llama_memory_seq_pos_max(llama_get_memory(c.ctx), 0) + 1; + TokenBatch& batch = c.fresh_batch(); + const size_t n_batch = static_cast(batch.n_batch()); + + for (size_t i = 0; i < n_tokens; i += n_batch) { + const size_t n = std::min(n_tokens - i, n_batch); + batch.clear(); + for (size_t j = 0; j < n; j++) { + if (auto e = batch.add(tokens[i + j], start + static_cast(i + j), 0, + i + j == n_tokens - 1)) { + err = std::move(*e); + return -1; + } + } + const int ret = decode_tracked(c, batch); + if (ret != 0) { + return ret; + } + } + return 0; +} + +// The message for a failed decode_append / decode_tracked. +static std::string decode_error(const char* what, int ret, const std::string& err) { + return err.empty() ? std::string(what) + " failed with code: " + std::to_string(ret) + : std::string(what) + " failed: " + err; } -// common_speculative_process only takes a common_batch since 60e9cf7a7; the -// llama_batch overload is gone. The MTP path still decodes the target through -// decode_tracked (llama_decode), so the spec is fed a mirror of that same batch -// rather than a second, separately-built one that could drift from it. `mirror` -// is reused across calls; clear() keeps its allocation. -static bool speculative_process(common_speculative* spec, common_batch& mirror, - const llama_batch& b) { - mirror.clear(); - for (int32_t i = 0; i < b.n_tokens; i++) { - mirror.add(b.token[i], b.pos[i], b.seq_id[i][0], b.logits[i] != 0); +// MTP: one entry into both the target's decode batch and the common_batch the +// speculative state is fed (common_speculative_process only takes a +// common_batch), so the two cannot drift apart. +static std::optional mtp_push(TokenBatch& b, common_batch& mirror, + llama_token id, llama_pos pos, + llama_seq_id seq_id, bool output) { + auto err = b.add(id, pos, seq_id, output); + if (!err) { + mirror.add(id, pos, seq_id, output); } - return common_speculative_process(spec, mirror); + return err; } // Same rationale as check_seq_id: an out-of-range logits index is a caller bug, @@ -458,22 +481,6 @@ struct MsgEnvGuard { operator ErlNifEnv*() const { return env; } }; -// Frees a llama_batch obtained from llama_batch_init on scope exit. Same -// rationale as MsgEnvGuard: several statements between the init and the free -// can throw (fine decoders, std::bad_alloc, detokenize failures), and an unwind -// past a bare llama_batch_init leaks the batch's token/pos/seq_id buffers. -// Binds to an existing local so the surrounding code keeps using `batch` -// directly. -struct BatchFreeGuard { - llama_batch& batch; - - explicit BatchFreeGuard(llama_batch& b) : batch(b) {} - ~BatchFreeGuard() { llama_batch_free(batch); } - - BatchFreeGuard(const BatchFreeGuard&) = delete; - BatchFreeGuard& operator=(const BatchFreeGuard&) = delete; -}; - // --- Cancellation --- fine::ResourcePtr cancel_flag_new(ErlNifEnv* env) { @@ -494,7 +501,7 @@ static bool cancel_abort_cb(void* data) { // Installs the flag as the context's abort callback for the duration of a // generation loop, so a cancel interrupts even a long prefill decode -// (llama_decode returns 2). Cleared on scope exit. +// (llama_process returns 2). Cleared on scope exit. struct AbortCallbackScope { llama_context* ctx; @@ -1557,40 +1564,17 @@ LLAMA_NIF(sampler_sample, ERL_NIF_DIRTY_JOB_CPU_BOUND); std::variant, fine::Error> decode(ErlNifEnv* env, fine::ResourcePtr ctx, std::vector token_ids) { - int n_tokens = static_cast(token_ids.size()); - if (n_tokens == 0) { + if (token_ids.empty()) { return fine::Ok(); } - // Explicit batch instead of llama_batch_get_one: get_one requests logits - // for the LAST TOKEN OF EVERY CHUNK, computing (and copying) a full-vocab - // logit row per n_batch tokens of prompt for nothing. Only the very last - // token needs logits. Positions continue seq 0 from wherever the context - // is, preserving get_one's append semantics for repeated decode calls. - llama_pos start = - llama_memory_seq_pos_max(llama_get_memory(ctx->ctx), 0) + 1; - - int n_batch = llama_n_batch(ctx->ctx); - llama_batch& batch = ctx->reserve_batch(std::min(n_tokens, n_batch)); - - for (int i = 0; i < n_tokens; i += n_batch) { - int n = std::min(n_tokens - i, n_batch); - bool is_last_chunk = (i + n >= n_tokens); - - batch.n_tokens = n; - - for (int j = 0; j < n; j++) { - batch.token[j] = static_cast(token_ids[i + j]); - batch.pos[j] = start + static_cast(i + j); - batch.n_seq_id[j] = 1; - batch.seq_id[j][0] = 0; - batch.logits[j] = (is_last_chunk && j == n - 1); - } - - int ret = decode_tracked(*ctx, batch); - if (ret != 0) { - return fine::Error(std::string("llama_decode failed with code: " + std::to_string(ret))); - } + // Positions continue seq 0 from wherever the context is, so repeated + // decode calls append. + std::vector tokens(token_ids.begin(), token_ids.end()); + std::string err; + int ret = decode_append(*ctx, tokens.data(), tokens.size(), err); + if (ret != 0) { + return fine::Error(decode_error("decode", ret, err)); } return fine::Ok(); @@ -1733,7 +1717,7 @@ embed_decode( } // `seq_id` goes to llama_memory_seq_rm below, which is OUTSIDE any batch, so - // upstream's llama_batch_allocr::init never sees it and + // upstream's llama_batch_ext_add_token never sees it and // GGML_ASSERT(seq_id == -1 || (seq_id >= 0 && ...)) at // llama-kv-cache.cpp:385 aborts the VM. Batch-carried seq ids are bounded by // upstream; ones that reach a memory op first are not. @@ -1746,23 +1730,19 @@ embed_decode( llama_memory_seq_rm(llama_get_memory(ctx->ctx), static_cast(seq_id), -1, -1); - // Build batch with explicit seq_id and position tracking - llama_batch batch = llama_batch_init(n_tokens, 0, 1); - BatchFreeGuard batch_guard(batch); - batch.n_tokens = n_tokens; - + // All tokens get embeddings, so the whole input has to fit one batch. + TokenBatch& batch = ctx->fresh_batch(); for (int i = 0; i < n_tokens; i++) { - batch.token[i] = static_cast(token_ids[i]); - batch.pos[i] = static_cast(i); - batch.n_seq_id[i] = 1; - batch.seq_id[i][0] = static_cast(seq_id); - batch.logits[i] = true; // all tokens get embeddings + if (auto err = batch.add(static_cast(token_ids[i]), static_cast(i), + static_cast(seq_id), true)) { + return fine::Error(decode_error("embed_decode", -1, *err)); + } } int ret = decode_tracked(*ctx, batch); if (ret != 0) { - return fine::Error(std::string("embed_decode failed with code: " + std::to_string(ret))); + return fine::Error(decode_error("embed_decode", ret, "")); } return fine::Ok(); @@ -1830,10 +1810,11 @@ LLAMA_NIF(get_embeddings, 0); // --- Batched embeddings: decode many sequences in a single batch --- // // Each {seq_id, token_ids} sequence is laid out at its own positions (0..len-1) -// under its own seq_id, so one llama_decode populates per-sequence pooled +// under its own seq_id, so one decode populates per-sequence pooled // embeddings retrievable via get_embeddings(ctx, seq_id, ...). The caller must // size the context with embeddings=true and n_seq_max >= number of sequences, -// and keep the total token count within n_batch/n_ubatch. +// and keep the total token count within n_batch/n_ubatch (an entry past +// n_batch is refused). std::variant, fine::Error> embed_batch_decode( ErlNifEnv* env, @@ -1869,27 +1850,22 @@ embed_batch_decode( static_cast(seq_id), -1, -1); } - llama_batch batch = llama_batch_init(total, 0, 1); - BatchFreeGuard batch_guard(batch); - batch.n_tokens = total; - - int idx = 0; + TokenBatch& batch = ctx->fresh_batch(); for (auto& [seq_id, tokens] : sequences) { int len = static_cast(tokens.size()); for (int i = 0; i < len; i++) { - batch.token[idx] = static_cast(tokens[i]); - batch.pos[idx] = static_cast(i); - batch.n_seq_id[idx] = 1; - batch.seq_id[idx][0] = static_cast(seq_id); - batch.logits[idx] = true; // all tokens get embeddings - idx++; + // all tokens get embeddings + if (auto err = batch.add(static_cast(tokens[i]), static_cast(i), + static_cast(seq_id), true)) { + return fine::Error(decode_error("embed_batch_decode", -1, *err)); + } } } int ret = decode_tracked(*ctx, batch); if (ret != 0) { - return fine::Error(std::string("embed_batch_decode failed with code: " + std::to_string(ret))); + return fine::Error(decode_error("embed_batch_decode", ret, "")); } return fine::Ok(); @@ -1910,28 +1886,26 @@ prefill( return fine::Error(std::string("empty token list")); } - int n_batch = llama_n_batch(ctx->ctx); - llama_batch& batch = ctx->reserve_batch(std::min(n_tokens, n_batch)); + TokenBatch& batch = ctx->fresh_batch(); + const int n_batch = batch.n_batch(); for (int i = 0; i < n_tokens; i += n_batch) { int n = std::min(n_tokens - i, n_batch); - bool is_last_chunk = (i + n >= n_tokens); - - batch.n_tokens = n; + batch.clear(); for (int j = 0; j < n; j++) { - batch.token[j] = static_cast(token_ids[i + j]); - batch.pos[j] = static_cast(i + j); - batch.n_seq_id[j] = 1; - batch.seq_id[j][0] = static_cast(seq_id); // Only request logits for the last token of the last chunk - batch.logits[j] = (is_last_chunk && j == n - 1); + if (auto err = batch.add(static_cast(token_ids[i + j]), + static_cast(i + j), + static_cast(seq_id), i + j == n_tokens - 1)) { + return fine::Error(decode_error("prefill decode", -1, *err)); + } } int ret = decode_tracked(*ctx, batch); if (ret != 0) { - return fine::Error(std::string("prefill decode failed with code: " + std::to_string(ret))); + return fine::Error(decode_error("prefill decode", ret, "")); } } @@ -1962,18 +1936,14 @@ decode_batch( const auto* vocab = ctx->model->vocab(); - // Build a single batch with all entries - llama_batch batch = llama_batch_init(n, 0, 1); - BatchFreeGuard batch_guard(batch); - batch.n_tokens = n; - + // Build a single batch with all entries; every one needs logits to sample. + TokenBatch& batch = ctx->fresh_batch(); for (int i = 0; i < n; i++) { auto& [seq_id, token_id, pos] = entries[i]; - batch.token[i] = static_cast(token_id); - batch.pos[i] = static_cast(pos); - batch.n_seq_id[i] = 1; - batch.seq_id[i][0] = static_cast(seq_id); - batch.logits[i] = true; // need logits for all entries to sample + if (auto err = batch.add(static_cast(token_id), static_cast(pos), + static_cast(seq_id), true)) { + return fine::Error(decode_error("decode_batch", -1, *err)); + } } int ret = decode_tracked(*ctx, batch); @@ -2025,18 +1995,16 @@ decode_token( int64_t pos, int64_t seq_id) { - llama_batch& batch = ctx->reserve_batch(1); - batch.n_tokens = 1; - batch.token[0] = static_cast(token_id); - batch.pos[0] = static_cast(pos); - batch.n_seq_id[0] = 1; - batch.seq_id[0][0] = static_cast(seq_id); - batch.logits[0] = true; + TokenBatch& batch = ctx->fresh_batch(); + if (auto err = batch.add(static_cast(token_id), static_cast(pos), + static_cast(seq_id), true)) { + return fine::Error(decode_error("decode_token", -1, *err)); + } int ret = decode_tracked(*ctx, batch); if (ret != 0) { - return fine::Error(std::string("decode_token failed with code: " + std::to_string(ret))); + return fine::Error(decode_error("decode_token", ret, "")); } return fine::Ok(); @@ -2056,22 +2024,19 @@ batch_eval( return fine::Error(std::string("empty entries list")); } - llama_batch& batch = ctx->reserve_batch(n); - batch.n_tokens = n; - + TokenBatch& batch = ctx->fresh_batch(); for (int i = 0; i < n; i++) { auto& [token_id, pos, seq_id, logits] = entries[i]; - batch.token[i] = static_cast(token_id); - batch.pos[i] = static_cast(pos); - batch.n_seq_id[i] = 1; - batch.seq_id[i][0] = static_cast(seq_id); - batch.logits[i] = logits; + if (auto err = batch.add(static_cast(token_id), static_cast(pos), + static_cast(seq_id), logits)) { + return fine::Error(decode_error("batch_eval", -1, *err)); + } } int ret = decode_tracked(*ctx, batch); if (ret != 0) { - return fine::Error(std::string("batch_eval failed with code: " + std::to_string(ret))); + return fine::Error(decode_error("batch_eval", ret, "")); } return fine::Ok(); @@ -2081,14 +2046,14 @@ LLAMA_NIF(batch_eval, ERL_NIF_DIRTY_JOB_CPU_BOUND); // --- Fused batch eval + sample (Server hot loop) --- // // One NIF call per Server tick: builds a batch from `entries` -// ({token, pos, seq_id, wants_logits}), runs llama_decode, then samples every +// ({token, pos, seq_id, wants_logits}), decodes it, then samples every // wants_logits entry whose seq_id has a sampler registered in `samplers`, // returning {seq_id, new_token, piece, is_eog}. The piece is empty for EOG // tokens. Unlike decode_batch above, samplers are per-sequence resources owned // by the caller — their grammar/penalty state advances across ticks and is // never reset or shared here. // -// KV-pressure policy (llama_decode == 1, "no KV slot found"), mirroring +// KV-pressure policy (decode returns 1, "no KV slot found"), mirroring // llama-server's update_slots(): first drop whole sequences listed in // `purgeable_seq_ids` (idle slots whose cache the caller is willing to lose), // retrying after each purge; once the purge list is exhausted, recursively @@ -2099,7 +2064,8 @@ LLAMA_NIF(batch_eval, ERL_NIF_DIRTY_JOB_CPU_BOUND); // other sequences keep going (the caller fails just that request). Purged and // failed seq ids plus the split count are returned so the caller can fix up // its bookkeeping and emit telemetry. Purgeable seqs must not have entries in -// the batch. +// the batch. An entry the batch refuses (see TokenBatch::add) returns -1 with +// `err` set. static int bes_decode_range( LlamaContext& lctx, @@ -2113,7 +2079,8 @@ static int bes_decode_range( int64_t& n_splits, std::vector& failed, std::vector>& results, - llama_batch& batch) // reserved by the caller for >= entries.size() tokens + TokenBatch& batch, + std::string& err) { // Skip entries whose sequence already failed this call — decoding past a // failed (missing) position would leave a hole in that sequence's KV. @@ -2131,15 +2098,15 @@ static int bes_decode_range( return 0; } - batch.n_tokens = static_cast(n); + batch.clear(); for (size_t i = 0; i < n; i++) { const auto& [token_id, pos, seq_id, logits] = entries[idxs[i]]; - batch.token[i] = static_cast(token_id); - batch.pos[i] = static_cast(pos); - batch.n_seq_id[i] = 1; - batch.seq_id[i][0] = static_cast(seq_id); - batch.logits[i] = logits; + if (auto e = batch.add(static_cast(token_id), static_cast(pos), + static_cast(seq_id), logits)) { + err = std::move(*e); + return -1; + } } int ret = decode_tracked(lctx, batch); @@ -2206,11 +2173,11 @@ static int bes_decode_range( size_t mid = begin + (end - begin) / 2; int rc = bes_decode_range(lctx, vocab, entries, begin, mid, samplers, purgeable, purge_idx, purged, n_splits, - failed, results, batch); + failed, results, batch, err); if (rc != 0) return rc; return bes_decode_range(lctx, vocab, entries, mid, end, samplers, purgeable, purge_idx, purged, n_splits, - failed, results, batch); + failed, results, batch, err); } return ret; @@ -2255,7 +2222,7 @@ batch_eval_sample( // `purgeable_seq_ids` reaches llama_memory_seq_rm inside bes_decode_range's // KV-pressure loop, outside any batch — same abort as embed_decode's. The // entries' seq ids are batch-carried and bounded by upstream's - // llama_batch_allocr::init; these are not, so they are rejected here rather + // llama_batch_ext_add_token; these are not, so they are rejected here rather // than filtered: a purge id the caller cannot name is a bookkeeping bug in // the caller, not a condition to paper over. for (int64_t seq_id : purgeable_seq_ids) { @@ -2264,15 +2231,15 @@ batch_eval_sample( } } - llama_batch& batch = ctx->reserve_batch(static_cast(entries.size())); + TokenBatch& batch = ctx->fresh_batch(); + std::string err; int rc = bes_decode_range(*ctx, vocab, entries, 0, entries.size(), smpls, purgeable_seq_ids, purge_idx, purged, - n_splits, failed, results, batch); + n_splits, failed, results, batch, err); if (rc != 0) { - return fine::Error(std::string( - "batch_eval_sample failed with code: " + std::to_string(rc))); + return fine::Error(decode_error("batch_eval_sample", rc, err)); } return fine::Ok(std::move(results), std::move(purged), n_splits, @@ -3067,18 +3034,21 @@ fine::Ok<> generate_mtp_tokens( // for the whole prompt. We ask for logits on the final prompt token only, // because we sample the first generated token from them below. common_batch spec_batch(ctx_tgt); - int n_batch = llama_n_batch(ctx_tgt); + TokenBatch& batch = sp.ctx_tgt->fresh_batch(); + const size_t n_batch = static_cast(batch.n_batch()); llama_pos n_past = 0; for (size_t i = 0; i < prompt.size(); i += n_batch) { - int n = std::min(static_cast(prompt.size() - i), n_batch); - bool is_last_chunk = (i + n >= prompt.size()); - - llama_batch batch = llama_batch_init(n, 0, 1); - BatchFreeGuard batch_guard(batch); - for (int j = 0; j < n; j++) { - const bool want_logits = is_last_chunk && j == n - 1; - batch_push(batch, prompt[i + j], static_cast(i + j), - seq_id, want_logits); + const size_t n = std::min(prompt.size() - i, n_batch); + + batch.clear(); + spec_batch.clear(); + for (size_t j = 0; j < n; j++) { + const bool want_logits = i + j == prompt.size() - 1; + if (auto err = mtp_push(batch, spec_batch, prompt[i + j], + static_cast(i + j), seq_id, want_logits)) { + send_error("prompt decode failed: " + *err); + return fine::Ok(); + } } int ret = decode_tracked(*sp.ctx_tgt, batch); @@ -3086,11 +3056,11 @@ fine::Ok<> generate_mtp_tokens( send_error("prompt decode failed: code=" + std::to_string(ret)); return fine::Ok(); } - bool proc_ok = speculative_process(sp.spec, spec_batch, batch); + bool proc_ok = common_speculative_process(sp.spec, spec_batch); if (!proc_ok) { fprintf(stderr, "MTP prefill: common_speculative_process returned false " - "at chunk i=%zu n=%d\n", i, n); + "at chunk i=%zu n=%zu\n", i, n); } } n_past = static_cast(prompt.size()); @@ -3282,18 +3252,21 @@ fine::Ok<> generate_mtp_tokens( // 2. Build the verification batch: [sampled, drafts...] at consecutive // positions starting at n_past, all with logits. const int n_verify = 1 + static_cast(drafts.size()); - llama_batch batch = llama_batch_init(n_verify, 0, 1); - BatchFreeGuard batch_guard(batch); - batch_push(batch, sampled, n_past, seq_id, true); - for (size_t i = 0; i < drafts.size(); i++) { - batch_push(batch, drafts[i], - n_past + 1 + static_cast(i), seq_id, true); + batch.clear(); + spec_batch.clear(); + for (int i = 0; i < n_verify; i++) { + const llama_token tok = i == 0 ? sampled : drafts[i - 1]; + if (auto err = mtp_push(batch, spec_batch, tok, + n_past + static_cast(i), seq_id, true)) { + send_error("verify decode failed: " + *err); + return fine::Ok(); + } } // 2b. Roll the draft ctx back to n_past so common_speculative_process // can re-decode the verify batch on it. common_speculative_draft // advances ctx_dft to roughly n_past + drafts.size() via internal - // AR decoding; without this rollback, the next llama_decode on + // AR decoding; without this rollback, the next decode on // ctx_dft would hit an "inconsistent sequence positions" abort // (M-RoPE requires the current pos_max to be < the batch's first // position). Mirrors the upstream server's seq_rm between draft @@ -3312,7 +3285,7 @@ fine::Ok<> generate_mtp_tokens( send_error("verify decode failed: code=" + std::to_string(ret)); return fine::Ok(); } - if (!speculative_process(sp.spec, spec_batch, batch)) { + if (!common_speculative_process(sp.spec, spec_batch)) { send_error("common_speculative_process failed"); return fine::Ok(); } @@ -3429,21 +3402,24 @@ fine::Ok<> generate_mtp_tokens( // target. Every row asks for output, as the verify batch did, // because the drafter's extraction follows the output rows. if (n_accepted_total > 0) { - llama_batch redo = llama_batch_init(n_accepted_total, 0, 1); - BatchFreeGuard redo_guard(redo); + batch.clear(); + spec_batch.clear(); for (int i = 0; i < n_accepted_total; i++) { llama_token tok = prompt[prompt.size() - n_accepted_total - 1 + i]; - batch_push(redo, tok, - n_past + static_cast(i), seq_id, - /*logits=*/ true); + if (auto err = mtp_push(batch, spec_batch, tok, + n_past + static_cast(i), seq_id, + /*output=*/ true)) { + send_error("rollback re-decode failed: " + *err); + return fine::Ok(); + } } - int ret = decode_tracked(*sp.ctx_tgt, redo); + int ret = decode_tracked(*sp.ctx_tgt, batch); if (ret != 0) { send_error("rollback re-decode failed: code=" + std::to_string(ret)); return fine::Ok(); } - if (!speculative_process(sp.spec, spec_batch, redo)) { + if (!common_speculative_process(sp.spec, spec_batch)) { send_error("common_speculative_process failed after rollback"); return fine::Ok(); } @@ -3462,7 +3438,7 @@ fine::Ok<> generate_mtp_tokens( // Close out us_other for this iter — captures the post-sample-loop // work plus any implicit GPU-sync wait that bleeds into the next - // iter from llama_decode's async submission on Metal. + // iter from llama_process's async submission on Metal. sp.us_other.fetch_add( std::chrono::duration_cast( std::chrono::steady_clock::now() - t_anchor).count(), @@ -3544,12 +3520,10 @@ fine::Ok<> generate_tokens( return fine::Ok(); } - // Process prompt in chunks - int n_batch = llama_n_batch(ctx); - for (size_t i = 0; i < prompt_tokens.size(); i += n_batch) { - int n = std::min(static_cast(prompt_tokens.size() - i), n_batch); - llama_batch batch = llama_batch_get_one(prompt_tokens.data() + i, n); - int ret = decode_tracked(*ctx_res, batch); + // Process the prompt in n_batch chunks + { + std::string err; + int ret = decode_append(*ctx_res, prompt_tokens.data(), prompt_tokens.size(), err); if (ret == 2) { // Aborted by the cancel flag mid-prefill — stop quietly. return fine::Ok(); @@ -3627,8 +3601,8 @@ fine::Ok<> generate_tokens( } // Decode next token - llama_batch batch = llama_batch_get_one(&new_token, 1); - int ret = decode_tracked(*ctx_res, batch); + std::string err; + int ret = decode_append(*ctx_res, &new_token, 1, err); if (ret == 2) { // Aborted by the cancel flag — stop quietly. return fine::Ok(); @@ -3684,18 +3658,14 @@ generate( return fine::Error(kNoLogitsMessage); } - // Process prompt in chunks of n_batch - int n_batch = llama_n_batch(ctx); - for (size_t i = 0; i < prompt_tokens.size(); i += n_batch) { - int n = std::min(static_cast(prompt_tokens.size() - i), n_batch); - llama_batch batch = llama_batch_get_one(prompt_tokens.data() + i, n); - int ret = decode_tracked(*ctx_res, batch); - if (ret == 2) { - return fine::Ok(std::string()); - } - if (ret != 0) { - return fine::Error(std::string("prompt decode failed with code: " + std::to_string(ret))); - } + // Process the prompt in n_batch chunks + std::string err; + int ret = decode_append(*ctx_res, prompt_tokens.data(), prompt_tokens.size(), err); + if (ret == 2) { + return fine::Ok(std::string()); + } + if (ret != 0) { + return fine::Error(decode_error("prompt decode", ret, err)); } // Generation loop @@ -3728,13 +3698,12 @@ generate( } // Decode the new token for next iteration - llama_batch batch = llama_batch_get_one(&new_token, 1); - int ret = decode_tracked(*ctx_res, batch); + ret = decode_append(*ctx_res, &new_token, 1, err); if (ret == 2) { break; } if (ret != 0) { - return fine::Error(std::string("generation decode failed with code: " + std::to_string(ret))); + return fine::Error(decode_error("generation decode", ret, err)); } } diff --git a/c_src/llama_cpp_ex/llama_nif.h b/c_src/llama_cpp_ex/llama_nif.h index a19fee3..524d1fc 100644 --- a/c_src/llama_cpp_ex/llama_nif.h +++ b/c_src/llama_cpp_ex/llama_nif.h @@ -6,9 +6,11 @@ #include "speculative.h" #include "decision.h" #include "input_tokenize.h" +#include #include #include #include +#include #include #include #include @@ -97,11 +99,73 @@ class LlamaModel { } }; +// A llama_batch_ext of single-sequence token entries, plus the per-entry output +// flags the opaque batch does not expose (decode_tracked records them). +// llama_batch_ext_init sizes it to the context's n_batch, so add() refuses an +// entry past that instead of llama_context::decode's +// GGML_ASSERT(n_tokens_all <= n_batch) aborting the VM. It also refuses a token +// id outside the vocabulary and a seq id outside [0, n_seq_max), which is what +// bounds the batch-carried ids coming from Elixir. +class TokenBatch { +public: + explicit TokenBatch(llama_context* ctx) + : ext_(llama_batch_ext_init(ctx)) + , n_batch_(llama_n_batch(ctx)) + , n_seq_max_(llama_n_seq_max(ctx)) {} + ~TokenBatch() { llama_batch_ext_free(ext_); } + + TokenBatch(const TokenBatch&) = delete; + TokenBatch& operator=(const TokenBatch&) = delete; + + void clear() { + llama_batch_ext_clear(ext_); + outputs_.clear(); + } + + // Appends one entry. Returns the reason on refusal; the batch is unchanged. + std::optional add(llama_token id, llama_pos pos, llama_seq_id seq_id, bool output) { + const int32_t idx = llama_batch_ext_add_token(ext_, seq_id, id); + if (idx < 0) { + return add_error(idx, id, seq_id); + } + llama_batch_ext_set_pos(ext_, idx, &pos); + if (output) { + llama_batch_ext_set_output_logits(ext_, idx, true); + } + outputs_.push_back(output ? 1 : 0); + return std::nullopt; + } + + llama_batch_ext* get() { return ext_; } + const std::vector& outputs() const { return outputs_; } + int32_t n_batch() const { return n_batch_; } + +private: + llama_batch_ext* ext_; + std::vector outputs_; + int32_t n_batch_; + uint32_t n_seq_max_; + + std::string add_error(int32_t code, llama_token id, llama_seq_id seq_id) const { + switch (code) { + case -1: + return "batch exceeds n_batch (" + std::to_string(n_batch_) + " tokens)"; + case -2: + return "invalid token id " + std::to_string(id); + case -3: + return "invalid seq_id " + std::to_string(seq_id) + + " (must be 0 <= seq_id < " + std::to_string(n_seq_max_) + ")"; + default: + return "llama_batch_ext_add_token failed with code: " + std::to_string(code); + } + } +}; + // RAII wrapper for llama_context* // Holds a ResourcePtr to the model to prevent premature GC. // // INVARIANT: a context is driven by a single process (the Server GenServer or -// one owning caller). The reusable `batch` below relies on that — decode-side +// one owning caller). The reusable `batch_` below relies on that — decode-side // NIFs must never run concurrently on the same context from multiple // processes. (A single process can't overlap NIF calls, so no locking.) class LlamaContext { @@ -115,11 +179,9 @@ class LlamaContext { // frees such a context before the one it points at. bool has_ctx_other = false; - // Reusable explicit batch for the decode-side NIFs (batch_eval, - // batch_eval_sample, decode_token, prefill), allocated once on first use - // and grown on demand instead of llama_batch_init/free per call. - llama_batch batch{}; - int32_t batch_capacity = 0; + // Reusable batch for every decode on this context, created on first use. + // Its capacity is fixed at n_batch, so one allocation serves all calls. + std::unique_ptr batch_; // Whether this context was created with kv_unified: true. Captured because // llama_kv_cache::seq_cp aborts (GGML_ASSERT(is_full), @@ -137,7 +199,7 @@ class LlamaContext { // caller, the decision engine, leaves either embeddings off or pooling NONE. bool pooled_embeddings = false; - // Shape of the last *successful* llama_decode on this context. + // Shape of the last *successful* decode on this context. // // `sampler_sample_at/3` takes its index straight from Elixir, and // llama_sampler_sample -> llama_get_logits_ith aborts on anything upstream's @@ -154,22 +216,10 @@ class LlamaContext { int32_t n_outputs_last = 0; std::vector logits_last; - void record_batch(const llama_batch& b) { - logits_last.assign(static_cast(b.n_tokens), 0); - if (!b.logits) { - // A null logits array means llama.cpp outputs the last token only - // (llama-batch.cpp:120-129). - n_outputs_last = b.n_tokens > 0 ? 1 : 0; - if (b.n_tokens > 0) logits_last.back() = 1; - return; - } - n_outputs_last = 0; - for (int32_t i = 0; i < b.n_tokens; i++) { - if (b.logits[i]) { - logits_last[static_cast(i)] = 1; - n_outputs_last++; - } - } + void record_batch(const TokenBatch& b) { + logits_last = b.outputs(); + n_outputs_last = static_cast( + std::count(logits_last.begin(), logits_last.end(), 1)); } void forget_batch() { @@ -201,22 +251,16 @@ class LlamaContext { live::add(this); } - // Returns the reusable batch with capacity for at least n tokens - // (per-token seq-id capacity 1 — all decode builders use single-seq - // entries). Contents are stale; the caller fills 0..n-1 and n_tokens. - llama_batch& reserve_batch(int32_t n) { - if (batch_capacity < n) { - if (batch_capacity > 0) llama_batch_free(batch); - batch = llama_batch_init(n, 0, 1); - batch_capacity = n; - } - return batch; + // The reusable batch, emptied. Contents of the previous call are dropped. + TokenBatch& fresh_batch() { + if (!batch_) batch_ = std::make_unique(ctx); + batch_->clear(); + return *batch_; } ~LlamaContext() { std::lock_guard lock(live::mutex()); live::remove(this); - if (batch_capacity > 0) llama_batch_free(batch); release(); // llama_free the draft first: gemma4-assistant still holds // cparams.ctx_other until that returns. Then drop the Fine keep. @@ -226,6 +270,7 @@ class LlamaContext { // Frees the llama context, leaving this object empty. Called by the // destructor, or by the exit hook for a context the BEAM still references. void release() { + batch_.reset(); if (ctx) llama_free(ctx); ctx = nullptr; } @@ -327,7 +372,7 @@ class LlamaSpeculative { std::atomic us_ckpt{0}; // Everything in the speculative iter NOT inside the three hot-path // timers above. On Metal this is dominated by implicit GPU-sync waits - // from the previous iter's async verify decode (llama_decode returns + // from the previous iter's async verify decode (llama_process returns // before the command buffer completes; the wait lands on the next // unrelated allocation in the next iter). Sum of draft+verify+sample // +other ≈ us_total. diff --git a/docs/architecture.md b/docs/architecture.md index ca0b34b..d275300 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -162,7 +162,7 @@ sequenceDiagram Ctx->>NIF: generate(ctx, sampler, tokens, 100) loop For each token (on DirtyCPU scheduler) - NIF->>NIF: llama_decode(batch) + NIF->>NIF: llama_process(DECODE, batch) NIF->>NIF: llama_sampler_sample(sampler, ctx, -1) NIF->>NIF: llama_sampler_accept(sampler, token) Note over NIF: Check for EOG token @@ -191,7 +191,7 @@ sequenceDiagram Stream->>Gen: spawn_link(generate_tokens NIF) loop Token generation - NIF->>NIF: llama_decode + llama_sampler_sample + NIF->>NIF: llama_process + llama_sampler_sample NIF-->>Stream: enif_send {ref, {:token, id, "text"}} Stream-->>User: "text" (via Enum.each) end @@ -260,8 +260,8 @@ graph TD D -->|"One tick = one forward pass"| E[batch_eval NIF] subgraph "C++ Layer" - E --> F[Build llama_batch
decode tokens + prefill chunks] - F --> G[Single llama_decode call] + E --> F[Build llama_batch_ext
decode tokens + prefill chunks] + F --> G[Single llama_process call] end G --> H[sampler_sample_at per slot] diff --git a/docs/release-guide.md b/docs/release-guide.md index e2acb98..3a36c53 100644 --- a/docs/release-guide.md +++ b/docs/release-guide.md @@ -78,7 +78,9 @@ Watch for two harder ones: The NIF uses these key APIs (grep `llama_nif.cpp` for the full list): - `llama_model_*`, `llama_context_*`, `llama_vocab_*` — model/context/vocab management - `llama_tokenize`, `llama_detokenize`, `llama_token_to_piece` — tokenization -- `llama_batch_*`, `llama_decode` — inference +- `llama_batch_ext_*`, `llama_process` — inference, through `TokenBatch` + (`llama_nif.h`); the MTP path also builds the `common_batch` that + `common_speculative_process` takes - `llama_sampler_*` — sampling chain - `llama_memory_*` — KV cache / memory management - `llama_get_embeddings_*`, `llama_pooling_type` — embeddings @@ -399,6 +401,24 @@ superseded by #30415 (a different patch with the same effect), and communicator holds it) conflicts with #29413 and needs a rebase, as does `cuda-mmq-src1-reuse`, which conflicts with #31296 and #31047. +Re-checked at `9581154a2` (b12447+16), covering the 95 commits from +`9daa0aa22`, as a source diff: #1, #2 and #3 are as above. +`ggml-cpu/CMakeLists.txt`, `ggml/include`, `ggml-rpc` and `ggml-rpc.h` were +not touched, so the RPC protocol stays 10.0.0, `GGML_OP_COUNT` 99, and a +v0.8.69 worker and a v0.8.71 client still talk. The `ggml-cuda.cu` diff is +#32385 (MUL_MAT_ID on AMD) and #31617 (warp primitives), none of it near +`ggml_backend_cuda_comm_init`. Not re-measured: no Spark for this bump. + +The same range deprecates the `llama_batch` API (#32306): `llama_batch_init`, +`llama_batch_get_one`, `llama_batch_free`, `llama_decode` and `llama_encode`. +The deprecated calls still work — `llama_decode` copies the batch into a +`llama_batch_ext` per call (`llama_batch_compat`) — but the NIF moved to +`llama_batch_ext` and `llama_process` rather than build with the warnings. +A `llama_batch_ext` from `llama_batch_ext_init(ctx)` holds at most `n_batch` +entries and checks token and seq ids as they are added, so its error codes +(-1 full, -2 token, -3 seq id) are now where the NIF learns about bad batch +contents. + | # | Upstream defect | Our workaround | Still needed? | |---|---|---|---| | 1 | `GGML_NATIVE=ON` makes ggml's `-mcpu=native` probe resolve to **base ARMv8-A** on Cortex-X925/A725 with GCC 13.3 — silently, with a soft CMake warning and exit 0. Costs every `sdot`/`smmla`/SVE kernel. | `LLAMA_CPU_ARM_ARCH` + `LLAMA_CUDA_ARCH` in the `Makefile`, which must be set together. See [DGX Spark](dgx-spark.md) and [Cross-Platform Builds](cross-platform-builds.md). | `scripts/spark/verify-build-flags.sh` on an aarch64 host. If a default build (no `LLAMA_CPU_ARM_ARCH`) now reports non-zero `sdot`/`smmla`, upstream fixed the probe. | @@ -566,7 +586,7 @@ mix hex.publish ### Compilation errors after upgrade - **Missing function**: check if the API was renamed or removed in `include/llama.h` -- **Struct field changes**: check `llama_model_params`, `llama_context_params`, `llama_batch` structs +- **Struct field changes**: check `llama_model_params`, `llama_context_params`, `common_batch` - **Common library changes**: `common/chat.h` is the most volatile dependency — check `common_chat_templates_inputs` and `common_chat_msg` ### Build downloads precompiled binary instead of compiling from source diff --git a/lib/llama_cpp_ex/mtp.ex b/lib/llama_cpp_ex/mtp.ex index bc9ec1e..a76bdf7 100644 --- a/lib/llama_cpp_ex/mtp.ex +++ b/lib/llama_cpp_ex/mtp.ex @@ -115,13 +115,16 @@ defmodule LlamaCppEx.MTP do * `:mtp` - a Multi-Token Prediction head, inside the target GGUF or in a sidecar passed as `:draft_model`. The draft context is built with `ctx_type: :mtp`. - * `:dflash` - a DFlash / DFlash 2 block-diffusion drafter (GGUF - architecture `dflash`), passed as `:draft_model`. It drafts a whole - block in one pass from the target's hidden states, and is the fastest - option measured on GB10: 3.3× plain decode on Qwen3.8-27B NVFP4 against - 2.1× for its MTP head (see `docs/dgx-spark.md`). + * `:dflash` - a DFlash block-diffusion drafter (GGUF architecture + `dflash`), passed as `:draft_model`. Upstream picks the head from the + drafter's metadata, so DFlash 2 and LiliCorr drafters run under this + type too. It drafts a whole block in one pass from the target's + hidden states, and is the fastest option measured on GB10: 3.3× plain + decode on Qwen3.8-27B NVFP4 against 2.1× for its MTP head (see + `docs/dgx-spark.md`). * `:dspark` - a DSpark drafter (DFlash plus a Markov head), passed as - `:draft_model`. + `:draft_model`. `init/2` refuses any other DFlash-family drafter for + this type; `:dflash` runs them all. * `:draft_model` - A separate `LlamaCppEx.Model` holding the drafter. Required for `:dflash` and `:dspark`. A DFlash drafter computes with the target's embeddings and LM head; its context gets a backend for every diff --git a/lib/llama_cpp_ex/server.ex b/lib/llama_cpp_ex/server.ex index 09303aa..23d9f71 100644 --- a/lib/llama_cpp_ex/server.ex +++ b/lib/llama_cpp_ex/server.ex @@ -85,7 +85,7 @@ defmodule LlamaCppEx.Server do ### `[:llama_cpp_ex, :server, :kv_pressure]` - Emitted when a forward pass hit KV-cache pressure (`llama_decode == 1`) and + Emitted when a forward pass hit KV-cache pressure (the decode returned 1) and the server recovered by purging idle slots' cached prefixes and/or splitting the batch. @@ -1686,7 +1686,7 @@ defmodule LlamaCppEx.Server do end # Idle slots whose retained prefix-cache KV may be dropped by the NIF under - # KV pressure (llama_decode == 1). Active slots are never purgeable. + # KV pressure (the decode returned 1). Active slots are never purgeable. defp purgeable_seq_ids(state) do for {seq_id, slot} <- state.slots, slot.state == :idle, slot.cached_pos > 0 do seq_id diff --git a/mix.exs b/mix.exs index 368785e..94e8c7d 100644 --- a/mix.exs +++ b/mix.exs @@ -133,7 +133,7 @@ end defmodule LlamaCppEx.MixProject do use Mix.Project - @version "0.8.70" + @version "0.8.71" @source_url "https://github.com/nyo16/llama_cpp_ex" def project do diff --git a/test/nif_guards_test.exs b/test/nif_guards_test.exs index 7e55854..1c47d93 100644 --- a/test/nif_guards_test.exs +++ b/test/nif_guards_test.exs @@ -77,12 +77,12 @@ defmodule LlamaCppEx.NIFGuardsTest do # * `:message` — `{:error, "invalid seq_id N (must be ...)"}`, for NIFs # already returning a string error. Adding a guard never changes a NIF's # error *type*. - # * `:upstream` — no local guard: the id is carried *inside a `llama_batch`*, - # and `llama_batch_allocr::init` - # (`vendor/llama.cpp/src/llama-batch.cpp:61-64`) range-checks batch seq ids - # and returns false, so `llama_decode` returns non-zero. The three blockers - # this enumeration was written for were distinct precisely because they - # reached `llama_memory_seq_rm` *outside* any batch. + # * `:upstream` — no local guard: the id is carried *inside a batch*, and + # `llama_batch_ext_add_token` (`vendor/llama.cpp/src/llama-batch.cpp`) + # range-checks it and returns an error code, which the NIF's + # `TokenBatch::add` turns into `{:error, "invalid seq_id ..."}`. The three + # blockers this enumeration was written for were distinct precisely + # because they reached `llama_memory_seq_rm` *outside* any batch. # * `:lookup` — no local guard: `llama_get_embeddings_seq` is a # `std::map::find` (`llama-context.cpp:921-928`) returning nullptr. # @@ -230,7 +230,7 @@ defmodule LlamaCppEx.NIFGuardsTest do Add each one with its outcome class. A seq_id that reaches a llama_memory_* call outside a batch needs a local guard - (:raise/:atom/:message); one carried inside a llama_batch is bounded + (:raise/:atom/:message); one carried inside a batch is bounded by upstream (:upstream). Guessing wrong is safe — the generated test will disagree with you. """ @@ -245,6 +245,31 @@ defmodule LlamaCppEx.NIFGuardsTest do end end + # `llama_context::decode` has GGML_ASSERT(n_tokens <= n_batch). A batch NIF + # handed more entries than that aborted the VM; the batch now refuses the + # entry past n_batch, and a token id outside the vocabulary, before decoding. + describe "batch contents" do + test "more entries than n_batch, or an unknown token, is refused", %{ + model: model, + tokens: tokens + } do + {:ok, ctx} = Context.create(model, n_ctx: 512, n_batch: 4, n_seq_max: @n_seq_max) + entries = for i <- 0..4, do: {hd(tokens), i, 0, i == 4} + + assert {:error, message} = NIF.batch_eval(ctx.ref, entries) + assert message =~ "n_batch" + assert NIF.memory_seq_pos_max(ctx.ref, 0) == -1 + + n_vocab = NIF.vocab_n_tokens(model.ref) + assert {:error, message} = NIF.decode_token(ctx.ref, n_vocab, 0, 0) + assert message =~ "invalid token id #{n_vocab}" + + # `tokens` is longer than n_batch, so this also decodes in chunks. + assert length(tokens) > 4 + assert_context_healthy(ctx, tokens, model) + end + end + # `vendor/llama.cpp/src/llama-kv-cache.cpp:502` calls `ggml_abort` on a partial # cross-sequence copy when the KV cache is not unified. `Server` guards it via # `cross_slot_sharing`; the NIF boundary did not, so any direct caller killed diff --git a/vendor/llama.cpp b/vendor/llama.cpp index 9daa0aa..9581154 160000 --- a/vendor/llama.cpp +++ b/vendor/llama.cpp @@ -1 +1 @@ -Subproject commit 9daa0aa22b8aa0143344d7549dee1f0e32f048bb +Subproject commit 9581154a2250cc9e6948e4c2a2fc4b4e19d1a57e