Repository navigation
server: preemption notices, asynchronous parks and exact concurrency (pin branch on b11577) - #269
danielhanchen wants to merge 118 commits into
Conversation
… on b11577 The change carried by #210, applied to upstream b11577 alone (first carried on b11540 by #260). #210 sat on b10871 with Inkling merged in, so its diff carried an old ggml-org#25731 and no longer composes with the nightly.
…dent, deficit-aware victim choice A parked head that has waited PREEMPT_ROTATE_US is owed the cells: a prompt that has never been resident waits until it is back, so a stream of short requests can no longer overtake it for good. Rotation now considers every generating resident, the longest included, and a held prompt that sits on cells, choosing the fewest cells that make room; a resumed slot is not rotated again before it has decoded 32 tokens, or one token and 2 s. Ordinary victim choice parks the single eligible slot that covers the shortfall where there is one, and LLAMA_SERVER_PREEMPT_POLICY=largest can now pick the largest slot. /props reports a preempt block (enabled, ram_mib, async, notices, recurrent, granularity). A pinned park buffer grows by half again instead of to the exact size, within --preempt-ram, and update_preemption() returns at once with --preempt-ram 0 and nothing parked.
…, /props and buffer growth
…ft in --preempt-ram Rounded to 1 MiB and taken whenever it fitted, the reserved slack filled a small budget: under --preempt-ram 2 MiB a 0.5 MiB state held 1 MiB, and the rotation that needs the head's and a resident's states at once was refused. Round to 64 KiB, and grow only where the budget still holds a second state of the grown size.
…n b11577 The decision size check moved into slot_prompt_rejected, so the n_past clamp upstream added beside it needs its own copy of the first decision position.
supports_op keeps the scheduler from sending a page table over RPC, but the server rebuilds whatever graph a client submits and computes it without asking, so a paged node would reach a backend that trusts its page ids and counts. graph_compute now rejects a FLASH_ATTN_EXT node with src[5] and drops that connection. test-rpc-paged-fa sends the graph from its own client against a CPU server: on the previous commit the paged graph computes, here it is refused and the server keeps serving.
…f asserting The paged pool files every cell under its position's page, and find_slot asserts the position is not negative. A sequence state with a negative position, from a corrupt or foreign file, reached that assert and aborted the process. state_read_meta now fails the restore through its error path, so llama_state_seq_set_data returns 0. test-exact-pages patches the first position of a saved sequence to -1: on the previous commit the restore aborts on the assert, here it returns 0 and the unpatched state still restores.
…rror return state_read_sinfo refuses a whole-context restore before reading a byte, so the assert in state_read_meta's whole-cache branch is not reachable in this mode. test-exact-pages now holds that in place: llama_state_set_data on a saved whole-context state returns 0 and the process lives.
…backend_device_i Appending event_query to ggml_backend_device_i changed its layout without a backend API version bump, so a backend module built against the old layout would load and the core would read the device's reg pointer as event_query. A fork-side bump would instead collide with upstream's next one. The member and every NULL initialiser of it are gone, so the interface is upstream's again; the CUDA backend returns its implementation from get_proc_address as "ggml_backend_dev_event_query", and ggml_backend_event_query() and ggml_backend_dev_supports_event_query() look it up there, keeping the blocking fallback for a backend that does not publish it. test-event-query checks that a CUDA device reports the query and that every device that does reads a completed event as complete.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
@codex review |
|
@codex security review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1161f5947b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if (const ggml_tensor * pages = ggml_cuda_fattn_pages(dst)) { | ||
| GGML_ASSERT(dst->src[0]->ne[0] == 256 && dst->src[2]->ne[0] == 256); | ||
| GGML_ASSERT(dst->src[1]->type == GGML_TYPE_F16 && dst->src[2]->type == GGML_TYPE_F16); | ||
| GGML_ASSERT(dst->src[3] && dst->src[0]->ne[3] == 1); | ||
| GGML_ASSERT(pages->type == GGML_TYPE_I32 && ggml_is_contiguous(pages)); |
There was a problem hiding this comment.
When a FLASH_ATTN_EXT operation carries src[5] but otherwise forms a valid ordinary-attention operation, ggml_cuda_flash_attn_ext_supported() ignores the page table and can advertise CUDA support for configurations such as 128-wide heads, non-F16 KV tensors, a missing mask, or a nonzero softcap. Dispatch then reaches these paged-path assertions and aborts instead of allowing the scheduler or caller to reject the operation. Mirror the paged-path constraints in the CUDA support check.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
I have left this one. The mismatch is real at the ggml level, but nothing in the tree can build such a node. The only producer of a paged FLASH_ATTN_EXT is llama-graph, which attaches the page table only when the exact-mode KV cache was built, and that constructor already refuses heads that are not 256 wide, MLA, non-F16 K/V, a non-unified or transposed cache, SWA, soft-capped models and layers not on CUDA; the KV mask is always present. The RPC server refuses any FLASH_ATTN_EXT carrying src[5], and test-backend-ops builds only the supported shape (3996/3996 on CUDA, the paged cases included).
| size_t preempt_ram_budget() const { | ||
| return params_base.preempt_ram_mib < 0 ? SIZE_MAX : (size_t) params_base.preempt_ram_mib * 1024 * 1024; |
There was a problem hiding this comment.
On a 32-bit build, a valid positive setting such as --preempt-ram 4096 wraps this multiplication to zero, while larger values wrap to a small unrelated budget. The server consequently drops and recomputes parked states as though no RAM were available, despite the configured multi-gigabyte allowance. Perform the conversion in a wider type and clamp or reject values that exceed SIZE_MAX.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
I have left this one. The multiplication can wrap where size_t is 32 bits, but the only effect is a smaller park budget: every enable check reads preempt_ram_mib != 0, so a wrapped value just makes preempt_fits_budget() refuse and the slot falls back to recompute, which is the existing and correct path. Output does not change and nothing is lost. llama-server is also not built for any 32-bit target in CI (the wasm job builds only test-backend-ops, Android is arm64-v8a only, the x86 release entries are commented out), and a 32-bit process could not hold a multi-gigabyte pinned budget anyway.
|
|
||
| preempt_wait_for_shift(); | ||
|
|
||
| llama_memory_update(ctx_tgt); |
There was a problem hiding this comment.
When context shifting or cache reuse has set preempt_shift_pending, preempt_wait_for_shift() clears that flag and this call ignores the result of llama_memory_update(). A false result also represents FAILED_PREPARE or FAILED_COMPUTE, so update_preemption() can immediately serialize and remove a sequence whose positions were changed but whose K values were not shifted, producing a corrupt restore and incorrect output. Propagate the failure, or distinguish it from NO_UPDATE, before allowing the sequence to be parked.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
I have left this one, because a false return from llama_memory_update cannot signal a failed shift here. No memory implementation's init_update produces FAILED_PREPARE or FAILED_COMPUTE: llama_kv_cache_context sets only SUCCESS or NO_UPDATE, recurrent memory always returns NO_UPDATE, and the composites only combine their children's status. So false means NO_UPDATE. The real failure, a K-shift graph that fails to allocate or compute inside llama_kv_cache::update, still returns true from apply() and memory_update(), so checking the bool would never catch it and would only block parks after a harmless no-op. llama_context::decode treats that case the same way (memory_update(false) with the result ignored), and the pending shift stays recorded for the next update to retry.
A grouped child continues from the parent's cells through seq_cp, which LLAMA_EXACT_CONCURRENCY refuses (a page belongs to one sequence), while copy_prompt_to() still clones the parent's prompt metadata. The child then decoded its suffix at the shared position with no KV prefix behind it. Each task now keeps its whole prompt in exact mode.
…eloaded The whole-context restore is already refused under LLAMA_EXACT_CONCURRENCY, but llama_state_get_size/get_data and llama_state_save_file still succeeded, so llama-completion --prompt-cache wrote a session that made the next run fail to load it and exit. The save now fails the same way, before the file is opened, so no session is left behind and the next run starts fresh. Per-sequence saves, which the server uses, are unchanged.
|
@codex review |
1 similar comment
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2bf0e4afed
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if (exact_pages && seq_id_src != seq_id_dst) { | ||
| LLAMA_LOG_ERROR("%s: exact concurrency does not support copying cells between " | ||
| "sequences (%d -> %d); ignoring the copy\n", | ||
| __func__, seq_id_src, seq_id_dst); | ||
| return; |
There was a problem hiding this comment.
With LLAMA_EXACT_CONCURRENCY and n_cmpl > 1, this no-op still leaves every child completion without the parent's KV prefix: the current tools/server/server-context.cpp:6350-6370 path calls copy_state_to(), which invokes seq_cp() and then clones the parent's prompt metadata before marking the child ready. The fresh evidence beyond the earlier decision-task report is that the follow-up disables only grouped decision tasks, while ordinary multi-completion requests still use this path; each child therefore decodes at the suffix position with missing prefix state and can return incorrect output. Reject n_cmpl > 1 in exact mode or independently restore each child's state.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
I have left this one, because exact mode already refuses n > 1 before any child exists. handle_completions_impl throws when task.params.n_cmpl > 1 && common_exact_concurrency(), ahead of the only add_child() call in the server, and that becomes a 400 invalid_request_error. n is a schema alias of n_cmpl, and every completion route (/completion, /v1/completions, /v1/chat/completions, infill, responses, messages) goes through that function, so copy_state_to() is never reached by an exact-mode child; decision prompt sharing, the only other child path, is off in exact mode since 40cd744. A live server with LLAMA_EXACT_CONCURRENCY=1 answers 400 to all four n=2 variants and logs no refused seq_cp.
| if (slot.preempt_is_out()) { | ||
| n_preempted_slots++; |
There was a problem hiding this comment.
When asynchronous preemption is enabled, preempt_is_out() is also true for SLOT_STATE_PREEMPTING and SLOT_STATE_RESTORING, so llamacpp:requests_preempted counts transfers as parked requests. In particular, a restoring request already owns KV cells and is no longer waiting for room, contrary to the metric's documented meaning and the /slots definition of is_preempted; monitoring and alerts therefore overstate the parked queue during every asynchronous restore. Count only state == SLOT_STATE_PREEMPTED here, while exposing transfers through the separate transfer state.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
The count is intended, so I have kept it and fixed the wording in 02dcf73. requests_preempted counts requests that are out of decoding because of preemption: parked, and those whose copy out or back in is still in flight. None of them produces tokens. Studio, the consumer we ship, reads it as "parked > 0" to excuse a silent stream, and an in-flight transfer is just as silent, so dropping those states would flag a stall during every asynchronous restore. /slots already separates the two with is_preempted and is_transferring for residency accounting. The HELP text and README said "parked, waiting for room"; they now describe what is counted.
The token was sampled before the park and post_decode discards the output, but a backend sampler runs on every output of its sequence and, at one output per sequence, draws its uniform from the stream's RNG. Every recompute park left the stream one draw ahead.
…onse A multi-prompt /v1/completions request is one ordinary task per prompt, each of which can be parked; the non-streaming OAI merge kept only the first result's preempt object, so parks of the other prompts were dropped.
|
@codex review |
1 similar comment
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b8f7b90d72
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if (exact_pages && seq_id_src != seq_id_dst) { | ||
| LLAMA_LOG_ERROR("%s: exact concurrency does not support copying cells between " | ||
| "sequences (%d -> %d); ignoring the copy\n", | ||
| __func__, seq_id_src, seq_id_dst); | ||
| return; |
There was a problem hiding this comment.
With LLAMA_EXACT_CONCURRENCY=1, this still reports success through the void public API while leaving the destination empty. Fresh evidence beyond the fixed server paths is examples/parallel/parallel.cpp: with --kv-unified --prompt-sharing, lines 277-279 copy the system prefix into each client and lines 330-351 then start each suffix at n_tokens_system; this no-op therefore makes every client decode at the suffix position without the system KV prefix and produces incorrect output. Either support independent destination pages or reject exact mode in every remaining cross-sequence-copy application instead of silently continuing.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
I have left this one. LLAMA_EXACT_CONCURRENCY is an opt-in for llama-server, where every cross-sequence copy is now either avoided (decision prompt sharing is off in exact mode) or refused up front (n > 1 answers 400). The remaining callers are the standalone examples and batched-bench, which are not an exact-mode surface: a page belongs to one sequence, so prompt sharing is exactly what the mode gives up. Running one of them with the variable set is not silent either, since every refused copy logs an error that names exact concurrency and the two sequences. Refusing rather than aborting is deliberate, so a stray copy cannot take a server down.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
… not only its name A CUDA, ROCm or MUSA module from before these kernels has the same registry name and API version, answers the paged attention probe through its dense path and has no batch-invariant dispatch. The check now also requires the ggml_backend_cuda_set_exact_decode_width extension that ships with them.
- decision-group accounting no longer depends on the parent staying resident: every member carries the shared length, and an idle slot's cached prompt counts as the member it last served - the sleeping /props answer carries the live async value, as the awake one - a state size of 0 (the size query's error return) is not parked as saved - a RAM restore does not call common_speculative_begin(), which reset the draft sampler's random stream though the per-sequence state never left - the replay test checks token counts, not prompt_ms against predicted_ms
Cells held by at least one sequence, a shared cell counted once and a partly used block in full, and optionally those held only by given sequences. The server's preemption planner reads its occupancy and idle-reclaim figures from it instead of rebuilding them from the task tree. -1 where a memory cannot tell (recurrent state, DSV4); the planner keeps its estimate there.
- occupancy and idle reclaim come from llama_memory_n_cells_used(), so a cleared, finished or idle family member can neither hide nor double the cells its siblings still share; the task-tree estimate is only the fallback - a child slot's stale cache is cleared at launch, after the prompt cache has kept it: the parent's prompt replaces it, and until then it could be neither used nor parked - the planner keeps only the prefix the batch builder reuses: none for a stateless task, and none under exact concurrency, where cached cells come from other ubatches than a cold prompt's - accepted tokens waiting for a replay after a checkpoint restore survive a park or a rewind - a cache-reuse shift is applied to the draft too before anything can park it
…et's The draft context splits each batch at its own ubatch size, so with another size its splits move with what else is in the batch.
- test-rpc-paged-fa: a client that fails in its own setup is not a refusal - test-paged-fa-supports also checks ROCm and MUSA devices - test-exact-pages fails on a model that does not load; a refused context is still a skip - test-exact-pages / test-exact-weights run against LLAMA_TEST_EXACT_MODEL (256-wide K/V heads), not the generic fixture they always skipped on - test-memory-cells-used registered against the generic fixture - server tests: the late-prompt test overlaps by observing the leader, new tests for an exact-mode prompt reusing no cached prefix
LLAMA_SERVER_PREEMPT_GRANULARITY plans in blocks the KV memory does not have, so the occupancy is not read from the cells while it differs from them.
Every wait for a precondition (a resident growing into the pool, a leader decoding N tokens, a stream starting, slots coming back) now runs as long as the server keeps prefilling or decoding, and fails only when the request it depends on ends first or nothing moves for 120 s. Two upper bounds that were fixed seconds (the stream length the head may wait behind, and the wait behind a resident that ends by itself) now scale with the speed measured in the same run. A slow runner only makes the tests slower; the assertions on the behaviour under test are unchanged.
…ny client Two 2000-token generations in 3840 cells overflowed only if the second started within ~180 tokens of the first; a slower client gave no park at all. 4000 each in 6144 cells (a model copy with a longer training context) overflow unless the second is ~1900 tokens late, and the shorter one then holds more than the 1 MiB budget, so the park still drops its cells. The planner-off variant sends both prompts in one request.
- server: guard the second preempt_apply_shift() (the --cache-reuse shift before the decode loop) like the first; a throwing shift now fails the requests instead of terminating the server. - server: charge a park's RAM per buffer, so spare target capacity no longer hides the draft state's allocation. - server: a generating slot bounds its draft reservation by its own prompt, which a context shift made shorter than the task's; a slot still prefilling or re-prefilling keeps the whole input as the bound. - llama: llama_memory_n_cells_used() counts an owner given twice once, and an occupancy-only query reads the cells' used count (one cell per block) or skips the rest of a decided block. - tests: test-rpc-paged-fa counts only a failure armed after setup as a refusal; test-exact-pages and test-exact-weights fail when no exact context can be made; test-exact-pages checks the two counts agree.
- server: victim and rotation estimates, and the n_cmpl > 1 family ended when nothing can be parked, count the cells that would actually be freed (llama_memory_n_cells_used with the sequences given), so a prefix shared with another sequence no longer inflates them; rotation reads its candidates together. - server: the resume projection charges a prompt held behind the head only for the cells it sits on. - server: the growth estimate keeps the capacity of a grow-only park buffer when the next state is smaller, and counts a held draft buffer. - server: exact concurrency disables the RAM prompt cache, since every prompt is prefilled in full there. - llama: the indexed hybrid memory refuses shared tokens and isolates prompts under exact concurrency, as the hybrid memory does. - tests: test-paged-fa-supports fails instead of skipping when a build with the flash attention kernels refuses a valid paged shape; the async test gate reads the log up to its last startup line; the M-RoPE refusal test gets a batch that passes the batch geometry check.
- llama: llama_memory_update() returns -1 when an update is still due after the call (a K-shift that failed to allocate or compute only logs); the server then fails the requests instead of parking a sequence whose shifted positions do not match its K. - server: a generating slot parked in RAM after a context shift projects its final size from its own prompt; the planner's kept prefix takes the batch builder's shared-prefix and decision bounds; the resident projection is summed in 64 bits. - llama: under exact concurrency, seq_id -1 (the whole cache) is refused on the per-sequence save paths too, before a file is opened, and a context without sequence memory (an encoder, whose batches are not isolated) is refused. - llama, ggml-cuda: document that the exact decode width is process-wide, so a context created later with a wider n_seq_max changes the numerics of contexts already decoding; GGML_CUDA_BATCH_INVARIANT_MAX_COLS pins it.
- A failed pre-decode shift (or any abort_all_slots() error) clears the slots' cached prompts, as a failed decode does: the batch builder had already counted tokens that never reached the cells, so an identical retry failed with an invalid batch. - A prompt held in full takes no extra cell in the live reservation and the victim estimate, as on restore: repeating a fully cached prompt no longer purges another slot's cache. - The victim fallbacks of both policies cover every resident that cannot be parked (n_cmpl > 1, TTS, realtime transcription), so an older text request beside a realtime transcription can still be parked. - Realtime transcriptions take their next audio rows before planning, and the reservation charges the rows a step holds (the first step is the whole prefix). - The --preempt-ram budget is computed in 64 bits and saturated, so a 32-bit build does not wrap 4096 MiB to 0.
- server: a context shift moves the aLoRA invocation start with its tokens, so a recompute park replays the adapted tokens with the adapter. - ggml-cuda: a captured CUDA graph is recaptured when the exact decode width (the batch-invariant column bound) has changed since its capture; the reused graph replayed the kernels chosen under the old bound. - llama: the MSA cache forwards the base cache's allocation granularity and, under exact concurrency, refuses shared tokens and isolates prompts, as llama_kv_cache does; exact concurrency also refuses encoder-decoder models, whose encode() packs sequences without isolation. - common, speculative: exact concurrency refuses -pps and a branching draft (-np > 1), whose prompt copies it refuses. - tests: the late-prompt test gives the follower a long leader to overlap.
- common: one helper, common_exact_concurrency_seq_copy_ok(), that every tool copying cells between sequences calls before loading a model; under exact concurrency it logs why and refuses. Called by lookahead (always), parallel and batched-bench (-pps) and speculative (-np > 1); the per-tool checks of round 13 move into it. The audited copiers are these four; the server's only copy is for n_cmpl > 1, which exact mode already rejects. - lookahead, parallel, speculative: a model or context that failed to load is reported instead of dereferenced (a refused exact context segfaulted lookahead on CPU). - tests: CTest runs each of the four copying tools under LLAMA_EXACT_CONCURRENCY=1 and expects the refusal. - llama: MSA attention does not read the page table, so exact concurrency refuses it at graph build, like the other layouts without one. - llama: context creation enforces the exact batch geometry itself (a batch holding a whole ubatch beside a decode step of every sequence), so library callers that skip the common init get the same refusal.
The common logger writes on its own thread and is never torn down, so a line logged just before a tool returns from main() can be lost (on Windows CI only the timestamp prefix of the refusal came out). Flush it on every refusal exit: common_exact_concurrency_seq_copy_ok(), the failed common_exact_concurrency_init() in argument parsing, the refused model, context and served-context paths of common_init_result, and the failed-load reports of lookahead, parallel and speculative.
- _wait_processing() waits while the server makes progress, like _await(), instead of a fixed 30 s deadline. - The started-slot test tokenizes its prompts first and counts parks from the responses, which a poll of /slots can miss; its "trimmed" log requirement is dropped, since the batch builder may correctly drop the stale cells first. - The partial-prefill park test uses 64-token chunks, a 1500-token follower tokenized before the resident starts, and sends it once the resident holds 1000 cells, so the follower is parked mid-prefill over a wide window. - The sliding-window test gives the follower ~500 steps before the stale cache would be purged, instead of ~40. - The test harness forwards --spec-ngram-mod-n-min / -n-max, which the n-gram reservation test sets. - test-rpc-paged-fa also requires the RPC server's own log of refusing the paged graph, so a client that fails locally (an error return or an abort before sending) is no longer read as the server's refusal.
- Every common_init_from_params() caller in examples/ and tools/ that used the model or context unchecked now reports a
missing one and returns (lookup, lookup-stats, lookup-create, speculative-simple, cvector-generator, mtmd-debug, and
the model-only checks of embedding, retrieval, finetune, results); exact concurrency refuses the context of a model
it cannot run and leaves the model loaded, which these dereferenced. CTest runs the four lookup / speculative-simple
tools under LLAMA_EXACT_CONCURRENCY=1 and expects the report.
- server: an empty prompt passes the up-front prompt checks, as the batch builder answers it with an empty response;
an embedding-only server answered {"prompt": [[]]} with a 500 for logits it never needed.
- common/speculative: DSpark and DFlash2 split one batch into equal blocks, so their drafting sequences share the
tightest per-call bound (common_speculative_n_draft_shared); other drafters keep a bound per sequence.
…g of a failed preemption test The round-15 widening left the busy request a 700-token tail after the wait, which the Windows CI runner (1.6 s a step on tinygemma3) could not finish inside the 600 s read timeout; it is not a hang (on Linux the request ends in 5-10 s). The test now waits for 40 decoded tokens of a 200-token busy request (a 160-token tail, as before round 15), with a stale cache of 450 so its cells, the stale cache and the follower's first chunk still overflow the pool and the planner trims the follower's started slot; both requests are bounded at 450 s and a busy request that does not finish fails with its own message. A conftest hook records each phase's report on the item, and the preemption fixture prints the last 300 lines of the server log when its test failed, which CI keeps nowhere else.
… say so The shared bound is the tightest per-call bound among the drafting sequences, so it is at most every one's own bound and no draft exceeds its caller's limit; the comment said each draft was truncated afterwards.
… check the speculative-simple draft context, validate the decode figure before any context - common_exact_concurrency_position_edit_ok() refuses a tool that would move or rescale cached cells (seq_add / seq_div); self-extend (--grp-attn-n > 1) is refused in common_exact_concurrency_init(), passkey after parsing, llama-completion with --context-shift after parsing - speculative-simple returns 1 when the speculative types need a draft context and it could not be created - llama_set_exact_decode_tokens() checks the explicit column bound for one sequence before any context exists, so a figure past it is refused instead of stored for good - tests: test-exact-decode-tokens and ctests for passkey, completion --grp-attn-n 2 and --context-shift
…cation tests' network calls - _wait_processing() takes alive= and every caller passes its request, so a request that ended first fails at once - the two-generation, trimmed-slot and FCFS arrival-order tests submit their prompts in one request, so they overlap whatever the client's speed - the remaining overlap tests check from the server log that the requests were in flight together, and those asserting a park first check that something was parked, failing with a precondition message otherwise - the cached-prefix test gates its FCFS outcome on /props preempt.policy - test_preempt_notify bounds its /slots GET and streamed POSTs - README: a parked stream is kept alive with : preempt-keepalive
… covering-victim test queues its prompts together under cost - task_prompt_rejected() checks as launch_slot_with_task() and then the batch builder do: the aLoRA set, the token ids, then the empty prompt, logits and sizes; a request the batch-builder checks refuse gets the sampler error first when its sampling parameters cannot be built, as before - test_embedding: an invalid token is a 400 and refused sampling parameters a 400 on an embedding-only server, as on base - test_one_victim_that_covers_the_shortfall_is_parked_instead_of_several_small_ones sends all four prompts in one request under LLAMA_ARG_PREEMPT_POLICY=cost and checks the one parked slot is the covering one
…ront prompt check task_prompt_rejected() also runs on the HTTP thread (tasks_prompt_rejected() before the tasks are posted). The aLoRA-set check moved into it read params_base.lora_adapters and their scales there, while SERVER_TASK_TYPE_SET_LORA replaces that vector on the server loop: a data race, seen as a spurious 400 "Cannot run multiple aLoRAs in a single request". The check is back where it was, only in launch_slot_with_task(), which runs on the server loop like every other reader and writer of the adapters (GET_LORA, SET_LORA, construct_lora_list), so no lock is needed and none is added. What the up-front check still reads off the server loop, none of it written after loading: the vocabulary and model (validate(), common_sampler_init() in the sampler trial, llama_model_n_ctx_train()), the context's sizes and flags (llama_n_ctx_seq, llama_n_batch, llama_n_ubatch, llama_get_memory, llama_get_causal_attn, which the server never sets), the constant params_base.kv_unified_per_slot, and the task itself. A request that both enables several aLoRAs and fails a check run up front now gets that check's error instead of the aLoRA one; both are refusals, and none of them is a 200.
…h geometry llama_set_exact_decode_tokens() raised the process-wide figure after checking only the explicit column bound, while every exact context's constructor requires n_batch >= n_ubatch + n_seq_max * figure. A context valid at the old figure (132/128/2 at 1) kept decoding after the figure grew to 8, though an identical new one is refused, and a prompt beside a neighbour was then split into other ubatches than alone. - each exact context with more than one sequence holds its slack, (n_batch - n_ubatch) / n_seq_max, in a process-wide multiset for its life (llama_exact_geometry_hold, a member, so a constructor that throws later releases it); the check and the record are one step under the lock the figure uses, so no figure grows in between - the setter refuses, logging and leaving the old figure, a figure past the tightest live slack - decode() and encode() refuse a step, before touching the batch or memory, when the figure has outgrown the context anyway - test-exact-decode-tokens -m MODEL (new ctest test-exact-decode-tokens-live-context): a vocab-only 132/128/2 context refuses 8 and 3, accepts 2; freed, 8 is accepted; an identical context is then refused and a 144/128/2 one accepted, which refuses 9
- ggml-hexagon.cpp: upstream replaced the per-op supports_op switch with htp_kparams_check() (ggml-org#30843); the page-table refusal is re-applied as a FLASH_ATTN_EXT case in the new switch, refusing src[5] before the kparams check - tests/CMakeLists.txt: upstream's behavior labels (ggml-org#31036) kept; the pin's tests added beside them with their labels unchanged
A parking llama-server that predates the stream park notices is silent for a park during prompt processing, headers included. The sync chat stream waits such a park out from a watchdog that shuts the socket down when nothing is parked, but a Winsock recv before headers cannot be woken that way, so the watchdog is off on Windows and a healthy park longer than the first-token window ended in ReadTimeout. Rather than reading the socket by hand there, do not ask that build to park on Windows. The notices shipped in the same llama.cpp change as --preempt-async (unslothai/llama.cpp#269), the one trace of them the --help probe can see before launch, so a Windows launch on a build with --preempt-ram but without --preempt-async names --preempt-ram 0 (zero, since a build from before #197 parks on a default budget) and the server never parks. With preemption on, Studio does the pausing, as on any server that does not park; otherwise the child runs as upstream. Notice builds, other platforms (where the watchdog covers the header wait) and a budget the user named are unchanged, so the existing header wait fallback still serves a build forced into parking by hand.
A parking llama-server from before the stream park notices (no --preempt-async, which came with them in unslothai/llama.cpp#269) is silent for a park, headers included. Every reader with a deadline then has to tell that silence from a stall on its own: the Windows header wait cannot, and Deep Research ended a healthy silent park as a timeout because it extends its deadline only on a park notice. Rather than teach each consumer to probe /metrics, Studio no longer turns that build's parking on anywhere: the launch names --preempt-ram 0 and Studio owns the pausing, which needs no notices. Notice builds and a budget the user named are unchanged. Only pre-notice development builds of #269 are affected.
The #260 change set (server-side preemption, preemption notices, asynchronous parks and exact concurrency) ported onto upstream b11577 (cbe28bb) as a pin branch, plus fixes for the four review findings left open on #260. It replaces #260 as the commit pinned for this feature: #260 on b11540 is refused by the pin-set sim on the current nightly tags (b11577 onward conflict in
common/common.cpp,src/llama-kv-cache.cppandtests/CMakeLists.txt). The base wasbase/upstream-cbe28bb3c; after the b11642 merge below it isbase/upstream-2639a3b4e(b11642), so the diff here is exactly what gets pinned.Commits
common/common.cppkeeps upstream's newcommon_batch_set_zero_state()and then the exact-concurrency block;src/llama-kv-cache.cppkeeps upstream's newembd_allsplit (split_seq) ahead of the isolating split;tests/CMakeLists.txtkeeps upstream's per-testLINK llamaand addstest-exact-buftthe same way. Otherwise line-for-line server: preemption notices, asynchronous parks and exact concurrency (pin branch on b11540) #260's 69 files.n_decision_firstin the prompt-reuse bound, whose declaration server: preemption notices, asynchronous parks and exact concurrency (pin branch on b11540) #260 had moved intoslot_prompt_rejected; it is declared again where that read is, or the server does not compile.graph_computerefuses aFLASH_ATTN_EXTthat carries a page table. The client-sidesupports_opcheck only kept the scheduler from choosing RPC; a direct client could still send one, and the CUDA path trusts the page ids.test-rpc-paged-fa(POSIX,GGML_RPC=ON) fails without it.LLAMA_EXACT_CONCURRENCYa restored state with a negative position fails the restore through the normal error return instead of aborting onubatch.pos[i] >= 0.test-exact-pagespatches a saved position to -1: before, the process aborted (rc 134); after, the restore returns 0 and the unpatched state still restores.state_read_sinfobefore anything is read, so the assert the review pointed at is unreachable;test-exact-pagesnow pins thatllama_state_set_datareturns 0 and the process keeps running.event_queryis no longer appended toggml_backend_device_i. That changed the interface layout without an API version bump, so an older backend module would load and have itsregpointer read as the callback; bumping the version in the fork would collide with upstream's next bump in the nightly merge. The CUDA backend now publishes the query fromget_proc_addressas"ggml_backend_dev_event_query",ggml_backend_event_query()andggml_backend_dev_supports_event_query()look it up there, and a backend that does not publish it keeps the blocking fallback.ggml-backend-impl.his upstream's again and every per-backendevent_query = NULLline is gone.test-event-querychecks CUDA reports it and CPU does not.Evidence
ctest -L main: 53 of 53. CUDA (sm_100) builds ofllama-server,test-exact-pages,test-event-queryandtest-state-seq-copy; on a B200,test-event-query,test-state-seq-copyandtest-exact-pages(Qwen3.5-0.8B) pass.Review follow-up (24bec04..f032338)
Scheduler, merged as
fix/sched-269:n_cmpl > 1request no longer turns an overflow into an error for every slot.-c 8192 -np 4 --kv-unified, parks went from 117 to 2.--preempt-policy fcfs|cost(LLAMA_ARG_PREEMPT_POLICY; the oldLLAMA_SERVER_PREEMPT_POLICYstill works), defaultfcfs. Under fcfs the newest running request is the victim and resumes go oldest first, the same order vLLM's scheduler uses.costkeeps the smallest-covering-victim choice.Models and backends, merged as
fix/models-269:supports_oprefuses pagedFLASH_ATTN_EXTshapes the paged kernel does not run (a 64-wide head with a page table passed the check and then hit the kernel assert).test-event-queryskips devices without events (GGML_CUDA_NO_PEER_COPY=ON).test-exact-buftcovers integrated GPUs.preemptrecord is sent only when the server can park, so responses without--preempt-rammatch upstream's shape.Evidence for f032338
CPU build,
tools/server/tests, 2 threads:unit/test_preempt.py, fcfs (default)unit/test_preempt.py,LLAMA_ARG_PREEMPT_POLICY=costunit/test_preempt_models.pyunit/test_preempt_notify.pyThe skips are 5 async-copy tests (the CPU backend copies synchronously) and 8 that need an MTP, hybrid, exact or other model file. The slow-marked tests ran.
--preempt-ram, tokens, status codes and errors match upstream b11577 on the fix branch, and the upstream default-flag server suite fails the same two tests on b11577 as here.llama-serverbuilds. On a shared B200, 4 chats at-c 8192 -np 4 --kv-unified --preempt-ram 3840: 2 parks for 4 x (2000 + 512) and 3 parks for 4 x (2992 + 1024), 0 rotations, 0 recomputes, 0 context errors in 3 repeats under each policy. Median aggregate throughput was 299 and 314 tok/s under fcfs, 158 and 110 under cost, and 261 and 183 running two chats at a time without preemption. The card was shared, so the spreads are wide and these are indicative only.Merge of b11612 (9b1a6a1)
The pin-set sim refused this pin on b11612 and b11617: upstream ggml-org#30023 (first in b11596) folded
tests/test-state-restore-fragmented.cppinto the newtest-llama-contextbinary, and this pin had added a neighbour check to that file. Merged b11612 into the branch: the old test and its CMake entry go as upstream removed them, the pin's own CMake entries stay, and the check that a fragmented restore leaves the neighbouring sequences byte-identical moved into upstream'sfragcase intests/llama-context/cases-kv.cpp. The base branch moved to b11612 with it.test-llama-context, 130 generated models, CPU (-t 4 -j 6)test-llama-context, CUDA (B200)test-event-query,test-state-seq-copy,test-exact-pages(Qwen3.5-0.8B), CUDAunit/test_preempt.py, fcfsunit/test_preempt.py, costtest_a_request_that_cannot_be_helped_gets_the_context_error_and_the_server_lives[ram-0], which passed in 2 reruns here and on f032338 (host load average near 500)unit/test_preempt_models.py,unit/test_preempt_notify.pyctest -L maintimedtest-llama-contextout at 900 s and at 3600 s on this host: with no thread count it runs 96 threads per model on a machine at load 500. With-t 4it finishes in one to three minutes, on b11612 and on this branch alike.Review round 3 (37254a7..ce0965c)
--preempt-ramset the context recorded a fence event after every decode, and the server scanned every slot every iteration, with nothing parked--kv-unified, but the planner summed lengths as one pool, so it parked a request for nothingllama_memory_shares_cells(): false for recurrent states, the DSV4 cache, and a KV cache with a stream per sequence; composite caches answer for their attention cache. False turns planning, the last-resort park and/props.preempt.enabledofftest_a_recompute_restore_does_not_count_its_replay_as_promptfailed on a T4: the cold first prefill outweighed the warm re-prefillsOn CPU at abf60fd:
test_preempt.py65 passed, 13 skipped under fcfs and under cost;test_preempt_models.py+test_preempt_notify.py17 passed;test-llama-context -t 4130 of 130. With--preempt-ramunset, tokens, status codes and errors match upstream b11612 (solo, n=2, three that fit, an over-long prompt, and the family pressure case).Merge of b11641 (49b2552)
The pin-set sim refused this pin on b11641: upstream ggml-org#30506 (LiliCorr drafting) gave the DFlash block a fixed size for LiliCorr, on the line where this pin bounds the block by the caller's remaining context. Kept both: the other drafters keep the per-call bound, and the LiliCorr lattice walk stops at it. The base branch moved to b11641.
test-llama-context -t 4, 130 generated modelstest-state-seq-copy,test-exact-pages,test-exact-weights,test-exact-buft,test-exact-geometry,test-llama-context-modelunit/test_preempt.py, fcfs / costunit/test_preempt_models.py+unit/test_preempt_notify.pyunit/test_speculative.py+unit/test_completion.pytest_completion_stream_with_openai_library_stops, which fails the same way on upstream without pinsReview round 4 (bc04bd4, d1766cc)
A park buffer can come back pageable after the load-time probe pinned 1 MiB, and that slot then parks synchronously, but the context kept charging the async margin and reporting
/props.preempt.async: true. bc04bd4 recomputes both from the slots when one falls back: the async margin while any slot still parks asynchronously,asynconly while every slot does.LLAMA_SERVER_PREEMPT_PAGEABLE=Nforces the Nth async park's buffer pageable fortest_a_slot_whose_park_buffer_comes_back_pageable_is_no_longer_reported_async. On a T4 that test still failed at bc04bd4:/propsis served from metadata copied at load, so it never saw the change. d1766cc keeps the state in an atomic that the HTTP thread reads. On a T4 at d1766cc the test passes under both policies.Merge of b11642 (6355265)
Upstream ggml-org#30498 (prompt cache keeps only restorable slot states) added
cache_ram_explicitbeside this pin's options incommon_paramsand reworded the--cache-ramREADME row above them; both sides kept. Itsserver-context.cppchange merged cleanly and only concerns which idle slots are saved to the prompt cache. On CPU at 6355265 (and again at d1766cc, 65 passed, 14 skipped under each policy):test_preempt.py65 passed, 14 skipped under fcfs and under cost (one cost run had one failure under host load near 1000 that did not recur in a full rerun),test_preempt_models.py+test_preempt_notify.py17,test-llama-context -t 4130 of 130, the state and exact-page tests pass, andunit/test_systemone.pyfailstest_systemone_shared_promptthe same way on plain b11642.Decode speed (a9f9160)
A 10-round A/A/B on a Colab L4 showed this branch decoding 1.0 to 1.2% slower than b11642 on the server, with or without
--preempt-ram, whilellama-benchshowed no difference. Padding showed it is code layout, not work: on hosts with the JCC erratum microcode (Skylake through Cascade Lake, which these cloud Xeons are), a branch crossing or ending on a 32-byte boundary misses the decoded uop cache, and where the per-token sampler loops land moves with unrelated code. Padding b11642's libllama by 16 bytes reproduced the slowdown; padding this branch by 16 bytes removed it. a9f9160 builds libllama with-Wa,-mbranches-within-32B-boundaries(GCC/binutils) or-mbranches-within-32B-boundaries(clang) on non-MSVC x86; object code differs only by padding (43 of 43 translation units identical with nops and padding prefixes dropped).Colab L4, Qwen3-0.6B Q8_0, 10 rounds in random order, fresh server per arm, median of 3 requests; gen tok/s, delta against the mean of the two base arms:
An A100 run showed the same pattern. On CPU at a9f9160:
test_preempt.py65 passed, 14 skipped under each policy, notify and models 17,test-llama-context130 of 130 (and 130 of 130 on CUDA).Merged prebuilt on a T4
The whole pin set of #211 on b11641 with this PR at 49b2552, rebuilt on Kaggle with the nightly's merge steps (tree hash equal to the local sim), CUDA sm_75: full build ok,
test-chatok,test-backend-opsMUL_MAT 2512/2512, MUL_MAT_ID 990/990, PAD 37/37, GET_ROWS 216/216,feature_matrix --gpuall 7 features, and this PR'stest_preempt.pyon the merged server 70 passed, 8 skipped under each policy.GPU runs
T4 x2 (sm_75, Kaggle) on d1766cc, CUDA build:
unit/test_preempt.py, fcfs (GPU 0) and cost (GPU 1)unit/test_preempt_notify.py+unit/test_preempt_models.pytest-event-query,test-state-seq-copy -m Qwen3-0.6B-Q8_0 -np 2 -ngl 99The 8 skips need an MTP, hybrid, M-RoPE or exact-concurrency model file.
T4 on 37254a7, upstream b11612 built the same way,
-c 4096 --parallel 4 --kv-unified -ngl 99, 4 concurrent requests of 1500 prompt tokens and 512 out:--preempt-ramunset--preempt-ram 2048Timing on the same card, arms rotated, 10 measurements each, ranges in brackets: a single request decodes at 177.9 [173.8-188.2] tok/s on b11612, 177.5 [175.1-187.7] with
--preempt-ramunset, 180.7 [175.9-186.2] with it set; prompt processing and a first batch of 4 x 1000 tokens are likewise within b11612's own spread.AMD Strix Halo (gfx1151, ROCm/HIP) on 37254a7 against b11612, Qwen3-0.6B Q8_0,
-c 4096 --parallel 4 --kv-unified, 4 chats that do not fit together: b11612 finished 0 of 4 with 10 context errors; this PR with--preempt-ramfinished 4 of 4 with 3 parks, and without it wrote the same text as b11612. Single-request decode 181.7 tok/s on b11612, 193.8 without and 182.0 with--preempt-ram(one machine, not an exclusive timing).Composing with master's pins on b11658 to b11669 (3a68a5c, f57ea1b)
Master now pins ggml-org#30520 (Nemotron 3.5, upstream from b11669), which rewrites the LiliCorr walk's loop header in
common/speculative.cpp. This pin inserted its per-call bound right under that header, so the pin-set sim refused it on b11658, b11662 and b11669. 3a68a5c moves the bound three lines down, after the row pointers are computed and before anything is read or pushed, so the behaviour is the same and the hunks no longer touch.f57ea1b, test helper only: upstream ggml-org#30580 (b11669) added a single-string
override_kvto the server tests'ServerProcess, beside this pin's list of the same name, so on b11669 the merged helper passed the whole list as one argument andtest_preempt_models.pycould not start its server. The pin's list is nowoverride_kvs.pin_contractandmerge_checkscleantest-chat,test-backend-opsMUL_MAT 2512, MUL_MAT_ID 1043, PAD 37, GET_ROWS 340test_preempt.py+test_preempt_notify.py+test_preempt_models.py+test_speculative.pytest_a_server_that_never_asked_for_parking_behaves_as_upstream, whose two requests never overlapped at load average 700; it passed 4 of 4 rerunstest_a_parked_head_is_not_overtaken_by_a_stream_of_short_requests, a CPU-only test, timed out when run without a thread cap at load 500 and passes in 42 s on the merged tree's CPU build with the suite's 2 threadsb11669 composing also clears the earlier blocker on that tag: ggml-org#25731 was repinned on master.
Merge of b11793 and review round 5 (18365c1, 27ce31b..5149d08)
18365c1 merges upstream b11793. The review of that head found:
preempt_parkablecheck used by victim choice, rotation and the test knob, plus a backstop inpreempt_park()test-server-preempt-kindfails before, passes after; a probe over the real selector picks a transcription or TTS slot before, never aftersupports_opclaimed paged FLASH_ATTN_EXT shapes the launch then asserts on (page-table width, cell count not in whole pages, short mask)supports_opand the launch assert; the load-time probe now builds the shape the graph usestest-paged-fa-supports: 6 bad shapes reported supported before, refused after, the 2 valid ones still supportedllama-kv-cache.cpp:1289)decode()refuses it as invalid inputtest-exact-pageson Qwen3.5-0.8B (CUDA): aborts before, returns -1 and passes aftertest-rpc-paged-faused the pid of a failedfork()(so cleanup couldkill(-1)), and two overflow tests relied on seeing the leader busy before sending the followerkill(-1)attempted before, error and exit 1 afterOn 5149d08: CPU
test_preempt.py+test_preempt_notify.py+test_preempt_models.py83 passed, 14 skipped, 1 timed out at the 600 s cap at load ~1000 (test_a_long_resident_that_never_shifts_is_rotated_out_for_an_aged_head[cost], both variants pass when rerun alone);test-llama-context132 passed, 2 skipped;test-server-preempt-kind,test-server-decision-exact,test-server-tokens,test-exact-buft,test-event-query,test-exact-geometryandtest-rpc-paged-fapass; on a B200,test-backend-ops -o FLASH_ATTN_EXT -p exact_pagesgives the same 3 passed / 3 not supported before and after. Pin-set sim of the prebuilt set with this head on b11786: ok, all pins intact.Review round 6 (5149d08..4e17297)
src[5])test-paged-fa-supports: 8-plane and 2-plane masks reported supported before, refused after; pagedtest-backend-opscases unchanged (3 OK, 3 not supported). zDNN read from source only (s390x)common_exact_batch_geometrysummed ubatch and decode width inint, so a huge width wrapped into an accepted geometryINT32_MAXtest-exact-geometrycase: accepted before, refused aftern_ctxcould not fit after a park); early trimming dropped the cached chunksn_cache_reuseneeds; later members of a decision group were not charged their private suffixes; the final-capacity sum could overflown_ctxtest_preempt.pytests (12 concurrent 256-token non-causal decisions on a 256-cell pool hit the context error before;cache_n50 instead of about 249 before), both pass after; the group and overflow cases by a probe compiled against the server code (kv_used160 vs 220 actual;n_final-2147483548), right aftertest_a_parked_head_is_not_overtaken_by_a_stream_of_short_requestsfailed once on the Windows runner (10.57 s against a fixed 10 s bound)bench.yml.disabledcarried action pins from the fork's own pinning pass.githubOn 4e17297: CPU
test_preempt.py+test_preempt_notify.py+test_preempt_models.py86 passed, 14 skipped, 0 failed;test-exact-geometry,test-server-preempt-kind,test-server-decision-exact,test-server-tokens,test-rpc-paged-fapass;test-llama-context132 passed, 2 skipped; CUDAtest-paged-fa-supportspasses. Pin-set sim of the prebuilt set with this head on b11786: ok.The
test-thread-safetyheap corruption seen once on the Windows Vulkan runner at 5149d08 is not on a path this PR changes there: every new library check sits behindLLAMA_EXACT_CONCURRENCY, which no CI job sets, and the CUDA change is not built on Vulkan.Review round 7 (4e17297..514a287)
ggml_backend_cuda_set_exact_decode_width, checked in one place for the weight check and the paged-attention probe; such a module is refused at loadtest-exact-buftcase with a registry that has the name but not the extension (refused); the real CUDA registry still passes on a B200family_n_shared, set inserver-decision.cpp); an idle slot counts as the member it last servedcommon_speculative_begin(), resetting the draft sampler's RNGdraft-simpledrafter parked every 8 tokens: token 12 differed before, identical after/propswhile sleeping returned the load-time async flag without the live correctionpreempt_async_livecorrectionprompt_nequal to the input lengthtest-rpc-paged-facounted a client that failed to start as a refusal, and derived its port from the PIDOn 514a287: CPU preempt suites 87 passed, 15 skipped, 0 failed;
test-exact-buft,test-exact-geometry,test-server-preempt-kind,test-server-decision-exact,test-server-tokens,test-paged-fa-supports,test-event-query,test-rpc-paged-fapass;test-llama-context132 passed, 2 skipped; CUDAtest-exact-buftandtest-paged-fa-supportspass. Pin-set sim of the prebuilt set with this head on b11786: ok.This pin branch is based on upstream (
base/upstream-824eaecf4) and goes into the prebuilt as a diff applied on top of an upstream tag; it is never merged into this fork's master, so master's workflow deletions do not apply here, and the PR changes nothing under.github.Review round 8 (514a287..8dd07ec)
Rounds 6 and 7 patched the task-tree estimate of shared decision-group cells and each review found new cases (an emptied parent or child, idle reclaim crediting cells a running sibling still holds). Round 8 replaces it with ground truth:
llama_memory_n_cells_used(): cells held by at least one sequence, read from the cells (a shared cell once, a partly used block in full), plus the cells held only by the sequences given (what removing them frees). The planner's occupancy and idle-reclaim figures come from it; the old estimate stays only where the memory returns -1 (recurrent state, DeepSeek V4). A child slot's stale cache is cleared at launch, after the prompt-cache savetest-memory-cells-used, 8 checkspreempt_detach()keeps draft and checkpoint while a replay is pending;preempt_apply_shift()updates both contexts first; the planner uses the batch builder's stateless ruletest-rpc-paged-facounted a client setup failure as a refusal;test-paged-fa-supportsskipped ROCm and MUSA;test-exact-pagesreported a missing model as a skip; the exact-mode model tests were registered with a fixture they cannot useLLAMA_TEST_EXACT_MODELLLAMA_SERVER_PREEMPT_GRANULARITYtest knob plans in pages the cache does not have/slotsshows no change for 120 s); the two fixed-second limits scale with speed measured in the same run; behaviour assertions unchangedOn 8dd07ec: CPU preempt suites 87 passed, 16 skipped, 0 failed;
test-memory-cells-used,test-exact-geometry,test-exact-buft,test-server-preempt-kind,test-server-decision-exact,test-server-tokens,test-paged-fa-supports,test-rpc-paged-fapass;test-llama-context132 passed, 2 skipped; CUDA (B200)test-paged-fa-supports,test-exact-weights,test-exact-pagesand the exact-mode no-reuse server test pass. Pin-set sim with this head on b11786: ok.Review round 9 (8dd07ec..463e1af)
--cache-reuseshift: the server died before; a 500 and a healthy server aftertest-rpc-paged-fastill passed a client that crashed or timed out in setup as a server refusal;test-exact-pagesandtest-exact-weightsskipped when the context could not be createdllama_memory_n_cells_used()counted a sequence given twice inseq_freeas two owners; an occupancy-only query scanned the whole pooltest-memory-cells-usedcases (seq 0 twice: 0 before, 20 after; plus seq 1: 130 before, 50 after);test-exact-pageschecks the fast and full counts agree (CUDA)On 463e1af: CPU preempt suites 87 passed, 16 skipped, 0 failed; ctest subset 14/14 (test-llama-context, test-llama-context-model, test-memory-cells-used, test-rpc-paged-fa, test-thread-safety, test-state-seq-copy, test-server-, test-exact-); CUDA
test-exact-pagesandtest-exact-weightspass. Pin-set sim with this head on b11786: ok.The
test-thread-safetyheap corruption (0xc0000374) on the Windows CPU and Vulkan runners is upstream's: the same CI (cpu) workflow run on this PR's base branch alone (base/upstream-824eaecf4, upstream b11793 with no change of this PR) failswindows / x64-openblasontest-thread-safetywith the same exit code.Review round 10 (463e1af..8256bfb)
llama_memory_n_cells_usedwithseq_free); rotation reads its candidates together; the old estimate stays as the fallbackLLAMA_EXACT_CONCURRENCYtest-paged-fa-supportspassed when a CUDA / ROCm device refused every valid paged shape; the async test gate could skip on a late startup banner; the M-RoPE rejection test had invalid batch geometrylistening on;-b 512 -ub 128A reported stale CUDA graph policy after the exact decode width grows did not reproduce (B200, Qwen3.5-0.8B: 0 of 1,986,560 logits differ between graphs on and off after the change), so that code is unchanged.
On 8256bfb: CPU preempt suites 87 passed, 16 skipped, 0 failed; ctest subset 15/15; CUDA
test-exact-pages,test-exact-weights,test-paged-fa-supportsand the 8 exact-mode server tests pass. Pin-set sim with this head on b11786: ok.Review round 11 (8256bfb..ac1dbf8)
llama_memory_update()returns -1 when a shift is still due after the call; the server fails the requests instead of parking--cache-reuseshift: 200 with the failure unnoticed before; a 500, a healthy/healthand a working next request afterpreempt_n_keepapplies the shared-prefix and decision boundstest-exact-pagescase: fails against the base library, passes afterRaising the exact decode width changes the CUDA kernel policy process-wide (it only grows), so a context created before another context raised it computes differently afterwards (max logit difference 0.31 in the round-10 run) and matches a fresh context. This is by design and now documented in
llama.handggml-cuda.cu: create every context before serving, as the server does, or pin the policy withGGML_CUDA_BATCH_INVARIANT_MAX_COLS.On ac1dbf8: CPU preempt suites 87 passed, 16 skipped, 0 failed; ctest subset 15/15; CUDA
test-exact-pages,test-exact-weightsand the 8 exact-mode server tests pass. Pin-set sim with this head on b11786: ok.Review round 12 (ac1dbf8..95ddf1f)
abort_all_slots()clears each released slot's cached prompt, as the ordinary decode-error path doescache_n127 then 0), kept after (127 and 127)size_tOn 95ddf1f: CPU preempt suites 87 passed, 16 skipped, 0 failed; ctest subset 15/15. Pin-set sim with this head on b11786: ok.
Review round 13 (95ddf1f..68ea605)
-ppsin batched-bench / parallel) and branching speculation (llama-speculative -np > 1) carried on after exact mode refused the copy, sampling stale logits-pps(incommon_exact_concurrency_init) andllama-speculative -np > 1up frontbatched-bench -ppsprinted a result after "ignoring the copy" before, exits 1 after;llama-speculative -np 2aborted (rc 134) before, exits 1 with the refusal afterllama_kv_cache::init_batchdoesencode()round 11 showed differs batched vs solo)On 68ea605: CPU preempt suites 87 passed, 16 skipped, 0 failed;
test-llama-context134 of 134 models; CUDA exact-mode server tests 8 passed. Pin-set sim with this head on b11786: ok.Review round 14 (68ea605..c520216)
-ppsand branching-speculation cases round 13 refused; in exact mode it ignored the 30 refused copies and failed with a misleading KV-capacity message, and on CPU it used the refused (null) context and crashedcommon_exact_concurrency_seq_copy_ok(), called right after argument parsing by every tool that copies cells between sequences (an audit ofllama_memory_seq_cpcallers: lookahead always, parallel and batched-bench with-pps, speculative with-np > 1; the round-13 checks now go through it); lookahead, parallel and speculative report a model or context that failed to load instead of using itLLAMA_EXACT_CONCURRENCY=1and expect the refusal; lookahead on CPU: segfault (rc 139) before, refusal (rc 1) after; on a B200: 30 ignored copies before, refused aftercommoninit, so a library caller could create a context whose batch cannot hold a ubatch beside a decode step of every sequencellama_init_from_modelapplies the same rule whenn_seq_max > 1On c520216: CPU preempt suites 87 passed, 16 skipped, 0 failed; ctest subset 18/19 (the remaining one, the full
test-llama-contextmodel sweep, timed out under host load; it runs without exact mode, which these changes do not touch); CUDA exact-mode server tests 8 passed. Pin-set sim with this head on b11786: ok.Review round 15 (c520216..9226f57)
common_log_flush()before every refusal exit (the seq-copy check, the init check in argument parsing, the model / context refusals incommon_init_result, and the failed-load reports in lookahead, parallel and speculative)_wait_processing()still had a fixed deadline; the partial-prefill, started-slot and sliding-window tests could miss their windows on a slow runner; the n-gram options were not forwarded by the test harnesstest-rpc-paged-fastill passed when the paged client failed locally before sendingOn 9226f57: CPU preempt suites 87 passed, 16 skipped, 0 failed; ctest subset 17/17 (including the four refusal tests and test-rpc-paged-fa). Pin-set sim with this head on b11786: ok.
Review round 16 (9226f57..b867ddc)
common_init_from_paramscaller in examples/ and tools/ checked: the six with no check (lookup, lookup-stats, lookup-create, speculative-simple, cvector-generator, mtmd-debug) and the four checking only the model (embedding, retrieval, finetune, results) now log, flush and return 1{"prompt":[[]]}with 500 (base: 200, empty content), with parking offtask_prompt_rejected()returns early for an empty prompt, as the batch builder doestest_embedding.py(bert-bge-small): 500 before, 200 after;prompt: []is still 400test-speculative-shared-bound(bounds 1 and 2 with n_max 3 give 1, non-drafting sequences ignored); the graph assert itself was not reproduced (no DSpark / DFlash2 model offline)On 839cacc: CPU preempt suites 87 passed, 16 skipped, 0 failed; ctest subset 22/22;
test_embedding.py37 passed. On ef9fc2b the full CPU preempt suite ran only at a host load average of 1,000 to 1,400: three timing-sensitive cases failed there (one passed on rerun; the policy test also failed with the test files of 839cacc under the same load), none of them in the earlier run at lower load. Pin-set sim with ef9fc2b on b11786: ok.Review round 17 (b867ddc..841ba3f)
llama_memory_seq_add/seq_div: a cell's page offset is its position), but SelfExtend (--grp-attn-n > 1),llama-completion --context-shiftand passkey's rolling context still issued them and then decoded at positions their cells did not havecommon_exact_concurrency_position_edit_ok()refuses them with the usual message and a log flush: SelfExtend incommon_exact_concurrency_init()(every tool that calls it), passkey and completion's context shift right after argument parsing; the server already turns context shift off when the memory cannot shift. The Android JNI example (no common parameter parsing) is not coveredtest-exact-position-edit-passkey,-completion-self-extend,-completion-context-shiftexpect the refusalcommon_speculative_init()throw (SIGABRT)has_dft()or MTP) and the draft model or context was not created: log, flush, return 1, as speculative and the server dollama_set_exact_decode_tokens()skipped its validation before any context existed, so a figure pastGGML_CUDA_BATCH_INVARIANT_MAX_COLSwas stored, could not be lowered again, and refused every later contexttest-exact-decode-tokens(bound 8: 9 refused with figure and width unchanged, 8 accepted, 9 after 8 refused, 1 and 0 keep 8, width 9 refused): 7 of 10 checks fail against the base library, all pass on this head_wait_processingignored whether its request had already ended (120 s of polling), an FCFS-only assertion also ran under the cost policy, and overlap / pressure was assumed rather than established_wait_processingtakes the request's liveness; the FCFS assertion reads the policy from/props; the arrival-order, trimmed-slot and two-generation tests send their prompts in one request; the remaining overlap tests check from the server log that the follower launched while the earlier request ran, and the policy tests that something parked, failing with "precondition not met" otherwise (no timing window widened); notification-test HTTP calls are bounded; the README names the: preempt-keepaliveheartbeatOn 841ba3f (host load average 520 to 1,040 throughout): CPU preempt suites 80 passed, 16 skipped, 0 failed under both the default and the cost policy; ctest 83 of 85 (
test-llama-contextandtest-llama-context-modeltimed out at 600 s under this load, as in earlier rounds, base included). Pin-set sim with this head on b11786: ok, 21 pins intact. Round 16's sliding-window test fix passed thewindowsCI job on b867ddc.Review round 18 (841ba3f..be076be)
Round 18 on 841ba3f: 10 APPROVE, 2 NEEDS_DISCUSSION (CUDA stalled in the reviewers' sandboxes; no defect named), 0 REQUEST_CHANGES.
test_embedding.py(bert-bge-small) now checks invalid token 400, empty prompt 200, a valid prompt needing logits 500 andrepeat_penalty: 0400, each as base answers it: the invalid-token case fails before (500), all pass after; the aLoRA ordering has no test (no aLoRA adapter here)test_one_victim_that_covers_the_shortfall_is_parked_instead_of_several_small_onessent its leader from another client thread after the residents started, so they could finish before it arrived, and ran under the default policy though it tests cost-policy victim choiceLLAMA_ARG_PREEMPT_POLICY=cost(checked through/props); it asserts every prompt predicted 300 tokens, that something parked, and that exactly one park of at least 1400 cells on the medium prompt's slot came before the first resumeOn be076be (host load average 620 to 1,360): CPU preempt suites 80 passed, 16 skipped, 0 failed under both the default and the cost policy; ctest
exact|server19/19;test_embedding.py37/37. Pin-set sim with this head on b11786: ok, 21 pins intact.Review round 19 (be076be..32abc8d)
Round 19 on be076be (4 reviewers): 2 APPROVE, 2 REQUEST_CHANGES, both on the same finding.
params_base.lora_adapterswhilePOST /lora-adaptersreplaces that vector on the server loop: a data race, seen by both reviewers as a wrong 400 "Cannot run multiple aLoRAs in a single request" under stress (1 in 500, 14 in 3,000 and 1 in 2,000 requests; base 0)launch_slot_with_task(), on the server loop, as in base; every other reader and writer of the adapters is on the server loop too, so no lock is needed. What the up-front check reads is now documented: the vocabulary and model, the context's sizes and flags, and the task, none of which the server loop changes after loading. A request that enables several aLoRAs and also fails an up-front check now gets that check's error (still a refusal, never a 200)test_embedding.py37/37 including the round-18 precedence casesOn 32abc8d (host load average 780 to 830): CPU preempt suite (default policy) 78 passed, 18 skipped, 0 failed (two more skips than before: they need
test-llama-archs, which this targeted build did not produce). Pin-set sim with this head on b11786: ok, 21 pins intact. The cost-policy suite and the ctests were not rerun this round.Review round 20 (32abc8d..d5b2f1c)
Round 20 on 32abc8d (4 reviewers): 3 APPROVE, 1 REQUEST_CHANGES.
llama_set_exact_decode_tokens()could raise the process-wide figure after an exact context was created, without rechecking that context's batch geometry (n_batch >= n_ubatch + n_seq_max x figure), which its constructor enforces; the context then kept decoding with a batch the constructor would refuse (reviewer, B200, Qwen3.5-0.8B: a 256-token prompt alone vs beside an 8-token neighbour differed in all 248,320 logits)decode()/encode()return -1 before touching the batch or memory if the figure ever outgrows a context anyway. The only caller of the setter (common_exact_concurrency_init()) runs before any context exists, so the server and tools are unaffectedtest-exact-decode-tokens-live-context(vocabulary-only load): a 132/128/2 context at figure 1, then 8 and 3 refused and 2 accepted while it lives, 8 accepted once it is freed, an identical context then refused and a 144/128/2 one refusing 9; against the previous library it accepts 8 and fails 4 of 9 checks, all pass on this headOn d5b2f1c: ctest
-R exact17/17; CPU preempt suite (default policy) 78 passed, 18 skipped, 0 failed. Pin-set sim with this head on b11786: ok, 21 pins intact. Not run here: the reviewer's B200 logits repro on this head (it should now stop at the setter), the backstop through a real weighted context (no current path reaches it, the setter refuses first), and the cost-policy suite.Refresh onto b11838 (d5b2f1c..c732f0f)
The prebuilt picks the newest upstream tag aged at least 6 h, now b11838, where d5b2f1c no longer merged in the pin-set simulation. c732f0f merges b11838 (4231b23) into the branch (an additive merge, nothing rewritten) and the base is now
base/upstream-4231b23dc.ggml/src/ggml-hexagon/ggml-hexagon.cpp: upstream's op-fusion refactor replaced the per-op switch insupports_opwithhtp_kparams_check(); the page-table refusal for FLASH_ATTN_EXT is re-applied as its own case on top (not compiled here: no Hexagon toolchain).tests/CMakeLists.txt: upstream's new test labels kept, this PR's tests re-added beside them with their labels unchanged.src/llama-context.cpp,src/llama-graph.cpp: merged without conflict.On c732f0f: ctest
exact|server20/20; CPU preempt suite (default policy) 78 passed, 18 skipped, 0 failed. Pin-set simulation on b11838 with this head: this PR merges additively and its pin contract holds (all of its lines present); the simulation still refuses two pins owned by other PRs of the set (ggml-org#25731, #266), which this refresh does not touch.Not shown
supports_op; that is read from source, not run.test-thread-safety; the sleeping/propsfix at runtime; a real pre-kernel GPU module (covered by a fake registry only); the round-8 speculative replay, cache-reuse shift and child-cache-clear fixes at runtime; a run on the Windows runner after 463e1af; the per-buffer park charge and the round-10 growth estimate on the async CUDA path in a full server run; the indexed hybrid memory exact-mode rules at runtime; a realtime audio model end to end; the MSA (MiniMax-M3) and encoder-decoder (T5) exact-mode refusals at runtime; a real aLoRA adapter across a context shift; the MSA refusal at runtime; the fulltest-llama-contextsweep on c520216.