Repository navigation
Conversation
A writable model is its own model patcher. It loads after the model it serves, so load_models_gpu([patcher]) alone gives it VBAR priority without nesting it as an additional model, and its storage policy comes from where anonymous files live rather than from that model's checkpoint.
Whether a model wants a writable cache is its own decision. Qwen Image 2.1 keeps its host prefix cache when its patcher is not dynamic; the generic sampler loop no longer skips non-dynamic patchers.
Writable models now stay loaded until node cleanup instead of being torn down inside the sampler. The caller registers the (patcher, model) pair as ephemeral; cleanup drops the patcher first so unpinning happens before the model frees its VBAR and anonymous backing file, and discards dirty pins so nothing is flushed into a file about to die. The sampler loop over patchers no longer needs a context stack.
A model state is a writable model's values as CPU tensors with its generation metadata, passed between nodes by value. Loading copies the values into the model's own backing: an anonymous file that dynamic VRAM pages from and writes back to, or plain CPU tensors. Flushing reads the loaded values back and drops the model; releasing drops it without them. Nothing writes a state's tensors in place, so a copy shares them. The sampler loads its writable model into a state and registers the state for release after the node, replacing writable_model.
The KV cache is a module of writable weights that CLIP.generate loads into an anonymous model state for the node, so it is paged like any other model instead of being allocated in VRAM up front. An off-chain MTP accept moves rows within the attention caches, so the step pins those storages in one pop while the GPU is still verifying; only the moves themselves run after the accept sync. Linear caches only record the path for the next forward and are not pinned.
Generation always records its cursor into the model state's metadata: cache length and capacity, the sampled-but-unfed token, the next RoPE position, each cache's commit cursor, the penalty history and RNG. A later generate call with that metadata restores the caches and feeds the pending token with the new prompt from the saved position. Resumed DeltaNet layers prefill through the decode path so their recurrent state survives; a tail cut by max_length or a stop token is rolled back so the cache ends exactly at the returned tokens.
Writable models flush into anonymous files on every platform, so fall back to an unlinked temporary file where memfd is unavailable.
Model State is a new core type: a writable model's values over a file, such as an LLM's KV cache. Open Model State opens a state file in the output folder, or starts an empty state when the file is missing or the node is set to always start fresh, and hands the state to a sampler. The node runs again whenever the file changes.
Writes a model state's values and metadata to a file in the output folder, typically the one it was opened from, and passes the state on so a loop can end on it.
CLIP.generate takes an optional model state to continue from. The TextGenerate node copies its input state so the input is left as it was and two generations from it are independent, applies the chat template's system prompt only when the state holds no conversation yet, and flushes the output state back to CPU only when another node consumes it; otherwise the loaded cache is released after the node as before. TextGenerateLTX2Prompt does not expose the state.
Covers loading, flushing, saving, copying and reopening a model state, continuing a Qwen generation from a saved state, and a CUDA script that checks plain, resident, pinned and file-only states against each other.
Name both nodes' state output "state", move Save Model State to model/merging next to the other save nodes, and say where it writes.
rattus128
force-pushed
the
synap5e/feat/qwen35-batched-continue
branch
from
November 9, 2026 08:54
0bc3f7a to
34ed07b
Compare
rattus128
force-pushed
the
prs/qwen35-kv-chat
branch
from
November 9, 2026 22:18
76724a5 to
6920543
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Continuing a saved model state used to feed the new tokens through the decode path 8 rows at a time, so a follow-up turn on a long chat spent most of its time there. Now the new tokens run as one prefill on top of the saved state:
A state saved with drafted rows not yet committed rewinds to its last committed token, so the next continue can batch as well. Continues of 8 tokens or fewer keep the decode path.
Next turn on a saved 24.9k-token state, Qwen3.5 w4a8, seconds (2 runs each):
--use-ck-attention)--use-ck-attention)The generated text is the same on both paths in every row.
A prompt longer than 4096 tokens is prefilled in pieces of at most 4096, each continuing the one before. One pass over a 41.8k-token prompt needs about 18 GB of working memory, which pages every weight out of VRAM on both a 24 GB and a 32 GB card, so the decode after it read the weights back from system memory.
512-token reply to a 41.8k-token prompt (whole call, then decode speed):
The RTX 4090 runs Linux with comfy-kitchen 0.2.40; the RTX 5090 runs Windows with comfy-kitchen at kijai/comfy-kitchen#303 (34bc583).