Repository navigation
Conversation
A writable model is its own model patcher. It loads after the model it serves, so load_models_gpu([patcher]) alone gives it VBAR priority without nesting it as an additional model, and its storage policy comes from where anonymous files live rather than from that model's checkpoint.
Whether a model wants a writable cache is its own decision. Qwen Image 2.1 keeps its host prefix cache when its patcher is not dynamic; the generic sampler loop no longer skips non-dynamic patchers.
Writable models now stay loaded until node cleanup instead of being torn down inside the sampler. The caller registers the (patcher, model) pair as ephemeral; cleanup drops the patcher first so unpinning happens before the model frees its VBAR and anonymous backing file, and discards dirty pins so nothing is flushed into a file about to die. The sampler loop over patchers no longer needs a context stack.
A model state is a writable model's values as CPU tensors with its generation metadata, passed between nodes by value. Loading copies the values into the model's own backing: an anonymous file that dynamic VRAM pages from and writes back to, or plain CPU tensors. Flushing reads the loaded values back and drops the model; releasing drops it without them. Nothing writes a state's tensors in place, so a copy shares them. The sampler loads its writable model into a state and registers the state for release after the node, replacing writable_model.
The KV cache is a module of writable weights that CLIP.generate loads into an anonymous model state for the node, so it is paged like any other model instead of being allocated in VRAM up front. An off-chain MTP accept moves rows within the attention caches, so the step pins those storages in one pop while the GPU is still verifying; only the moves themselves run after the accept sync. Linear caches only record the path for the next forward and are not pinned.
Generation always records its cursor into the model state's metadata: cache length and capacity, the sampled-but-unfed token, the next RoPE position, each cache's commit cursor, the penalty history and RNG. A later generate call with that metadata restores the caches and feeds the pending token with the new prompt from the saved position. Resumed DeltaNet layers prefill through the decode path so their recurrent state survives; a tail cut by max_length or a stop token is rolled back so the cache ends exactly at the returned tokens.
Writable models flush into anonymous files on every platform, so fall back to an unlinked temporary file where memfd is unavailable.
Model State is a new core type: a writable model's values over a file, such as an LLM's KV cache. Open Model State opens a state file in the output folder, or starts an empty state when the file is missing or the node is set to always start fresh, and hands the state to a sampler. The node runs again whenever the file changes.
Writes a model state's values and metadata to a file in the output folder, typically the one it was opened from, and passes the state on so a loop can end on it.
CLIP.generate takes an optional model state to continue from. The TextGenerate node copies its input state so the input is left as it was and two generations from it are independent, applies the chat template's system prompt only when the state holds no conversation yet, and flushes the output state back to CPU only when another node consumes it; otherwise the loaded cache is released after the node as before. TextGenerateLTX2Prompt does not expose the state.
Covers loading, flushing, saving, copying and reopening a model state, continuing a Qwen generation from a saved state, and a CUDA script that checks plain, resident, pinned and file-only states against each other.
Name both nodes' state output "state", move Save Model State to model/merging next to the other save nodes, and say where it writes.
A continued turn with more than SPEC_ROWS new tokens prefilled 8 rows at a time through the decode kernels. It now prefills in one batch: GatedDeltaNet starts its conv window and delta rule from the cached state, and the causal mask covers the cached prefix plus the new rows. The int8 cache's new rows attend the quantized prefix through the decode kernel and each other causally, merged by log-sum-exp, then the quantized pages grow one page per update through the bf16 window. A saved state used to end with the last step's accepted rows still uncommitted in the DeltaNet state. It now ends before them and stores them in pending, so the next turn re-feeds them.
Agent-style chat sends a long shared prompt with a short new part each turn. With prefix_cache on (or COMFY_LLM_PREFIX_CACHE=on), TextGenerate keeps one model state per conversation, takes the state with the longest matching token prefix, resumes from the DeltaNet checkpoint at the last message boundary inside it, and prefills only the rest in batched segments. Image positions are keyed by a hash of the image, so image prompts match too. States stay loaded between calls. Past a size or count budget the least recently used one is saved to a file and loaded again on its next match. The shared start (up to the end of the system message) is saved once per system prompt, and a new conversation starts from that file. The mode applies per call; env follows COMFY_LLM_PREFIX_CACHE (off by default). Attention uses the bf16 KV cache while the cache is on, since the int8 cache can't continue a prefill from a prefix.
vram, gpu, cpu and disk map to on and stats to off. LLMPrefixCacheStats returns the last generate's cache stats as JSON.
Drop the forced bf16 KV cache: prefills now continue an int8 prefix. Cache keys include the KV cache type, the attention tensors are trimmed and padded along an int8 cache's page dimension too, and the shared-start file keeps its pages. An int8 state resumes only where its bf16 window still holds the two pages before the resume point. The rope capacity comes from the cache entry (an int8 cache's key tensor is its window). The in-use flag is only set on a dynamic-VRAM patcher, so the cache also runs on a CPU device. Add unit tests for the cache: a next turn, a spilled conversation and a new conversation from the shared-start file, each on bf16 and int8.
Cleanup now skips unreached raw module groups, resets PREFETCH_QUEUES before walking them, keeps cleaning after one failure and re-raises the first error, and always drops _prefetch markers. The Qwen3.5 MTP draft step passes its group as a list. Regression tests in tests-unit/comfy_test/model_prefetch_cleanup_test.py.
For comfy-kitchen#303's 3- and 5-bit widths of AsymW4A8Int8Layout.
For our evals and demos only, until comfy-kitchen#303 merges and is released.
The causal merge repeated the keys and values to every query head in float32 and scored 512 rows against the whole continue at once: several GB of temporaries for a continue of 30k rows. It now scores each key/value head's query heads in one matmul and sizes the row chunks to a fixed float32 budget. The prefix attention is written into one preallocated output instead of concatenated.
A one-pass prefill of a 41k-row prompt needs about 18 GB of working memory, which pages every weight out of VRAM; the decode that follows reads part of them back from system memory each step. Each piece now continues the one before through the batched continue.
So a long segment goes in pieces too, and a short one through the decode path.
From ComfyUI#17557.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
An integration build of the Qwen3.5 local-LLM work that is still under review, so evals and demos run against one branch. It stacks ComfyUI#17360 (model states, int8 KV cache, MTP drafts), rattus128#12 (batched continue of a saved state, long prompts prefilled in pieces) and ComfyUI#17525 (prompt state cache), plus:
Not for merge as is: each part lands through its own PR.