Skip to content

Continue a saved Qwen3.5 state with one batched prefill - #12

Draft
synap5e wants to merge 17 commits into
rattus128:prs/qwen35-kv-chatfrom
Comfy-Org:synap5e/feat/qwen35-batched-continue
Draft

synap5e wants to merge 17 commits into
rattus128:prs/qwen35-kv-chatfrom
Comfy-Org:synap5e/feat/qwen35-batched-continue

Conversation

@synap5e

@synap5e synap5e commented Nov 5, 2026 •

Copy link
Copy Markdown

Continuing a saved model state used to feed the new tokens through the decode path 8 rows at a time, so a follow-up turn on a long chat spent most of its time there. Now the new tokens run as one prefill on top of the saved state:

  • the attention mask covers the saved rows;
  • the DeltaNet layers start from the saved conv and recurrent state;
  • the int8 KV cache attends its stored pages through the decode kernel, then merges in the new rows' attention over each other.

A state saved with drafted rows not yet committed rewinds to its last committed token, so the next continue can batch as well. Continues of 8 tokens or fewer keep the decode path.

Next turn on a saved 24.9k-token state, Qwen3.5 w4a8, seconds (2 runs each):

KV cache path +1.4k tokens, RTX 4090 +1.4k tokens, RTX 5090 +5.5k tokens, RTX 4090 +5.5k tokens, RTX 5090
bf16 decode 15.5 / 14.7 25.4 / 25.4 48.3 / 52.3 82.4 / 81.6
bf16 batched 6.8 / 6.7 8.2 / 6.8 8.2 / 8.8 9.1 / 9.2
int8 (--use-ck-attention) decode 15.0 / 14.3 22.0 / 22.4 49.1 / 46.2 80.3 / 80.5
int8 (--use-ck-attention) batched 4.4 / 4.3 4.8 / 4.9 7.2 / 7.1 6.7 / 7.5

The generated text is the same on both paths in every row.

A prompt longer than 4096 tokens is prefilled in pieces of at most 4096, each continuing the one before. One pass over a 41.8k-token prompt needs about 18 GB of working memory, which pages every weight out of VRAM on both a 24 GB and a 32 GB card, so the decode after it read the weights back from system memory.

512-token reply to a 41.8k-token prompt (whole call, then decode speed):

KV cache prefill RTX 4090 RTX 5090
bf16 one pass 160 s, 3.8 tok/s 126 / 131 s, 5.4 tok/s
bf16 pieces 29 s, 58 tok/s 29 / 29 s, 91 tok/s
int8 one pass 101 s, 6.9 tok/s 43 / 65 s, 99 tok/s
int8 pieces 26 s, 78 tok/s 32 / 31 s, 114 tok/s

The RTX 4090 runs Linux with comfy-kitchen 0.2.40; the RTX 5090 runs Windows with comfy-kitchen at kijai/comfy-kitchen#303 (34bc583).

A writable model is its own model patcher. It loads after the model it
serves, so load_models_gpu([patcher]) alone gives it VBAR priority
without nesting it as an additional model, and its storage policy comes
from where anonymous files live rather than from that model's
checkpoint.
Whether a model wants a writable cache is its own decision. Qwen Image
2.1 keeps its host prefix cache when its patcher is not dynamic; the
generic sampler loop no longer skips non-dynamic patchers.
Writable models now stay loaded until node cleanup instead of being
torn down inside the sampler. The caller registers the (patcher, model)
pair as ephemeral; cleanup drops the patcher first so unpinning
happens before the model frees its VBAR and anonymous backing file,
and discards dirty pins so nothing is flushed into a file about to die.
The sampler loop over patchers no longer needs a context stack.
A model state is a writable model's values as CPU tensors with its
generation metadata, passed between nodes by value. Loading copies the
values into the model's own backing: an anonymous file that dynamic VRAM
pages from and writes back to, or plain CPU tensors. Flushing reads the
loaded values back and drops the model; releasing drops it without
them. Nothing writes a state's tensors in place, so a copy shares them.

The sampler loads its writable model into a state and registers the
state for release after the node, replacing writable_model.
The KV cache is a module of writable weights that CLIP.generate loads
into an anonymous model state for the node, so it is paged like any
other model instead of being allocated in VRAM up front.

An off-chain MTP accept moves rows within the attention caches, so the
step pins those storages in one pop while the GPU is still verifying;
only the moves themselves run after the accept sync. Linear caches only
record the path for the next forward and are not pinned.
Generation always records its cursor into the model state's metadata:
cache length and capacity, the sampled-but-unfed token, the next RoPE
position, each cache's commit cursor, the penalty history and RNG. A
later generate call with that metadata restores the caches and feeds the
pending token with the new prompt from the saved position. Resumed
DeltaNet layers prefill through the decode path so their recurrent state
survives; a tail cut by max_length or a stop token is rolled back so the
cache ends exactly at the returned tokens.
Writable models flush into anonymous files on every platform, so fall
back to an unlinked temporary file where memfd is unavailable.
Model State is a new core type: a writable model's values over a file,
such as an LLM's KV cache. Open Model State opens a state file in the
output folder, or starts an empty state when the file is missing or the
node is set to always start fresh, and hands the state to a sampler. The
node runs again whenever the file changes.
Writes a model state's values and metadata to a file in the output
folder, typically the one it was opened from, and passes the state on
so a loop can end on it.
CLIP.generate takes an optional model state to continue from. The
TextGenerate node copies its input state so the input is left as it
was and two generations from it are independent, applies the chat
template's system prompt only when the state holds no conversation
yet, and flushes the output state back to CPU only when another node
consumes it; otherwise the loaded cache is released after the node as
before. TextGenerateLTX2Prompt does not expose the state.
Covers loading, flushing, saving, copying and reopening a model state,
continuing a Qwen generation from a saved state, and a CUDA script that
checks plain, resident, pinned and file-only states against each other.
Name both nodes' state output "state", move Save Model State to
model/merging next to the other save nodes, and say where it writes.
@rattus128
rattus128 force-pushed the synap5e/feat/qwen35-batched-continue branch from 0bc3f7a to 34ed07b Compare November 9, 2026 08:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants