Skip to content

Retain managed model states between text generation runs - #13

Merged
rattus128 merged 5 commits into
prs/qwen35-kv-chatfrom
dev/persistent-model-state-review
Nov 9, 2026
Merged

rattus128 merged 5 commits into
prs/qwen35-kv-chatfrom
dev/persistent-model-state-review

Conversation

@rattus128

@rattus128 rattus128 commented Nov 9, 2026 •

Copy link
Copy Markdown
Owner

Scope

  • Retain a CLIP-owned managed state model between generations, with independent input snapshots and normal RAM/VRAM accounting.
  • Reuse resident named saves. SaveModelState.writethrough defaults to true; false defers persistence until lifecycle writeback and is not crash-durable while pending.
  • Track snapshot revisions independently of ModelPatcher's patch UUIDs; model state no longer changes either patcher UUID field. Only the active state can discard dynamic writes.
  • Preserve generation's metadata reference when forking a saved handle.

Persistence only. Batched continuation and long-prompt prefill remain in #12. Both PRs independently target prs/qwen35-kv-chat; neither depends on the other. No dependency or cache-capacity changes. All tests remain in the final commit.

Validation

  • 81 CPU tests passed on Linux and Windows; Ruff and diff checks passed.
  • Wookie: Windows 11 build 26200, driver 617.42, PyTorch 2.11.0+cu130, comfy-aimdo 0.6.0, comfy-kitchen 0.2.41. Tested the RTX 5080 16 GB and RTX 5060 8 GB separately, isolated by GPU UUID.
  • Three dynamic-state lifecycle trials per GPU passed, covering resident reuse, snapshot switching, RAM/VRAM eviction, deferred writeback, collection, owner changes, and release ownership.
  • Per GPU: all 60 writable-cache cases and two persistent-continuation cases passed, with graphs enabled/disabled, BF16/INT8 caches, and MTP enabled/disabled. Tokens and captured outputs matched exactly against the plain restoration path.
  • Real qwen3.5_9b_w4a8g16_int8_head_embed_mtp.safetensors workflows passed: immediate/deferred saves on the 5080 with graphs enabled; deferred saves on the 5060 with graphs disabled. Covered recall, matching branch outputs, and continuation after explicit model unload.
  • The unused-copy release regression fails against the previous PR head and passes with this change.

The new GPU validation is Windows-only and uses the 9B checkpoint; earlier 27B results are not claimed as a rerun of this revision.

@rattus128
rattus128 force-pushed the dev/persistent-model-state-review branch from 4313df7 to b1913ef Compare November 9, 2026 07:50
@rattus128 rattus128 changed the title Retain model states and batch saved-state Qwen prefill Retain managed model states between text generation runs Nov 9, 2026
@rattus128
rattus128 force-pushed the dev/persistent-model-state-review branch from b1913ef to 156e544 Compare November 9, 2026 12:19
@rattus128
rattus128 marked this pull request as ready for review November 9, 2026 14:51
@rattus128
rattus128 merged commit 76724a5 into prs/qwen35-kv-chat Nov 9, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant