Skip to content

KV snapshots on the switch: save idle slots on the drain, restore them on the next start - #115

Open
T-Crypt wants to merge 10 commits into
mainfrom
T-Crypt/glm-g2-kvsnap
Open

T-Crypt wants to merge 10 commits into
mainfrom
T-Crypt/glm-g2-kvsnap

Conversation

@T-Crypt

@T-Crypt T-Crypt commented Oct 21, 2026

Copy link
Copy Markdown
Owner

OME B51PJKA. README "Next: v0.3.x": KV snapshots wired into the drain and start paths.

Opt-in per engine with kv_snapshots (the directory the engine's --slot-save-path points at) and kv_snapshots_mib (16 GiB by default). An engine without it is unchanged.

  • Drain (internal/runner/kvsave.go): after the slot boundary, before the park step or anything stops the engine, the runner saves each idle slot's prompt cache, records it in the ledger, and evicts down to the budget. It is bounded at 60 s, keeps the library free floor, and stops at once if the engine exits. A failure is one line, and the stop ladder runs as before.
  • Start (internal/gateway/kvrestore.go): only a start this gateway ran restores. The snapshots are warmed into the page cache while the start waits; once the engine answers, its empty slots restore the newest snapshot of that model before ensure returns. Bounded at 30 s.
  • Identity is the one model the engine lists on /v1/models. An engine that lists none or several (llama-swap) never snapshots, so nothing crosses models.
  • Slot calls carry the engine's own credential (api_key_env / api_key_file), never the node key.
  • A snapshot holds the conversation's tokens: the ledger is written owner-only and the README asks for a private directory.

Proof:

  • Config, kvsnap, runner and gateway tests (save on drain, restore on start, unsupported engine, timeouts, llama-swap skip).
  • Full proof green after merging main: build, gofmt, vet, go test ./..., go test -race on kvsnap/runner/gateway, Windows build and vet, golangci-lint (0 issues).

Not done here: a live save and restore against a real llama-server on aphotic. That check gates the release notes for this feature.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant