Repository navigation
feat: run snapshot and restore on the restate Environment object with durable chain state - #377
Conversation
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
fuse | 179dc8b | Commit Preview URL Branch Preview URL |
Nov 15 2026, 06:15 AM |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| if st.Pending != nil || st.Dropped { | ||
| return orchestrator.ChainHold{Busy: true}, nil | ||
| } | ||
| now, err := nowMillis(ctx) | ||
| if err != nil { | ||
| return orchestrator.ChainHold{}, err | ||
| } | ||
| st.HeldUntil = now + lease.Milliseconds() |
There was a problem hiding this comment.
HoldChain grants another hold while HeldUntil is still in the future. Two concurrent live migrations can receive the same head and base. The durable branch no longer takes the shared chain.mu, so both can take competing diffs or consume the same base, causing migration to fail. Either caller can also release the other's hold and let checkpoints resume.
Reject an active hold, and tie each release to the hold that acquired it.
| if !found || record.State != SnapshotStateDeleting { | ||
| return nil | ||
| } | ||
| if err := fm.deleteSnapshotArtifact(ctx, record); err != nil && !IsNotFound(err) { | ||
| return err |
There was a problem hiding this comment.
SnapshotCollect checks only that the row is deleting, not whether a move has since pinned it. The queued deletion can wait behind another environment operation while an already-planned move acquires its pin and starts reading. The collector then deletes the source despite that active pin, breaking the copy.
Check pins again when collection runs, and make deletion and new moves agree on whether the snapshot is still available.
| if st.Pending != nil || st.HeldUntil > now { | ||
| st.Dropped = true | ||
| restate.Set(ctx, chainKey, st) | ||
| return nil |
There was a problem hiding this comment.
If a VM is destroyed during a migration hold and the process dies before ReleaseChain, this branch leaves Dropped set after the hold expires. The gone-VM sweep sends retirement once and removes its mirror, while checkpointTick returns immediately for dropped chains. A pending copy that never reports back has the same problem.
Schedule a durable follow-up for the deadline. Without it, destroyed VMs can leave their standby copies permanently pinned and using disk.
…eckpoint chain can reuse
…the restate environment object
…int restarts mid-snapshot and mid-restore
…eanup, and re-check pins before snapshot gc deletes
988de82 to
179dc8b
Compare
Runs snapshot and restore on the restate
Environment/{vm_id}object, moves the checkpoint chain onto that object's state, and hands snapshot gc deletions to restate. #2 of the restate stack (#346 -> #345 -> #349 -> #348 -> #342).Environment/{vm_id}.Snapshot(cold, live, diff, shrunk): begin (checks, quota, lineage) ->Runcheckpoint (keyed by feat: idempotency keys on host-agent mutations so durable steps can retry after a lost response #370, so a retry or replay gets the first checkpoint back) -> rowcreating->Runlist ->ready. a record that cannot be finished runs the compensation: the host artifact is deleted and the row markederror(snapshot.failed). the host agent picks snapshot ids, so the row is written right after the keyed checkpoint rather than before it; the journal plus the key give the same "no untracked artifact" guarantee without an agent api change.Environment/{vm_id}.Restore: validate -> lookup ->restoring->Runrestore ->ready; a restart resumes it. a host restore that fails for good marks the rowerror.Environment/{vm_id}.Checkpoint. the chain (head, base, standby) lives in object state, the base is pinned inArtifactPin(chain:<vm_id>) instead of memory, andfm.chainsis only a mirror for the standby picker. the copy to the standby is a one-wayArtifactsmove that reports back toChainMoved(newnotifyon the move job), so a long copy never holds the environment object and blocks a destroy. the tick still walks vms one at a time by waiting for each copy to settle.HoldChain(1h lease) instead ofchain.mu; a busy chain makes it take a full seed. a destroy drops the chain (deferred while a copy or hold is in flight). a copy that never reports back is forgotten after 2h without deleting anything, which keeps feat: run artifact moves as a durable restate service with pull pins in durable state #376's unsettled-move rule.ArtifactPin), and on restate sendsCollectSnapshot(idempotency-keyed per marking), retried under theENVIRONMENTpolicy before the row goeserror.snapshotInProcess): they are ported later, and a handler on the same object must not call it.FUSE_RESTATE_MODE=offkeeps the in-process paths; the snapshot/restore code is split into the same step methods both modes use.No api surface change (fields, states, status codes), so no sdk/openapi changes. Docs: snapshots concept page, orchestrator-deploy (retry policy row, stale "nothing runs on it yet"), restate runbook.
verified locally
gofmt -l .,go build ./...,go vet ./..., golangci-lint v2.12.2 (0 issues),go test ./... -count=1,-raceon durable/orchestrator; live/checkpoint/snapshot unit tests.bun run build.not run
kill -9between host checkpoint and row write,kill -9during restore,live-snapshot-agent.sh11/11 with mode self.Stacked on #376.
Closes #345
Fix overlapping migration holds and deletion of newly pinned snapshots before merging.
Summary
Moves snapshots, restores, checkpoint chains, and retention cleanup onto Restate.
Diagram
sequenceDiagram participant Caller participant Environment participant Host participant Artifacts participant ArtifactPin Caller->>Environment: Snapshot or Checkpoint Environment->>Host: Keyed checkpoint Host-->>Environment: Checkpoint ID Environment->>Environment: Save snapshot record opt Background checkpoint Environment->>Artifacts: Send Move Artifacts->>ArtifactPin: Pin source and destination Artifacts->>Host: Copy checkpoint Artifacts-->>Environment: ChainMoved Environment->>ArtifactPin: Pin chain base Environment->>Environment: Save chain endReviews (1) · Last reviewed commit: "docs: snapshot, restore and checkpoint c..." · Reviewed by Greptile