feat: run snapshot and restore on the restate Environment object with durable chain state

docs/content/docs/cloud/orchestrator-deploy.mdx

@@ -71,15 +71,15 @@ range covers the fleet.
 | `FUSE_RESTATE_INGRESS_URL` | | | Restate ingress URL, required unless the mode is `off`. `http://127.0.0.1:18080` with `ops/restate/compose.yaml`. |
 | `FUSE_RESTATE_API_KEY` / `FUSE_RESTATE_IDENTITY_KEY` | | | Mode `cloud` only: the Cloud API key and the `publickeyv1_...` signing key. Other cloud settings are on [Restate Cloud](/docs/cloud/restate-cloud#settings). |
 | `ORCH_RESTATE_LISTEN` | | `127.0.0.1:9080` | Restate handler endpoint (h2c). Only restate-server, or the TLS proxy in front of it in cloud public mode, should reach it. |
-| `FUSE_RESTATE_RETRY_<SERVICE>_INITIAL_INTERVAL` / `_MAX_INTERVAL` / `_MAX_ATTEMPTS` | | `500ms` / `30s` / `10` | Invocation retry policy per restate service; `<SERVICE>` is `ENVIRONMENT`, `HOST`, `ARTIFACTS` or `ARTIFACTPIN`. The `ARTIFACTS` policy also bounds the retries of an artifact move's pull. |
+| `FUSE_RESTATE_RETRY_<SERVICE>_INITIAL_INTERVAL` / `_MAX_INTERVAL` / `_MAX_ATTEMPTS` | | `500ms` / `30s` / `10` | Invocation retry policy per restate service; `<SERVICE>` is `ENVIRONMENT`, `HOST`, `ARTIFACTS` or `ARTIFACTPIN`. The `ARTIFACTS` policy also bounds the retries of an artifact move's pull; the `ENVIRONMENT` policy bounds the host steps of a destroy, snapshot or restore, and the retries of a snapshot gc deletion. |
 
 With `FIRECRACKER_BASE_URL` unset, the orchestrator runs the in-memory stub provider
 (fine for local dev, not for production). With `DATABASE_URL` unset, state is
 in-memory and does not survive a restart.
 
 ## Restate
 
-Restate is optional and off by default; nothing runs on it yet. To run it next to the orchestrator:
+Restate is optional and off by default. With it on, destroys, artifact moves, snapshots, restores, the checkpoint chain and snapshot gc run on it. To run it next to the orchestrator:
 
 ```bash
 docker compose -f ops/restate/compose.yaml up -d

docs/content/docs/concepts/snapshots.mdx

@@ -101,6 +101,17 @@ Restore is **in-place only** for both kinds: it restores the snapshot onto its
 original VM, which must still be running. There is no cross-host restore, clone,
 or standalone restore into a new VM.
 
+### With restate on
+
+With `FUSE_RESTATE_MODE=self` or `cloud`, a snapshot and a restore each run as
+one durable operation on the environment, one at a time per environment. If the
+orchestrator restarts mid-way, the operation picks up where it stopped: a
+snapshot whose host checkpoint was taken still gets its record, and a restore
+finishes instead of leaving the snapshot in `restoring`. A snapshot whose record
+cannot be written is undone: the host artifact is deleted and the snapshot is
+left in `error` with the cause in `last_error`. The API, its fields and its
+status codes are the same in both modes.
+
 ### The guest clock after a live restore
 
 A resumed guest's clock is frozen at the moment the snapshot was taken, and

docs/runbooks/restate.mdx

@@ -158,6 +158,34 @@ forgets its chain instead of deleting it.
 q "SELECT service_key, key, value_utf8 FROM state WHERE service_name = 'ArtifactPin'"
 ```
 
+snapshots and restores run as `Environment/<vm_id>/Snapshot` and
+`Environment/<vm_id>/Restore`. the host checkpoint carries the host-agent
+idempotency key, so a restart between it and the row write records the
+checkpoint already taken. a snapshot whose record cannot be finished deletes
+the host artifact and leaves the row `error` (`snapshot.failed` event). a
+restore resumes at its first unfinished step after a restart.
+
+the checkpoint loop runs each vm's tick as `Environment/<vm_id>/Checkpoint`.
+the chain (head, base, standby) is that object's state, so a restart keeps it,
+and its base is pinned in `ArtifactPin` with holder `chain:<vm_id>`. the copy
+to the standby is a one-way `Artifacts` move that reports back to
+`Environment/<vm_id>/ChainMoved`, so a long copy does not hold the
+environment's other ops. a delta migrate takes `HoldChain` (a 1h lease) for
+the whole migrate; a tick on a held chain does nothing, and a second migrate
+finds it busy and takes a full seed. a destroy drops the
+chain. a copy that never reports back (a killed `Artifacts` invocation) is
+forgotten after 2h without deleting anything.
+
+```sh
+q "SELECT service_key, value_utf8 FROM state WHERE service_name = 'Environment' AND key = 'chain'"
+```
+
+retention gc marks an expired snapshot `deleting` and sends
+`Environment/<vm_id>/CollectSnapshot`, which retries the host delete under the
+`ENVIRONMENT` policy and only then moves the row to `error`
+(`snapshot.gc_failed`); the next sweep marks it again. gc skips a snapshot any
+pin holds, in process or in `ArtifactPin`.
+
 copies are metered by `orchestrator_artifact_move_duration_seconds`,
 `orchestrator_artifact_move_bytes_total` and
 `orchestrator_artifact_move_failures_total`, labeled `source_host` and

internal/durable/artifacts.go

@@ -53,6 +53,15 @@ func moveAbortTimeout(pullTimeout time.Duration) time.Duration {
 // only a killed invocation leaves them, for an operator to release.
 func move(ctx restate.ObjectContext, fleet Artifacts, policy RetryPolicy, job orchestrator.ArtifactMoveJob) (string, error) {
 	id, err := moveSteps(ctx, fleet, policy, job)
+	if job.Notify != "" && (err == nil || restate.IsTerminalError(err)) {
+		// a checkpoint chain's copy: its vm's object waits for this instead
+		// of the caller.
+		res := orchestrator.ChainMoveResult{LocalID: job.LocalID, ID: id}
+		if err != nil {
+			res.Err = err.Error()
+		}
+		restate.ObjectSend(ctx, EnvironmentObject, job.Notify, "ChainMoved").Send(res)
+	}
 	if err != nil && restate.IsTerminalError(err) {
 		// the code is what tells the waiting caller that the move itself
 		// ended, whichever restate-server version answers.