[andrewn6]

Runs token rotation on the restate Environment/{vm_id} object, so a crash between the guest upload and the state store write no longer leaves the stored token different from the guest's. #3 of the restate stack (#346 -> #345 -> #349 -> #348 -> #342).

• Environment/{vm_id}.RotateToken: generate -> Run upload -> Run persist. generate is a Run, so a replay reuses the token it already made instead of generating a second one.
• generate returns ciphertext only (RotatedCredentials: the token, cert and key sealed under TOKEN_ENCRYPTION_KEY, plus the token as the vm row stores it). the upload and persist steps decrypt inside themselves, so neither the plaintext token nor the tls key enters the journal.
• upload sends the three credential files one at a time in path order, so a retried step carries the same file under each host-agent idempotency key (<step key>/1..3).
• persist writes the state store synchronously and retries with no attempt bound (the guest already holds the new token, so giving up would leave the old one stored); it replaces persistVMBackground on this path. it also sets the token back on the environment handle, which a restart between upload and persist leaves on the old one.
• refusals: vm not running and no encryption key cross restate as code 487 and come back with the fleet's own message; vm not found keeps its 480 sentinel (404); an agent 4xx on upload is final. the api status codes do not change (not running is still the documented 500).
• the client waits without the ingress request timeout, since a rotation can queue behind a long snapshot on the same object; the api request (or durableWait for an approved hitl run) bounds it.
• metrics: orchestrator_operations_total{op="rotate_token",result} and the duration histogram, in both modes; step metrics as rotate_token.{generate,upload,persist}.
• triggers are unchanged: the rotate-token api action and an approved hitl rotate-token both call FleetManager.RotateToken. fork and migrate still mint their own credentials in process; nothing here touches them.
• FUSE_RESTATE_MODE=off keeps the in-process path (same steps, background persist as before).

No api surface change (fields, states, status codes, messages), so no sdk/openapi changes. Docs: docs/runbooks/restate.mdx (handler, journal query, key-change note), docs/content/docs/cloud/orchestrator-deploy.mdx (retry policy row, what runs on restate), docs/content/docs/cloud/monitoring.mdx (rotate_token op). Checked and left alone: concepts/environments.mdx, cloud/security.mdx, the cli rotate-token page, openapi (behavior unchanged).

verified locally

• gofmt -l ., go build ./..., go vet ./..., golangci-lint v2.12.2 (0 issues), go test ./... -count=1, -race on durable/orchestrator.
• durable integration tests against a local restate-server 1.6.2: a persist failing past the policy's attempts still ends stored; upload retried inside its step with generate run once; refusal message and vm-not-found sentinel crossing restate; endpoint restart while persist runs (connections dropped, new endpoint on the same address): the restarted endpoint runs only persist, and the stored token equals the guest's first token. while held before persist, the test reads the invocation's journal through admin sql (sys_journal joined to sys_invocation), asserts it holds the generate step's ciphertext (so the check is not vacuous) and does not hold the plaintext token. a mutation returning the plaintext token from generate makes that test fail.
• orchestrator unit tests: mode off rotation, durable routing, the persist step after a simulated restart, refusals, metrics in both modes.
• docs bun run build; bun run lint fails only on pre-existing biome formatting in files this PR does not touch.

not run

• the phantom e2e: kill -9 of the orchestrator between upload and persist, then exec into the vm, and restate sql over sys_journal on a real rotation.
• the known gap stays: fused reads its token file once at startup, so a rotation takes effect in the running guest only after fused restarts (same as before this PR).

Stacked on #377.

Closes #349

[cloudflare-workers-and-pages]

Deploying with    Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Preview URL	Updated (UTC)
✅ Deployment successful! View logs	fuse	8cc53a7	Commit Preview URL Branch Preview URL	Nov 15 2026, 07:37 AM

[greptile-apps]

Retries can skip credential files

After a partial upload, retrying this step can report success without writing every credential file. Run gives requests numbered keys, but uploadFiles assigns those numbers through concurrent goroutines over a map. A retry can assign /1 to a different file. The host agent compares only the HTTP method and URL, which are identical for all three uploads, so it returns the earlier file's cached success without writing the requested file.

Rotation can then persist a token that never reached the guest. Give each credential path a stable request key across attempts; sorting the map alone does not fix the race between concurrent requests.

[greptile-apps]

New tokens can remain unsaved

If Postgres stays unavailable through the configured retry budget, this persist step ends with a final failure even though the guest already has the new credentials. Nothing schedules the same token write again. A later orchestrator restart loads the old token from Postgres while the guest's files hold the new one, leaving the crash-safety fix incomplete.

Keep the same pending token recoverable until it is saved, rather than ending the rotation when these retries run out.