[andrewn6]

summary

approved hitl actions and intervention deadlines run on restate in modes self and cloud. an approval is a keyed Intervention/{id}.Run invocation, so an action interrupted by a restart is carried on instead of settled as "may or may not have run", and every pending intervention's deadline is a durable timer.

• Intervention virtual object (keyed by intervention id), handlers Run and Expire. the fleet side is one method per step (HitlRunBegin, HitlRunInProcess, HitlRunFinish, HitlExpire) behind the durable.Hitl interface, bound in server.go; SetDurableHitl wires the client.
• run routes: exec, computer, drain, fork and migrate run in one in-process step that writes the result to the row (storeRunResult) itself. destroy, snapshot and token rotation are called on Environment/{env} (Destroy, Snapshot, RotateToken) as handler-to-handler calls from the Intervention key (no same-key call, so no deadlock); their steps are not re-run inline, and the run writes what they returned.
• exec runs once: the in-process step carries the durable.Run idempotency key (feat: idempotency keys on host-agent mutations so durable steps can retry after a lost response #370), so a retry after kill -9 gets the agent's first answer back instead of running the command again.
• at most once for the rest: computer, drain, fork and migrate have no host-side key (re-running a fork under the same keys would replay answers for a different vm id), so the step writes a start mark into the sealed result column before acting. an attempt that finds the mark stores the "may or may not have run" error instead of acting twice. a result whose write failed is kept in memory for the step's retry, so a db blip does not turn a finished action into "maybe".
• deadlines: raising a pending intervention (ask, gate, signal) sends Intervention/{id}.Expire with the deadline as its delay. it applies on_timeout through GetIntervention's existing expiry path; an on_timeout: allow approval is sent to Run like a human's. an early fire (clock skew) re-sends itself for the rest. the reconcile expiry sweep stays as the backstop.
• resume pass: in modes self/cloud recoverOrphanRuns is not used. the first reconcile sends every approved-without-result action to Run again (keyed hitl-run:<id>, so one in flight is not started twice), covering a crash between the decision's row write and its send. a failed send sets a flag and the next reconcile resends.
• metrics (both modes): orchestrator_hitl_decision_seconds{kind,outcome}, orchestrator_hitl_decider_duration_seconds{verdict} (error when the call failed), orchestrator_hitl_run_duration_seconds{action,result} (approval to result).
• FUSE_RESTATE_MODE=off keeps the in-process run, the hitlRuns claim and recoverOrphanRuns unchanged.

design choices

• no awakeable, no awakeable id column. the decision already lives on the intervention row (invariant 1) and DecideIntervention writes it there. an invocation waiting on an awakeable or a workflow promise would sleep for the whole deadline (up to 24h), which invariant 4 rules out, and resolving the awakeable has the same crash window between the row write and the restate call as a keyed send. so the decision is a one-way keyed send of Run once the row says approve, and on_timeout is a delayed send of Expire. no invocation is ever waiting on a human; no migration, no api change.
• waiters read the row. hitlWaiters stays as the in-process wake hint: the handler endpoint runs in the orchestrator process (self and cloud, tunnel or public), so storeRunResult from the durable step wakes it. after a restart the sdk's ?wait= re-poll reads the row and is woken when the resumed run stores its result. attaching to the invocation would add nothing the row does not already give.
• nothing secret in the journal: the journaled plan is only the route, env id and snapshot options. exec output and computer screenshots are written to the row inside the step and never returned from it (checked against sys_journal in the integration test).
• failure modes: the run's steps retry without an attempt bound under the INTERVENTION policy's backoff (FUSE_RESTATE_RETRY_INTERVENTION_*), since they only fail on the state store. an endpoint unreachable past the invocation policy behaves like the other ported services (restate's on-max-attempts handling); the resume pass's resend attaches to that invocation rather than starting a second one.

deferred

• decider as a Run: askDecider still runs in the gating api request with its 5s timeout and fall-through to a human. it runs before the pending row exists and the agent's request fails if the orchestrator dies mid-call, so there is nothing to resume; journaling it would need the gate itself to become an invocation. latency and errors are now metered.
• signal detection (lastCall/quietFor for quiet stalls, the expiry-ahead-of-reap check) stays in reconcile memory. the signal interventions' own deadlines do get the durable timer.

local verification

• gofmt, go build ./..., go vet ./..., go test ./... -count=1, -race on durable, orchestrator and metrics, golangci-lint v2.12.2: clean.
• durable integration tests against restate-server 1.6.2 (ports 58080/59070), whole internal/durable suite under -race:
  • TestHitlRunSurvivesRestartBeforeRun: run sent, endpoint killed before it starts, restarted endpoint runs it once; a second send (the resume pass) starts nothing.
  • TestHitlRunExecNotRerunAfterRestart: endpoint killed after the agent ran the exec and before the result write; the retry carries the same Idempotency-Key, the fake agent replays, the command ran once; the journal holds no output.
  • TestHitlExpireSurvivesRestart: deadline timer armed, endpoint killed, fires on the restarted one; an early fire re-arms.
  • TestHitlRunSnapshotCallsEnvironment: an approved snapshot runs through Environment.Snapshot and the run stores its id.
• orchestrator unit tests: approval goes to restate and not in process, waiter woken by the step's result, at-most-once start mark, routes, resume pass, resend after a failed send, on_timeout: allow through the timer, metrics.
• docs: bun run build passes; bun run lint fails identically on the base branch (biome config/format of untouched files).

not run

• not run on phantom or any live host: the hitl e2e scripts (hitl-ask, hitl-exec, hitl-access, hitl-takeover, hitl-computer) with mode self and a real kill -9 are unrun.
• the restart tests use a fake fleet and a fake agent, not a real FleetManager behind restate.

Docs: docs/content/docs/guides/human-in-the-loop.mdx (restart behavior with restate on), docs/content/docs/cloud/monitoring.mdx (hitl metrics), docs/content/docs/cloud/orchestrator-deploy.mdx (INTERVENTION retry policy). checked and left alone: concepts/reconcile.mdx, cloud/tracing.mdx. no api surface change, so no sdk or openapi changes.

Stacked on #379.

Closes #342

Confidence Score: 4/5

The exec replay window must be bounded before merging because a long outage can cause an approved command to run twice.

Findings

1.  Approved commands can repeat ▶
2.  Failed sends build up ▶
3.  Deadline timers fire late ▶
4.  Approved actions miss lifecycle metrics ▶

Summary

Moves approved HITL actions and pending deadlines onto Restate while keeping the in-process path when Restate is off.

• Approved actions can resume across orchestrator restarts.
• Pending intervention deadlines run as durable timers.
• HITL metrics show how long decisions and actions take.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Human decision or deadline] --> B[Save decision on intervention]
  B --> C[Send keyed Intervention.Run]
  C --> D{Action}
  D --> E[Exec or other in-process action]
  D --> F[Environment durable handler]
  E --> G[Save result and wake waiters]
  F --> G
  H[Pending intervention] --> I[Delayed Intervention.Expire]
  I --> A
  J[First reconcile after restart] --> C

Loading

_{Reviews (1) · Last reviewed commit: "docs: hitl approvals across restarts wit..." · Reviewed by Greptile}

[cloudflare-workers-and-pages]

Deploying with    Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Preview URL	Updated (UTC)
✅ Deployment successful! View logs	fuse	3360d72	Commit Preview URL Branch Preview URL	Nov 15 2026, 07:38 AM

[greptile-apps]

Approved commands can repeat

HitlRunInProcess skips the start mark for exec and relies on the host's saved answer, which expires after 24 hours by default. If the orchestrator dies after a command finishes but before storing its result, resuming after that window runs the command again under the same key. A command with lasting effects can therefore happen twice.

Persist a replay deadline or keep the host's answer until the intervention is settled.

[greptile-apps]

Failed sends build up

hitlSend limits active requests but creates unlimited goroutines waiting for a slot. During a Restate outage, failed run sends ask the next reconcile to enqueue every unfinished approval again. If that backlog exceeds what eight requests can send between reconciles, duplicate waiters keep accumulating and consume memory.

Use a bounded queue and avoid queuing the same intervention more than once.

[greptile-apps]

Deadline timers fire late

scheduleExpiry calculates the remaining time before the send waits for a slot. Restate starts that full delay only when it receives the request, so time spent in the local queue is added to the deadline. A slow or backed-up ingress leaves readers or reconcile to enforce the actual deadline.

Calculate time.Until(in.Deadline) inside the send callback instead.

[greptile-apps]

Approved actions miss lifecycle metrics

These direct calls to Environment skip DestroyVM, CreateSnapshot, and RotateToken, whose wrappers call startOperation. The durable handlers record only step metrics. With Restate enabled, these approved actions disappear from orchestrator_operations_total and orchestrator_operations_duration_seconds, so existing dashboards and failure alerts miss them.

Preserve the operation measurements around these routes without rerunning the actions.