[andrewn6]

Runs hitl webhook deliveries on restate, so a notification survives an orchestrator restart and a receiver that is down for hours, and delivery state is visible. #4 of the restate stack (#346 -> #345 -> #349 -> #348 -> #342).

• Webhooks.Deliver (a plain service, not the Environment object): one invocation is one attempt at one delivery to one endpoint. recordAudit enqueues it with a one-way ingress send instead of go deliver (sends run off the request goroutine, at most 8 in flight, keeping the request id).
• retries: a failed attempt does not sleep. it sends the next attempt to itself with WithDelay (1s, then x4, capped at 1h) and ends, so a delivery failing for a day is a chain of short invocations, each on the deployment current when it starts (epic invariant 4). once the next attempt would land more than 24h after the event, it writes a hitl_webhook dead letter keyed by delivery id and ends. a receiver's non-2xx is a journaled attempt result, not a step error, so the WEBHOOKS retry policy only covers restate failing to run an attempt.
• nothing secret in the journal: the input holds the notification (event, intervention/env ids, gate, on, deadline, intervention link), the endpoint host, and the url + hmac secret sealed under TOKEN_ENCRYPTION_KEY. the url is sealed too, since a webhook url can carry a token in its path or query (slack-style). the post step unseals, signs at send time and posts; its journaled result is status + error with the url stripped (*url.Error reduced to its cause). an endpoint that no longer decrypts is dead-lettered at once.
• why sealed rather than resolving the endpoint from the vm at send time: the cancel-on-destroy notification and any retry after a destroy have no vm left to resolve from. same approach as feat: run token rotation as one restate journal with only ciphertext journaled #378's RotatedCredentials.
• fallbacks: FUSE_RESTATE_MODE=off keeps the in-process 0/1/4/16s goroutine (removal is [restate] make restate required and remove the in-process op paths #353). with restate on but no TOKEN_ENCRYPTION_KEY, or a send restate does not take, the delivery also goes in process.
• metrics by endpoint host (host:port only, bounded and with no path/query): orchestrator_hitl_webhook_attempts_total, _failures_total, _last_status (0 = no answer) in both modes; _backlog and _dead_letters_total with restate on. the backlog is per process: after a restart a delivery waiting out its backoff rejoins it on its next attempt. the legacy path's failure log now names the host, not the full url.
• deliveries are at least once: a kill mid-post re-posts, so receivers should dedupe on intervention_id + event (documented).

No api surface change (fields, states, status codes, messages), so no sdk/openapi changes. Docs: docs/content/docs/guides/human-in-the-loop.mdx (delivery semantics in both modes), docs/content/docs/cloud/monitoring.mdx (webhook metrics + dead-letter alert), docs/content/docs/cloud/orchestrator-deploy.mdx (WEBHOOKS retry service, what runs on restate), docs/runbooks/restate.mdx (handler, chain, journal and pending queries). Checked and left alone: openapi and the api reference (no change), cloud/security.mdx.

verified locally

• gofmt -l ., go build ./..., go vet ./..., golangci-lint v2.12.2 (0 issues), go test ./... -count=1, -race on durable/orchestrator/metrics.
• durable integration tests against a local restate-server 1.6.2: the acceptance run (receiver down, handler endpoint killed after the first failed attempt and replaced on the same address, receiver brought up: delivered exactly once, signature verifies); a receiver that never answers dead-letters after the window with attempts numbered 1..n; a sealed endpoint that no longer decrypts dead-letters after one post. the acceptance test reads the chain's journal through admin sql (sys_journal joined to sys_invocation, narrowed to invocations carrying the delivery id), asserts it holds the delivery id, intervention id and the sealed endpoint (not vacuous), and that it holds neither the url's path/query token nor the signing secret; putting the url in a journaled field makes it fail.
• orchestrator unit tests: durable routing with the delivery json free of url and secret, the post step signing a content-free body, failure results without the url, sealed refusal, dead letter row and backlog going back to 0, fallback to in process on a failed send and without an encryption key, metrics in mode off.
• docs bun run build; bun run lint fails only on pre-existing biome formatting in files this PR does not touch.

not run

• the hitl e2e on phantom: kill -9 of the orchestrator with a receiver down, then the receiver back, and restate sql over a real delivery's journal.
• a restate outage longer than the WEBHOOKS invocation policy (10 attempts) while a dead-letter write is failing pauses that invocation, as for the other services.

Stacked on #378.

Closes #348

Confidence Score: 4/5

The PR appears safe to merge, with non-blocking fixes recommended for backlog metrics and delays during fallback.

Findings

1.  Finished deliveries remain counted ▶
2.  Older counts overwrite newer ones ▶
3.  Fallback retries block other notifications ▶

Summary

Moves HITL webhook delivery onto Restate with sealed endpoints, delayed retries, and dead letters.

• HITL webhook deliveries can retry across orchestrator restarts.
• Webhook metrics show delivery results by endpoint host.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Audit event] --> B{Restate and encryption key?}
  B -->|No| C[In-process retries]
  B -->|Yes| D[Seal endpoint and enqueue]
  D -->|Send fails| C
  D --> E[Webhooks.Deliver]
  E --> F[Unseal, sign, and post]
  F -->|Delivered| G[Finish]
  F -->|Failed within retry window| H[Send delayed next attempt]
  H --> E
  F -->|Window ended or endpoint unreadable| I[Write dead letter]

Loading

_{Reviews (1) · Last reviewed commit: "docs: durable hitl webhook delivery, its..." · Reviewed by Greptile}

[cloudflare-workers-and-pages]

Deploying with    Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Preview URL	Updated (UTC)
✅ Deployment successful! View logs	fuse	b25c84c	Commit Preview URL Branch Preview URL	Nov 15 2026, 07:38 AM

[greptile-apps]

Finished deliveries remain counted

With load-balanced Restate Cloud replicas, notifyWebhook adds a delivery to the sending process's backlog, but another replica can finish it. WebhookPost and WebhookDeadLetter only remove entries from their own process, so the sender keeps counting that delivery forever. Later retries can leave entries on several replicas.

Count pending deliveries in shared state, or use a measure that does not require the same process to see both ends.

[greptile-apps]

Older counts overwrite newer ones

webhookPending releases webhookMu before publishing the count. Concurrent updates can publish in reverse order: an addition computes 1, a completion publishes 0, then the addition publishes its stale 1. The gauge stays wrong until another update, even with one process.

Keep the count change and its publication in the same locked section.

[greptile-apps]

Fallback retries block other notifications

A failed SendWebhook keeps its send slot while fm.webhooks.deliver runs all four fallback attempts. With a down receiver, that can take about 61 seconds beyond the ingress timeout. Eight such failures block every later notification from trying Restate or its fallback, including notifications for healthy receivers.

Release the send slot before running the fallback.