release: flowsafe 0.18.0, fleet-control 0.3.3

docs/do-runner-design.md

@@ -138,7 +138,76 @@ The optional `flowsafe.runLifecycle` record stores deadlines, trusted economic-s
 
 Economic settlement projections enter through internal `StartRunOptions.economicOperations` or `ResumeRunOptions.economicOperations`. The host fixes them at the execution-leg boundary before the leg becomes cancellable. Public HTTP bodies cannot supply them. The runtime exposes no dynamic mid-leg mutation because an external snapshot update could race Mastra's own snapshot writes and lose a disputed marker.
 
-The runner uses a short-lived Durable Object alarm only to reconcile an interrupted run-owner reservation. Alarms do not drive workflow execution; starts and approval resumes arrive by request.
+The run's Durable Object alarm serves two duties: reconciling an interrupted run-owner reservation, and the per-suspension deadlines described below. It is armed at the earlier of the two due times. Starts and approval resumes still arrive by request; the alarm drives execution only when a suspension deadline expires.
+
+### Per-suspension deadlines
+
+A suspended step can carry its own deadline. When it elapses before the awaited signal arrives, the run's own Durable Object resumes the run.
+
+A step arms one by adding the reserved key to the payload it hands Mastra's `suspend()`:
+
+```typescript
+import {
+  isSuspensionTimeoutResumeData,
+  SUSPENSION_DEADLINE_PAYLOAD_KEY,
+} from '@proofoftech/flowsafe/do-runner';
+import { z } from 'zod';
+
+const gate = createStep({
+  id: 'gate',
+  inputSchema: z.object({ topic: z.string() }),
+  outputSchema: z.object({ topic: z.string(), settledBy: z.string() }),
+  execute: async ({ inputData, resumeData, suspend }) => {
+    if (!resumeData) {
+      return suspend({
+        reason: 'awaiting approval',
+        [SUSPENSION_DEADLINE_PAYLOAD_KEY]: 900_000,
+      });
+    }
+    return {
+      topic: inputData.topic,
+      settledBy: isSuspensionTimeoutResumeData(resumeData)
+        ? 'timeout'
+        : 'signal',
+    };
+  },
+});
+```
+
+The value is relative milliseconds, a safe integer between `MIN_SUSPENSION_DEADLINE_MS` and `MAX_SUSPENSION_DEADLINE_MS` (365 days). A step declaring a Zod `suspendSchema` must declare the reserved field, or use a loose object (`z.looseObject()`, or `.passthrough()` on a `z.object()`): Mastra validates the suspend payload and substitutes the parsed output, so a strict `z.object()` strips the key and nothing arms. A malformed or out-of-range value never fails the suspension — the suspension is already persisted — it is logged and left unarmed.
+
+Only a top-level suspended step can arm one. Mastra reports a step suspended inside a nested workflow under its nested path, but keys the suspend payload, `suspendedAt`, and `resumeCount` by the enclosing top-level step, so a nested suspension has no fence of its own and an entry that cannot be fenced must never be armed. A nested path carrying the reserved key is therefore refused and logged, not armed. The refusal holds on both projections of that suspension: the live result reports the nested path, while the snapshot-rehydrated one collapses it to the enclosing step and is recognized by the nesting marker Mastra persists in the payload (`__workflow_meta.path`, pinned by a tripwire test).
+
+A top-level step id that contains a dot does arm normally. Entries are keyed by the dot-joined suspended path, which is exactly how both projections key the payload and the two fence fields, so `['a.b']` from a live result and `['a','b']` from a rehydrated snapshot describe the same entry. Such an id must not coincide with a nested path, though: a top-level step `'a.b'` suspended alongside a nested workflow `'a'` whose inner step `'b'` suspends produces one key for two suspensions, which Mastra's own snapshot namespace also collides. Neither projection can then say which suspension an entry belongs to, so every deadline on that key is refused rather than armed. A refusal is logged by whichever boundary reads a projection that shows it, so a request visible on only one projection is still refused everywhere and reported where it shows. The constraint is run-design-wide, not deadline-specific: the same dot-joined key space is what the approval bridge fences approval records on, so a top-level id that coincides with a nested path muddies every consumer of that namespace. A payload that carries `__workflow_meta.path` itself implies an ambiguous or nested key — that field is Mastra's nesting marker, and an author must not set it.
+
+The expiring resume delivers one flowsafe-defined envelope as its resume data:
+
+```json
+{
+  "flowsafe.suspensionTimeout": {
+    "step": "gate",
+    "deadlineAt": 1751883300000,
+    "expiredAt": 1751883300118
+  }
+}
+```
+
+Branch on `isSuspensionTimeoutResumeData()` rather than the literal key. A step that declares a `resumeSchema` must accept this shape as well as its signal shape: Mastra validates resume data before the engine runs, so a schema that rejects the envelope makes every timeout resume throw, and the deadline is dropped once the retry budget is spent. The envelope is the runner's to mint — a resume request that carries the reserved key is refused with a 400, so a caller cannot drive a step's timeout branch while provenance still names them as the requester.
+
+Behavior worth knowing before relying on it:
+
+- The deadline is `suspendedAt + deadlineMs`, taken from the suspension itself, so repeated reconciliation cannot walk it forward.
+- Before resuming, the object re-reads authoritative state and proceeds only while the run is still suspended at that step with the same `suspendedAt` and `resumeCount`. A real signal that arrived first drops the entry. Only a read that SUCCEEDED is evidence about the run. Mastra answers a state read from the in-memory run an isolate still holds whenever storage is unavailable or the row lookup comes back empty, and that answer carries the run object's own status — `pending` until something updates it — with no suspended paths and no request context. Mastra marks it (`isFromInMemory`), a persisted row never carries the mark, and the runner's authoritative read refuses it, so neither a lagging read replica nor a storage fault can discard a live deadline or spend its retry budget. A summary that reads back self-inconsistently — `suspended` with no suspended paths at all — is refused too, as a second line of defence for that shape whatever produced it.
+- A failure INSIDE the deadline duty that leaves an entry unconsumed — a failing timeout resume, a run that reads back absent or self-inconsistent at the fence check — is charged to that entry's retry ledger, backed off, and abandoned after five failures; every alarm is armed at least a second out, so an entry that stays due backs off instead of re-arming at its own past deadline, which Cloudflare would fire immediately. Only a wake whose authoritative read SUCCEEDED may spend that budget, with one standing exception in the other direction: a wake whose stored record names a run other than the object's own `idFromName` identity is refused before it reads anything, and that refusal IS charged — charging is what walks a foreign record to a tombstone in five wakes and quiets it, and no object reachable through the exported topology (`host-kit/do-run-topology.ts`) can be addressed under a name its record disagrees with. A read that did not succeed — Mastra's in-memory fallback, or a read that threw, whether from a workflow a later deploy unregistered or from a storage fault — charges nothing and keeps the 60 second watchdog cadence until it heals: bounded to one wake and one read per minute, never the floor, never a delete. A read that succeeded and found NOTHING is the other case: the store answered, and staleness windows are sub-second against a fifteen-minute budget, so a genuinely absent run is charged and terminates quietly as a tombstone. The uncharged side has one bound of its own — an entry that has been DUE for 24 hours with the run's state continuously unreadable is abandoned under its own log (`abandoned after 24 h of unreadable run state`), and abandoning it converges the wake, so an object with nothing else armed deletes its alarm instead of heart-beating forever for a registration a deploy dropped for good. That clock starts when an entry falls due, never when it is armed, and one failed read stamps every entry due at that wake: a record clears about a day after its last entry came due, not a day per entry, while an entry armed further out is left untouched until its own turn. A wake that cannot read its record, BUILD its runtime, or write its ledger keeps the same cadence for the same reason: none of those faults is evidence about any entry, and a misconfigured binding throws from `build(env)` on every wake, so charging it would tombstone every live deadline of the run in five wakes.
+- A wake with a record in hand that cannot read authoritative state keeps that record: it re-arms nothing, keeps the 60 second cadence, and the entry is charged only once a wake that could read finds it due. On a wake with NO record, a read that succeeds with null converges instead — there is nothing to keep, so the wake arms nothing and deletes the alarm. Both of those are bounded and self-healing, though during a storage incident the affected population is every suspended run whose object takes a wake. Two cases are neither. A wake with NO record whose read THROWS has no record to keep, no entry to charge and none to stamp, so it keeps the 60 second heartbeat with no terminator until the read heals or the isolate is evicted. That is accepted rather than converged — converging it would delete the alarm of a run whose deadline record a failed boundary write never landed, which is the failure that retry wake exists for — and reaching it takes a retry wake followed by the run's workflow itself going unreadable for good. A wake whose `build(env)` throws is the other, and it needs no record at all: a misconfigured binding fails before any entry is in hand, so the wake charges nothing and stamps nothing — not even the 24 hour clock, which runs only on entries a failed READ found due — and it too keeps the 60 second heartbeat with no terminator until the deployment is fixed. That is deliberate: the fault is the host's and says nothing about any deadline, and the alternative is tombstoning every live deadline of the run over a configuration mistake.
+- The resume records `requestedByKind: 'system'` and the reserved principal id `flowsafe-suspension-deadline`, and broadcasts the new summary like any other resume.
+- A timeout resume is NOT an approval decision. It mints no grant and records no reviewer. A step that gates a privileged action must treat the timeout branch as a denial, an escalation, or a no-op — never as consent.
+- An armed deadline is not observable through `RunSummary` in v1. The record lives in the run object's own storage, no route projects it, and the bounds that describe it are exported so a consumer can validate its own `deadlineMs` before arming, while `MAX_SUSPENSION_DEADLINES_PER_RUN` is exported as an operational figure rather than as something to check a deadline against. The workerd spike needs a test-only introspection route on its Durable Object precisely because nothing else can see the armed state.
+- A timeout resume takes no host route, so the hooks a resume normally passes through do not fire for it: `RunRouterOptions.beforeResume` cannot vet it (the run object resumes itself), and `RunRouterOptions.reconcileApprovals` / `reconcileApprovalsForSummary` do not run at that moment. An approval record filed for the expired suspension therefore stays open until a later host status read reconciles it, where the `(suspendedAt, resumeCount)` binding it already carries shows the suspension moved on. Nothing decides that approval, and nothing acts on a decision arriving late.
+- A `foreach` step arms one deadline for the step as a whole, because one suspended path with one fence is all either projection reports for it. A default `foreach` is sequential: one iteration is suspended at a time, so each iteration's suspension gets its own deadline and clearing a whole `foreach` by timeout takes one deadline per item. With `concurrency` above 1, up to `concurrency` iterations suspend at once behind that one path, one `suspendedAt`, and the FIRST suspended iteration's payload, so only that iteration's `deadlineMs` is read per batch; the timeout resume delivers the envelope to every iteration suspended at that moment (at most `concurrency` of them), iterations not yet started then suspend afresh with their own deadline, and clearing the whole `foreach` takes `ceil(items / concurrency)` deadlines.
+- Wake precision is the Durable Object alarm's, near the requested time rather than exact. There is no maintenance-sweep backstop, so a lost alarm is a lost deadline; run-level `deadlineMs` remains the swept mechanism. The record itself is a separate best-effort write made after Mastra has persisted the suspension, not part of it: a write that fails leaves a 60 second retry wake. Every wake that does not resume — that retry wake included — ends by re-deriving the run's deadlines from an authoritative read; the stored record is only the identity fallback for an object that carries no name, never the source the wake trusts over authoritative state. One re-derivation is not made from an authoritative read: the terminal deadline route's non-finalizing branch — the CAS-stale answer, and the already-cleaned-up replay — reconciles from `result.summary`, which `timeOutAsPrincipal` produced with a post-persist `getWorkflowRunById`, the read Mastra can answer from its in-memory fallback. A marked answer there would derive nothing and clear a record for a run that is still suspended. It is not guarded mechanically because reaching it takes that second read flipping to the fallback inside one held run lock, after the snapshot read the branch opens with has already succeeded on the same store; and if it ever did, the run's own next boundary or wake re-derives what was dropped, since the record is bookkeeping about a suspension the snapshot still holds.
+- One entry fires per wake, at most 32 entries are armed per run, and a failing wake backs off and is abandoned after five failures — abandoned for that suspension: the spent entry stays in the record as a tombstone, never selected and never armed again, so reconciliation cannot re-derive the same suspension a fresh budget; a later suspension of the same step starts a fresh budget. A tombstone counts against the 32-entry cap, which stays bounded per run, and abandonment is therefore cap-conditional: entries are capped in deadline order, a tombstone belongs to an older suspension than the live ones around it, so past 32 concurrently armed suspensions a tombstone holds its slot while the newest live deadline is the one refused — and a tombstone that the cap does splice out re-derives with a fresh budget. A record left holding only tombstones arms no alarm at all and is cleared by the next lifecycle boundary or wake that reads a run which is no longer suspended there; for a run whose state is gone entirely, nothing reads it again and the record persists (about 150 bytes per entry, bounded by the cap). Both are accepted. The stored record, its parser, and the wake arithmetic are not exported: they are the run object's own state, and only the object that owns the alarm can act on them.
+- Scope is `DurableObjectRunner`-hosted workflow runs. The durable-agent runner has its own resume path and does not arm suspension deadlines.
 
 ## Run summary
 

docs/flowsafe-architecture.md

@@ -79,6 +79,8 @@ The runtime does not mint `breakwater.isolationScope`. One deployment serves one
 
 The snapshot remains Mastra's workflow state. Flowsafe does not invent a second workflow-state document inside Durable Object storage.
 
+The one exception is wake scheduling. A suspended step can arm a per-suspension deadline through a reserved key in the payload it passes Mastra's `suspend()`; the run's object derives that deadline from the authoritative summary, keeps it in its own storage, and multiplexes it with run-owner recovery onto its single alarm. When a deadline expires the object resumes the run itself under a reserved system principal, with an envelope the step distinguishes from a real signal, after re-checking that the run is still suspended at the same step and suspension. That record and the arithmetic over it stay inside the run's object: only the holder of the alarm can act on them, so they are not part of the package surface. See [Durable Object runner design](do-runner-design.md) for the arming contract, the `suspendSchema` and `resumeSchema` caveats, the top-level-step-only limitation, and the failure modes.
+
 ## Approval architecture
 
 `D1ApprovalStoreFactory.store()` returns one memoized store for the deployment database. Request routes, runtime grant derivation, and alarm maintenance duties share that store.

docs/maintainer-guide.md

@@ -91,6 +91,10 @@ CI tests the declared supported peer version as part of the normal gate. A separ
 
 Treat a red canary as a release investigation even though it does not block a merge. Update the declared peer range only after tests, workerd proofs, package tarball probes, and migration notes pass.
 
+Per-suspension deadlines couple to one undocumented Mastra behavior: a step arms a deadline through a reserved key in the payload it hands `suspend()`, which only reaches flowsafe because Mastra substitutes the schema-parsed suspend payload into the run summary (verified in the declared minimum peer, 1.50.0). A change there — a different substitution, a different key for a nested suspension, or resume-data validation moving — silently disarms every deadline. Tripwire tests in `packages/flowsafe/src/do-runner/runtime.test.ts` pin the observed behavior: the reserved key surviving a schema that declares it, being stripped by a strict schema that does not, surviving a loose schema, and a nested suspension being refused rather than armed. Check them on every Mastra upgrade and treat a failure as a behavior change to document, never as a test to relax.
+
+Rolling this release back is not symmetric: 0.17.x has no deadline reader, so the first alarm a downgraded run object takes deletes the alarm and orphans every armed record. Re-upgrading heals only runs that later receive another lifecycle boundary — which excludes exactly the runs a suspension deadline exists for, since a suspended run waiting on a signal has no boundary but its own wake. Prefer rolling forward; if a downgrade is unavoidable, treat every deadline armed before it as lost.
+
 ## Public documentation
 
 `pnpm docs:check` validates local links and anchors, package export coverage, TypeDoc entry coverage, npm-safe package links, orphaned public pages, stale internal markers, and manifest-backed Node engine and peer-dependency claims. `pnpm docs:api` builds all supported API surfaces, including the React UI in its own TypeScript program.

docs/security-threat-model.md

@@ -71,7 +71,7 @@ Trusted merges apply sanitized external or stored context first, then workflow,
 
 The Worker chooses the object name through `idFromName()`. Each object reasserts the addressed identity:
 
-- runner object: `workflowId:runId`;
+- runner object: `workflowId:runId` — and the runner object also derives `workflowId:runId` from that same `id.name` on any alarm wake that does not resume, in preference to its own stored record, which is consulted only when the object carries no name; the name is therefore a trusted input to its wake, not only an assertion target;
 - thread object: server-minted `threadId` and internal principal header;
 - hub object: fixed deployment singleton name;
 - provider host object: fixed deployment singleton name.

packages/agent-starter/CHANGELOG.md

@@ -1,5 +1,12 @@
 # anchorage-agent-starter
 
+## 0.0.14
+
+### Patch Changes
+
+- Updated dependencies [e7fb658]
+  - @proofoftech/flowsafe@0.18.0
+
 ## 0.0.13
 
 ### Patch Changes

packages/agent-starter/package.json

@@ -1,6 +1,6 @@
 {
   "name": "anchorage-agent-starter",
-  "version": "0.0.13",
+  "version": "0.0.14",
   "private": true,
   "description": "Production-shaped Cloudflare Workers starter for durable, approval-gated Anchorage agents",
   "type": "module",