From e7fb6581a2924e81c5a88492c2bbf38b4e202ffb Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Thu, 20 Aug 2026 09:45:20 +0400 Subject: [PATCH 1/2] feat(flowsafe): per-suspension deadlines on the run Durable Object A suspended step can arm a deadline with the reserved 'flowsafe.deadlineMs' key in its suspend() payload. The run's own DO persists a fenced record (dot-joined step key, suspendedAt + resumeCount), multiplexes it with run-owner recovery on its single alarm, and on expiry resumes the run itself with the 'flowsafe.suspensionTimeout' envelope under requestedByKind 'system' (isSuspensionTimeoutResumeData to branch). A resume carrying the reserved key is refused 400 on both the workflow and agent-host routes. Correctness rules hardened along the way: - Only a read that succeeded may conclude anything: runtime.authoritativeStatus throws RunStateUnreadableError on Mastra's isFromInMemory fallback (mapped to 503); recoverStartAttempt refuses to conclude or delete off that fabricated state (previously a real row delete behind a lagging read). - Unsuccessful reads, runtime build failures, and ledger-write failures keep an uncharged 60 s watchdog; a due entry continuously unreadable for 24 h is abandoned under its own log. A successful null charges: five failed wakes tombstone the entry for that suspension (a later suspension starts fresh). - Nested suspensions and ambiguous dot-joined keys are refused on both projections; a foreign record's ledger is never inherited by the merge. - Everything after a successful resume is best-effort (broadcast + reconcile). Ships with a real-workerd spike scenario (arm, kill + restart, the restarted object's alarm resumes with the envelope), ~100 new tests each pinned by a negative control, docs, threat-model and maintainer-guide updates, and a flowsafe minor changeset (fleet-control pairs via updateInternalDependencies). Co-Authored-By: Claude Fable 5 --- .changeset/suspension-deadline.md | 13 + docs/do-runner-design.md | 71 +- docs/flowsafe-architecture.md | 2 + docs/maintainer-guide.md | 4 + docs/security-threat-model.md | 2 +- packages/flowsafe/README.md | 40 + packages/flowsafe/scripts/spike-verify.mjs | 298 +- packages/flowsafe/spike/worker.ts | 155 + .../src/agent-host/thread-host.test.ts | 184 +- .../flowsafe/src/agent-host/thread-host.ts | 43 +- .../src/do-runner/do-error-response.ts | 8 + .../src/do-runner/durable-object.test.ts | 3440 ++++++++++++++++- .../flowsafe/src/do-runner/durable-object.ts | 1022 ++++- packages/flowsafe/src/do-runner/index.ts | 24 + .../flowsafe/src/do-runner/runtime.test.ts | 436 +++ packages/flowsafe/src/do-runner/runtime.ts | 74 + .../src/do-runner/suspension-deadline.test.ts | 1099 ++++++ .../src/do-runner/suspension-deadline.ts | 584 +++ .../flowsafe/src/do-runner/thread-do.test.ts | 36 +- packages/flowsafe/src/do-runner/thread-do.ts | 25 +- .../src/signals/thread-do-routes.test.ts | 60 +- 21 files changed, 7570 insertions(+), 50 deletions(-) create mode 100644 .changeset/suspension-deadline.md create mode 100644 packages/flowsafe/src/do-runner/suspension-deadline.test.ts create mode 100644 packages/flowsafe/src/do-runner/suspension-deadline.ts diff --git a/.changeset/suspension-deadline.md b/.changeset/suspension-deadline.md new file mode 100644 index 0000000..7e11368 --- /dev/null +++ b/.changeset/suspension-deadline.md @@ -0,0 +1,13 @@ +--- +"@proofoftech/flowsafe": minor +--- + +Time out one suspension. A workflow step can now arm a deadline on the suspension it creates by adding the reserved `flowsafe.deadlineMs` key to the payload it passes Mastra's `suspend()`. When that deadline elapses before the awaited signal arrives, the run's own Durable Object resumes the run itself and delivers the `flowsafe.suspensionTimeout` envelope to the expired step, which tells it apart from a real signal with the exported `isSuspensionTimeoutResumeData()` guard. The deadline is persisted in the run's Durable Object storage, so it survives eviction and hibernation, and the resume is fenced on the exact suspension it was armed against: a genuine signal that arrives first drops the deadline instead of resuming twice. + +This is backward compatible. Nothing arms unless the reserved key is present, so existing runs and existing suspensions behave exactly as before. A host that forwards a `timeoutMs` field today is unaffected: that field was ignored and remains ignored. + +Authoring notes. A step that declares a Zod `suspendSchema` must declare the reserved field, or use a loose object (`z.looseObject()`, or `.passthrough()` on a `z.object()`): Mastra validates the suspend payload and substitutes the parsed result, so a strict `z.object()` strips the key and no deadline arms. A step that declares a `resumeSchema` must accept the timeout envelope as well as its own signal shape, or every timeout resume fails that validation and the deadline is abandoned. Only a top-level suspended step can arm a deadline: a step suspended inside a nested workflow is reported under its nested path while its suspension time is recorded against the enclosing step, so there is no way to fence the resume, and the deadline is refused and logged instead of armed. A top-level step id containing a dot arms normally, but it must not coincide with a nested path: a step `'a.b'` suspended alongside a nested workflow `'a'` whose inner step `'b'` suspends gives one key for two suspensions, and every deadline on that key is refused and logged rather than armed against the wrong one. A `foreach` step arms one deadline for the step as a whole, because that is all the run summary reports for it: run sequentially, which is the default, each iteration suspends on its own and gets its own deadline, so clearing a whole loop by timeout takes one deadline per item; with `concurrency` above 1 the timeout resume delivers the envelope to every iteration suspended at that moment (at most `concurrency` of them), iterations not yet started suspend afresh with their own deadline, only the first suspended iteration's value is read per batch, and clearing the whole loop takes one deadline per batch of `concurrency` items. Values must be a whole number of milliseconds between `MIN_SUSPENSION_DEADLINE_MS` and `MAX_SUSPENSION_DEADLINE_MS` (365 days); a value outside that range is reported in the runner log and left unarmed rather than failing the suspension that Mastra has already persisted. + +Operational caveats. Wake precision is a Durable Object alarm's — near the requested time, not exact — and there is no maintenance-sweep backstop for suspension deadlines, so a lost alarm is a lost deadline. Run-level `deadlineMs` remains the swept mechanism when a hard expiry is required. One deadline fires per wake, a run arms at most `MAX_SUSPENSION_DEADLINES_PER_RUN` of them, and a failing timeout resume backs off before it is abandoned for that suspension — the spent entry is kept, never retried and never re-armed, so the same suspension cannot earn itself a fresh budget, while a later suspension of the same step starts one. Only a wake whose authoritative read succeeded spends that budget: a read that did not succeed keeps a 60 second retry cadence and charges nothing, so a storage incident cannot abandon a live deadline — unless an entry has been due for a full day with the state unreadable throughout, after which that entry is abandoned under its own log (`abandoned after 24 h of unreadable run state`). A run whose state has been readable and genuinely absent across five wakes is charged and abandoned the ordinary way, and either abandonment is permanent for the suspension it was armed against. The routes that must not answer from a fabricated read fail closed instead: while a pending owner recovery cannot read authoritative state, `POST /runs` and `GET /runs/:workflowId/:runId/dispatch-status` return 503 for that run. Each previously answered from the fabricated read in its own way: `dispatch-status` returned 200 with a fabricated summary after deleting the very row it could not see, and `POST /runs` refused the run with a 500 (`existing run … has no matching committed owner`) — the same recovery had deleted the row and released its claim, while the fabricated read still reported the run as existing. A start whose own body throws is the one path that answers otherwise: it logs the recovery read it could not make, leaves the journal armed for a later wake, and rethrows the original start failure, so the caller sees that error's own status — a 500 for an unclassified start fault, or the error's own code such as a 400 for a malformed `deadlineMs` — rather than the 503. The same guard covers agent-host owner recovery, which now retries fail-closed — keeping its journal, its reservation and its run record — instead of possibly deleting a row behind a lagging read. The agent-host routes that drive that recovery fail closed in the taxonomy of the surface they escape to rather than reporting a run they could not read. Those that escape to the thread Durable Object's own shell, such as the dispatch read (`GET /_flowsafe/agent-host/runs/:agentId/:runId?resourceId=…&dispatch=1`) and the start route (`POST /_flowsafe/agent-host/start`), answer 503 from that shell; the schedule-wake route (`POST /signal/schedule`, which recovers a lost dispatch receipt through that same recovery) answers 502 with an `internal error` body from the signals router instead of settling a dispatch receipt from a read it could not make. An armed deadline is not observable through `RunSummary` in this release: the record is the run object's own wake state, no route projects it, and the exported bounds exist so a consumer can validate its own `deadlineMs` before arming rather than to read back what is armed, while `MAX_SUSPENSION_DEADLINES_PER_RUN` is exported as the operational figure a host plans against. + +A timeout resume is not an approval decision. It records `requestedByKind: 'system'` with the reserved `flowsafe-suspension-deadline` principal id, mints no connector grant, and names no reviewer. Only the runner mints the envelope: a resume request whose resume data carries the reserved `flowsafe.suspensionTimeout` key is rejected with a 400 on both the workflow run route and the agent-host resume route, so no caller can present itself to a step as an expired deadline. A step that gates a privileged action must treat its timeout branch as a denial, an escalation, or a no-op — never as consent. Because the run object resumes itself, a timeout resume passes through no host route: `RunRouterOptions.beforeResume` does not vet it and approval reconciliation does not run at that moment, so an approval record filed for the expired suspension stays open until a later host status read supersedes it against the suspension it was bound to. Suspension deadlines cover workflow runs hosted by `DurableObjectRunner`; durable agents keep their own resume path. diff --git a/docs/do-runner-design.md b/docs/do-runner-design.md index 8ce54ea..89c2c84 100644 --- a/docs/do-runner-design.md +++ b/docs/do-runner-design.md @@ -138,7 +138,76 @@ The optional `flowsafe.runLifecycle` record stores deadlines, trusted economic-s Economic settlement projections enter through internal `StartRunOptions.economicOperations` or `ResumeRunOptions.economicOperations`. The host fixes them at the execution-leg boundary before the leg becomes cancellable. Public HTTP bodies cannot supply them. The runtime exposes no dynamic mid-leg mutation because an external snapshot update could race Mastra's own snapshot writes and lose a disputed marker. -The runner uses a short-lived Durable Object alarm only to reconcile an interrupted run-owner reservation. Alarms do not drive workflow execution; starts and approval resumes arrive by request. +The run's Durable Object alarm serves two duties: reconciling an interrupted run-owner reservation, and the per-suspension deadlines described below. It is armed at the earlier of the two due times. Starts and approval resumes still arrive by request; the alarm drives execution only when a suspension deadline expires. + +### Per-suspension deadlines + +A suspended step can carry its own deadline. When it elapses before the awaited signal arrives, the run's own Durable Object resumes the run. + +A step arms one by adding the reserved key to the payload it hands Mastra's `suspend()`: + +```typescript +import { + isSuspensionTimeoutResumeData, + SUSPENSION_DEADLINE_PAYLOAD_KEY, +} from '@proofoftech/flowsafe/do-runner'; +import { z } from 'zod'; + +const gate = createStep({ + id: 'gate', + inputSchema: z.object({ topic: z.string() }), + outputSchema: z.object({ topic: z.string(), settledBy: z.string() }), + execute: async ({ inputData, resumeData, suspend }) => { + if (!resumeData) { + return suspend({ + reason: 'awaiting approval', + [SUSPENSION_DEADLINE_PAYLOAD_KEY]: 900_000, + }); + } + return { + topic: inputData.topic, + settledBy: isSuspensionTimeoutResumeData(resumeData) + ? 'timeout' + : 'signal', + }; + }, +}); +``` + +The value is relative milliseconds, a safe integer between `MIN_SUSPENSION_DEADLINE_MS` and `MAX_SUSPENSION_DEADLINE_MS` (365 days). A step declaring a Zod `suspendSchema` must declare the reserved field, or use a loose object (`z.looseObject()`, or `.passthrough()` on a `z.object()`): Mastra validates the suspend payload and substitutes the parsed output, so a strict `z.object()` strips the key and nothing arms. A malformed or out-of-range value never fails the suspension — the suspension is already persisted — it is logged and left unarmed. + +Only a top-level suspended step can arm one. Mastra reports a step suspended inside a nested workflow under its nested path, but keys the suspend payload, `suspendedAt`, and `resumeCount` by the enclosing top-level step, so a nested suspension has no fence of its own and an entry that cannot be fenced must never be armed. A nested path carrying the reserved key is therefore refused and logged, not armed. The refusal holds on both projections of that suspension: the live result reports the nested path, while the snapshot-rehydrated one collapses it to the enclosing step and is recognized by the nesting marker Mastra persists in the payload (`__workflow_meta.path`, pinned by a tripwire test). + +A top-level step id that contains a dot does arm normally. Entries are keyed by the dot-joined suspended path, which is exactly how both projections key the payload and the two fence fields, so `['a.b']` from a live result and `['a','b']` from a rehydrated snapshot describe the same entry. Such an id must not coincide with a nested path, though: a top-level step `'a.b'` suspended alongside a nested workflow `'a'` whose inner step `'b'` suspends produces one key for two suspensions, which Mastra's own snapshot namespace also collides. Neither projection can then say which suspension an entry belongs to, so every deadline on that key is refused rather than armed. A refusal is logged by whichever boundary reads a projection that shows it, so a request visible on only one projection is still refused everywhere and reported where it shows. The constraint is run-design-wide, not deadline-specific: the same dot-joined key space is what the approval bridge fences approval records on, so a top-level id that coincides with a nested path muddies every consumer of that namespace. A payload that carries `__workflow_meta.path` itself implies an ambiguous or nested key — that field is Mastra's nesting marker, and an author must not set it. + +The expiring resume delivers one flowsafe-defined envelope as its resume data: + +```json +{ + "flowsafe.suspensionTimeout": { + "step": "gate", + "deadlineAt": 1751883300000, + "expiredAt": 1751883300118 + } +} +``` + +Branch on `isSuspensionTimeoutResumeData()` rather than the literal key. A step that declares a `resumeSchema` must accept this shape as well as its signal shape: Mastra validates resume data before the engine runs, so a schema that rejects the envelope makes every timeout resume throw, and the deadline is dropped once the retry budget is spent. The envelope is the runner's to mint — a resume request that carries the reserved key is refused with a 400, so a caller cannot drive a step's timeout branch while provenance still names them as the requester. + +Behavior worth knowing before relying on it: + +- The deadline is `suspendedAt + deadlineMs`, taken from the suspension itself, so repeated reconciliation cannot walk it forward. +- Before resuming, the object re-reads authoritative state and proceeds only while the run is still suspended at that step with the same `suspendedAt` and `resumeCount`. A real signal that arrived first drops the entry. Only a read that SUCCEEDED is evidence about the run. Mastra answers a state read from the in-memory run an isolate still holds whenever storage is unavailable or the row lookup comes back empty, and that answer carries the run object's own status — `pending` until something updates it — with no suspended paths and no request context. Mastra marks it (`isFromInMemory`), a persisted row never carries the mark, and the runner's authoritative read refuses it, so neither a lagging read replica nor a storage fault can discard a live deadline or spend its retry budget. A summary that reads back self-inconsistently — `suspended` with no suspended paths at all — is refused too, as a second line of defence for that shape whatever produced it. +- A failure INSIDE the deadline duty that leaves an entry unconsumed — a failing timeout resume, a run that reads back absent or self-inconsistent at the fence check — is charged to that entry's retry ledger, backed off, and abandoned after five failures; every alarm is armed at least a second out, so an entry that stays due backs off instead of re-arming at its own past deadline, which Cloudflare would fire immediately. Only a wake whose authoritative read SUCCEEDED may spend that budget, with one standing exception in the other direction: a wake whose stored record names a run other than the object's own `idFromName` identity is refused before it reads anything, and that refusal IS charged — charging is what walks a foreign record to a tombstone in five wakes and quiets it, and no object reachable through the exported topology (`host-kit/do-run-topology.ts`) can be addressed under a name its record disagrees with. A read that did not succeed — Mastra's in-memory fallback, or a read that threw, whether from a workflow a later deploy unregistered or from a storage fault — charges nothing and keeps the 60 second watchdog cadence until it heals: bounded to one wake and one read per minute, never the floor, never a delete. A read that succeeded and found NOTHING is the other case: the store answered, and staleness windows are sub-second against a fifteen-minute budget, so a genuinely absent run is charged and terminates quietly as a tombstone. The uncharged side has one bound of its own — an entry that has been DUE for 24 hours with the run's state continuously unreadable is abandoned under its own log (`abandoned after 24 h of unreadable run state`), and abandoning it converges the wake, so an object with nothing else armed deletes its alarm instead of heart-beating forever for a registration a deploy dropped for good. That clock starts when an entry falls due, never when it is armed, and one failed read stamps every entry due at that wake: a record clears about a day after its last entry came due, not a day per entry, while an entry armed further out is left untouched until its own turn. A wake that cannot read its record, BUILD its runtime, or write its ledger keeps the same cadence for the same reason: none of those faults is evidence about any entry, and a misconfigured binding throws from `build(env)` on every wake, so charging it would tombstone every live deadline of the run in five wakes. +- A wake with a record in hand that cannot read authoritative state keeps that record: it re-arms nothing, keeps the 60 second cadence, and the entry is charged only once a wake that could read finds it due. On a wake with NO record, a read that succeeds with null converges instead — there is nothing to keep, so the wake arms nothing and deletes the alarm. Both of those are bounded and self-healing, though during a storage incident the affected population is every suspended run whose object takes a wake. Two cases are neither. A wake with NO record whose read THROWS has no record to keep, no entry to charge and none to stamp, so it keeps the 60 second heartbeat with no terminator until the read heals or the isolate is evicted. That is accepted rather than converged — converging it would delete the alarm of a run whose deadline record a failed boundary write never landed, which is the failure that retry wake exists for — and reaching it takes a retry wake followed by the run's workflow itself going unreadable for good. A wake whose `build(env)` throws is the other, and it needs no record at all: a misconfigured binding fails before any entry is in hand, so the wake charges nothing and stamps nothing — not even the 24 hour clock, which runs only on entries a failed READ found due — and it too keeps the 60 second heartbeat with no terminator until the deployment is fixed. That is deliberate: the fault is the host's and says nothing about any deadline, and the alternative is tombstoning every live deadline of the run over a configuration mistake. +- The resume records `requestedByKind: 'system'` and the reserved principal id `flowsafe-suspension-deadline`, and broadcasts the new summary like any other resume. +- A timeout resume is NOT an approval decision. It mints no grant and records no reviewer. A step that gates a privileged action must treat the timeout branch as a denial, an escalation, or a no-op — never as consent. +- An armed deadline is not observable through `RunSummary` in v1. The record lives in the run object's own storage, no route projects it, and the bounds that describe it are exported so a consumer can validate its own `deadlineMs` before arming, while `MAX_SUSPENSION_DEADLINES_PER_RUN` is exported as an operational figure rather than as something to check a deadline against. The workerd spike needs a test-only introspection route on its Durable Object precisely because nothing else can see the armed state. +- A timeout resume takes no host route, so the hooks a resume normally passes through do not fire for it: `RunRouterOptions.beforeResume` cannot vet it (the run object resumes itself), and `RunRouterOptions.reconcileApprovals` / `reconcileApprovalsForSummary` do not run at that moment. An approval record filed for the expired suspension therefore stays open until a later host status read reconciles it, where the `(suspendedAt, resumeCount)` binding it already carries shows the suspension moved on. Nothing decides that approval, and nothing acts on a decision arriving late. +- A `foreach` step arms one deadline for the step as a whole, because one suspended path with one fence is all either projection reports for it. A default `foreach` is sequential: one iteration is suspended at a time, so each iteration's suspension gets its own deadline and clearing a whole `foreach` by timeout takes one deadline per item. With `concurrency` above 1, up to `concurrency` iterations suspend at once behind that one path, one `suspendedAt`, and the FIRST suspended iteration's payload, so only that iteration's `deadlineMs` is read per batch; the timeout resume delivers the envelope to every iteration suspended at that moment (at most `concurrency` of them), iterations not yet started then suspend afresh with their own deadline, and clearing the whole `foreach` takes `ceil(items / concurrency)` deadlines. +- Wake precision is the Durable Object alarm's, near the requested time rather than exact. There is no maintenance-sweep backstop, so a lost alarm is a lost deadline; run-level `deadlineMs` remains the swept mechanism. The record itself is a separate best-effort write made after Mastra has persisted the suspension, not part of it: a write that fails leaves a 60 second retry wake. Every wake that does not resume — that retry wake included — ends by re-deriving the run's deadlines from an authoritative read; the stored record is only the identity fallback for an object that carries no name, never the source the wake trusts over authoritative state. One re-derivation is not made from an authoritative read: the terminal deadline route's non-finalizing branch — the CAS-stale answer, and the already-cleaned-up replay — reconciles from `result.summary`, which `timeOutAsPrincipal` produced with a post-persist `getWorkflowRunById`, the read Mastra can answer from its in-memory fallback. A marked answer there would derive nothing and clear a record for a run that is still suspended. It is not guarded mechanically because reaching it takes that second read flipping to the fallback inside one held run lock, after the snapshot read the branch opens with has already succeeded on the same store; and if it ever did, the run's own next boundary or wake re-derives what was dropped, since the record is bookkeeping about a suspension the snapshot still holds. +- One entry fires per wake, at most 32 entries are armed per run, and a failing wake backs off and is abandoned after five failures — abandoned for that suspension: the spent entry stays in the record as a tombstone, never selected and never armed again, so reconciliation cannot re-derive the same suspension a fresh budget; a later suspension of the same step starts a fresh budget. A tombstone counts against the 32-entry cap, which stays bounded per run, and abandonment is therefore cap-conditional: entries are capped in deadline order, a tombstone belongs to an older suspension than the live ones around it, so past 32 concurrently armed suspensions a tombstone holds its slot while the newest live deadline is the one refused — and a tombstone that the cap does splice out re-derives with a fresh budget. A record left holding only tombstones arms no alarm at all and is cleared by the next lifecycle boundary or wake that reads a run which is no longer suspended there; for a run whose state is gone entirely, nothing reads it again and the record persists (about 150 bytes per entry, bounded by the cap). Both are accepted. The stored record, its parser, and the wake arithmetic are not exported: they are the run object's own state, and only the object that owns the alarm can act on them. +- Scope is `DurableObjectRunner`-hosted workflow runs. The durable-agent runner has its own resume path and does not arm suspension deadlines. ## Run summary diff --git a/docs/flowsafe-architecture.md b/docs/flowsafe-architecture.md index a26860c..32dd454 100644 --- a/docs/flowsafe-architecture.md +++ b/docs/flowsafe-architecture.md @@ -79,6 +79,8 @@ The runtime does not mint `breakwater.isolationScope`. One deployment serves one The snapshot remains Mastra's workflow state. Flowsafe does not invent a second workflow-state document inside Durable Object storage. +The one exception is wake scheduling. A suspended step can arm a per-suspension deadline through a reserved key in the payload it passes Mastra's `suspend()`; the run's object derives that deadline from the authoritative summary, keeps it in its own storage, and multiplexes it with run-owner recovery onto its single alarm. When a deadline expires the object resumes the run itself under a reserved system principal, with an envelope the step distinguishes from a real signal, after re-checking that the run is still suspended at the same step and suspension. That record and the arithmetic over it stay inside the run's object: only the holder of the alarm can act on them, so they are not part of the package surface. See [Durable Object runner design](do-runner-design.md) for the arming contract, the `suspendSchema` and `resumeSchema` caveats, the top-level-step-only limitation, and the failure modes. + ## Approval architecture `D1ApprovalStoreFactory.store()` returns one memoized store for the deployment database. Request routes, runtime grant derivation, and alarm maintenance duties share that store. diff --git a/docs/maintainer-guide.md b/docs/maintainer-guide.md index f39e155..d751daa 100644 --- a/docs/maintainer-guide.md +++ b/docs/maintainer-guide.md @@ -91,6 +91,10 @@ CI tests the declared supported peer version as part of the normal gate. A separ Treat a red canary as a release investigation even though it does not block a merge. Update the declared peer range only after tests, workerd proofs, package tarball probes, and migration notes pass. +Per-suspension deadlines couple to one undocumented Mastra behavior: a step arms a deadline through a reserved key in the payload it hands `suspend()`, which only reaches flowsafe because Mastra substitutes the schema-parsed suspend payload into the run summary (verified in the declared minimum peer, 1.50.0). A change there — a different substitution, a different key for a nested suspension, or resume-data validation moving — silently disarms every deadline. Tripwire tests in `packages/flowsafe/src/do-runner/runtime.test.ts` pin the observed behavior: the reserved key surviving a schema that declares it, being stripped by a strict schema that does not, surviving a loose schema, and a nested suspension being refused rather than armed. Check them on every Mastra upgrade and treat a failure as a behavior change to document, never as a test to relax. + +Rolling this release back is not symmetric: 0.17.x has no deadline reader, so the first alarm a downgraded run object takes deletes the alarm and orphans every armed record. Re-upgrading heals only runs that later receive another lifecycle boundary — which excludes exactly the runs a suspension deadline exists for, since a suspended run waiting on a signal has no boundary but its own wake. Prefer rolling forward; if a downgrade is unavoidable, treat every deadline armed before it as lost. + ## Public documentation `pnpm docs:check` validates local links and anchors, package export coverage, TypeDoc entry coverage, npm-safe package links, orphaned public pages, stale internal markers, and manifest-backed Node engine and peer-dependency claims. `pnpm docs:api` builds all supported API surfaces, including the React UI in its own TypeScript program. diff --git a/docs/security-threat-model.md b/docs/security-threat-model.md index e1cce54..0329e01 100644 --- a/docs/security-threat-model.md +++ b/docs/security-threat-model.md @@ -71,7 +71,7 @@ Trusted merges apply sanitized external or stored context first, then workflow, The Worker chooses the object name through `idFromName()`. Each object reasserts the addressed identity: -- runner object: `workflowId:runId`; +- runner object: `workflowId:runId` — and the runner object also derives `workflowId:runId` from that same `id.name` on any alarm wake that does not resume, in preference to its own stored record, which is consulted only when the object carries no name; the name is therefore a trusted input to its wake, not only an assertion target; - thread object: server-minted `threadId` and internal principal header; - hub object: fixed deployment singleton name; - provider host object: fixed deployment singleton name. diff --git a/packages/flowsafe/README.md b/packages/flowsafe/README.md index fb90242..f3141f8 100644 --- a/packages/flowsafe/README.md +++ b/packages/flowsafe/README.md @@ -168,6 +168,46 @@ The terminal snapshot commits before lifecycle cleanup. Cleanup abandons open ap Monitor `nextDeadlineAt`, `lastDeadlineAt`, `lastDeadlineAttemptAt`, and `lastDeadlineError` on the maintenance health projection. The sweep is bounded by `maintenance.deadlineLimit`, which defaults to 100. +### Time out one suspension + +A suspended step can carry its own deadline. It arms one by adding the reserved `SUSPENSION_DEADLINE_PAYLOAD_KEY` to the payload it passes to Mastra's `suspend()`. When the deadline elapses before the awaited signal arrives, the run's own Durable Object resumes that step with a payload the step can tell apart from a real signal. `createStep` below is the import-swapped factory `init()` returns: + +```typescript +import { + isSuspensionTimeoutResumeData, + SUSPENSION_DEADLINE_PAYLOAD_KEY, +} from '@proofoftech/flowsafe/do-runner'; +import { z } from 'zod'; + +const gate = createStep({ + id: 'gate', + inputSchema: z.object({ topic: z.string() }), + outputSchema: z.object({ topic: z.string(), settledBy: z.string() }), + execute: async ({ inputData, resumeData, suspend }) => { + if (!resumeData) { + return suspend({ + reason: 'awaiting approval', + [SUSPENSION_DEADLINE_PAYLOAD_KEY]: 900_000, + }); + } + return { + topic: inputData.topic, + settledBy: isSuspensionTimeoutResumeData(resumeData) + ? 'timeout' + : 'signal', + }; + }, +}); +``` + +The value is relative milliseconds between `MIN_SUSPENSION_DEADLINE_MS` and `MAX_SUSPENSION_DEADLINE_MS`. A step that declares a Zod `suspendSchema` must declare the reserved field or use a loose object, because Mastra substitutes the parsed suspend payload and a strict schema strips unknown keys. A step that declares a `resumeSchema` must accept the timeout envelope as well as its own signal shape, or the timeout resume fails validation and the deadline is abandoned after the runner's retries. The timeout resume delivers `SUSPENSION_TIMEOUT_RESUME_KEY` wrapping the expired step, its deadline, and the expiry time; branch on `isSuspensionTimeoutResumeData()` instead of the literal key, and build the same envelope in your own tests from that key and the `SuspensionTimeoutEnvelope` and `SuspensionTimeoutResumeData` types. Only the runner mints it: a resume request that carries the reserved key is rejected. It records `requestedByKind: 'system'` with the reserved `SUSPENSION_DEADLINE_PRINCIPAL_ID`, and it is not an approval decision: it mints no grant and records no reviewer. + +Only a top-level suspended step can arm a deadline. A step suspended inside a nested workflow is reported under the nested path while its suspension time is recorded against the enclosing step, so there is nothing to fence the resume against; the deadline is refused and logged instead of armed. + +Wake precision is the Durable Object alarm's, and there is no maintenance-sweep backstop for it, so treat a suspension deadline as best-effort near its due time. Run-level `deadlineMs` remains the swept mechanism. `MAX_SUSPENSION_DEADLINES_PER_RUN` caps how many deadlines one run arms; the stored record, its parser, and the wake arithmetic stay inside the run's Durable Object, because nothing outside it can act on that state safely. Suspension deadlines apply to `DurableObjectRunner`-hosted workflow runs; durable agents keep their own resume path. + +Read the [Durable Object runner design](https://github.com/ProofOfTechOrg/anchorage/blob/main/docs/do-runner-design.md) for the suspension fence, the stored record, and the failure modes. + ### REST surface The default approval base is `/api/approvals`. diff --git a/packages/flowsafe/scripts/spike-verify.mjs b/packages/flowsafe/scripts/spike-verify.mjs index b137933..1cdda10 100644 --- a/packages/flowsafe/scripts/spike-verify.mjs +++ b/packages/flowsafe/scripts/spike-verify.mjs @@ -8,7 +8,10 @@ // roles, wrong agents/bindings, absent raw resume, stale decision replay, // and restart-evicted stream replay with authoritative status fallback. // The remaining scenarios retain the lower-level workflow, stream, background -// task, signal, goal, schedule, and webhook compatibility proofs. +// task, signal, goal, schedule, and webhook compatibility proofs, plus the +// per-suspension deadline scenario: a run arms its own Durable Object alarm +// inside its Mastra suspend payload, the process is killed, and the restarted +// object resumes the run ITSELF with the reserved timeout envelope. // // Auth rides the worker's host-kit seam: every request presents one of the // LOCAL-ONLY spike bearer tokens (spike/worker.ts SPIKE_ACTORS). The C probe @@ -54,6 +57,21 @@ const AGENT_RUN_BODY = { workflowId: 'demo-agent-gate', inputData: { topic: 'launch' }, }; +// Suspension deadlines (T1-T3): the run's own DO alarm resumes a suspension +// whose awaited signal never arrives. `deadlineMs` rides the run input so ONE +// spike workflow drives both sides of the fence. The TIMEOUT deadline has to +// outlive every step between arming and the kill — a wake that fired while the +// process was still alive would prove nothing about surviving process death — +// and still elapse inside the kill+restart window, so it adds no wall clock of +// its own; hence 10s rather than the module's 1s floor. The SIGNAL deadline +// only has to outlive its own immediate resume, which settles the entry. +const DEADLINE_WORKFLOW_ID = 'demo-deadline'; +const DEADLINE_STEP = 'wait-signal'; +// The reserved arming key, as it appears on the wire (do-runner exports it to +// TypeScript authors as SUSPENSION_DEADLINE_PAYLOAD_KEY). +const DEADLINE_PAYLOAD_KEY = 'flowsafe.deadlineMs'; +const TIMEOUT_DEADLINE_MS = 10_000; +const SIGNAL_DEADLINE_MS = 30_000; const GUARDED_AGENT_ID = 'spike-guarded-agent'; const GUARDED_AGENT_START_PATH = `/agents/${GUARDED_AGENT_ID}/runs`; const AUTH = { @@ -273,6 +291,44 @@ function guardedStatusPath(run) { return `/agents/${GUARDED_AGENT_ID}/runs/${encodeURIComponent(run.threadId)}/${encodeURIComponent(run.runId)}`; } +// --- Suspension deadline (T1-T3) helpers ----------------------------------- + +// Read the wake state the run's OWN Durable Object holds: the persisted fenced +// entry plus the single alarm its two duties share (spike/worker.ts +// handleSuspensionDeadlineProbe). Nothing on the run surface exposes either. +async function armedDeadline(runId) { + const { status, body } = await http( + 'GET', + `/deadline/armed?workflowId=${DEADLINE_WORKFLOW_ID}&runId=${encodeURIComponent(runId)}`, + ); + assert(status === 200, `armed-deadline probe -> ${status}`, body); + assert( + body.status === 200, + `run DO armed-deadline route -> ${body.status}`, + body, + ); + return body.armed; +} + +// A wake settles its consumed entry AFTER the resume it just ran, so a status +// that already reads 'success' can precede that write by a few milliseconds. +// Poll for the settled state instead of racing the tail of the wake, and return +// the last observation either way so a real failure still reports what it saw. +async function settledDeadline(runId, timeoutMs) { + const deadline = Date.now() + timeoutMs; + let armed; + while (Date.now() < deadline) { + armed = await armedDeadline(runId); + if (armed.record === null && armed.alarmAt === null) return armed; + await sleep(250); + } + return armed; +} + +function deadlineRunPath(runId) { + return `/runs/${DEADLINE_WORKFLOW_ID}/${encodeURIComponent(runId)}`; +} + // --- Track E (M-007) webhook probe helpers --------------------------------- // GitHub's X-Hub-Signature-256 = 'sha256=' + hex HMAC-SHA256(secret, rawBody). @@ -1973,6 +2029,237 @@ async function main() { }, ); + // --- Suspension deadlines (T1-T3): the run resumes ITSELF ----------------- + // The one mechanism the unit tests can only assert against a hand-written + // DurableObjectState stub: a step suspends with a deadline inside its own + // Mastra suspend payload, the run's OWN Durable Object persists a fenced + // entry and arms its single alarm for it, the process DIES, and the restarted + // object wakes itself and resumes the run with the reserved timeout envelope + // under the system principal — with no client ever calling resume. + const signalRun = await step( + 'T1 signal fence: a real signal before the deadline resumes the run as its ' + + 'human requester and settles the armed entry', + async () => { + const started = await http('POST', '/runs', { + headers: AUTH.operator, + body: { + workflowId: DEADLINE_WORKFLOW_ID, + inputData: { topic: 'launch', deadlineMs: SIGNAL_DEADLINE_MS }, + }, + }); + assert( + started.status === 200 && started.body.status === 'suspended', + 'signal-fence run suspended at its timed step', + { status: started.status, body: started.body }, + ); + assert( + JSON.stringify(started.body.suspended?.[0]) === + JSON.stringify([DEADLINE_STEP]), + `suspended[0] is ['${DEADLINE_STEP}']`, + started.body.suspended, + ); + const resumed = await http( + 'POST', + `${deadlineRunPath(started.body.runId)}/resume`, + { + headers: AUTH.operator, + body: { + step: started.body.suspended[0], + resumeData: { signal: 'launch' }, + }, + }, + ); + assert( + resumed.status === 200 && resumed.body.status === 'success', + 'the signalled run resumed to success', + resumed.body, + ); + // The step branched on the exported guard, so this is the negative half + // of the SAME contract T3 proves: a real signal must not look like a + // timeout to the step that receives it. + assert( + resumed.body.result?.resumedBy === 'signal' && + resumed.body.result?.timeoutStep === undefined, + 'the step saw a real signal, not a timeout envelope', + resumed.body.result, + ); + assert( + resumed.body.requestedBy === 'opal' && + resumed.body.requestedByKind === 'human', + 'provenance names the human whose signal advanced the run', + resumed.body, + ); + // The fence path: the resume settles the armed entry, so there is nothing + // left for a later wake to resume a run that has already moved on. + const armed = await armedDeadline(started.body.runId); + assert( + armed.record === null && armed.alarmAt === null, + 'the signalled run keeps no armed deadline and no alarm', + armed, + ); + return { runId: started.body.runId }; + }, + ); + + const deadlineRun = await step( + 'T2 deadline arm: a suspension carrying the reserved deadline key arms the ' + + "run's own DO alarm with a fenced entry", + async () => { + const started = await http('POST', '/runs', { + headers: AUTH.operator, + body: { + workflowId: DEADLINE_WORKFLOW_ID, + inputData: { topic: 'launch', deadlineMs: TIMEOUT_DEADLINE_MS }, + }, + }); + assert( + started.status === 200 && started.body.status === 'suspended', + 'timed run suspended at its timed step', + { status: started.status, body: started.body }, + ); + // The arming value travels inside MASTRA's suspend payload (the step + // declares no suspendSchema, so nothing strips the reserved key), and the + // DO derives its record from this authoritative summary. + assert( + started.body.suspendPayload?.[DEADLINE_STEP]?.[DEADLINE_PAYLOAD_KEY] === + TIMEOUT_DEADLINE_MS, + 'the reserved deadline key survived into the authoritative summary', + started.body.suspendPayload, + ); + const suspendedAt = started.body.suspendedAt?.[DEADLINE_STEP]; + assert( + Number.isSafeInteger(suspendedAt), + 'the summary carries the suspendedAt fence the entry is armed against', + started.body.suspendedAt, + ); + + const armed = await armedDeadline(started.body.runId); + const entry = armed.record?.entries?.[0]; + assert( + armed.record?.workflowId === DEADLINE_WORKFLOW_ID && + armed.record?.runId === started.body.runId && + armed.record?.entries?.length === 1 && + entry?.step === DEADLINE_STEP && + entry?.deadlineAt === suspendedAt + TIMEOUT_DEADLINE_MS && + entry?.suspendedAt === suspendedAt && + entry?.resumeCount === 0 && + entry?.attempts === undefined, + 'the run DO persisted ONE entry, fenced to this suspension and due at ' + + 'suspendedAt + deadlineMs (never now + deadlineMs)', + armed, + ); + // ONE alarm serves both DO duties, so this also proves the suspension due + // time WON the min() — it is not the 60s run-owner recovery wake. + assert( + typeof armed.alarmAt === 'number' && + armed.alarmAt <= entry.deadlineAt + 5_000, + 'the object armed its single alarm for the suspension deadline', + armed, + ); + return { runId: started.body.runId, deadlineAt: entry.deadlineAt }; + }, + ); + + await step( + 'T3 deadline wake: the armed alarm survives a workerd kill+restart and the ' + + 'run resumes ITSELF with the timeout envelope under the system principal', + async () => { + // Still suspended on THIS process, so the wake has not fired yet: + // whatever resumes this run has to come from the restarted object. + const before = await http('GET', deadlineRunPath(deadlineRun.runId), { + headers: AUTH.viewer, + }); + assert( + before.status === 200 && before.body.status === 'suspended', + 'the timed run is still suspended when its process is killed', + before.body, + ); + + // Captured BEFORE the kill and asserted against the envelope below: the + // pre-kill read above proves the run had not resumed YET, but the wake + // could still fire in the gap between that read and the kill. Only an + // expiry stamped at or after the kill proves the RESTARTED process's + // timer did the work rather than the original one's. + const killedAt = Date.now(); + await killServer(currentServer); + // The deadline elapses with NOTHING running: no isolate, no timer, no + // client. Only the persisted entry and the object's own alarm survive. + await launchServer( + 'deadline-alarm', + stateDir, + join(tmpDir, 'deadline-alarm.log'), + ); + assert( + !/address already in use/i.test(currentServer.chunks.join('')), + 'deadline-alarm log must not contain "address already in use" (orphan trap)', + ); + + const pollDeadlineRun = async (timeoutMs) => { + const deadline = Date.now() + timeoutMs; + let seen; + while (Date.now() < deadline) { + seen = await http('GET', deadlineRunPath(deadlineRun.runId), { + headers: AUTH.viewer, + }); + if (seen.body.status === 'success') return seen; + // 'running' is the resume the wake is executing right now; only a + // terminal status other than success means the wake went wrong. + if (['failed', 'cancelled', 'timed_out'].includes(seen.body.status)) { + throw new Error( + `timed run terminated without succeeding: ${JSON.stringify(seen.body)}`, + ); + } + await sleep(250); + } + throw new Error( + `the suspension deadline never resumed the run: ${JSON.stringify(seen?.body)}`, + ); + }; + const resumed = await pollDeadlineRun(60_000); + assert( + resumed.body.result?.resumedBy === 'timeout' && + resumed.body.result?.timeoutStep === DEADLINE_STEP && + resumed.body.result?.deadlineAt === deadlineRun.deadlineAt && + resumed.body.result?.expiredAt >= deadlineRun.deadlineAt && + resumed.body.result?.expiredAt >= killedAt && + deadlineRun.deadlineAt > killedAt && + resumed.body.requestedBy === 'flowsafe-suspension-deadline' && + resumed.body.requestedByKind === 'system', + 'the alarm armed before the kill fired on the RESTARTED process and ' + + 'resumed the run itself: the step saw the reserved timeout envelope ' + + '(isSuspensionTimeoutResumeData) for its own step and deadline, under ' + + 'the system principal, with no client calling resume, the deadline ' + + 'was still in the future when the process died, and the wake acted ' + + 'after the kill rather than in the gap before it', + { killedAt, body: resumed.body }, + ); + // The consumed entry is settled, so the wake cannot repeat: one resume + // per expired deadline, and no alarm left behind for a finished run. + const armed = await settledDeadline(deadlineRun.runId, 10_000); + assert( + armed.record === null && armed.alarmAt === null, + 'the consumed entry was settled and the shared alarm cleared', + armed, + ); + // The T1 run is untouched across the same restart: its entry was settled + // by the real signal, so no timeout resume can reach it — the run still + // reads as the human signal advanced it, and its step ran once. + const signalled = await http('GET', deadlineRunPath(signalRun.runId), { + headers: AUTH.viewer, + }); + assert( + signalled.status === 200 && + signalled.body.status === 'success' && + signalled.body.result?.resumedBy === 'signal' && + signalled.body.requestedBy === 'opal' && + signalled.body.requestedByKind === 'human', + 'the signalled run kept its human resume across the restart (the fence ' + + 'held: no timeout resume reached it)', + signalled.body, + ); + }, + ); + await step( 'S deployment sentinel mismatch: a freshly started Worker refuses a D1 ' + 'provisioned for another deployment', @@ -2038,8 +2325,13 @@ try { '(E-S2), a signed webhook matched its subscription row and landed a ' + 'notification in the thread inbox (E-S1), a poll provider rehydrated its ' + 'subscriptions from D1 after a kill+restart and fired delivery (E-S3). ' + - 'Finally, a fresh Worker refused a D1 sentinel stamped for another ' + - 'deployment with 503 before authentication or routing.', + 'Per-suspension deadlines (T1-T3): a real signal resumed its run as the ' + + 'human requester and settled the armed entry, a suspension carrying the ' + + "reserved deadline key armed the run DO's own fenced wake, and after a " + + 'kill+restart that wake fired on the restarted object and resumed the run ' + + 'ITSELF with the timeout envelope under the system principal — no client ' + + 'called resume. Finally, a fresh Worker refused a D1 sentinel stamped for ' + + 'another deployment with 503 before authentication or routing.', ); } catch (error) { exitCode = 1; diff --git a/packages/flowsafe/spike/worker.ts b/packages/flowsafe/spike/worker.ts index b4e6204..e2e3ad4 100644 --- a/packages/flowsafe/spike/worker.ts +++ b/packages/flowsafe/spike/worker.ts @@ -126,9 +126,12 @@ import { type InitResult, init, isPathSafeId, + isSuspensionTimeoutResumeData, mintThreadId, type RunnerRuntime, resourceIdFromKey, + SUSPENSION_DEADLINE_PAYLOAD_KEY, + SUSPENSION_TIMEOUT_RESUME_KEY, stampDeploymentIdentityRequest, ThreadDurableObject, type ThreadScope, @@ -568,6 +571,13 @@ const WORKFLOWS: ReadonlyArray = [ 'agent tool-call approval gate (R-003 suspend shape) -> grant-gated publish (Track A)', sampleInput: { topic: 'launch' }, }, + { + id: 'demo-deadline', + title: 'Demo suspension deadline', + description: + 'one step that suspends with a per-suspension deadline, so the run resumes itself when the awaited signal never arrives', + sampleInput: { topic: 'launch', deadlineMs: 10_000 }, + }, ]; const scheduleTargetPolicy = createScheduleTargetPolicy({ workflows: [...WORKFLOWS, { id: 'sched-echo' }], @@ -814,10 +824,120 @@ function defineWorkflows(env: Env): RunnerRuntime { .then(schedEcho) .commit(); + // Per-suspension deadline probe (T1-T3): a step arms its own durable wake by + // putting SUSPENSION_DEADLINE_PAYLOAD_KEY in the payload it hands Mastra's + // suspend(); the run's OWN Durable Object derives a fenced entry from the + // authoritative summary, persists it in DO storage, and arms its single alarm + // for it. When the wake fires it resumes THIS step with the reserved timeout + // envelope under the system principal. The step therefore records WHICH kind + // of resume reached it, through the exported guard rather than a string + // literal, so spike-verify can assert the ENVELOPE arrived — not merely that + // the run advanced. + // + // It declares NEITHER a suspendSchema (which would have to declare the + // reserved key, or Mastra's parse substitution strips it) NOR a resumeSchema + // (which would have to accept the timeout envelope, or the resume is + // refused) — the simplest correct arming shape. `deadlineMs` rides the run + // input so ONE workflow drives both sides of the fence: a short deadline that + // must fire itself, and a longer one whose real signal must settle the entry + // first. + const deadlineInputSchema = z.object({ + topic: z.string(), + deadlineMs: z.number().int(), + }); + const deadlineOutputSchema = z.object({ + topic: z.string(), + resumedBy: z.enum(['timeout', 'signal']), + timeoutStep: z.string().optional(), + deadlineAt: z.number().optional(), + expiredAt: z.number().optional(), + }); + const deadlineWait = createStep({ + id: 'wait-signal', + inputSchema: deadlineInputSchema, + outputSchema: deadlineOutputSchema, + execute: async ({ inputData, resumeData, suspend }) => { + if (!resumeData) { + return suspend({ + [SUSPENSION_DEADLINE_PAYLOAD_KEY]: inputData.deadlineMs, + awaiting: 'external launch signal', + }); + } + if (isSuspensionTimeoutResumeData(resumeData)) { + const timeout = resumeData[SUSPENSION_TIMEOUT_RESUME_KEY]; + return { + topic: inputData.topic, + resumedBy: 'timeout' as const, + timeoutStep: timeout.step, + deadlineAt: timeout.deadlineAt, + expiredAt: timeout.expiredAt, + }; + } + return { topic: inputData.topic, resumedBy: 'signal' as const }; + }, + }); + createWorkflow({ + id: 'demo-deadline', + inputSchema: deadlineInputSchema, + outputSchema: deadlineOutputSchema, + }) + .then(deadlineWait) + .commit(); + return runtime; } +/** + * DO storage key holding one run's armed suspension deadlines. Deliberately NOT + * on the package's public surface — it is the alarm's own bookkeeping — so the + * spike mirrors the constant to introspect it. + * + * The trade, stated plainly: a duplicated literal can drift from the module's + * own, and exporting the key to stop that would put a Durable Object's private + * wake state on the public surface for every consumer, where reading it proves + * nothing and writing it corrupts the alarm. Drift is also the cheaper failure: + * this key is read in ONE place, the route below, and spike-verify's T2 + * assertion then observes no armed record where it requires one and fails + * loudly. Silent success is not reachable. + */ +const SPIKE_SUSPENSION_DEADLINE_STORAGE_KEY = 'flowsafe:suspension-deadline:v1'; + +/** The run-DO route the T1-T3 probes read that armed state through. */ +const SPIKE_SUSPENSION_DEADLINE_PATH = '/spike/suspension-deadline'; + export class DemoRunner extends DurableObjectRunner { + /** + * Spike-only introspection of the state the suspension-deadline duty owns: + * the armed record in THIS object's storage and the single alarm both DO + * duties share. Nothing on the run surface exposes either (correctly — a + * client has no business reading an object's wake schedule), so spike-verify + * would otherwise have to infer "armed" from the resume that follows it. The + * runner's own routes all live under /runs, so a /spike/ path cannot collide + * with one, and deployment identity is verified exactly as the inherited + * fetch does. + */ + async fetch(request: Request): Promise { + if (new URL(request.url).pathname !== SPIKE_SUSPENSION_DEADLINE_PATH) { + return super.fetch(request); + } + try { + await verifyDurableObjectDeploymentRequest(request, this.state, this.env); + const storage = this.state?.storage; + // getAlarm is not part of the structural storage subset do-runner + // declares (it never reads the alarm back); workerd always provides it. + const alarms = storage as unknown as + | { getAlarm(): Promise } + | undefined; + return json({ + alarmAt: (await alarms?.getAlarm()) ?? null, + record: + (await storage?.get(SPIKE_SUSPENSION_DEADLINE_STORAGE_KEY)) ?? null, + }); + } catch (error) { + return doErrorResponse(error); + } + } + protected build(env: Env): RunnerRuntime { return defineWorkflows(env); } @@ -2385,6 +2505,36 @@ async function handleScheduleProbe( return null; } +// --- Suspension deadline probe (T1-T3) ------------------------------------- +// LOCAL-ONLY worker-level probe reading the wake state a run's OWN DO holds for +// its suspension deadlines: the persisted fenced entry and the single alarm the +// object's two duties share. spike-verify asserts a suspension ARMS one before +// the process is killed, and that a run resumed by a real signal keeps none — +// neither is observable on the run surface, and both are exactly what the +// hand-written DurableObjectState stub in the unit tests cannot prove. +async function handleSuspensionDeadlineProbe( + request: Request, + env: Env, +): Promise { + const url = new URL(request.url); + if (request.method !== 'GET' || url.pathname !== '/deadline/armed') { + return null; + } + const workflowId = url.searchParams.get('workflowId'); + const runId = url.searchParams.get('runId'); + if (!isPathSafeId(workflowId) || !isPathSafeId(runId)) { + return json({ error: 'path-safe workflowId and runId required' }, 404); + } + // The SAME address the run topology uses (idFromName(`${workflowId}:${runId}`)), + // so this reads the storage of the very object that serves the run. + const response = await fetchDeploymentObject( + runStub(env, workflowId, runId), + env, + `http://do${SPIKE_SUSPENSION_DEADLINE_PATH}`, + ); + return json({ status: response.status, armed: await response.json() }); +} + const handler: ExportedHandler = { async fetch( request: CfRequest, @@ -2454,6 +2604,11 @@ const handler: ExportedHandler = { const sigpProbe = await handleSignalProviderProbe(routed, env); if (sigpProbe) return sigpProbe; + // Suspension deadline probe (T1-T3 armed-wake introspection) — local, + // unauthenticated, ahead of the routers. + const deadlineProbe = await handleSuspensionDeadlineProbe(routed, env); + if (deadlineProbe) return deadlineProbe; + // Track E webhook ingress: the github webhook route TERMINATES on the Worker, // signature-authed (not bearer), route-absent when its secret is unset. const webhookResponse = await webhookRouter(env)(routed); diff --git a/packages/flowsafe/src/agent-host/thread-host.test.ts b/packages/flowsafe/src/agent-host/thread-host.test.ts index eb085f5..e694ac8 100644 --- a/packages/flowsafe/src/agent-host/thread-host.test.ts +++ b/packages/flowsafe/src/agent-host/thread-host.test.ts @@ -20,7 +20,12 @@ import type { ScheduleSourceStore, ThreadScope, } from '../do-runner/index.js'; -import { resourceIdFromKey } from '../do-runner/index.js'; +import { + doErrorResponse, + RunStateUnreadableError, + resourceIdFromKey, + SUSPENSION_TIMEOUT_RESUME_KEY, +} from '../do-runner/index.js'; import { D1SchedulesStorage, type ScheduleDatabase, @@ -682,6 +687,49 @@ describe('createThreadAgentHost owner recovery', () => { expect(alarmAt()).toBeUndefined(); }); + it('retains every recovery record and rearms when the authoritative read does not succeed', async () => { + const { host, scope, state, resources, resourceAccess, alarmAt } = harness( + ['writer'], + { + // The runner-side guard reaches here: recoverStartAttempt refuses a + // read that did not reach storage rather than reporting the fabricated + // 'pending' shell it would otherwise see. + runtime: { + recoverStartAttempt: vi.fn(async () => { + throw new RunStateUnreadableError( + 'durable-agentic-loop', + 'acme_run', + ); + }), + }, + }, + ); + const settle = vi.spyOn(resourceAccess, 'settleReservation'); + seedRecoveryState(state, 'acme_run', ownerRecovery('acme_run')); + await resources.reserveAll( + [ + { kind: 'thread', resourceId: 'acme_thread' }, + { kind: 'resource', resourceId: RESOURCE_ID }, + { kind: 'run', resourceId: 'acme_run' }, + ], + { kind: 'human', id: 'operator-1' }, + 'token-acme_run', + ); + + await expect( + host.recoverOwnership(scope.init.runtime, scope.threadId), + ).rejects.toBeInstanceOf(RunStateUnreadableError); + + // #then — fail closed: an unreadable read is not evidence the attempt was + // abandoned, so nothing is deleted, nothing is settled, and the journal + // stays armed for a wake that can read. + expect(state.has(TEST_OWNER_RECOVERY_KEY)).toBe(true); + expect(state.has(TEST_RUN_RECORD_KEY)).toBe(true); + expect(state.has(THREAD_BINDING_KEY)).toBe(true); + expect(settle).not.toHaveBeenCalled(); + expect(alarmAt()).toBeDefined(); + }); + it('keeps an unthreaded nonterminal journal armed, then releases ephemeral claims at terminal state', async () => { const { host, scope, state, resources, alarmAt, setSummary } = harness(); const owner = { kind: 'human' as const, id: 'operator-1' }; @@ -1574,6 +1622,52 @@ describe('createThreadAgentHost', () => { expect(state.has('flowsafe:agent-run:v1:acme_run')).toBe(false); }); + it('keeps run metadata and names the failure when a failed start cannot read authoritative state', async () => { + // #given — the same failed start, with the read that would tell an + // interrupted start apart from a failed one refusing to answer from state + // it could not reach. + const { host, scope, state, alarmAt } = harness(['writer'], { + runtime: { + recoverStartAttempt: vi.fn(async () => { + throw new RunStateUnreadableError('durable-agentic-loop', 'acme_run'); + }), + }, + }); + mocked.stream.mockRejectedValue(new Error('model unavailable')); + const logged: string[] = []; + const log = vi + .spyOn(console, 'error') + .mockImplementation((...args: unknown[]) => { + logged.push(String(args[0])); + }); + + // #when + try { + await expect( + host.start(scope, { + agentId: 'writer', + threadId: 'acme_thread', + resourceId: RESOURCE_ID, + runId: 'acme_run', + prompt: 'go', + entryPath: 'http.start', + }), + // #then — the caller still sees the ORIGINAL start failure: the read + // concluded nothing, so it cannot reclassify one. + ).rejects.toThrow('model unavailable'); + } finally { + log.mockRestore(); + } + + // #then — and the failure is named rather than swallowed, because it is + // the reason the metadata above survives and a wake is left to retry it. + expect(logged).toContain( + 'interrupted start could not read authoritative state', + ); + expect(state.has('flowsafe:agent-run:v1:acme_run')).toBe(true); + expect(alarmAt()).toBeDefined(); + }); + it('does not create a reusable binding for an unthreaded ephemeral run', async () => { const fixture = harness(); const { host, scope, state } = fixture; @@ -2222,6 +2316,57 @@ describe('createThreadAgentHost', () => { }); }); + it('fails dispatch status closed when the pending recovery cannot read authoritative state', async () => { + // #given — a dispatch-status read whose pending owner recovery runs first, + // with the read that would settle it refusing to answer from state it + // could not reach. + const { host, scope, state, resources, resourceAccess } = harness( + ['writer'], + { + runtime: { + recoverStartAttempt: vi.fn(async () => { + throw new RunStateUnreadableError( + 'durable-agentic-loop', + 'acme_run', + ); + }), + }, + }, + ); + const settle = vi.spyOn(resourceAccess, 'settleReservation'); + seedRecoveryState(state, 'acme_run', ownerRecovery('acme_run')); + await resources.reserveAll( + [ + { kind: 'thread', resourceId: 'acme_thread' }, + { kind: 'resource', resourceId: RESOURCE_ID }, + { kind: 'run', resourceId: 'acme_run' }, + ], + { kind: 'human', id: 'operator-1' }, + 'token-acme_run', + ); + + // #when + const raised = await host + .route( + new Request( + `https://thread/_flowsafe/agent-host/runs/writer/acme_run?resourceId=${RESOURCE_ID}&dispatch=1`, + ), + scope, + ) + .catch((error: unknown) => error); + + // #then — the route escapes to the Durable Object shell, which answers the + // retryable 503 this release documents rather than a 200 assembled from a + // read that never happened. The recovery stays owed: journal, run record + // and reservation all survive for a wake that can read. + expect(raised).toBeInstanceOf(RunStateUnreadableError); + expect(doErrorResponse(raised).status).toBe(503); + expect(settle).not.toHaveBeenCalled(); + expect(state.has(TEST_OWNER_RECOVERY_KEY)).toBe(true); + expect(state.has(TEST_RUN_RECORD_KEY)).toBe(true); + expect(await resources.owner('run', 'acme_run')).toBeUndefined(); + }); + it('rejects a snapshot whose thread correlation does not match the addressed DO', async () => { const { host, scope, setSnapshot } = harness(); await host.start(scope, { @@ -2384,6 +2529,43 @@ describe('createThreadAgentHost', () => { expect(mocked.resumeViaRuntime).not.toHaveBeenCalled(); }); + it('refuses an approval resume that forges the suspension-timeout envelope', async () => { + const fixture = harness(); + seedSuspendedApprovalRun(fixture); + + await expect( + fixture.host.route( + new Request('https://thread/_flowsafe/agent-host/resume', { + method: 'POST', + headers: { 'content-type': 'application/json' }, + body: JSON.stringify({ + agentId: 'writer', + threadId: 'acme_thread', + resourceId: RESOURCE_ID, + runId: 'acme_run', + entryPath: 'approval.resume', + requestedBy: 'reviewer-1', + resumeData: { + [SUSPENSION_TIMEOUT_RESUME_KEY]: { + step: 'tool', + deadlineAt: 1, + expiredAt: 2, + }, + }, + }), + }), + fixture.scope, + ), + // #then — this route forwards client resume data verbatim as a human + // requester, so without the guard a caller could drive a step's timeout + // branch. Only a run object's alarm mints that envelope. + ).rejects.toMatchObject({ + status: 400, + message: expect.stringContaining(SUSPENSION_TIMEOUT_RESUME_KEY), + }); + expect(mocked.resumeViaRuntime).not.toHaveBeenCalled(); + }); + it('resumes an unthreaded suspended run without requiring or inventing memory', async () => { const { host, scope, state, setSummary, setSnapshot } = harness(); setSummary({ diff --git a/packages/flowsafe/src/agent-host/thread-host.ts b/packages/flowsafe/src/agent-host/thread-host.ts index f3e63d1..816a505 100644 --- a/packages/flowsafe/src/agent-host/thread-host.ts +++ b/packages/flowsafe/src/agent-host/thread-host.ts @@ -50,6 +50,7 @@ import { resourceIdFromKey, type ScheduleSourceAgentTarget, type ScheduleSourceStore, + SUSPENSION_TIMEOUT_RESUME_KEY, type ThreadScope, } from '../do-runner/index.js'; import { mastraRegistryEntries } from '../do-runner/mastra-registry.js'; @@ -355,6 +356,29 @@ function requestedBy(value: unknown): string { return value; } +/** + * The reserved suspension-timeout envelope is minted by a run object's alarm and + * by nothing else. Agent runs never arm a suspension deadline, so a forged + * envelope here could only mislead a step — but this route forwards client + * resume data verbatim under `requestedByKind: 'human'`, and the guarantee the + * feature sells is that no caller can present itself to a step as an expired + * deadline. The KEY is refused, exactly as the workflow resume route refuses it, + * so a step that reads the key directly cannot be fooled either. + */ +function resumeData(value: unknown): unknown { + if ( + value !== null && + typeof value === 'object' && + SUSPENSION_TIMEOUT_RESUME_KEY in value + ) { + throw new AgentHostRequestError( + 400, + `resume data must not carry the reserved '${SUSPENSION_TIMEOUT_RESUME_KEY}' key`, + ); + } + return value; +} + function createFifoLock(): (operation: () => Promise) => Promise { let tail = Promise.resolve(); return async (operation: () => Promise): Promise => { @@ -1546,8 +1570,17 @@ export function createThreadAgentHost( ref.runId, recovery.token, ); - } catch { - // Unknown authoritative state: retain metadata and fail closed. + } catch (recoverError) { + // Unknown authoritative state: retain metadata and fail closed — + // and logged, never swallowed silently, because this read is the + // only thing that could tell an interrupted start apart from a + // failed one, and its own failure is why the journal is left + // armed for a wake that can read. The caller still sees the + // ORIGINAL start error. + console.error( + 'interrupted start could not read authoritative state', + recoverError, + ); } if ( summary === null || @@ -1747,6 +1780,10 @@ export function createThreadAgentHost( const durable = current.agents.get(module.meta.id); if (!durable) throw new Error('guarded agent was not registered'); const requesterId = requestedBy(body.requestedBy); + const resumeFields = + 'resumeData' in body + ? { resumeData: resumeData(body.resumeData) } + : {}; const step = typeof body.step === 'string' || (Array.isArray(body.step) && @@ -1772,7 +1809,7 @@ export function createThreadAgentHost( runId: ref.runId, requestedBy: requesterId, ...(step !== undefined ? { step } : {}), - ...('resumeData' in body ? { resumeData: body.resumeData } : {}), + ...resumeFields, ...(snapshotExecution.threaded ? { memory: { diff --git a/packages/flowsafe/src/do-runner/do-error-response.ts b/packages/flowsafe/src/do-runner/do-error-response.ts index 54c37ee..cb18876 100644 --- a/packages/flowsafe/src/do-runner/do-error-response.ts +++ b/packages/flowsafe/src/do-runner/do-error-response.ts @@ -18,6 +18,7 @@ import { RunAlreadyExistsError, RunLifecycleBlockedError, RunNotSuspendedError, + RunStateUnreadableError, RunTerminalConflictError, UnknownRunError, UnknownWorkflowError, @@ -52,6 +53,13 @@ function statusOf(error: unknown): number | undefined { // problem, not the caller's — 503 so monitors separate a wiring fault from // a code fault. Fail closed: nothing below this line runs for one. if (error instanceof DeploymentIdentityError) return 503; + // An authoritative read that did not reach storage: the same shape of answer + // as the misprovisioning above — the caller asked for nothing wrong, the + // condition is the operator's, and it clears on its own — so it is retryable + // rather than a 500 that reads as a code fault. A 404 or a 200 with a + // fabricated summary would be worse than either: both invite the caller to + // conclude something from a read that never happened. + if (error instanceof RunStateUnreadableError) return 503; if ( error instanceof UnknownWorkflowError || error instanceof UnknownRunError diff --git a/packages/flowsafe/src/do-runner/durable-object.test.ts b/packages/flowsafe/src/do-runner/durable-object.test.ts index e91b2ea..3ae4477 100644 --- a/packages/flowsafe/src/do-runner/durable-object.test.ts +++ b/packages/flowsafe/src/do-runner/durable-object.test.ts @@ -1,6 +1,10 @@ // SPDX-License-Identifier: Apache-2.0 import type { DurableObjectState } from '@cloudflare/workers-types'; import { InMemoryStore } from '@mastra/core/storage'; +import type { + DefaultEngineType, + ExecuteFunction, +} from '@mastra/core/workflows'; import { describe, expect, it, vi } from 'vitest'; import { z } from 'zod'; import { @@ -32,11 +36,28 @@ import { DurableObjectRunner, type DurableObjectRunOwner, type DurableObjectRunOwnershipStore, + nextDutyAlarmAt, } from './durable-object.js'; import { EXECUTION_PRINCIPAL_HEADER } from './execution-principal-header.js'; import { init } from './init.js'; -import type { RunnerRuntime, RunSummary } from './runtime.js'; +import { + type RunnerRuntime, + RunStateUnreadableError, + type RunSummary, +} from './runtime.js'; import type { ScheduleSourceStore } from './schedule-source.js'; +import { + isSuspensionTimeoutResumeData, + MAX_SUSPENSION_DEADLINE_ATTEMPTS, + MAX_SUSPENSION_DEADLINE_MS, + MIN_SUSPENSION_DEADLINE_MS, + SUSPENSION_DEADLINE_PAYLOAD_KEY, + SUSPENSION_DEADLINE_PRINCIPAL_ID, + SUSPENSION_DEADLINE_STORAGE_KEY, + SUSPENSION_TIMEOUT_RESUME_KEY, + type SuspensionDeadlineEntry, + type SuspensionDeadlineRecord, +} from './suspension-deadline.js'; interface TestEnv extends DeploymentIdentityEnv { storage: InMemoryStore; @@ -98,6 +119,20 @@ function makeProductionEnv( }; } +/** + * The two run-state reads a RunnerRuntime stub has to answer, over one + * implementation. Every stub in this file goes in through + * `as unknown as RunnerRuntime`, so TypeScript sees nothing when a method is + * missing: a stub carrying only `status` would send each alarm-driven test + * down the unreadable-state path — no resume, no charge, watchdog cadence — + * and pass anyway. Two spies rather than one so a test can pin WHICH read a + * path made: a wake reads `authoritativeStatus`, an HTTP route reads + * `status`. + */ +function statusStub(read: RunnerRuntime['status']) { + return { status: vi.fn(read), authoritativeStatus: vi.fn(read) }; +} + function gatedRuntime(storage: InMemoryStore): RunnerRuntime { const { createWorkflow, createStep, runtime } = init({ storage }); const gate = createStep({ @@ -206,8 +241,10 @@ function recoveryStorage(events: string[] = []): { state: DurableObjectState; storage: DurableKeyValueStorage; values: Map; + alarms: number[]; } { const values = new Map(); + const alarms: number[] = []; const storage: DurableKeyValueStorage = { async get(key: string): Promise { events.push(`get:${key}`); @@ -221,8 +258,11 @@ function recoveryStorage(events: string[] = []): { events.push(`delete:${key}`); return values.delete(key); }, - async setAlarm(): Promise { + async setAlarm(scheduledTime: number | Date): Promise { events.push('setAlarm'); + alarms.push( + scheduledTime instanceof Date ? scheduledTime.getTime() : scheduledTime, + ); }, async deleteAlarm(): Promise { events.push('deleteAlarm'); @@ -232,6 +272,7 @@ function recoveryStorage(events: string[] = []): { state: { storage } as unknown as DurableObjectState, storage, values, + alarms, }; } @@ -250,7 +291,7 @@ describe('DurableObjectRunner.fetch', () => { it('rejects a start without a trusted execution principal before runtime or ownership work', async () => { const reserve = vi.fn(async () => true); const runtime = { - status: vi.fn(async () => null), + ...statusStub(async () => null), start: vi.fn(), } as unknown as RunnerRuntime; const runner = new TestRunner(undefined, { @@ -302,7 +343,7 @@ describe('DurableObjectRunner.fetch', () => { it('rejects a new resume leg without requester kind before runtime work', async () => { const runtime = { resume: vi.fn(), - status: vi.fn(async () => null), + ...statusStub(async () => null), } as unknown as RunnerRuntime; const runner = new TestRunner(undefined, { ...makeProductionEnv(), @@ -525,7 +566,7 @@ describe('DurableObjectRunner.fetch', () => { ); const runtime = { start, - status: vi.fn(async () => null), + ...statusStub(async () => null), } as unknown as RunnerRuntime; const env = makeProductionEnv(); env.owners = owners; @@ -596,7 +637,7 @@ describe('DurableObjectRunner.fetch', () => { }); const runtime = { start: vi.fn(), - status: vi.fn(async () => null), + ...statusStub(async () => null), } as unknown as RunnerRuntime; const env = makeProductionEnv(); env.owners = owners; @@ -705,7 +746,11 @@ describe('DurableObjectRunner.fetch', () => { await expect(runner.alarm()).rejects.toThrow("belongs to 'globex'"); expect(events[0]).toBe('setAlarm'); - expect(events.filter((event) => event === 'setAlarm')).toHaveLength(2); + // The watchdog is armed before the identity read and is the ONLY arm on + // this path: re-arming after the failure would let a due suspension + // deadline pull the next wake to the one-second floor, on a wake whose + // deadline duty cannot run at all. + expect(events.filter((event) => event === 'setAlarm')).toHaveLength(1); expect(events).not.toContain('deleteAlarm'); }); @@ -823,7 +868,7 @@ describe('DurableObjectRunner.fetch', () => { start: vi.fn(async () => { throw new Error('injected pre-snapshot failure'); }), - status: vi.fn(async () => null), + ...statusStub(async () => null), recoverStartAttempt: vi.fn(async () => null), } as unknown as RunnerRuntime; const runner = new TestRunner(state, { @@ -848,6 +893,62 @@ describe('DurableObjectRunner.fetch', () => { ); }); + it('keeps the journal and names the failure when an interrupted start cannot read authoritative state', async () => { + // #given — the same failed start, with the read that would tell an + // interrupted start apart from a failed one refusing to answer from state + // it could not reach. + const { state, values, alarms } = recoveryStorage(); + const reserve = vi.fn(async () => true); + const settle = vi.fn(async () => undefined); + const runtime = { + start: vi.fn(async () => { + throw new Error('injected pre-snapshot failure'); + }), + ...statusStub(async () => null), + recoverStartAttempt: vi.fn(async () => { + throw new RunStateUnreadableError('gated', 'run-blind-start'); + }), + } as unknown as RunnerRuntime; + const runner = new TestRunner(state, { + ...makeProductionEnv(new InMemoryStore(), { reserve, settle }), + runtime, + }); + const logged: string[] = []; + const log = vi + .spyOn(console, 'error') + .mockImplementation((...args: unknown[]) => { + logged.push(String(args[0])); + }); + const before = Date.now(); + + // #when + const response = await runner + .fetch( + post('/runs', { + workflowId: 'gated', + runId: 'run-blind-start', + inputData: { topic: 't' }, + }), + ) + .finally(() => log.mockRestore()); + + // #then — the caller still sees the ORIGINAL start failure, never the + // read's: a read that concluded nothing cannot reclassify one. + expect(response.status).toBe(500); + await expect(response.json()).resolves.toMatchObject({ + error: 'injected pre-snapshot failure', + }); + // #then — and the failure is named rather than swallowed, because it is + // the reason the attempt is left unsettled with its journal armed for a + // wake that can read. + expect(logged).toContain( + 'interrupted start could not read authoritative state', + ); + expect(settle).not.toHaveBeenCalled(); + expect(values.has('flowsafe:run-owner-recovery:v1')).toBe(true); + expect(alarms.at(-1)).toBeGreaterThanOrEqual(before + 60_000); + }); + it('does not execute when the owner reservation conflicts', async () => { const { state, values } = recoveryStorage(); const reserve = vi.fn(async () => false); @@ -920,7 +1021,7 @@ describe('DurableObjectRunner.fetch', () => { persisted = summary; return summary; }), - status: vi.fn(async () => persisted), + ...statusStub(async () => persisted), } as unknown as RunnerRuntime; const runner = new TestRunner(undefined, { ...makeProductionEnv(), @@ -1880,3 +1981,3324 @@ describe('DurableObjectRunner.fetch', () => { expect(resumeFrame.summary.status).toBe('success'); }); }); + +const TIMED_DEADLINE_MS = 900_000; +const RETRY_DEADLINE_MS = 1_800_000; +const DOTTED_STEP = 'wait.signal'; + +// The one execute shape every single-step timed workflow shares, so the +// fixtures below pass the behaviour and nothing else. +type TimedStepExecute = ExecuteFunction< + unknown, + { topic: string }, + { topic: string; settledBy: string }, + unknown, + unknown, + DefaultEngineType +>; + +// A gate that arms a deadline on its suspension and reports which kind of +// resume woke it, so the DO tests below can prove the timeout envelope reached +// the step rather than only that a resume happened. `onSettle` counts the +// settling gates' post-suspension executions for countedTimedRuntime. +function timedRuntime( + storage: InMemoryStore, + onSettle?: () => void, +): RunnerRuntime { + const { createWorkflow, createStep, runtime } = init({ storage }); + const timedStep = (id: string, execute: TimedStepExecute) => + createStep({ + id, + inputSchema: z.object({ topic: z.string() }), + outputSchema: z.object({ topic: z.string(), settledBy: z.string() }), + execute, + }); + const singleStepWorkflow = ( + id: string, + step: ReturnType, + ): void => { + createWorkflow({ + id, + inputSchema: z.object({ topic: z.string() }), + outputSchema: z.object({ topic: z.string(), settledBy: z.string() }), + }) + .then(step) + .commit(); + }; + const settling: TimedStepExecute = async ({ + inputData, + resumeData, + suspend, + }) => { + if (!resumeData) { + return suspend({ + reason: 'awaiting approval', + [SUSPENSION_DEADLINE_PAYLOAD_KEY]: TIMED_DEADLINE_MS, + }); + } + onSettle?.(); + return { + topic: inputData.topic, + settledBy: isSuspensionTimeoutResumeData(resumeData) + ? 'timeout' + : 'signal', + }; + }; + const relaying = + (firstDeadlineMs: number, resumedDeadlineMs: number): TimedStepExecute => + async ({ resumeData, suspend }) => + suspend({ + reason: resumeData ? 'awaiting the next signal' : 'awaiting approval', + [SUSPENSION_DEADLINE_PAYLOAD_KEY]: resumeData + ? resumedDeadlineMs + : firstDeadlineMs, + }); + singleStepWorkflow('timed', timedStep('gate', settling)); + singleStepWorkflow( + 'timed-escalating', + timedStep('gate', async ({ inputData, resumeData, suspend }) => { + if (!resumeData) { + return suspend({ + reason: 'awaiting approval', + [SUSPENSION_DEADLINE_PAYLOAD_KEY]: TIMED_DEADLINE_MS, + }); + } + if (isSuspensionTimeoutResumeData(resumeData)) { + return suspend({ + reason: 'escalated after timeout', + [SUSPENSION_DEADLINE_PAYLOAD_KEY]: RETRY_DEADLINE_MS, + }); + } + return { topic: inputData.topic, settledBy: 'signal' }; + }), + ); + // A TOP-LEVEL step id that contains a dot. The rehydrated projection the + // alarm reads splits the stored key on every dot, so this is the id whose + // entry the wake has to recognize from [['wait','signal']]. + singleStepWorkflow('timed-dotted', timedStep(DOTTED_STEP, settling)); + // Arms a value below MIN_SUSPENSION_DEADLINE_MS on every suspension, so each + // lifecycle boundary re-derives the identical rejection. + singleStepWorkflow( + 'timed-unarmable', + timedStep('gate', async ({ suspend }) => + suspend({ + reason: 'awaiting approval', + [SUSPENSION_DEADLINE_PAYLOAD_KEY]: MIN_SUSPENSION_DEADLINE_MS - 1, + }), + ), + ); + // Suspends again on every resume, so a RESUME boundary — not only a start — + // can be the first one this run ever has a deadline to arm. + singleStepWorkflow( + 'timed-relay', + timedStep('gate', relaying(TIMED_DEADLINE_MS, RETRY_DEADLINE_MS)), + ); + // The reverse relay: the re-suspension's deadline is SHORTER than the first + // one's, so a stale record left by a failed resume-boundary write points at + // a far-future time while the current suspension is due much sooner. + singleStepWorkflow( + 'timed-relay-shortening', + timedStep('gate', relaying(RETRY_DEADLINE_MS, TIMED_DEADLINE_MS)), + ); + return runtime; +} + +// A top-level step id containing a dot, alongside a nested workflow whose inner +// step makes the SAME dot-joined key: 'a.b' as a step id, and 'a' > 'b' as a +// path. Mastra keys its snapshot namespace by that joined path, so the two +// suspensions are indistinguishable there, and an entry armed for one of them +// could resume the other. +function collidingRuntime(storage: InMemoryStore): { + runtime: RunnerRuntime; + settled: () => string[]; +} { + const settled: string[] = []; + const { createWorkflow, createStep, runtime } = init({ storage }); + const suspending = (id: string, label: string) => + createStep({ + id, + inputSchema: z.looseObject({}), + outputSchema: z.looseObject({}), + execute: async ({ resumeData, suspend }) => { + if (!resumeData) { + return suspend({ + reason: 'awaiting approval', + [SUSPENSION_DEADLINE_PAYLOAD_KEY]: TIMED_DEADLINE_MS, + }); + } + settled.push(label); + return { settledBy: label }; + }, + }); + const inner = createWorkflow({ + id: 'a', + inputSchema: z.looseObject({}), + outputSchema: z.looseObject({}), + }) + .then(suspending('b', 'nested')) + .commit(); + createWorkflow({ + id: 'timed-collision', + inputSchema: z.looseObject({}), + outputSchema: z.looseObject({}), + }) + .parallel([suspending('a.b', 'top-level'), inner]) + .commit(); + return { runtime, settled: () => settled }; +} + +// A `foreach` whose every iteration arms a deadline, recording the iterations a +// timeout resume actually reached. Mastra reports the whole foreach as ONE +// suspended path with one fence, so bounded work per wake has to be judged on +// the iterations rather than on the path. +function foreachRuntime( + storage: InMemoryStore, + workflowId: string, + options?: { concurrency: number }, +): { runtime: RunnerRuntime; timedOut: () => number[] } { + const timedOut: number[] = []; + const { createWorkflow, createStep, runtime } = init({ storage }); + const gate = createStep({ + id: 'gate', + inputSchema: z.object({ item: z.number() }), + outputSchema: z.object({ item: z.number(), settledBy: z.string() }), + execute: async ({ inputData, resumeData, suspend }) => { + if (!resumeData) { + return suspend({ + reason: 'awaiting approval', + // Concurrent iterations each ask for a DIFFERENT deadline, so the + // claim that only the first suspended iteration's value is read per + // batch is measured rather than assumed. A sequential foreach arms + // one iteration at a time and needs no such spread. + [SUSPENSION_DEADLINE_PAYLOAD_KEY]: + TIMED_DEADLINE_MS + (options ? inputData.item * 60_000 : 0), + }); + } + if (isSuspensionTimeoutResumeData(resumeData)) { + timedOut.push(inputData.item); + } + return { item: inputData.item, settledBy: 'timeout' }; + }, + }); + const workflow = createWorkflow({ + id: workflowId, + inputSchema: z.array(z.object({ item: z.number() })), + outputSchema: z.array( + z.object({ item: z.number(), settledBy: z.string() }), + ), + }); + (options ? workflow.foreach(gate, options) : workflow.foreach(gate)).commit(); + return { runtime, timedOut: () => timedOut }; +} + +function timedEnv(): TestEnv { + const env = makeProductionEnv(); + env.runtime = timedRuntime(env.storage); + return env; +} + +// The same `timed` gate, counting how often its post-suspension body ran, so a +// race between a wake and a real signal can be judged on the one thing that +// matters: the gated action must not execute twice. +function countedTimedRuntime(storage: InMemoryStore): { + runtime: RunnerRuntime; + settled: () => number; +} { + let settled = 0; + return { + runtime: timedRuntime(storage, () => { + settled += 1; + }), + settled: () => settled, + }; +} + +function storedDeadlines( + values: Map, +): SuspensionDeadlineRecord | undefined { + return values.get(SUSPENSION_DEADLINE_STORAGE_KEY) as + | SuspensionDeadlineRecord + | undefined; +} + +/** One stored entry by step, for a record asserted entry by entry. */ +function storedEntry( + values: Map, + step: string, +): SuspensionDeadlineEntry | undefined { + return storedDeadlines(values)?.entries.find((entry) => entry.step === step); +} + +/** Rewind every armed deadline into the past — the wake condition, no clock. */ +function elapseDeadlines( + values: Map, + entry: Partial = {}, +): void { + const stored = storedDeadlines(values); + if (!stored) throw new Error('no suspension deadline is armed'); + values.set(SUSPENSION_DEADLINE_STORAGE_KEY, { + ...stored, + entries: stored.entries.map((current) => ({ + ...current, + deadlineAt: Date.now() - 1, + ...entry, + })), + }); +} + +async function startTimed( + runner: TestRunner, + runId: string, + workflowId = 'timed', +): Promise { + const response = await runner.fetch( + post('/runs', { workflowId, runId, inputData: { topic: 't' } }), + ); + if (response.status !== 200) { + throw new Error(`start failed: ${await response.text()}`); + } + return (await response.json()) as RunSummary; +} + +function suspendedFence(runId: string, suspendedAt: number): RunSummary { + return { + runId, + status: 'suspended', + suspended: [['gate']], + suspendPayload: { + gate: { [SUSPENSION_DEADLINE_PAYLOAD_KEY]: TIMED_DEADLINE_MS }, + }, + suspendedAt: { gate: suspendedAt }, + }; +} + +/** The entry a boundary derives from suspendedFence's shape of suspension. */ +function armedEntry( + step: string, + suspendedAt: number, +): SuspensionDeadlineEntry { + return { + step, + deadlineAt: suspendedAt + TIMED_DEADLINE_MS, + suspendedAt, + resumeCount: 0, + }; +} + +/** Seed the stored deadline record exactly as a prior boundary would have. */ +function seedDeadlines( + values: Map, + runId: string, + entries: SuspensionDeadlineEntry[], + workflowId = 'timed', +): void { + values.set(SUSPENSION_DEADLINE_STORAGE_KEY, { + version: 1, + workflowId, + runId, + entries, + }); +} + +/** + * A DurableObjectState over `storage` whose put fails for the deadline key + * only. `once` lets the first matching write fail and the rest pass; `armed: + * false` starts with writes passing until fail() arms the failure; stop() + * ends a persistent failure. + */ +function deadlineWriteFailures( + storage: DurableKeyValueStorage, + options: { name?: string; once?: boolean; armed?: boolean } = {}, +): { state: DurableObjectState; fail: () => void; stop: () => void } { + let failing = options.armed ?? true; + const state = { + ...(options.name === undefined ? {} : { id: { name: options.name } }), + storage: { + ...storage, + put: async (key: string, value: unknown) => { + if (failing && key === SUSPENSION_DEADLINE_STORAGE_KEY) { + if (options.once) failing = false; + throw new Error('injected deadline write failure'); + } + return storage.put(key, value); + }, + }, + } as unknown as DurableObjectState; + return { + state, + fail: () => { + failing = true; + }, + stop: () => { + failing = false; + }, + }; +} + +/** + * Blind the workflows store's row read — the exact seam Mastra falls back from + * — and hand back the restore. Deliberately NOT the Workflow method: stubbing + * that would FABRICATE the degraded state, and the point of the tests below is + * that a REAL suspended run held by a REAL runtime produces it. File-local + * rather than shared: runtime.test.ts keeps its own copy beside its own + * fixtures, which is cheaper than a shared module for eight lines. + */ +async function blindWorkflowRow(storage: InMemoryStore): Promise<() => void> { + const store = (await storage.getStore('workflows')) as unknown as { + getWorkflowRunById: (args: unknown) => Promise; + }; + const original = store.getWorkflowRunById; + store.getWorkflowRunById = async () => null; + return () => { + store.getWorkflowRunById = original; + }; +} + +/** Count the real row deletions a wake performs, and restore the store. */ +async function countRowDeletes( + storage: InMemoryStore, +): Promise<{ calls: () => number; restore: () => void }> { + const store = (await storage.getStore('workflows')) as unknown as { + deleteWorkflowRunById: (args: unknown) => Promise; + }; + const original = store.deleteWorkflowRunById; + let calls = 0; + store.deleteWorkflowRunById = async (args: unknown) => { + calls += 1; + return original.call(store, args); + }; + return { + calls: () => calls, + restore: () => { + store.deleteWorkflowRunById = original; + }, + }; +} + +describe('DurableObjectRunner suspension deadlines', () => { + it('arms the record and the alarm at the suspension fence plus the deadline', async () => { + const { state, values, alarms } = recoveryStorage(); + const runner = new TestRunner(state, timedEnv()); + + const started = await startTimed(runner, 'run-armed'); + + const suspendedAt = started.suspendedAt?.gate as number; + expect(started.status).toBe('suspended'); + expect(storedDeadlines(values)).toEqual({ + version: 1, + workflowId: 'timed', + runId: 'run-armed', + entries: [ + { + step: 'gate', + deadlineAt: suspendedAt + TIMED_DEADLINE_MS, + suspendedAt, + resumeCount: 0, + }, + ], + }); + expect(alarms.at(-1)).toBe(suspendedAt + TIMED_DEADLINE_MS); + }); + + it('arms nothing for a suspension without the reserved key', async () => { + const { state, values, alarms } = recoveryStorage(); + const runner = new TestRunner(state, makeProductionEnv()); + + await startGated(runner); + + expect(storedDeadlines(values)).toBeUndefined(); + expect(alarms).not.toHaveLength(0); + }); + + it('keeps the alarm armed on a wake with no recovery journal but a pending deadline', async () => { + const events: string[] = []; + const { state, values, alarms } = recoveryStorage(events); + const runner = new TestRunner(state, timedEnv()); + const started = await startTimed(runner, 'run-pending'); + const dueAt = (started.suspendedAt?.gate as number) + TIMED_DEADLINE_MS; + events.length = 0; + + // #when — the watchdog wake fires while no recovery journal exists. Before + // the alarm was multiplexed this branch deleted the alarm outright, which + // dropped the still-pending suspension deadline. + await runner.alarm(); + + expect(values.has('flowsafe:run-owner-recovery:v1')).toBe(false); + expect(storedDeadlines(values)?.entries).toHaveLength(1); + expect(events).not.toContain('deleteAlarm'); + expect(alarms.at(-1)).toBe(dueAt); + }); + + it('arms the recovery watchdog when it falls due before a far-future deadline', async () => { + const { state, values, alarms } = recoveryStorage(); + const farFuture = Date.now() + 10 * 24 * 60 * 60 * 1_000; + seedDeadlines(values, 'run-far-future', [ + { ...armedEntry('gate', Date.now()), deadlineAt: farFuture }, + ]); + const runner = new TestRunner(state, timedEnv()); + + const before = Date.now(); + await startTimed(runner, 'run-far-future'); + + // #then — the recovery arm that opens the start route took the earlier of + // the two due times, and did not push the far-future deadline out. + expect(alarms[0]).toBeGreaterThanOrEqual(before + 60_000); + expect(alarms[0]).toBeLessThan(farFuture); + }); + + it('re-arms to the pending deadline when run-owner recovery clears', async () => { + const { state, values, alarms } = recoveryStorage(); + const dueAt = Date.now() + TIMED_DEADLINE_MS; + seedDeadlines(values, 'run-cleared', [ + { ...armedEntry('gate', 1), deadlineAt: dueAt }, + ]); + values.set('flowsafe:run-owner-recovery:v1', { + version: 1, + workflowId: 'timed', + runId: 'run-cleared', + token: 'attempt-token', + }); + const env = makeProductionEnv(); + // An abandoned start attempt: recovery releases the claim and clears the + // journal, which is the branch that used to delete the alarm outright. + env.runtime = { + recoverStartAttempt: vi.fn(async () => null), + ...statusStub(async () => null), + } as unknown as RunnerRuntime; + env.owners = { + owner: async () => undefined, + reserveAll: async () => true, + settleReservation: async () => undefined, + }; + const runner = new TestRunner(state, env); + + await runner.alarm(); + + // #then — clearing the journal must not delete the alarm the deadline needs + expect(values.has('flowsafe:run-owner-recovery:v1')).toBe(false); + expect(alarms.at(-1)).toBe(dueAt); + }); + + it('resumes the suspended step with the timeout envelope under a system principal', async () => { + const sent: string[] = []; + const { storage, values } = recoveryStorage(); + const state = { + id: { name: 'timed:run-timeout' }, + storage, + getWebSockets: () => [{ send: (data: string) => sent.push(data) }], + } as unknown as DurableObjectState; + const runner = new TestRunner(state, timedEnv()); + await startTimed(runner, 'run-timeout'); + elapseDeadlines(values); + + await runner.alarm(); + + const frame = JSON.parse(sent.at(-1) ?? '{}') as { + type: string; + summary: RunSummary; + }; + expect(frame.type).toBe('run'); + expect(frame.summary).toMatchObject({ + status: 'success', + // The step branched on isSuspensionTimeoutResumeData, so the envelope + // survived the resume path intact. + result: { settledBy: 'timeout' }, + requestedBy: SUSPENSION_DEADLINE_PRINCIPAL_ID, + requestedByKind: 'system', + }); + // #then — a terminal run has no suspension left to arm + expect(storedDeadlines(values)).toBeUndefined(); + }); + + it('drops a stale entry without resuming when the suspension fence moved', async () => { + const { state, values } = recoveryStorage(); + const resume = vi.fn(); + const movedAt = Date.now() - TIMED_DEADLINE_MS + 60_000; + const runtime = { + // A real signal already resumed and re-suspended this step, so its + // suspendedAt is not the one the entry was armed against. + ...statusStub(async () => suspendedFence('run-stale', movedAt)), + resume, + } as unknown as RunnerRuntime; + // Armed against suspendedAt 1, so its long-past deadline is due and its + // fence no longer matches the moved-on suspension the status reports. + seedDeadlines(values, 'run-stale', [armedEntry('gate', 1)]); + const runner = new TestRunner(state, { ...timedEnv(), runtime }); + const logged: string[] = []; + const log = vi + .spyOn(console, 'error') + .mockImplementation((...args: unknown[]) => { + logged.push(String(args[0])); + }); + + try { + await runner.alarm(); + } finally { + log.mockRestore(); + } + + expect(resume).not.toHaveBeenCalled(); + // #then — the stale entry is gone, and the wake re-derived from the summary + // it already held rather than only deleting: the CURRENT suspension is what + // ends up armed. A wake that merely dropped would leave a suspended run + // with a derivable deadline, no record and no further wake. + expect(storedDeadlines(values)?.entries).toEqual([ + armedEntry('gate', movedAt), + ]); + // #then — a discarded deadline is observable; it used to vanish in silence + expect( + logged.some((message) => + message.includes( + "suspension deadline for step 'gate' of run 'run-stale' dropped", + ), + ), + ).toBe(true); + }); + + it('records a backoff attempt and never rethrows when the timeout resume fails', async () => { + const { state, values, alarms } = recoveryStorage(); + const runtime = { + ...statusStub(async () => suspendedFence('run-retry', 1)), + resume: vi.fn(async () => { + throw new Error('injected resume failure'); + }), + } as unknown as RunnerRuntime; + seedDeadlines(values, 'run-retry', [armedEntry('gate', 1)]); + const runner = new TestRunner(state, { ...timedEnv(), runtime }); + const log = vi.spyOn(console, 'error').mockImplementation(() => {}); + + try { + // A thrown alarm() is retried by workerd, which would re-run the resume. + await expect(runner.alarm()).resolves.toBeUndefined(); + } finally { + log.mockRestore(); + } + + const entry = storedDeadlines(values)?.entries[0]; + expect(entry?.attempts).toBe(1); + expect(entry?.nextAttemptAt).toBeGreaterThanOrEqual(Date.now() + 59_000); + expect(alarms.at(-1)).toBe(entry?.nextAttemptAt); + }); + + it('abandons a spent entry as a tombstone for exactly its own suspension', async () => { + const events: string[] = []; + const { state, values, alarms } = recoveryStorage(events); + let resumeFails = true; + const resume = vi.fn(async () => { + if (resumeFails) throw new Error('injected resume failure'); + return { + ...suspendedFence('run-exhausted', 1), + resumeCount: { gate: 1 }, + }; + }); + const runtime = { + ...statusStub(async () => suspendedFence('run-exhausted', 1)), + resume, + } as unknown as RunnerRuntime; + seedDeadlines(values, 'run-exhausted', [ + { + ...armedEntry('gate', 1), + attempts: MAX_SUSPENSION_DEADLINE_ATTEMPTS - 1, + nextAttemptAt: Date.now() - 1, + }, + ]); + const runner = new TestRunner(state, { ...timedEnv(), runtime }); + const logged: string[] = []; + const log = vi + .spyOn(console, 'error') + .mockImplementation((...args: unknown[]) => { + logged.push(String(args[0])); + }); + + try { + await runner.alarm(); + + // #then — the budget is spent, and the entry stays as a TOMBSTONE: kept + // for its suspension with the floor dropped, never selected again. + expect(logged).toContain( + `suspension deadline for step 'gate' of run 'run-exhausted' dropped after ${MAX_SUSPENSION_DEADLINE_ATTEMPTS} failed wakes`, + ); + expect(storedDeadlines(values)?.entries).toEqual([ + { + ...armedEntry('gate', 1), + attempts: MAX_SUSPENSION_DEADLINE_ATTEMPTS, + }, + ]); + expect(resume).toHaveBeenCalledTimes(1); + + // #when — the next nothing-due wake re-derives from authoritative state, + // which still shows the abandoned suspension's armed payload. Removing + // the entry instead of tombstoning it made this the resurrection path: + // a clean-ledger entry, armed at the one-second floor, retried forever. + const armsBefore = alarms.length; + const before = Date.now(); + await runner.alarm(); + + // #then — no resurrection: the merge carried the tombstone, the step + // did not run again, and nothing was armed at the floor for it. + expect(resume).toHaveBeenCalledTimes(1); + expect(storedDeadlines(values)?.entries).toEqual([ + { + ...armedEntry('gate', 1), + attempts: MAX_SUSPENSION_DEADLINE_ATTEMPTS, + }, + ]); + for (const at of alarms.slice(armsBefore)) { + expect(at).toBeGreaterThanOrEqual(before + 59_000); + } + expect(events.at(-1)).toBe('deleteAlarm'); + + // #when — a real signal resumes the step and it suspends again + resumeFails = false; + const response = await runner.fetch( + post('/runs/timed/run-exhausted/resume', { + step: 'gate', + resumeData: { approvedBy: 'bob' }, + }), + ); + + // #then — the moved fence dropped the tombstone: the LATER suspension of + // the same step is a fresh entry with a fresh budget. + expect(response.status).toBe(200); + expect(storedDeadlines(values)?.entries).toEqual([ + { ...armedEntry('gate', 1), resumeCount: 1 }, + ]); + } finally { + log.mockRestore(); + } + }); + + it("quiets a foreign record by charging it and then reclaims the record from the object's own name", async () => { + // #given — this object is `run-mine`, and the deadline key holds a record + // written for `run-other`: a stale record left by a namespace reused under + // another id, or one hand-written into storage. + const events: string[] = []; + const { storage, values } = recoveryStorage(events); + const state = { + id: { name: 'timed:run-mine' }, + storage, + } as unknown as DurableObjectState; + const mineAt = Date.now(); + const foreignAt = Date.now() - TIMED_DEADLINE_MS - 1; + const resume = vi.fn(); + const env = makeProductionEnv(); + env.runtime = { + ...statusStub(async () => suspendedFence('run-mine', mineAt)), + resume, + } as unknown as RunnerRuntime; + seedDeadlines(values, 'run-other', [armedEntry('foreign', foreignAt)]); + const runner = new TestRunner(state, env); + const logged: string[] = []; + const causes = new Map(); + const stamped: (number | undefined)[] = []; + const log = vi + .spyOn(console, 'error') + .mockImplementation((...args: unknown[]) => { + logged.push(String(args[0])); + causes.set(String(args[0]), args[1]); + }); + + // #when — the foreign entry falls due on every wake, with its backoff + // rewound so the whole budget is spent inside the test + try { + for (let wake = 0; wake < MAX_SUSPENSION_DEADLINE_ATTEMPTS + 1; wake++) { + const record = storedDeadlines(values); + if (record) { + values.set(SUSPENSION_DEADLINE_STORAGE_KEY, { + ...record, + entries: record.entries.map((entry) => + entry.nextAttemptAt === undefined + ? entry + : { ...entry, nextAttemptAt: Date.now() - 1 }, + ), + }); + } + await expect(runner.alarm()).resolves.toBeUndefined(); + for (const entry of storedDeadlines(values)?.entries ?? []) { + stamped.push(entry.unreadableSince); + } + } + } finally { + log.mockRestore(); + } + + // #then — the identity assert refused before any workflow body ran, and it + // refused with the entry in hand: the failure is CHARGED to the foreign + // entry, wake after wake, which is what walks that record to a tombstone + // instead of leaving it due forever. + expect(resume).not.toHaveBeenCalled(); + expect(causes.get('suspension deadline wake failed')).toBeInstanceOf(Error); + expect( + (causes.get('suspension deadline wake failed') as Error).message, + ).toContain('DO identity mismatch'); + for ( + let attempt = 1; + attempt < MAX_SUSPENSION_DEADLINE_ATTEMPTS; + attempt += 1 + ) { + expect(logged).toContain( + `suspension deadline wake for step 'foreign' of run 'run-other' failed (attempt ${attempt})`, + ); + } + expect(logged).toContain( + `suspension deadline for step 'foreign' of run 'run-other' dropped after ${MAX_SUSPENSION_DEADLINE_ATTEMPTS} failed wakes`, + ); + // #then — and never stamped: an identity mismatch is not an unreadable + // read, so the 24 h clock never starts on it. + expect(stamped).not.toHaveLength(0); + expect(stamped.filter((mark) => mark !== undefined)).toEqual([]); + // #then — the spent record arms nothing, and the wake after it re-derives + // from `id.name` rather than from the record: the object reclaims the key + // for its OWN run and the foreign tombstone goes with the fence it named. + expect(events).toContain('deleteAlarm'); + expect(storedDeadlines(values)).toEqual({ + version: 1, + workflowId: 'timed', + runId: 'run-mine', + entries: [armedEntry('gate', mineAt)], + }); + }); + + it("starts a clean ledger when a foreign record's tombstone collides with this run's own suspension", async () => { + // #given — this object is `run-mine` and the deadline key holds a record + // stamped for `run-other` whose SPENT entry matches every key the merge + // carries a ledger on: the same step, the same fence, the same deadline. + const events: string[] = []; + const { storage, values, alarms } = recoveryStorage(events); + const state = { + id: { name: 'timed:run-mine' }, + storage, + } as unknown as DurableObjectState; + const mineAt = Date.now(); + const resume = vi.fn(); + const env = makeProductionEnv(); + env.runtime = { + ...statusStub(async () => suspendedFence('run-mine', mineAt)), + resume, + } as unknown as RunnerRuntime; + seedDeadlines(values, 'run-other', [ + { + ...armedEntry('gate', mineAt), + attempts: MAX_SUSPENSION_DEADLINE_ATTEMPTS, + }, + ]); + const runner = new TestRunner(state, env); + + // #when — a tombstone is never due, so the wake reconciles from `id.name` + // and rewrites the key under this object's own run + await expect(runner.alarm()).resolves.toBeUndefined(); + + // #then — the reclaimed record carries this run's own entry and nothing + // else: a live deadline born at the budget would never fire, and its own + // arm would delete the alarm that was its last chance to. + expect(storedDeadlines(values)).toEqual({ + version: 1, + workflowId: 'timed', + runId: 'run-mine', + entries: [armedEntry('gate', mineAt)], + }); + expect(resume).not.toHaveBeenCalled(); + expect(alarms.at(-1)).toBe(mineAt + TIMED_DEADLINE_MS); + expect(events.at(-1)).toBe('setAlarm'); + }); + + it('deletes a foreign record when the reclaiming run derives no deadline of its own', async () => { + // #given — this object is `run-mine`, the deadline key holds a record + // stamped for `run-other` whose entry is far from due, and this run has + // already finished: the reconcile has nothing to write in its place. + const events: string[] = []; + const { storage, values } = recoveryStorage(events); + const state = { + id: { name: 'timed:run-mine' }, + storage, + } as unknown as DurableObjectState; + const resume = vi.fn(); + const env = makeProductionEnv(); + env.runtime = { + ...statusStub(async () => ({ runId: 'run-mine', status: 'success' })), + resume, + } as unknown as RunnerRuntime; + seedDeadlines(values, 'run-other', [ + armedEntry('foreign', Date.now() + 10 * TIMED_DEADLINE_MS), + ]); + const runner = new TestRunner(state, env); + + // #when — nothing is due, so the wake reconciles rather than resuming + await expect(runner.alarm()).resolves.toBeUndefined(); + + // #then — the write decides on the record that is STORED, not on the one + // this run may inherit a ledger from: a foreign record is present, so the + // empty derivation DELETES the key and the alarm goes with it. Deciding + // that on the inherited record instead would read as nothing-stored and + // skip the write, stranding the foreign entry armed forever and waking + // this object for a run it does not own. + expect(storedDeadlines(values)).toBeUndefined(); + expect(events).toContain(`delete:${SUSPENSION_DEADLINE_STORAGE_KEY}`); + expect(events.at(-1)).toBe('deleteAlarm'); + expect(resume).not.toHaveBeenCalled(); + }); + + it('keeps the record and its wake when a nothing-due wake reads Mastra in-memory fallback state', async () => { + const events: string[] = []; + const { storage: doStorage, values, alarms } = recoveryStorage(events); + const state = { + id: { name: 'timed:run-blinded-idle' }, + storage: doStorage, + } as unknown as DurableObjectState; + const env = timedEnv(); + const runner = new TestRunner(state, env); + await startTimed(runner, 'run-blinded-idle'); + const armed = storedDeadlines(values)?.entries; + expect(armed).toHaveLength(1); + + // #when — the real producer of the degraded read: the row lookup comes + // back empty (a replica behind, a store handle gone) while this isolate + // still holds the suspended Run, so Mastra answers from memory instead. + const restore = await blindWorkflowRow(env.storage); + const logged: string[] = []; + const log = vi + .spyOn(console, 'error') + .mockImplementation((...args: unknown[]) => { + logged.push(String(args[0])); + }); + events.length = 0; + const before = Date.now(); + + try { + await runner.alarm(); + } finally { + log.mockRestore(); + restore(); + } + + // #then — nothing concluded from a read that never reached storage. The + // fallback reports 'pending' for a run that has never been resumed, so the + // self-consistency backstop calls it readable: reconciling from it derived + // nothing and DELETED the record and the alarm of a run that is still + // suspended, silently. + expect(storedDeadlines(values)?.entries).toEqual(armed); + expect(events).not.toContain('deleteAlarm'); + expect(events.filter((event) => event.startsWith('delete:'))).toEqual([]); + expect(alarms.at(-1)).toBeGreaterThanOrEqual(before + 60_000); + expect( + logged.filter((message) => + message.includes('could not read authoritative state'), + ), + ).toHaveLength(1); + + // #then — and the run really was suspended the whole time + const status = await runner.fetch( + deploymentIdentityRequest('http://do/runs/timed/run-blinded-idle'), + ); + expect(((await status.json()) as RunSummary).status).toBe('suspended'); + }); + + it('charges nothing and runs nothing when a due wake reads Mastra in-memory fallback state', async () => { + const events: string[] = []; + const { storage: doStorage, values, alarms } = recoveryStorage(events); + const state = { + id: { name: 'timed:run-blinded-due' }, + storage: doStorage, + } as unknown as DurableObjectState; + const env = timedEnv(); + const runner = new TestRunner(state, env); + const started = await startTimed(runner, 'run-blinded-due'); + const suspendedAt = started.suspendedAt?.gate as number; + elapseDeadlines(values); + const due = storedDeadlines(values)?.entries[0]; + const restore = await blindWorkflowRow(env.storage); + const log = vi.spyOn(console, 'error').mockImplementation(() => {}); + const before = Date.now(); + + try { + await runner.alarm(); + } finally { + log.mockRestore(); + restore(); + } + + // #then — the entry is intact and UNCHARGED, and the wake kept the + // watchdog cadence: charging here would walk a live deadline to its + // tombstone inside a quarter of an hour of degraded reads. + const entry = storedDeadlines(values)?.entries[0]; + expect(entry).toMatchObject({ ...due }); + expect(entry).not.toHaveProperty('attempts'); + expect(entry?.unreadableSince).toBeGreaterThanOrEqual(before); + expect(events).not.toContain('deleteAlarm'); + expect(alarms.at(-1)).toBeGreaterThanOrEqual(before + 60_000); + + // #then — nothing resumed: the run is still suspended at the same fence + const mid = await runner.fetch( + deploymentIdentityRequest('http://do/runs/timed/run-blinded-due'), + ); + const midSummary = (await mid.json()) as RunSummary; + expect(midSummary.status).toBe('suspended'); + expect(midSummary.suspendedAt?.gate).toBe(suspendedAt); + + // #when — the read heals, which is all this wake was ever waiting for + await runner.alarm(); + + // #then — the deadline it declined to spend still fires + const settled = await runner.fetch( + deploymentIdentityRequest('http://do/runs/timed/run-blinded-due'), + ); + expect(await settled.json()).toMatchObject({ + status: 'success', + result: { settledBy: 'timeout' }, + requestedBy: SUSPENSION_DEADLINE_PRINCIPAL_ID, + }); + expect(storedDeadlines(values)).toBeUndefined(); + }); + + it('classifies a read that threw as unreadable state and keeps the thrown error as its cause', async () => { + // #given — a due entry whose authoritative read throws something that is + // not already a RunStateUnreadableError: a storage fault, as a driver + // surfaces one. + const events: string[] = []; + const { storage, values } = recoveryStorage(events); + const state = { + id: { name: 'timed:run-read-threw' }, + storage, + } as unknown as DurableObjectState; + const thrown = new Error('D1_ERROR: network connection lost'); + const resume = vi.fn(); + const env = makeProductionEnv(); + env.runtime = { + ...statusStub(async () => { + throw thrown; + }), + resume, + } as unknown as RunnerRuntime; + seedDeadlines(values, 'run-read-threw', [ + armedEntry('gate', Date.now() - TIMED_DEADLINE_MS - 1), + ]); + const runner = new TestRunner(state, env); + const logged: string[] = []; + const causes: unknown[] = []; + const log = vi + .spyOn(console, 'error') + .mockImplementation((...args: unknown[]) => { + logged.push(String(args[0])); + causes.push(args[1]); + }); + + // #when + try { + await expect(runner.alarm()).resolves.toBeUndefined(); + } finally { + log.mockRestore(); + } + + // #then — the wake reported the one conclusion it may draw from a failed + // read, and the error it reported carries the original as `cause`: the + // classification is what the alarm acts on, and the fault underneath it is + // still the thing an operator has to diagnose. + const raised = + causes[ + logged.indexOf( + 'suspension deadline wake could not read authoritative state', + ) + ]; + expect(raised).toBeInstanceOf(RunStateUnreadableError); + expect((raised as RunStateUnreadableError).cause).toBe(thrown); + expect(resume).not.toHaveBeenCalled(); + }); + + it('deletes no run row and settles nothing when recovery reads in-memory fallback state', async () => { + const events: string[] = []; + const { storage: doStorage, values, alarms } = recoveryStorage(events); + const state = { + id: { name: 'timed:run-blinded-recovery' }, + storage: doStorage, + } as unknown as DurableObjectState; + const settle = vi.fn(async () => undefined); + const env = makeProductionEnv(new InMemoryStore(), { + reserve: async () => true, + settle, + }); + env.runtime = timedRuntime(env.storage); + const token = 'attempt-token'; + // The interrupted start, exactly as one is left behind: claim reserved and + // journaled, Mastra's suspension persisted, no boundary left but recovery. + await env.owners.reserveAll( + [{ kind: 'run', resourceId: 'run-blinded-recovery' }], + OWNER_PRINCIPAL, + token, + ); + await (env.runtime as RunnerRuntime).start('timed', { + runId: 'run-blinded-recovery', + inputData: { topic: 't' }, + requestedBy: OWNER_PRINCIPAL.id, + requestedByKind: OWNER_PRINCIPAL.kind, + attemptToken: token, + }); + values.set('flowsafe:run-owner-recovery:v1', { + version: 1, + workflowId: 'timed', + runId: 'run-blinded-recovery', + token, + }); + const runner = new TestRunner(state, env); + const deletes = await countRowDeletes(env.storage); + const restore = await blindWorkflowRow(env.storage); + const logged: string[] = []; + const log = vi + .spyOn(console, 'error') + .mockImplementation((...args: unknown[]) => { + logged.push(String(args[0])); + }); + const before = Date.now(); + + try { + // #then — the wake does not rethrow: workerd retries a thrown alarm up + // to six times, which would be a retry storm inside the very incident + // that caused it. + await expect(runner.alarm()).resolves.toBeUndefined(); + } finally { + log.mockRestore(); + restore(); + deletes.restore(); + } + + // #then — no conclusion drawn from a read that never reached storage. The + // fallback presents no provenance and a 'pending' status, which used to + // walk straight into the abandoned-shell branch and DELETE a live row and + // its snapshot behind a lagging read. + expect(deletes.calls()).toBe(0); + expect(settle).not.toHaveBeenCalled(); + expect(values.has('flowsafe:run-owner-recovery:v1')).toBe(true); + expect( + logged.filter((message) => + message.includes('could not read authoritative state'), + ), + ).toHaveLength(2); + expect(alarms.at(-1)).toBeGreaterThanOrEqual(before + 60_000); + + // #then — the run survived, and the recovery it is still owed happens on a + // later wake once the read heals + const status = await runner.fetch( + deploymentIdentityRequest('http://do/runs/timed/run-blinded-recovery'), + ); + expect(((await status.json()) as RunSummary).status).toBe('suspended'); + + await runner.alarm(); + + expect(values.has('flowsafe:run-owner-recovery:v1')).toBe(false); + expect(settle).toHaveBeenCalledTimes(1); + }); + + it('answers dispatch-status with a retryable 503 while the run state cannot be read', async () => { + const { state, values } = recoveryStorage(); + const env = makeProductionEnv(); + env.runtime = { + ...statusStub(async () => null), + recoverStartAttempt: vi.fn(async () => { + throw new RunStateUnreadableError('timed', 'run-unreadable-route'); + }), + } as unknown as RunnerRuntime; + // The interrupted start this route settles, with the read that would + // settle it refusing to answer from state it could not reach. + values.set('flowsafe:run-owner-recovery:v1', { + version: 1, + workflowId: 'timed', + runId: 'run-unreadable-route', + token: 'attempt-token', + }); + const runner = new TestRunner(state, env); + + const response = await runner.fetch( + deploymentIdentityRequest( + 'http://do/runs/timed/run-unreadable-route/dispatch-status', + ), + ); + + // #then — the operator's problem and retryable, like a misprovisioned + // deployment: a 500 would read as a code fault, and answering 200 from the + // fabricated summary is the failure the guard exists to prevent. + expect(response.status).toBe(503); + await expect(response.json()).resolves.toMatchObject({ + error: expect.stringContaining('state is not readable'), + }); + expect(values.has('flowsafe:run-owner-recovery:v1')).toBe(true); + }); + + it('answers a start with a retryable 503 while the pending recovery cannot read authoritative state', async () => { + // #given — a start whose object still holds an interrupted start's + // journal, with the read that would settle it refusing to answer from + // state it could not reach. The recovery runs BEFORE the existing-run + // check, so nothing downstream of it sees a fabricated read. + const { state, values } = recoveryStorage(); + const env = makeProductionEnv(); + const start = vi.fn(); + env.runtime = { + ...statusStub(async () => null), + start, + recoverStartAttempt: vi.fn(async () => { + throw new RunStateUnreadableError('timed', 'run-start-unreadable'); + }), + } as unknown as RunnerRuntime; + values.set('flowsafe:run-owner-recovery:v1', { + version: 1, + workflowId: 'timed', + runId: 'run-start-unreadable', + token: 'attempt-token', + }); + const runner = new TestRunner(state, env); + + // #when + const response = await runner.fetch( + post('/runs', { + workflowId: 'timed', + runId: 'run-start-unreadable', + inputData: { topic: 't' }, + }), + ); + + // #then — retryable, and no second run: this route used to refuse with a + // 500 (`has no matching committed owner`) once the same recovery had + // deleted the row and released its claim behind the lagging read. The + // journal survives for a wake that can read it. + expect(response.status).toBe(503); + await expect(response.json()).resolves.toMatchObject({ + error: expect.stringContaining('state is not readable'), + }); + expect(start).not.toHaveBeenCalled(); + expect(values.has('flowsafe:run-owner-recovery:v1')).toBe(true); + }); + + it('stamps the unreadable clock once and clears it on the first read that succeeds', async () => { + const { state, values } = recoveryStorage(); + const armedAt = Date.now() - TIMED_DEADLINE_MS - 1; + let readable = false; + const movedAt = Date.now(); + const env = makeProductionEnv(); + const resume = vi.fn(); + env.runtime = { + ...statusStub(async () => { + if (!readable) throw new Error('D1: network error'); + // A real signal moved the suspension on while reads were failing, so + // the wake reconciles instead of resuming — which is what lets the + // stamp's fate be observed on an entry that survives. + return suspendedFence('run-clock', movedAt); + }), + resume, + } as unknown as RunnerRuntime; + seedDeadlines(values, 'run-clock', [armedEntry('gate', armedAt)]); + const runner = new TestRunner(state, env); + const log = vi.spyOn(console, 'error').mockImplementation(() => {}); + + try { + await runner.alarm(); + const stamped = storedDeadlines(values)?.entries[0]?.unreadableSince; + expect(stamped).toBeGreaterThan(0); + + // #when — the read keeps failing + await runner.alarm(); + + // #then — the clock is not restamped: rewriting it every wake would push + // the day it allows out forever, and nothing would ever bound the run. + expect(storedDeadlines(values)?.entries[0]?.unreadableSince).toBe( + stamped, + ); + + readable = true; + await runner.alarm(); + } finally { + log.mockRestore(); + } + + // #then — a read that SUCCEEDED is the clearing event: the reconcile + // rebuilds the entry from what it derived, and the stamp does not ride + // along. + expect(resume).not.toHaveBeenCalled(); + expect(storedDeadlines(values)?.entries).toEqual([ + armedEntry('gate', movedAt), + ]); + }); + + it('abandons an entry whose run state has been unreadable for a day', async () => { + const events: string[] = []; + const { state, values, alarms } = recoveryStorage(events); + const armedAt = Date.now() - TIMED_DEADLINE_MS - 1; + const env = makeProductionEnv(); + const resume = vi.fn(); + env.runtime = { + ...statusStub(async () => { + throw new Error('unknown workflow'); + }), + resume, + } as unknown as RunnerRuntime; + // A day of wakes that never charge anything: the workflow registration a + // deploy dropped for good, which no read will ever heal. + seedDeadlines(values, 'run-day', [ + { + ...armedEntry('gate', armedAt), + unreadableSince: Date.now() - 86_400_000 - 1, + }, + ]); + const runner = new TestRunner(state, env); + const logged: string[] = []; + const log = vi + .spyOn(console, 'error') + .mockImplementation((...args: unknown[]) => { + logged.push(String(args[0])); + }); + const before = Date.now(); + + try { + await runner.alarm(); + const armsAfterAbandon = alarms.length; + + // #then — a tombstone, under its own log: this entry was never charged a + // single failed resume, so the drop log would name a budget it never + // spent. + expect(logged).toContain( + "suspension deadline for step 'gate' of run 'run-day' abandoned after 24 h of unreadable run state", + ); + expect(storedDeadlines(values)?.entries).toEqual([ + { + ...armedEntry('gate', armedAt), + attempts: MAX_SUSPENSION_DEADLINE_ATTEMPTS, + }, + ]); + // #then — and the heartbeat ENDS, which is the whole point of the bound: + // giving up on the entry and then waking for it every minute forever + // would be no bound at all. Nothing else is armed, so the alarm goes. + expect(events.at(-1)).toBe('deleteAlarm'); + + // #when — a wake is forced anyway (workerd redelivery, or another duty) + await runner.alarm(); + + // #then — the tombstone is never selected, so nothing is retried and + // nothing is armed at the one-second floor for it + expect(resume).not.toHaveBeenCalled(); + for (const at of alarms.slice(armsAfterAbandon)) { + expect(at).toBeGreaterThanOrEqual(before + 60_000); + } + } finally { + log.mockRestore(); + } + }); + + it('restarts a future-dated unreadable clock at the wake that reads it', async () => { + // #given — a stamp AHEAD of the wake reading it: a clock that stepped + // backwards between two wakes, or a hand-written record. Left as read, its + // elapsed time can never pass the limit and the entry keeps an uncharged + // heartbeat forever. + const { state, values } = recoveryStorage(); + const armedAt = Date.now() - TIMED_DEADLINE_MS - 1; + const env = makeProductionEnv(); + env.runtime = { + ...statusStub(async () => { + throw new Error('unknown workflow'); + }), + resume: vi.fn(), + } as unknown as RunnerRuntime; + seedDeadlines(values, 'run-future-clock', [ + { + ...armedEntry('gate', armedAt), + unreadableSince: Date.now() + 10 * 86_400_000, + }, + ]); + const runner = new TestRunner(state, env); + const logged: string[] = []; + const log = vi + .spyOn(console, 'error') + .mockImplementation((...args: unknown[]) => { + logged.push(String(args[0])); + }); + const before = Date.now(); + + try { + // #when — a wake reads the future stamp + await runner.alarm(); + + // #then — it is corrected to this wake, so the day the bound allows + // actually starts running + const restarted = storedDeadlines(values)?.entries[0]?.unreadableSince; + expect(restarted).toBeGreaterThanOrEqual(before); + expect(restarted).toBeLessThanOrEqual(Date.now()); + + // #when — that day passes with the read still failing + values.set(SUSPENSION_DEADLINE_STORAGE_KEY, { + ...storedDeadlines(values), + entries: [ + { + ...armedEntry('gate', armedAt), + unreadableSince: Date.now() - 86_400_000 - 1, + }, + ], + }); + await runner.alarm(); + } finally { + log.mockRestore(); + } + + // #then — the entry matures and is abandoned like any other, instead of + // heart-beating for a run nothing will ever read + expect(logged).toContain( + "suspension deadline for step 'gate' of run 'run-future-clock' abandoned after 24 h of unreadable run state", + ); + expect(storedDeadlines(values)?.entries).toEqual([ + { + ...armedEntry('gate', armedAt), + attempts: MAX_SUSPENSION_DEADLINE_ATTEMPTS, + }, + ]); + }); + + it('runs one unreadable-state clock over the whole due batch and leaves entries that are not due alone', async () => { + const events: string[] = []; + const { state, values, alarms } = recoveryStorage(events); + const now = Date.now(); + const alphaAt = now - TIMED_DEADLINE_MS - 10_000; + const bravoAt = now - TIMED_DEADLINE_MS - 5_000; + const alpha = armedEntry('alpha', alphaAt); + // Due as well, and carrying a live ledger from an earlier failed resume: + // the stamp must ride alongside a ledger without disturbing it. + const bravo = { + ...armedEntry('bravo', bravoAt), + attempts: 2, + nextAttemptAt: now - 1, + }; + const future = armedEntry('future', now); + const env = makeProductionEnv(); + const resume = vi.fn(); + env.runtime = { + ...statusStub(async () => { + throw new Error('D1: network error'); + }), + resume, + } as unknown as RunnerRuntime; + seedDeadlines(values, 'run-batch', [alpha, bravo, future]); + const runner = new TestRunner(state, env); + const logged: string[] = []; + const log = vi + .spyOn(console, 'error') + .mockImplementation((...args: unknown[]) => { + logged.push(String(args[0])); + }); + + try { + await runner.alarm(); + + // #then — one failed read is one fact about the run, so every entry due + // at this wake starts its day together. A per-entry clock would make an + // N-entry record take N days to clear instead of one. + const clock = storedEntry(values, 'alpha')?.unreadableSince as number; + expect(clock).toBeGreaterThanOrEqual(now); + expect(storedEntry(values, 'bravo')).toEqual({ + ...bravo, + unreadableSince: clock, + }); + // #then — and the entry that is NOT due is byte-identical: its own day + // begins when it falls due, not when a sibling's did. + expect(storedEntry(values, 'future')).toEqual(future); + expect(resume).not.toHaveBeenCalled(); + + // #when — the read never heals and the day the bound allows passes + const stored = storedDeadlines(values) as SuspensionDeadlineRecord; + values.set(SUSPENSION_DEADLINE_STORAGE_KEY, { + ...stored, + entries: stored.entries.map((entry) => + entry.unreadableSince === undefined + ? entry + : { ...entry, unreadableSince: entry.unreadableSince - 86_400_001 }, + ), + }); + await runner.alarm(); + + // #then — the whole batch is abandoned in ONE wake, each under its own + // log, while the entry that never came due is untouched and still holds + // the object's alarm. + expect(storedEntry(values, 'alpha')).toEqual({ + ...alpha, + attempts: MAX_SUSPENSION_DEADLINE_ATTEMPTS, + }); + expect(storedEntry(values, 'bravo')).toEqual({ + ...armedEntry('bravo', bravoAt), + attempts: MAX_SUSPENSION_DEADLINE_ATTEMPTS, + }); + expect(storedEntry(values, 'future')).toEqual(future); + expect( + logged.filter((message) => message.includes('abandoned after 24 h')), + ).toHaveLength(2); + expect(events).not.toContain('deleteAlarm'); + expect(alarms.at(-1)).toBe(future.deadlineAt); + } finally { + log.mockRestore(); + } + }); + + it('clears the unreadable stamp from a lifecycle boundary while the wake read is still failing', async () => { + const { storage, values } = recoveryStorage(); + const state = { + id: { name: 'timed:run-boundary' }, + storage, + } as unknown as DurableObjectState; + const now = Date.now(); + const alphaAt = now - TIMED_DEADLINE_MS - 1; + const bravoAt = now; + // The summary the resume of the SIBLING step hands back: a live projection + // of a run still suspended at both fences, which is evidence about the run + // even while its row read keeps failing. + const bothSuspended: RunSummary = { + runId: 'run-boundary', + status: 'suspended', + suspended: [['alpha'], ['bravo']], + suspendPayload: { + alpha: { [SUSPENSION_DEADLINE_PAYLOAD_KEY]: TIMED_DEADLINE_MS }, + bravo: { [SUSPENSION_DEADLINE_PAYLOAD_KEY]: TIMED_DEADLINE_MS }, + }, + suspendedAt: { alpha: alphaAt, bravo: bravoAt }, + }; + const env = makeProductionEnv(); + env.runtime = { + ...statusStub(async () => { + throw new Error('D1: network error'); + }), + resume: vi.fn(async () => bothSuspended), + } as unknown as RunnerRuntime; + seedDeadlines(values, 'run-boundary', [ + armedEntry('alpha', alphaAt), + armedEntry('bravo', bravoAt), + ]); + const runner = new TestRunner(state, env); + const log = vi.spyOn(console, 'error').mockImplementation(() => undefined); + + try { + await runner.alarm(); + expect(storedEntry(values, 'alpha')?.unreadableSince).toBeGreaterThan(0); + + // #when — a boundary lands that this wake had nothing to do with: an + // HTTP resume of the sibling step + const response = await runner.fetch( + post('/runs/timed/run-boundary/resume', { + step: ['bravo'], + resumeData: { approvedBy: 'ops' }, + }), + ); + + // #then — the boundary's own summary is evidence the run is real, so the + // merge rebuilds both entries without the stamp: the day is not counted + // through a run a host keeps touching. + expect(response.status).toBe(200); + expect(storedDeadlines(values)?.entries).toEqual([ + armedEntry('alpha', alphaAt), + armedEntry('bravo', bravoAt), + ]); + } finally { + log.mockRestore(); + } + }); + + it('keeps its cadence when the unreadable stamp itself cannot be written', async () => { + const { storage, values, alarms } = recoveryStorage(); + const { state } = deadlineWriteFailures(storage, { + name: 'timed:run-stamp-write-fail', + }); + const armedAt = Date.now() - TIMED_DEADLINE_MS - 1; + const armed = armedEntry('gate', armedAt); + const env = makeProductionEnv(); + const resume = vi.fn(); + env.runtime = { + ...statusStub(async () => { + throw new Error('D1: network error'); + }), + resume, + } as unknown as RunnerRuntime; + seedDeadlines(values, 'run-stamp-write-fail', [armed]); + const runner = new TestRunner(state, env); + const logged: string[] = []; + const log = vi + .spyOn(console, 'error') + .mockImplementation((...args: unknown[]) => { + logged.push(String(args[0])); + }); + const before = Date.now(); + + try { + for (let wake = 0; wake < 3; wake += 1) { + await expect(runner.alarm()).resolves.toBeUndefined(); + } + } finally { + log.mockRestore(); + } + + // #then — the 24 h bound is conditional on this write landing, and a + // storage layer that cannot write the key converges nothing anyway: the + // failure is logged, the record is untouched, nothing is charged, and the + // wake keeps the watchdog cadence it would have kept regardless. + expect(storedDeadlines(values)?.entries).toEqual([armed]); + expect(resume).not.toHaveBeenCalled(); + expect( + logged.filter((message) => + message.includes('unreadable-state marker failed'), + ), + ).toHaveLength(3); + expect(logged.some((message) => message.includes('failed (attempt'))).toBe( + false, + ); + for (const at of alarms) { + expect(at).toBeGreaterThanOrEqual(before + 60_000); + } + }); + + it('reaches the tombstone in five charges when readable and unreadable wakes alternate', async () => { + const { state, values } = recoveryStorage(); + const armedAt = Date.now() - TIMED_DEADLINE_MS - 1; + const armed = armedEntry('gate', armedAt); + let readable = false; + const env = makeProductionEnv(); + const resume = vi.fn(async () => { + throw new Error('resume failed'); + }); + env.runtime = { + ...statusStub(async () => { + readable = !readable; + if (!readable) throw new Error('D1: network error'); + return suspendedFence('run-oscillating', armedAt); + }), + resume, + } as unknown as RunnerRuntime; + seedDeadlines(values, 'run-oscillating', [armed]); + const runner = new TestRunner(state, env); + const log = vi.spyOn(console, 'error').mockImplementation(() => undefined); + + try { + for (let wake = 0; wake < 24; wake += 1) { + // Only the backoff a charge wrote is rewound, so the entry stays due + // at its original fence: the question is whether an interleaved stamp + // ever resets the ledger, not how long the backoff takes. + const stored = storedDeadlines(values) as SuspensionDeadlineRecord; + values.set(SUSPENSION_DEADLINE_STORAGE_KEY, { + ...stored, + entries: stored.entries.map((entry) => + entry.nextAttemptAt === undefined + ? entry + : { ...entry, nextAttemptAt: Date.now() - 1 }, + ), + }); + await runner.alarm(); + } + } finally { + log.mockRestore(); + } + + // #then — a stamp is not a reprieve: the failed resumes still add up to + // the budget, the entry ends as a tombstone carrying no clock, and no + // sixth resume runs after it. + expect(resume).toHaveBeenCalledTimes(MAX_SUSPENSION_DEADLINE_ATTEMPTS); + expect(storedDeadlines(values)?.entries).toEqual([ + { ...armed, attempts: MAX_SUSPENSION_DEADLINE_ATTEMPTS }, + ]); + }); + + it('spends no abandonment budget on a wake that cannot build its runtime', async () => { + const events: string[] = []; + const { state, values, alarms } = recoveryStorage(events); + const armedAt = Date.now() - TIMED_DEADLINE_MS - 1; + const armed = armedEntry('gate', armedAt); + // A misconfigured binding: build(env) throws on EVERY wake and nothing + // memoizes the failure, so this is the shape that would walk a live + // deadline to its tombstone in five wakes if it were charged. + class UnbuildableRunner extends TestRunner { + protected build(): RunnerRuntime { + throw new Error('WORKFLOWS binding is not configured'); + } + } + seedDeadlines(values, 'run-unbuildable', [armed]); + const runner = new UnbuildableRunner(state, makeProductionEnv()); + const logged: string[] = []; + const log = vi + .spyOn(console, 'error') + .mockImplementation((...args: unknown[]) => { + logged.push(String(args[0])); + }); + const before = Date.now(); + + try { + for (let wake = 0; wake < 3; wake += 1) { + await expect(runner.alarm()).resolves.toBeUndefined(); + } + } finally { + log.mockRestore(); + } + + // #then — a fault that says nothing about this entry spends nothing on it: + // the record is untouched, no ledger, no clock, no tombstone. + expect(storedDeadlines(values)?.entries).toEqual([armed]); + // #then — and the wake keeps the watchdog cadence until the binding is + // fixed, never the one-second arm floor a still-due entry would take. + expect(events).not.toContain('deleteAlarm'); + for (const at of alarms) { + expect(at).toBeGreaterThanOrEqual(before + 60_000); + } + expect( + logged.filter((message) => message === 'suspension deadline wake failed'), + ).toHaveLength(3); + }); + + it('keeps an unwritable record and its cadence on a nothing-due wake', async () => { + const { storage, values, alarms } = recoveryStorage(); + const { state } = deadlineWriteFailures(storage, { + name: 'timed:run-idle-write-fail', + }); + const armedAt = Date.now(); + const armed = armedEntry('gate', armedAt); + const env = makeProductionEnv(); + env.runtime = { + ...statusStub(async () => suspendedFence('run-idle-write-fail', armedAt)), + } as unknown as RunnerRuntime; + seedDeadlines(values, 'run-idle-write-fail', [armed]); + const runner = new TestRunner(state, env); + const logged: string[] = []; + const log = vi + .spyOn(console, 'error') + .mockImplementation((...args: unknown[]) => { + logged.push(String(args[0])); + }); + const before = Date.now(); + + try { + for (let wake = 0; wake < 3; wake += 1) { + await expect(runner.alarm()).resolves.toBeUndefined(); + } + } finally { + log.mockRestore(); + } + + // #then — the re-derive read fine and derived the same entry; only the + // write failed. Nothing is due, so there is no entry to charge and none is + // charged: the record survives untouched on the watchdog cadence, never at + // the arm floor. + expect(storedDeadlines(values)?.entries).toEqual([armed]); + expect(logged).toContain('suspension deadline wake failed'); + expect(logged.some((message) => message.includes('failed (attempt'))).toBe( + false, + ); + for (const at of alarms) { + expect(at).toBeGreaterThanOrEqual(before + 60_000); + } + }); + + it('discards a malformed stored record and converges instead of throwing every wake', async () => { + const { state, values } = recoveryStorage(); + values.set(SUSPENSION_DEADLINE_STORAGE_KEY, { + version: 1, + workflowId: 'timed/forged', + runId: 'run-malformed', + entries: [], + }); + const runner = new TestRunner(state, timedEnv()); + const log = vi.spyOn(console, 'error').mockImplementation(() => {}); + + try { + await expect(runner.alarm()).resolves.toBeUndefined(); + } finally { + log.mockRestore(); + } + + expect(values.has(SUSPENSION_DEADLINE_STORAGE_KEY)).toBe(false); + }); + + it('re-arms at the new fence when the timeout resume suspends again', async () => { + const { state, values, alarms } = recoveryStorage(); + const runner = new TestRunner(state, timedEnv()); + const started = await startTimed( + runner, + 'run-escalating', + 'timed-escalating', + ); + const firstSuspendedAt = started.suspendedAt?.gate as number; + elapseDeadlines(values); + + await runner.alarm(); + + // #then — the entry is fenced on the ESCALATED suspension: a new resume + // ordinal, and the escalation's own deadline offset. Not `suspendedAt` + // alone: the re-suspension can land in the same millisecond as the first. + const entry = storedDeadlines(values)?.entries[0]; + expect(entry?.suspendedAt).toBeGreaterThanOrEqual(firstSuspendedAt); + expect(entry?.resumeCount).toBe(1); + expect(entry?.deadlineAt).toBe( + (entry?.suspendedAt as number) + RETRY_DEADLINE_MS, + ); + expect(alarms.at(-1)).toBe(entry?.deadlineAt); + }); + + it('clears the record when the run is terminated', async () => { + const events: string[] = []; + const { state, values } = recoveryStorage(events); + const runner = new TestRunner(state, timedEnv()); + await startTimed(runner, 'run-terminated'); + expect(storedDeadlines(values)?.entries).toHaveLength(1); + + const response = await runner.fetch( + post('/runs/timed/run-terminated/terminate', {}), + ); + + expect(response.status).toBe(200); + expect(storedDeadlines(values)).toBeUndefined(); + expect(events.at(-1)).toBe('deleteAlarm'); + }); + + it('clears the record on a terminal deadline route that transitions nothing', async () => { + const { state, values } = recoveryStorage(); + const env = makeProductionEnv(); + // A deadline request whose compare-and-swap no longer matches: the route + // returns the current terminal summary without finalizing, and it is the + // one terminal path that used to leave the record armed for a run that can + // never suspend again. + env.runtime = { + cancelActiveExecution: vi.fn(async () => undefined), + ...statusStub(async () => ({ runId: 'run-noop', status: 'timed_out' })), + timeOutAsPrincipal: vi.fn(async () => ({ + summary: { runId: 'run-noop', status: 'timed_out' }, + transitioned: false, + casMatched: false, + cleanup: { revision: 2, status: 'timed_out', cleanupCompleted: true }, + })), + } as unknown as RunnerRuntime; + seedDeadlines(values, 'run-noop', [armedEntry('gate', Date.now())]); + const runner = new TestRunner(state, env); + + const response = await runner.fetch( + post( + '/runs/timed/run-noop/deadline', + { expectedRevision: 1 }, + { + kind: 'system', + id: 'maintenance', + purpose: 'run-deadline-maintenance', + }, + ), + ); + + expect(response.status).toBe(200); + expect(storedDeadlines(values)).toBeUndefined(); + }); + + it('charges nothing and keeps the watchdog when a due wake cannot read authoritative state', async () => { + const { state, values, alarms } = recoveryStorage(); + const resume = vi.fn(); + const env = makeProductionEnv(); + // A run whose workflow a deploy unregistered, or a D1 fault: the read + // throws before the fence can be checked, so nothing consumes the entry + // and it stays due. + env.runtime = { + ...statusStub(async () => { + throw new Error('unknown workflow'); + }), + resume, + } as unknown as RunnerRuntime; + const armedAt = Date.now() - TIMED_DEADLINE_MS - 1; + const armed = armedEntry('gate', armedAt); + seedDeadlines(values, 'run-hot-loop', [armed]); + const runner = new TestRunner(state, env); + const logged: string[] = []; + const log = vi + .spyOn(console, 'error') + .mockImplementation((...args: unknown[]) => { + logged.push(String(args[0])); + }); + const before = Date.now(); + + try { + for (let wake = 0; wake < 3; wake += 1) { + await expect(runner.alarm()).resolves.toBeUndefined(); + } + } finally { + log.mockRestore(); + } + + // #then — Cloudflare fires an alarm set in the past immediately, so a past + // arm anywhere in this sequence is the observable hot loop. Every arm here + // is the watchdog's instead: the wake converged nothing. + expect(alarms.filter((at) => at <= before)).toEqual([]); + for (const at of alarms) { + expect(at).toBeGreaterThanOrEqual(before + 60_000); + } + expect(resume).not.toHaveBeenCalled(); + // #then — and NOT charged. A ~15 minute incident would otherwise walk this + // entry through its five failures and tombstone it, abandoning a live + // deadline permanently for the suspension it belongs to. + const entry = storedDeadlines(values)?.entries[0]; + expect(entry).toMatchObject({ ...armed }); + expect(entry).not.toHaveProperty('attempts'); + expect(entry).not.toHaveProperty('nextAttemptAt'); + // #then — the only bound on an unreadable run: stamped by the first such + // wake, and by that one only, so the day it allows cannot be pushed out. + expect(entry?.unreadableSince).toBeGreaterThanOrEqual(before); + expect(entry?.unreadableSince).toBeLessThanOrEqual(Date.now()); + expect( + logged.filter((message) => + message.includes('could not read authoritative state'), + ), + ).toHaveLength(3); + expect(logged).not.toContain('suspension deadline wake failed'); + }); + + it('does not charge a bookkeeping failure to a resume that succeeded', async () => { + const { storage, values, alarms } = recoveryStorage(); + const { state, fail } = deadlineWriteFailures(storage, { + once: true, + armed: false, + }); + const runner = new TestRunner(state, timedEnv()); + const started = await startTimed( + runner, + 'run-reconcile-fail', + 'timed-escalating', + ); + const armedAt = started.suspendedAt?.gate as number; + elapseDeadlines(values); + const logged: string[] = []; + const log = vi + .spyOn(console, 'error') + .mockImplementation((...args: unknown[]) => { + logged.push(String(args[0])); + }); + + try { + // #when — the resume runs the step and re-suspends it, and only the + // bookkeeping write that records the NEW deadline fails. + fail(); + await runner.alarm(); + } finally { + log.mockRestore(); + } + + // #then — the failure belongs to the reconciliation, not to the resume: + // charging it would record a failed resume that in fact succeeded, and the + // next wake could run the step a second time. + expect(logged).toContain('suspension deadline reconciliation failed'); + expect(logged.some((message) => message.includes('failed (attempt'))).toBe( + false, + ); + const entry = storedDeadlines(values)?.entries[0]; + expect(entry?.suspendedAt).toBe(armedAt); + expect(entry).not.toHaveProperty('attempts'); + // #then — the stale entry still has a wake, and it is not in the past + expect(alarms.at(-1)).toBeGreaterThan(armedAt); + + // #when — nothing else happens to this run: no client call, only its own + // next wake, which is all a suspended run is guaranteed to get. + const recovery = vi.spyOn(console, 'error').mockImplementation(() => {}); + try { + await runner.alarm(); + } finally { + recovery.mockRestore(); + } + + // #then — the escalated suspension the failed write lost is armed now. The + // wake found its own entry stale and reconciled from authoritative state + // instead of merely deleting it, which is what makes the loss temporary + // rather than permanent. + const escalated = storedDeadlines(values)?.entries; + expect(escalated).toHaveLength(1); + expect(escalated?.[0]?.suspendedAt).toBeGreaterThanOrEqual(armedAt); + expect(escalated?.[0]?.resumeCount).toBe(1); + expect(escalated?.[0]?.deadlineAt).toBe( + (escalated?.[0]?.suspendedAt as number) + RETRY_DEADLINE_MS, + ); + const status = await runner.fetch( + deploymentIdentityRequest( + 'http://do/runs/timed-escalating/run-reconcile-fail', + ), + ); + expect(((await status.json()) as RunSummary).status).toBe('suspended'); + }); + + it('does not charge a broadcast failure to a resume that ran the step', async () => { + // #given — a subscribed run-channel socket, and a step whose timeout + // branch completes the run with a value JSON cannot encode. The frame is + // built by JSON.stringify outside safeSend's per-socket tolerance, so + // building it throws after the resume has already run the step. + const events: string[] = []; + const { storage, values } = recoveryStorage(events); + const sent: string[] = []; + const state = { + id: { name: 'timed:run-broadcast-throws' }, + storage, + getWebSockets: () => [ + { + send: (frame: string) => { + sent.push(frame); + }, + }, + ], + } as unknown as DurableObjectState; + const armedAt = Date.now() - TIMED_DEADLINE_MS - 1; + let current: RunSummary = suspendedFence('run-broadcast-throws', armedAt); + const env = makeProductionEnv(); + const resume = vi.fn(async () => { + current = { + runId: 'run-broadcast-throws', + status: 'success', + result: { amount: 10n }, + }; + return current; + }); + env.runtime = { + ...statusStub(async () => current), + resume, + } as unknown as RunnerRuntime; + seedDeadlines(values, 'run-broadcast-throws', [ + armedEntry('gate', armedAt), + ]); + const runner = new TestRunner(state, env); + const logged: string[] = []; + const log = vi + .spyOn(console, 'error') + .mockImplementation((...args: unknown[]) => { + logged.push(String(args[0])); + }); + + // #when — the deadline fires, and the run keeps waking afterwards + try { + for (let wake = 0; wake < 3; wake += 1) { + await expect(runner.alarm()).resolves.toBeUndefined(); + } + } finally { + log.mockRestore(); + } + + // #then — the step ran exactly once. Charging the broadcast would record a + // failed resume that in fact succeeded, and the next wake would find the + // entry still due and run the expired step's body again. + expect(resume).toHaveBeenCalledTimes(1); + expect(sent).toEqual([]); + expect(logged).toContain('suspension deadline broadcast failed'); + expect(logged).not.toContain('suspension deadline wake failed'); + expect(logged.some((message) => message.includes('failed (attempt'))).toBe( + false, + ); + // #then — and the bookkeeping after it still ran: the run is terminal, so + // the record is cleared and the object stops waking for it. + expect(storedDeadlines(values)).toBeUndefined(); + expect(events.at(-1)).toBe('deleteAlarm'); + }); + + it('retries rather than dropping the entry when the run is momentarily unreadable', async () => { + const { state, values, alarms } = recoveryStorage(); + const resume = vi.fn(); + const env = makeProductionEnv(); + // A read replica that has not caught up with a snapshot this object wrote + // looks exactly like "no such run"; treating it as proof that a signal got + // there first would discard a live deadline. + env.runtime = { + ...statusStub(async () => null), + resume, + } as unknown as RunnerRuntime; + seedDeadlines(values, 'run-unreadable', [armedEntry('gate', 1)]); + const runner = new TestRunner(state, env); + const log = vi.spyOn(console, 'error').mockImplementation(() => {}); + + try { + await runner.alarm(); + } finally { + log.mockRestore(); + } + + expect(resume).not.toHaveBeenCalled(); + const entry = storedDeadlines(values)?.entries[0]; + expect(entry?.attempts).toBe(1); + expect(alarms.at(-1)).toBe(entry?.nextAttemptAt); + }); + + it('drops the entry when only the resumeCount fence moved', async () => { + const { state, values } = recoveryStorage(); + const resume = vi.fn(); + const env = makeProductionEnv(); + // Same step, same suspension time, one resume further on: a real signal + // resumed and re-suspended within the same millisecond. + env.runtime = { + ...statusStub(async () => ({ + ...suspendedFence('run-ordinal', 1), + resumeCount: { gate: 1 }, + })), + resume, + } as unknown as RunnerRuntime; + seedDeadlines(values, 'run-ordinal', [armedEntry('gate', 1)]); + const runner = new TestRunner(state, env); + const log = vi.spyOn(console, 'error').mockImplementation(() => {}); + + try { + await runner.alarm(); + } finally { + log.mockRestore(); + } + + expect(resume).not.toHaveBeenCalled(); + // #then — the ordinal alone settles it: the entry armed at resumeCount 0 is + // gone, replaced by one fenced on the resume that superseded it. + expect(storedDeadlines(values)?.entries).toEqual([ + { ...armedEntry('gate', 1), resumeCount: 1 }, + ]); + }); + + it('resumes one due entry per wake and keeps the other armed', async () => { + const { state, values, alarms } = recoveryStorage(); + const gateSuspendedAt = Date.now() - TIMED_DEADLINE_MS - 2; + const otherSuspendedAt = gateSuspendedAt + 1; + const bothSuspended: RunSummary = { + runId: 'run-pair', + status: 'suspended', + suspended: [['gate'], ['other']], + suspendPayload: { + gate: { [SUSPENSION_DEADLINE_PAYLOAD_KEY]: TIMED_DEADLINE_MS }, + other: { [SUSPENSION_DEADLINE_PAYLOAD_KEY]: TIMED_DEADLINE_MS }, + }, + suspendedAt: { gate: gateSuspendedAt, other: otherSuspendedAt }, + }; + const resumedSteps: unknown[] = []; + const env = makeProductionEnv(); + env.runtime = { + ...statusStub(async () => bothSuspended), + resume: vi.fn( + async (_workflowId, _runId, options: { step?: unknown }) => { + resumedSteps.push(options.step); + // The step the first wake did not touch is still suspended, so its + // deadline must survive the wake that consumed the other one. + return resumedSteps.length === 1 + ? { ...bothSuspended, suspended: [['other']] } + : { runId: 'run-pair', status: 'success' }; + }, + ), + } as unknown as RunnerRuntime; + seedDeadlines(values, 'run-pair', [ + armedEntry('gate', gateSuspendedAt), + armedEntry('other', otherSuspendedAt), + ]); + const runner = new TestRunner(state, env); + const before = Date.now(); + + await runner.alarm(); + + // #then — bounded work per wake: the earliest entry only, and the entry + // left behind is re-armed at the floor rather than at its own past + // deadline, which Cloudflare would fire immediately. + expect(resumedSteps).toEqual([['gate']]); + expect(storedDeadlines(values)?.entries).toEqual([ + armedEntry('other', otherSuspendedAt), + ]); + expect(alarms.at(-1)).toBeGreaterThan(before); + expect(alarms.at(-1)).toBeLessThanOrEqual(Date.now() + 1_000); + + await runner.alarm(); + + expect(resumedSteps).toEqual([['gate'], ['other']]); + expect(storedDeadlines(values)).toBeUndefined(); + }); + + it('serves the deadline duty in a wake whose recovery journal is poisoned', async () => { + const { state, values, alarms } = recoveryStorage(); + const runner = new TestRunner(state, timedEnv()); + await startTimed(runner, 'run-both-duties'); + elapseDeadlines(values); + // A journal this instance cannot parse: run-owner recovery throws, and its + // throw is what asks workerd to retry the wake. The deadline duty must + // still have run, or one poisoned journal would strand every deadline the + // run ever arms. + values.set('flowsafe:run-owner-recovery:v1', { version: 2 }); + const before = Date.now(); + + await expect(runner.alarm()).rejects.toThrow( + 'stored run owner recovery is malformed', + ); + + const status = await runner.fetch( + deploymentIdentityRequest('http://do/runs/timed/run-both-duties'), + ); + expect(await status.json()).toMatchObject({ + status: 'success', + result: { settledBy: 'timeout' }, + requestedBy: SUSPENSION_DEADLINE_PRINCIPAL_ID, + }); + expect(storedDeadlines(values)).toBeUndefined(); + // #then — the recovery duty still owns its own retry wake + expect(alarms.at(-1)).toBeGreaterThanOrEqual(before + 60_000); + }); + + it('runs the expired step body once when a real resume races the wake', async () => { + const { state, values } = recoveryStorage(); + const env = makeProductionEnv(); + const counted = countedTimedRuntime(env.storage); + env.runtime = counted.runtime; + const runner = new TestRunner(state, env); + await startTimed(runner, 'run-race'); + elapseDeadlines(values); + + // #when — the wake and a genuine signal arrive together. The operation + // lock is a FIFO mutex held across the whole alarm body, so the resume + // cannot land between the fence check and the timeout resume. + const wake = runner.alarm(); + const signal = runner.fetch( + post('/runs/timed/run-race/resume', { + step: 'gate', + resumeData: { approvedBy: 'bob' }, + }), + ); + const [, response] = await Promise.all([wake, signal]); + + // #then — the gated action ran exactly once, and the loser is told so + expect(counted.settled()).toBe(1); + expect(response.status).toBe(409); + expect(storedDeadlines(values)).toBeUndefined(); + }); + + it('arms, fences and resumes a top-level step id containing a dot', async () => { + const { state, values } = recoveryStorage(); + const runner = new TestRunner(state, timedEnv()); + + const started = await startTimed(runner, 'run-dotted', 'timed-dotted'); + const suspendedAt = started.suspendedAt?.[DOTTED_STEP] as number; + + // #then — the live summary reports the id whole, and the entry is keyed by + // the same joined path the payload and the fence are keyed by. + expect(started.suspended).toEqual([[DOTTED_STEP]]); + expect(storedDeadlines(values)?.entries).toEqual([ + armedEntry(DOTTED_STEP, suspendedAt), + ]); + + // #when — the wake fences against the REHYDRATED projection, which splits + // that id back into segments. + const rehydrated = await runner.fetch( + deploymentIdentityRequest('http://do/runs/timed-dotted/run-dotted'), + ); + expect(((await rehydrated.json()) as RunSummary).suspended).toEqual([ + ['wait', 'signal'], + ]); + elapseDeadlines(values); + const logged: string[] = []; + const log = vi + .spyOn(console, 'error') + .mockImplementation((...args: unknown[]) => { + logged.push(String(args[0])); + }); + + try { + await runner.alarm(); + } finally { + log.mockRestore(); + } + + // #then — the deadline fired: the entry was recognized, not treated as a + // moved fence and dropped, which is how it used to disappear in silence. + const status = await runner.fetch( + deploymentIdentityRequest('http://do/runs/timed-dotted/run-dotted'), + ); + expect(await status.json()).toMatchObject({ + status: 'success', + result: { settledBy: 'timeout' }, + requestedBy: SUSPENSION_DEADLINE_PRINCIPAL_ID, + }); + expect(logged).toEqual([]); + expect(storedDeadlines(values)).toBeUndefined(); + }); + + it('arms the deadline of a run whose start was interrupted', async () => { + const events: string[] = []; + const { state, values } = recoveryStorage(events); + const env = timedEnv(); + const token = 'attempt-token'; + // The start leg as an interrupted one leaves it: the claim is reserved and + // journaled, Mastra has persisted the suspension, and the isolate died + // before the route could reconcile. Recovery is the only boundary left. + await env.owners.reserveAll( + [{ kind: 'run', resourceId: 'run-interrupted' }], + OWNER_PRINCIPAL, + token, + ); + const started = await (env.runtime as RunnerRuntime).start('timed', { + runId: 'run-interrupted', + inputData: { topic: 't' }, + requestedBy: OWNER_PRINCIPAL.id, + requestedByKind: OWNER_PRINCIPAL.kind, + attemptToken: token, + }); + expect(started.status).toBe('suspended'); + values.set('flowsafe:run-owner-recovery:v1', { + version: 1, + workflowId: 'timed', + runId: 'run-interrupted', + token, + }); + const runner = new TestRunner(state, env); + + await runner.alarm(); + + // #then — the recovery wake armed from the summary recoverStartAttempt read + // back. Without it the journal clears, #armNextAlarm finds no record and + // DELETES the alarm, and a suspended run is left with no wake at all. + expect(values.has('flowsafe:run-owner-recovery:v1')).toBe(false); + expect(storedDeadlines(values)?.entries).toEqual([ + armedEntry('gate', started.suspendedAt?.gate as number), + ]); + expect(events.at(-1)).not.toBe('deleteAlarm'); + }); + + it('keeps the recovery cadence when a wake cannot verify its deployment', async () => { + const { state, values, alarms } = recoveryStorage(); + const env = timedEnv(); + // A namespace bound to another deployment's database: verification throws on + // every wake, so the deadline duty never runs and its ledger can never + // converge. Arming at the past deadline would then wake this object every + // second for as long as the misbinding lasts. + env.DB = deploymentIdentityDatabase('globex'); + const armed = armedEntry('gate', 1); + seedDeadlines(values, 'run-misbound', [armed]); + const runner = new TestRunner(state, env); + const before = Date.now(); + + for (let wake = 0; wake < 3; wake += 1) { + await expect(runner.alarm()).rejects.toThrow("belongs to 'globex'"); + } + + // #then — every arm is the pre-feature 60s recovery watchdog, never the + // one-second arm floor, and the untouched entry is still armed for when the + // binding is fixed. + expect(alarms).toHaveLength(3); + for (const at of alarms) { + expect(at).toBeGreaterThanOrEqual(before + 60_000); + } + expect(storedDeadlines(values)?.entries).toEqual([armed]); + }); + + it('charges the entry the wake was working on when its own re-arm fails', async () => { + const { storage, values } = recoveryStorage(); + let setAlarmCalls = 0; + const state = { + storage: { + ...storage, + setAlarm: async (at: number | Date) => { + setAlarmCalls += 1; + // 1 = the watchdog, 2 = the re-arm inside the FIRST entry's charge. + if (setAlarmCalls === 2) throw new Error('transient alarm failure'); + return storage.setAlarm?.(at); + }, + }, + } as unknown as DurableObjectState; + const gateSuspendedAt = Date.now() - TIMED_DEADLINE_MS - 2; + const otherSuspendedAt = gateSuspendedAt + 1; + const env = makeProductionEnv(); + env.runtime = { + ...statusStub(async () => ({ + runId: 'run-pair', + status: 'suspended', + suspended: [['gate'], ['other']], + suspendPayload: { + gate: { [SUSPENSION_DEADLINE_PAYLOAD_KEY]: TIMED_DEADLINE_MS }, + other: { [SUSPENSION_DEADLINE_PAYLOAD_KEY]: TIMED_DEADLINE_MS }, + }, + suspendedAt: { gate: gateSuspendedAt, other: otherSuspendedAt }, + })), + resume: vi.fn(async () => { + throw new Error('injected resume failure for gate'); + }), + } as unknown as RunnerRuntime; + seedDeadlines(values, 'run-pair', [ + armedEntry('gate', gateSuspendedAt), + armedEntry('other', otherSuspendedAt), + ]); + const runner = new TestRunner(state, env); + const logged: string[] = []; + const log = vi + .spyOn(console, 'error') + .mockImplementation((...args: unknown[]) => { + logged.push(String(args[0])); + }); + + try { + await expect(runner.alarm()).resolves.toBeUndefined(); + } finally { + log.mockRestore(); + } + + // #then — 'gate' failed, so 'gate' pays. Re-selecting a due entry after the + // failure would charge 'other' for a resume it never attempted: pushed a + // minute late, a fifth of its budget spent, and named in the operator log. + const entries = storedDeadlines(values)?.entries ?? []; + expect(entries.find((entry) => entry.step === 'other')).toEqual( + armedEntry('other', otherSuspendedAt), + ); + expect(entries.find((entry) => entry.step === 'gate')?.attempts).toBe(1); + expect( + logged.filter((message) => message.includes('failed (attempt 1)')), + ).toEqual([ + "suspension deadline wake for step 'gate' of run 'run-pair' failed (attempt 1)", + ]); + }); + + it('touches no deadline storage for a run that arms nothing', async () => { + const events: string[] = []; + const { state } = recoveryStorage(events); + const runner = new TestRunner(state, makeProductionEnv()); + const started = await startGated(runner); + + await runner.fetch( + post(`/runs/gated/${started.runId}/resume`, { + step: 'gate', + resumeData: { approvedBy: 'bob' }, + }), + ); + await runner.fetch(post(`/runs/gated/${started.runId}/terminate`, {})); + + // #then — a consumer that never arms a deadline pays reads, never writes: + // an unconditional delete at each lifecycle boundary would charge every + // existing runner user for a feature it does not use. + expect( + events.filter( + (event) => + event.endsWith(SUSPENSION_DEADLINE_STORAGE_KEY) && + !event.startsWith('get:'), + ), + ).toEqual([]); + }); + + it('reports an unarmable deadline once, not at every boundary', async () => { + const { state } = recoveryStorage(); + const runner = new TestRunner(state, timedEnv()); + const logged: string[] = []; + const log = vi + .spyOn(console, 'error') + .mockImplementation((...args: unknown[]) => { + logged.push(String(args[0])); + }); + + try { + // The step re-suspends with the same out-of-range value on every resume, + // so start and resume derive the identical rejection. + const started = await startTimed( + runner, + 'run-unarmable', + 'timed-unarmable', + ); + expect(started.status).toBe('suspended'); + const resumed = await runner.fetch( + post('/runs/timed-unarmable/run-unarmable/resume', { + step: 'gate', + resumeData: { approvedBy: 'bob' }, + }), + ); + expect(resumed.status).toBe(200); + } finally { + log.mockRestore(); + } + + // #then — the operator is told once. Repeating an unchanged rejection at + // every boundary buries the rest of the log under one author mistake. + expect( + logged.filter((message) => + message.startsWith('suspension deadline not armed for step'), + ), + ).toEqual([ + `suspension deadline not armed for step 'gate' of run 'run-unarmable': ${SUSPENSION_DEADLINE_PAYLOAD_KEY} must be between ${MIN_SUSPENSION_DEADLINE_MS} and ${MAX_SUSPENSION_DEADLINE_MS} ms`, + ]); + }); + + it('refuses a client resume that forges the timeout envelope', async () => { + const runner = makeRunner(); + const started = await startGated(runner); + + const response = await runner.fetch( + post(`/runs/gated/${started.runId}/resume`, { + step: 'gate', + resumeData: { + [SUSPENSION_TIMEOUT_RESUME_KEY]: { + step: 'gate', + deadlineAt: 1, + expiredAt: 2, + }, + }, + }), + ); + + // #then — a caller allowed to resume must not be able to drive the timeout + // branch while provenance still names them as the requester + expect(response.status).toBe(400); + expect((await response.json()) as { error: string }).toMatchObject({ + error: expect.stringContaining(SUSPENSION_TIMEOUT_RESUME_KEY), + }); + const status = await runner.fetch( + deploymentIdentityRequest(`http://do/runs/gated/${started.runId}`), + ); + expect(((await status.json()) as RunSummary).status).toBe('suspended'); + }); + + it('leaves a retry wake when the first deadline write of a start fails', async () => { + const events: string[] = []; + const { storage, values, alarms } = recoveryStorage(events); + const { state } = deadlineWriteFailures(storage, { + name: 'timed:run-first-arm', + once: true, + }); + const runner = new TestRunner(state, timedEnv()); + const log = vi.spyOn(console, 'error').mockImplementation(() => {}); + const before = Date.now(); + let started: RunSummary; + + try { + started = await startTimed(runner, 'run-first-arm'); + } finally { + log.mockRestore(); + } + const suspendedAt = started.suspendedAt?.gate as number; + + // #then — the start succeeded, the run is suspended with a derivable + // deadline, and nothing recorded it. Settling the reservation used to + // happen first and re-arm from storage, finding neither record nor journal + // and DELETING the alarm: no record, no wake, deadline lost forever. + expect(started.status).toBe('suspended'); + expect(storedDeadlines(values)).toBeUndefined(); + expect(events).not.toContain('deleteAlarm'); + expect(alarms.at(-1)).toBeGreaterThanOrEqual(before + 60_000); + + // #when — that retry wake fires, which is all this run is guaranteed + const recovery = vi.spyOn(console, 'error').mockImplementation(() => {}); + try { + await runner.alarm(); + } finally { + recovery.mockRestore(); + } + + // #then — with no record to work from, the wake re-derived from the run's + // own authoritative state and armed what the failed write lost + expect(storedDeadlines(values)?.entries).toEqual([ + armedEntry('gate', suspendedAt), + ]); + expect(alarms.at(-1)).toBe(suspendedAt + TIMED_DEADLINE_MS); + }); + + it('leaves a retry wake when a resume boundary is the first arm and it fails', async () => { + const events: string[] = []; + const { storage, values, alarms } = recoveryStorage(events); + const { state, stop } = deadlineWriteFailures(storage, { + name: 'timed-relay:run-relay', + }); + const runner = new TestRunner(state, timedEnv()); + const log = vi.spyOn(console, 'error').mockImplementation(() => {}); + const before = Date.now(); + let resumed: RunSummary; + + try { + await startTimed(runner, 'run-relay', 'timed-relay'); + // #when — a real signal resumes and the step suspends again. Nothing was + // ever armed for this run, so unlike a later boundary there is no stale + // record whose own alarm would heal the write that fails here. + const response = await runner.fetch( + post('/runs/timed-relay/run-relay/resume', { + step: 'gate', + resumeData: { approvedBy: 'bob' }, + }), + ); + expect(response.status).toBe(200); + resumed = (await response.json()) as RunSummary; + } finally { + log.mockRestore(); + } + const suspendedAt = resumed.suspendedAt?.gate as number; + + expect(resumed.status).toBe('suspended'); + expect(storedDeadlines(values)).toBeUndefined(); + expect(events).not.toContain('deleteAlarm'); + expect(alarms.at(-1)).toBeGreaterThanOrEqual(before + 60_000); + + stop(); + await runner.alarm(); + + // #then — the wake armed the re-suspension's own deadline, fenced on the + // resume ordinal the signal produced + expect(storedDeadlines(values)?.entries).toEqual([ + { + step: 'gate', + deadlineAt: suspendedAt + RETRY_DEADLINE_MS, + suspendedAt, + resumeCount: 1, + }, + ]); + expect(alarms.at(-1)).toBe(suspendedAt + RETRY_DEADLINE_MS); + }); + + it('keeps a wake when an interrupted start cannot record its deadline', async () => { + const events: string[] = []; + const { storage, values, alarms } = recoveryStorage(events); + const { state, stop } = deadlineWriteFailures(storage, { + name: 'timed:run-recovery-arm', + }); + const env = timedEnv(); + const token = 'attempt-token'; + await env.owners.reserveAll( + [{ kind: 'run', resourceId: 'run-recovery-arm' }], + OWNER_PRINCIPAL, + token, + ); + const started = await (env.runtime as RunnerRuntime).start('timed', { + runId: 'run-recovery-arm', + inputData: { topic: 't' }, + requestedBy: OWNER_PRINCIPAL.id, + requestedByKind: OWNER_PRINCIPAL.kind, + attemptToken: token, + }); + values.set('flowsafe:run-owner-recovery:v1', { + version: 1, + workflowId: 'timed', + runId: 'run-recovery-arm', + token, + }); + const runner = new TestRunner(state, env); + const log = vi.spyOn(console, 'error').mockImplementation(() => {}); + const before = Date.now(); + + try { + await runner.alarm(); + } finally { + log.mockRestore(); + } + + // #then — recovery cleared the journal it exists to clear, and left the + // wake the failed write needs instead of re-arming from a record that was + // never written, which would have deleted the alarm outright + expect(values.has('flowsafe:run-owner-recovery:v1')).toBe(false); + expect(storedDeadlines(values)).toBeUndefined(); + expect(events).not.toContain('deleteAlarm'); + expect(alarms.at(-1)).toBeGreaterThanOrEqual(before + 60_000); + + stop(); + await runner.alarm(); + + const suspendedAt = started.suspendedAt?.gate as number; + expect(storedDeadlines(values)?.entries).toEqual([ + armedEntry('gate', suspendedAt), + ]); + }); + + it('writes nothing on a no-record wake for a run that is not suspended', async () => { + const events: string[] = []; + const { storage, values } = recoveryStorage(events); + const state = { + id: { name: 'timed:run-finished' }, + storage, + } as unknown as DurableObjectState; + const env = makeProductionEnv(); + const reads = statusStub(async () => ({ + runId: 'run-finished', + status: 'success', + })); + env.runtime = { ...reads } as unknown as RunnerRuntime; + const runner = new TestRunner(state, env); + + await runner.alarm(); + + // #then — the wake re-derives from the run this object is named for, and a + // run that is not suspended derives nothing: no write, and no alarm left + // for an object with no duty pending. + expect(reads.authoritativeStatus).toHaveBeenCalledWith( + 'timed', + 'run-finished', + ); + expect(storedDeadlines(values)).toBeUndefined(); + expect( + events.filter( + (event) => + event.endsWith(SUSPENSION_DEADLINE_STORAGE_KEY) && + !event.startsWith('get:'), + ), + ).toEqual([]); + // #then — the record key is read exactly TWICE: the duty's own read, and + // the converged arm at the end of the body. The reconcile in between pays + // no read of its own, because it is told this wake already looked and + // found nothing — which `??` could not tell from "no record supplied". + expect( + events.filter( + (event) => event === `get:${SUSPENSION_DEADLINE_STORAGE_KEY}`, + ), + ).toHaveLength(2); + expect(events.at(-1)).toBe('deleteAlarm'); + }); + + it('keeps the recovery cadence when the retry ledger itself cannot be written', async () => { + const { storage, values, alarms } = recoveryStorage(); + const { state } = deadlineWriteFailures(storage); + const armedAt = Date.now() - TIMED_DEADLINE_MS - 1; + const armed = armedEntry('gate', armedAt); + const env = makeProductionEnv(); + env.runtime = { + ...statusStub(async () => suspendedFence('run-ledger', armedAt)), + resume: vi.fn(async () => { + throw new Error('injected resume failure'); + }), + } as unknown as RunnerRuntime; + seedDeadlines(values, 'run-ledger', [armed]); + const runner = new TestRunner(state, env); + const logged: string[] = []; + const log = vi + .spyOn(console, 'error') + .mockImplementation((...args: unknown[]) => { + logged.push(String(args[0])); + }); + const before = Date.now(); + + try { + for (let wake = 0; wake < 3; wake += 1) { + await expect(runner.alarm()).resolves.toBeUndefined(); + } + } finally { + log.mockRestore(); + } + + // #then — the ledger that would back this entry off is the write that + // keeps failing, so the wake converges nothing and keeps the recovery + // cadence. Arming from a still-due entry instead would land on the + // one-second floor and wake this object every second while storage is out. + expect(alarms.filter((at) => at < before + 60_000)).toEqual([]); + expect(logged).toContain('suspension deadline ledger update failed'); + expect(storedDeadlines(values)?.entries).toEqual([armed]); + }); + + it('keeps the recovery cadence when an unwritable ledger shares a wake with a poisoned journal', async () => { + const { storage, values, alarms } = recoveryStorage(); + const { state } = deadlineWriteFailures(storage); + const armedAt = Date.now() - TIMED_DEADLINE_MS - 1; + const env = makeProductionEnv(); + env.runtime = { + ...statusStub(async () => suspendedFence('run-poisoned-ledger', armedAt)), + resume: vi.fn(async () => { + throw new Error('injected resume failure'); + }), + } as unknown as RunnerRuntime; + // The recovery duty rethrows to ask workerd for a retry, and the re-arm on + // that path takes the min with the stored entry — so it is the second way + // into the same one-second spin. + values.set('flowsafe:run-owner-recovery:v1', { version: 2 }); + seedDeadlines(values, 'run-poisoned-ledger', [armedEntry('gate', armedAt)]); + const runner = new TestRunner(state, env); + const log = vi.spyOn(console, 'error').mockImplementation(() => {}); + const before = Date.now(); + + try { + for (let wake = 0; wake < 3; wake += 1) { + await expect(runner.alarm()).rejects.toThrow( + 'stored run owner recovery is malformed', + ); + } + } finally { + log.mockRestore(); + } + + expect(alarms.filter((at) => at < before + 60_000)).toEqual([]); + }); + + it('arms nothing and runs no step body when two suspensions share one key', async () => { + const { storage, values } = recoveryStorage(); + const state = { + id: { name: 'timed-collision:run-collision' }, + storage, + } as unknown as DurableObjectState; + const env = makeProductionEnv(); + const colliding = collidingRuntime(env.storage); + env.runtime = colliding.runtime; + const runner = new TestRunner(state, env); + const logged: string[] = []; + const log = vi + .spyOn(console, 'error') + .mockImplementation((...args: unknown[]) => { + logged.push(String(args[0])); + }); + + try { + const started = await startTimed( + runner, + 'run-collision', + 'timed-collision', + ); + expect(started.status).toBe('suspended'); + // #when — the only wake this run gets fires, and it reads the OTHER + // projection: the live boundary above saw two paths joining to 'a.b', + // this one sees 'a.b' and the nested 'a' whose marker implies it. + await runner.alarm(); + } finally { + log.mockRestore(); + } + + // #then — refused on both projections, so no entry exists to deliver one + // suspension's timeout envelope to the other suspension's step + expect(storedDeadlines(values)).toBeUndefined(); + expect(colliding.settled()).toEqual([]); + expect(logged).toContain( + "suspension deadline not armed for step 'a.b' of run 'run-collision': ambiguous suspended step path", + ); + }); + + it('resumes one foreach iteration per wake and re-arms from the new fence', async () => { + const { state, values, alarms } = recoveryStorage(); + const env = makeProductionEnv(); + const each = foreachRuntime(env.storage, 'timed-foreach'); + env.runtime = each.runtime; + const runner = new TestRunner(state, env); + const before = Date.now(); + + const response = await runner.fetch( + post('/runs', { + workflowId: 'timed-foreach', + runId: 'run-foreach', + inputData: [{ item: 0 }, { item: 1 }], + }), + ); + expect(((await response.json()) as RunSummary).status).toBe('suspended'); + elapseDeadlines(values); + + await runner.alarm(); + + // #then — a default foreach is sequential: one iteration is suspended at a + // time, so the expired deadline belongs to iteration 0, and the iteration + // that follows suspends as a NEW suspension with its own fence and its own + // deadline. Clearing a whole foreach by timeout therefore takes one + // deadline per item. + expect(each.timedOut()).toEqual([0]); + const entry = storedDeadlines(values)?.entries[0]; + expect(entry?.resumeCount).toBe(1); + expect(entry?.deadlineAt).toBe( + (entry?.suspendedAt as number) + TIMED_DEADLINE_MS, + ); + expect(alarms.at(-1)).toBe(entry?.deadlineAt); + + await runner.alarm(); + await runner.alarm(); + + // #then — the consumed suspension never fires again, and no wake lands on + // the one-second arm floor, which is what a repeat fire would look like + expect(each.timedOut()).toEqual([0]); + expect(storedDeadlines(values)?.entries).toEqual([entry]); + expect(alarms.filter((at) => at <= before)).toEqual([]); + }); + + it('resumes every suspended iteration of a concurrent foreach in one wake', async () => { + const { state, values } = recoveryStorage(); + const env = makeProductionEnv(); + const each = foreachRuntime(env.storage, 'timed-foreach-concurrent', { + concurrency: 3, + }); + env.runtime = each.runtime; + const runner = new TestRunner(state, env); + + const response = await runner.fetch( + post('/runs', { + workflowId: 'timed-foreach-concurrent', + runId: 'run-foreach-concurrent', + inputData: [{ item: 0 }, { item: 1 }, { item: 2 }], + }), + ); + expect(((await response.json()) as RunSummary).status).toBe('suspended'); + // #then — all three iterations are suspended, and both projections report + // ONE path with one fence and iteration 0's payload, so one entry is armed + // for the whole foreach — at ITERATION 0's deadline, even though every + // iteration asked for a different one. + const armed = storedDeadlines(values)?.entries; + expect(armed).toHaveLength(1); + expect(armed?.[0]?.deadlineAt).toBe( + (armed?.[0]?.suspendedAt as number) + TIMED_DEADLINE_MS, + ); + elapseDeadlines(values); + + await runner.alarm(); + await runner.alarm(); + + // #then — that one resume delivers the envelope to every iteration still + // suspended, so the foreach clears in a single wake rather than one + // iteration at a time, and the settled entry cannot fire again + expect(each.timedOut()).toEqual([0, 1, 2]); + expect(storedDeadlines(values)).toBeUndefined(); + const status = await runner.fetch( + deploymentIdentityRequest( + 'http://do/runs/timed-foreach-concurrent/run-foreach-concurrent', + ), + ); + expect(((await status.json()) as RunSummary).status).toBe('success'); + }); + + it('clears a concurrent foreach with more items than concurrency batch by batch', async () => { + const { state, values } = recoveryStorage(); + const env = makeProductionEnv(); + const each = foreachRuntime(env.storage, 'timed-foreach-batched', { + concurrency: 2, + }); + env.runtime = each.runtime; + const runner = new TestRunner(state, env); + + const response = await runner.fetch( + post('/runs', { + workflowId: 'timed-foreach-batched', + runId: 'run-foreach-batched', + inputData: [{ item: 0 }, { item: 1 }, { item: 2 }], + }), + ); + expect(((await response.json()) as RunSummary).status).toBe('suspended'); + const firstFence = storedDeadlines(values)?.entries[0]; + elapseDeadlines(values); + + await runner.alarm(); + + // #then — the timeout resume delivered the envelope to every iteration + // suspended AT THAT MOMENT: the first batch of `concurrency` items, not + // the whole foreach. Iteration 2 had not started, so it suspends afresh + // with its own fence and its own deadline, and the run stays suspended. + expect(each.timedOut()).toEqual([0, 1]); + const entry = storedDeadlines(values)?.entries[0]; + expect(entry?.resumeCount).toBe(1); + expect(entry).not.toEqual(firstFence); + // #then — and the new entry carries the value ITERATION 2 asked for: each + // batch reads the deadline of the first iteration suspended in it, not the + // one the first batch was armed with. + expect(firstFence?.deadlineAt).toBe( + (firstFence?.suspendedAt as number) + TIMED_DEADLINE_MS, + ); + expect(entry?.deadlineAt).toBe( + (entry?.suspendedAt as number) + TIMED_DEADLINE_MS + 2 * 60_000, + ); + const mid = await runner.fetch( + deploymentIdentityRequest( + 'http://do/runs/timed-foreach-batched/run-foreach-batched', + ), + ); + expect(((await mid.json()) as RunSummary).status).toBe('suspended'); + elapseDeadlines(values); + + await runner.alarm(); + + // #then — clearing 3 items at concurrency 2 took ceil(3 / 2) deadlines + expect(each.timedOut()).toEqual([0, 1, 2]); + expect(storedDeadlines(values)).toBeUndefined(); + const status = await runner.fetch( + deploymentIdentityRequest( + 'http://do/runs/timed-foreach-batched/run-foreach-batched', + ), + ); + expect(((await status.json()) as RunSummary).status).toBe('success'); + }); + + it.each([ + ['a path-unsafe segment', 'a/b:c'], + ['three segments', 'timed:run:extra'], + ['no separator', 'nocolon'], + ['an empty runId', 'timed:'], + ['an empty workflowId', ':run'], + ])('never lets an object name with %s steer a status read on a no-record wake', async (_label, name) => { + // 'a/b:c' used to reach status('a/b', 'c'): every other entry point + // validates with isPathSafeId before touching the runtime, and a record + // written from an unvalidated name would discard itself on read-back. + // The other four shapes were already skipped — regression pins. + const events: string[] = []; + const { storage } = recoveryStorage(events); + const state = { id: { name }, storage } as unknown as DurableObjectState; + const env = makeProductionEnv(); + const reads = statusStub(async () => null); + env.runtime = { ...reads } as unknown as RunnerRuntime; + const runner = new TestRunner(state, env); + + await runner.alarm(); + + // The wake's own read is authoritativeStatus, so a pin left on `status` + // would pass however the name were used. + expect(reads.authoritativeStatus).not.toHaveBeenCalled(); + expect(reads.status).not.toHaveBeenCalled(); + expect(events.at(-1)).toBe('deleteAlarm'); + }); + + it('converges to no alarm on a no-record wake without an object name', async () => { + const events: string[] = []; + const { state, values } = recoveryStorage(events); + const env = makeProductionEnv(); + const reads = statusStub(async () => null); + env.runtime = { ...reads } as unknown as RunnerRuntime; + const runner = new TestRunner(state, env); + + await runner.alarm(); + + // #then — without a name and without a record there is no identity to + // re-derive from, so the retry wake ends in a delete and a deadline whose + // record was never written is lost. Production is safe from this: the + // exported topology always addresses the runner object by idFromName. + expect(reads.authoritativeStatus).not.toHaveBeenCalled(); + expect(reads.status).not.toHaveBeenCalled(); + expect(storedDeadlines(values)).toBeUndefined(); + expect(events.at(-1)).toBe('deleteAlarm'); + }); + + it('keeps the record and its wake when a nothing-due wake reads a degraded summary', async () => { + const { state, values, alarms } = recoveryStorage(); + const env = makeProductionEnv(); + // Mastra's in-memory fallback: storage unavailable while the isolate still + // holds the run — 'suspended' with no suspended paths and no fences. + const reads = statusStub(async () => ({ + runId: 'run-degraded', + status: 'suspended', + suspended: [], + })); + const resume = vi.fn(); + env.runtime = { ...reads, resume } as unknown as RunnerRuntime; + const entry = armedEntry('gate', Date.now()); + seedDeadlines(values, 'run-degraded', [entry]); + const logged: string[] = []; + const log = vi + .spyOn(console, 'error') + .mockImplementation((...args: unknown[]) => { + logged.push(String(args[0])); + }); + const runner = new TestRunner(state, env); + + try { + await runner.alarm(); + } finally { + log.mockRestore(); + } + + // #then — a degraded read is not evidence: reconciling from it would have + // derived nothing and wiped the record of a run that is still suspended. + // The record stays, the alarm re-arms at the entry's own deadline, and the + // operator is told once. + expect(reads.authoritativeStatus).toHaveBeenCalledWith( + 'timed', + 'run-degraded', + ); + // A silent degradation to the watchdog path would look the same from the + // record alone, so the resume is pinned too: nothing ran. + expect(resume).not.toHaveBeenCalled(); + expect(storedDeadlines(values)?.entries).toEqual([entry]); + expect(alarms.at(-1)).toBe(entry.deadlineAt); + // #then — under its own wording: this refusal charges nothing, and the + // resume path's identically-shaped refusal charges the entry's budget, so + // the two must not read the same in an operator's log. + expect( + logged.filter((message) => message.includes('self-inconsistent')), + ).toEqual([ + "run 'run-degraded' of workflow 'timed' state read back self-inconsistent; keeping the record", + ]); + }); + + it('charges the ledger instead of wiping the record when a due wake reads a degraded summary', async () => { + const { state, values } = recoveryStorage(); + const env = makeProductionEnv(); + const reads = statusStub(async () => ({ + runId: 'run-degraded-due', + status: 'suspended', + suspended: [], + })); + const resume = vi.fn(); + env.runtime = { ...reads, resume } as unknown as RunnerRuntime; + const due = armedEntry('gate', Date.now() - TIMED_DEADLINE_MS - 1); + const unrelated = armedEntry('other', Date.now()); + seedDeadlines(values, 'run-degraded-due', [due, unrelated]); + const logged: string[] = []; + const log = vi + .spyOn(console, 'error') + .mockImplementation((...args: unknown[]) => { + logged.push(String(args[0])); + }); + const before = Date.now(); + const runner = new TestRunner(state, env); + + try { + await runner.alarm(); + } finally { + log.mockRestore(); + } + + // #then — the degraded summary matches no fence, so without the guard the + // 'moved on' reconcile would have deleted the due entry AND the unrelated + // far-future one. Instead the wake is charged as unreadable and retried. + expect(resume).not.toHaveBeenCalled(); + expect(logged.some((message) => message.includes('has moved on'))).toBe( + false, + ); + const entries = storedDeadlines(values)?.entries ?? []; + expect(entries.find((entry) => entry.step === 'other')).toEqual(unrelated); + const charged = entries.find((entry) => entry.step === 'gate'); + expect(charged?.attempts).toBe(1); + expect(charged?.nextAttemptAt).toBeGreaterThanOrEqual(before + 59_000); + }); + + it('re-arms to the current suspension when a retry wake holds only a stale record', async () => { + const { storage, values, alarms } = recoveryStorage(); + const { state, fail } = deadlineWriteFailures(storage, { + name: 'timed-relay-shortening:run-shortening', + once: true, + armed: false, + }); + const runner = new TestRunner(state, timedEnv()); + const started = await startTimed( + runner, + 'run-shortening', + 'timed-relay-shortening', + ); + const firstSuspendedAt = started.suspendedAt?.gate as number; + const staleDeadlineAt = firstSuspendedAt + RETRY_DEADLINE_MS; + expect(storedDeadlines(values)?.entries).toEqual([ + { ...armedEntry('gate', firstSuspendedAt), deadlineAt: staleDeadlineAt }, + ]); + const log = vi.spyOn(console, 'error').mockImplementation(() => {}); + let resumed: RunSummary; + + try { + // #when — a real signal re-suspends the step with a SHORTER deadline and + // the write recording it fails once: the record still holds the old + // far-future entry, and the failed reconcile left a 60 s retry wake. + fail(); + const response = await runner.fetch( + post('/runs/timed-relay-shortening/run-shortening/resume', { + step: 'gate', + resumeData: { approvedBy: 'bob' }, + }), + ); + expect(response.status).toBe(200); + resumed = (await response.json()) as RunSummary; + + await runner.alarm(); + } finally { + log.mockRestore(); + } + + // #then — the retry wake found nothing due and re-derived anyway: without + // that, it would have re-armed to the stale far-future entry and the + // 15-minute deadline would never fire. + const suspendedAt = resumed.suspendedAt?.gate as number; + expect(storedDeadlines(values)?.entries).toEqual([ + { ...armedEntry('gate', suspendedAt), resumeCount: 1 }, + ]); + expect(alarms.at(-1)).toBe(suspendedAt + TIMED_DEADLINE_MS); + expect(alarms.at(-1)).toBeLessThan(staleDeadlineAt); + }); + + it('arms a parallel suspension missed by a failed write on the next nothing-due wake', async () => { + const { state, values, alarms } = recoveryStorage(); + const gateSuspendedAt = Date.now(); + const otherSuspendedAt = Date.now() + 1; + const env = makeProductionEnv(); + env.runtime = { + ...statusStub(async () => ({ + runId: 'run-parallel', + status: 'suspended', + suspended: [['gate'], ['other']], + suspendPayload: { + gate: { [SUSPENSION_DEADLINE_PAYLOAD_KEY]: TIMED_DEADLINE_MS }, + other: { [SUSPENSION_DEADLINE_PAYLOAD_KEY]: TIMED_DEADLINE_MS }, + }, + suspendedAt: { gate: gateSuspendedAt, other: otherSuspendedAt }, + })), + } as unknown as RunnerRuntime; + // Only 'gate' ever made it into the record: 'other' suspended in parallel + // and the boundary write that would have armed it failed. + seedDeadlines(values, 'run-parallel', [ + armedEntry('gate', gateSuspendedAt), + ]); + const runner = new TestRunner(state, env); + + await runner.alarm(); + + expect(storedDeadlines(values)?.entries).toEqual([ + armedEntry('gate', gateSuspendedAt), + armedEntry('other', otherSuspendedAt), + ]); + expect(alarms.at(-1)).toBe(gateSuspendedAt + TIMED_DEADLINE_MS); + }); + + it('clears a stale record and its alarm when the run moved on with nothing to arm', async () => { + const events: string[] = []; + const { state, values } = recoveryStorage(events); + const env = makeProductionEnv(); + env.runtime = { + ...statusStub(async () => ({ + runId: 'run-moved-on', + status: 'success', + })), + } as unknown as RunnerRuntime; + seedDeadlines(values, 'run-moved-on', [armedEntry('gate', Date.now())]); + const runner = new TestRunner(state, env); + + await runner.alarm(); + + // #then — no permanent heartbeat: a run with no duty left converges to no + // record and no alarm on its first nothing-due wake. + expect(storedDeadlines(values)).toBeUndefined(); + expect(events.at(-1)).toBe('deleteAlarm'); + }); + + it('keeps the backoff of a re-derived entry instead of the past deadline or the floor', async () => { + const { state, values, alarms } = recoveryStorage(); + const suspendedAt = Date.now() - TIMED_DEADLINE_MS - 1; + const nextAttemptAt = Date.now() + 300_000; + const entry = { + ...armedEntry('gate', suspendedAt), + attempts: 3, + nextAttemptAt, + }; + const resume = vi.fn(); + const env = makeProductionEnv(); + env.runtime = { + ...statusStub(async () => suspendedFence('run-backed-off', suspendedAt)), + resume, + } as unknown as RunnerRuntime; + seedDeadlines(values, 'run-backed-off', [entry]); + const runner = new TestRunner(state, env); + + await runner.alarm(); + + // #then — the nothing-due re-derive carried the ledger across the merge: + // the entry keeps its three failures and arms at its own backoff floor, + // never at the long-past deadline the clean re-derived entry would carry. + expect(resume).not.toHaveBeenCalled(); + expect(storedDeadlines(values)?.entries).toEqual([entry]); + expect(alarms.at(-1)).toBe(nextAttemptAt); + }); + + it('reconciles idempotently on a wake just before the deadline and resumes once after it', async () => { + const { state, values, alarms } = recoveryStorage(); + // Authoritative state whose derived deadline lands 150 ms ahead of the + // wake — workerd can deliver an alarm marginally early. + const suspendedAt = Date.now() + 150 - TIMED_DEADLINE_MS; + const resume = vi.fn(async () => ({ + runId: 'run-early', + status: 'success', + })); + const env = makeProductionEnv(); + env.runtime = { + ...statusStub(async () => suspendedFence('run-early', suspendedAt)), + resume, + } as unknown as RunnerRuntime; + const entry = armedEntry('gate', suspendedAt); + seedDeadlines(values, 'run-early', [entry]); + const before = Date.now(); + const runner = new TestRunner(state, env); + + await runner.alarm(); + + // #then — no early resume and no ledger charge: the wake re-derived the + // byte-identical entry and armed at the one-second floor past it. + expect(resume).not.toHaveBeenCalled(); + expect(storedDeadlines(values)?.entries).toEqual([entry]); + expect(alarms.at(-1)).toBeGreaterThanOrEqual(before + 1_000); + + elapseDeadlines(values); + await runner.alarm(); + + // #then — the deadline still fires exactly once + expect(resume).toHaveBeenCalledTimes(1); + expect(storedDeadlines(values)).toBeUndefined(); + }); + + it('keeps the recovery cadence when a no-record wake cannot read authoritative state', async () => { + const events: string[] = []; + const { storage, alarms } = recoveryStorage(events); + const state = { + id: { name: 'timed:run-status-throws' }, + storage, + } as unknown as DurableObjectState; + const env = makeProductionEnv(); + const reads = statusStub(async () => { + throw new Error('workflow no longer registered'); + }); + const resume = vi.fn(); + env.runtime = { ...reads, resume } as unknown as RunnerRuntime; + const runner = new TestRunner(state, env); + const log = vi.spyOn(console, 'error').mockImplementation(() => {}); + const before = Date.now(); + + try { + for (let wake = 0; wake < 3; wake += 1) { + await expect(runner.alarm()).resolves.toBeUndefined(); + } + } finally { + log.mockRestore(); + } + + // #then — one status read per wake at the watchdog cadence: never the + // one-second floor, never a delete, nothing executed. A throw keeps the + // cadence because it cannot distinguish an unregistered workflow — which + // the next deploy self-heals — from a storage fault. + expect(reads.authoritativeStatus).toHaveBeenCalledTimes(3); + expect(alarms).toHaveLength(3); + for (const at of alarms) { + expect(at).toBeGreaterThanOrEqual(before + 60_000); + } + expect(events).not.toContain('deleteAlarm'); + expect(resume).not.toHaveBeenCalled(); + }); + + it('keeps a valid record on the recovery cadence when its nothing-due wake cannot read state', async () => { + const events: string[] = []; + const { storage, values, alarms } = recoveryStorage(events); + const state = { + id: { name: 'timed:run-throws-recorded' }, + storage, + } as unknown as DurableObjectState; + const env = makeProductionEnv(); + env.runtime = { + ...statusStub(async () => { + throw new Error('workflow no longer registered'); + }), + } as unknown as RunnerRuntime; + const entry = armedEntry('gate', Date.now()); + seedDeadlines(values, 'run-throws-recorded', [entry]); + const runner = new TestRunner(state, env); + const logged: string[] = []; + const log = vi + .spyOn(console, 'error') + .mockImplementation((...args: unknown[]) => { + logged.push(String(args[0])); + }); + const before = Date.now(); + + try { + for (let wake = 0; wake < 3; wake += 1) { + await expect(runner.alarm()).resolves.toBeUndefined(); + } + } finally { + log.mockRestore(); + } + + // #then — the re-derive read is wrapped exactly as the resume path's is, + // so a wake that charges nothing says so: an operator greps one line for + // an unreadable window, not the charged-failure wording. + expect(logged).toEqual([ + 'suspension deadline wake could not read authoritative state', + 'suspension deadline wake could not read authoritative state', + 'suspension deadline wake could not read authoritative state', + ]); + + // #then — the record SURVIVES: the re-derive could not read authoritative + // state, so nothing overwrote or dropped the valid entry, and every arm is + // the watchdog's, not the entry's far-future deadline it could not verify. + expect(storedDeadlines(values)?.entries).toEqual([entry]); + expect(alarms).toHaveLength(3); + for (const at of alarms) { + expect(at).toBeGreaterThanOrEqual(before + 60_000); + } + expect(events).not.toContain('deleteAlarm'); + }); +}); + +describe('nextDutyAlarmAt', () => { + const NOW = 1_751_882_400_000; + + it('has no arm when neither duty is pending', () => { + expect(nextDutyAlarmAt(undefined, undefined, NOW)).toBeUndefined(); + }); + + it('arms the one pending duty', () => { + expect(nextDutyAlarmAt(NOW + 5_000, undefined, NOW)).toBe(NOW + 5_000); + expect(nextDutyAlarmAt(undefined, NOW + 60_000, NOW)).toBe(NOW + 60_000); + }); + + it('takes the earlier of the two duties', () => { + expect(nextDutyAlarmAt(NOW + 5_000, NOW + 60_000, NOW)).toBe(NOW + 5_000); + expect(nextDutyAlarmAt(NOW + 90_000, NOW + 60_000, NOW)).toBe(NOW + 60_000); + }); + + it('floors an already-due time a second out instead of arming in the past', () => { + // Cloudflare fires a past alarm immediately; the floor is the anti-spin + // guarantee for an entry that is already due. + expect(nextDutyAlarmAt(NOW - 5_000, undefined, NOW)).toBe(NOW + 1_000); + expect(nextDutyAlarmAt(NOW - 5_000, NOW + 60_000, NOW)).toBe(NOW + 1_000); + }); +}); diff --git a/packages/flowsafe/src/do-runner/durable-object.ts b/packages/flowsafe/src/do-runner/durable-object.ts index 549269e..f50ea08 100644 --- a/packages/flowsafe/src/do-runner/durable-object.ts +++ b/packages/flowsafe/src/do-runner/durable-object.ts @@ -8,7 +8,9 @@ // enforced by RunnerRuntime's per-run FIFO locks; routing one DO instance per run // (idFromName(`${workflowId}:${runId}`)) makes those locks authoritative, // since all traffic for a run lands on one instance. The object's alarm is -// reserved for recovering run-owner claims that outlive an interrupted start. +// reserved for recovering run-owner claims that outlive an interrupted start +// and for the per-suspension deadlines of this run (suspension-deadline.ts); +// both duties share one wake, armed at the earliest of the two due times. import { decodeExecutionPrincipal, @@ -32,6 +34,7 @@ import { type RunLifecycleCas, type RunLifecycleTransitionResult, type RunnerRuntime, + RunStateUnreadableError, type RunSummary, UnknownRunError, } from './runtime.js'; @@ -40,6 +43,25 @@ import { type ScheduleSourceStore, type ScheduleSourceWorkflowTarget, } from './schedule-source.js'; +import { + dueSuspensionDeadline, + isReadableRunSummary, + isSuspensionDeadlineDue, + MAX_SUSPENSION_DEADLINE_ATTEMPTS, + MAX_SUSPENSION_DEADLINES_PER_RUN, + mergeSuspensionDeadlines, + nextSuspensionDeadlineAt, + parseSuspensionDeadlineRecord, + type RejectedSuspensionDeadline, + SUSPENSION_DEADLINE_PRINCIPAL_ID, + SUSPENSION_DEADLINE_STORAGE_KEY, + SUSPENSION_TIMEOUT_RESUME_KEY, + type SuspensionDeadlineEntry, + type SuspensionDeadlineRecord, + suspensionDeadlinesOf, + suspensionTimeoutResumeData, + tombstoned, +} from './suspension-deadline.js'; export interface DurableObjectRunOwner { readonly kind: ExecutionPrincipalKind; @@ -82,6 +104,37 @@ export interface DurableObjectRunLifecycleHooks { const RUN_OWNER_RECOVERY_KEY = 'flowsafe:run-owner-recovery:v1'; const RUN_OWNER_RECOVERY_DELAY_MS = 60_000; +const SUSPENSION_DEADLINE_RETRY_MS = 60_000; +// How long a run's state may stay unreadable before the entries due under it +// are given up on. A read that never succeeds is never charged, so without this +// bound one run whose workflow registration a deploy dropped for good would +// keep a 60 s heartbeat forever. A day is four orders of magnitude past any +// credible storage incident, so nothing but a permanent fault reaches it. +// +// The bound is per DUE BATCH, not per entry: one failed read stamps every entry +// due at that wake, so a record of any size clears about a day after its last +// entry falls due. An entry armed far in the future starts its own day only +// when it comes due — until then it is not this wake's business and is left +// untouched. +const SUSPENSION_DEADLINE_UNREADABLE_LIMIT_MS = 86_400_000; +// Cloudflare runs an alarm scheduled in the past immediately. An entry that is +// already due therefore cannot be armed at its own deadline: the wake would +// re-arm at the same past time and spin. Every arm is floored this far out, so +// a wake that fails to consume its entry costs one wake per second at worst +// until the retry ledger backs it off or drops it. The ledger is what ends +// that, so the floor only bounds a wake that CHARGES one. The three classes +// that never do converge nothing and keep the 60 s recovery cadence instead: a +// wake whose authoritative read did not succeed (uncharged by design — it is +// no evidence about the entry), a wake that could not BUILD its runtime (a +// misconfigured binding is a fault of the host, and says nothing about any +// entry either), and a wake that could not read its record or write its ledger +// at all. The first has one exception: a read that has not succeeded for a day +// abandons the entries due under it, and abandoning IS a convergence — that +// wake arms from what is left, which for a record holding nothing else is the +// delete that ends the heartbeat. Otherwise an unrelated entry's wake is +// deferred to the watchdog — up to a minute late, and only while the fault +// lasts. +const SUSPENSION_DEADLINE_ARM_FLOOR_MS = 1_000; interface RunOwnerRecovery { version: 1; @@ -119,10 +172,12 @@ class DurableObjectRunIdentityError extends DoStatusError { export abstract class DurableObjectRunner { protected readonly env: TEnv; - /** Absent in Node tests; present under workerd and reserved for alarms. */ + /** Absent in Node tests; present under workerd for storage and the alarm. */ protected readonly state?: DurableObjectRunnerState; #runtime?: RunnerRuntime; #operationTail = Promise.resolve(); + /** `step\0reason` of every suspension deadline this object has reported. */ + #reportedSuspensionRejections = new Set(); constructor(state: DurableObjectRunnerState | undefined, env: TEnv) { this.state = state; @@ -201,6 +256,27 @@ export abstract class DurableObjectRunner { return value; } + /** + * The reserved timeout envelope is minted by the alarm and by nothing else. + * A caller allowed to resume could otherwise drive a step's timeout branch + * while provenance still names them as the requester, which would make the + * one contract this feature sells — a timeout resume is distinguishable from + * a real signal — untrue. The KEY is refused, not just a well-formed + * envelope, so a step that reads the key directly cannot be fooled either. + */ + #resumeData(value: unknown): unknown { + if ( + value !== null && + typeof value === 'object' && + SUSPENSION_TIMEOUT_RESUME_KEY in value + ) { + throw new InvalidRunRequestError( + `resume data must not carry the reserved '${SUSPENSION_TIMEOUT_RESUME_KEY}' key`, + ); + } + return value; + } + #trustedExecutionPrincipal(request: Request): ExecutionPrincipal { const encoded = request.headers.get(EXECUTION_PRINCIPAL_HEADER); const principal = encoded ? decodeExecutionPrincipal(encoded) : undefined; @@ -331,27 +407,119 @@ export abstract class DurableObjectRunner { } } - async #armRunOwnerRecovery(recovery: RunOwnerRecovery): Promise { + /** + * The arming path for this object's one alarm, which two independent duties + * share: run-owner recovery and the run's suspension deadlines. Every arm + * that reflects CONVERGED state routes through here, or through the + * #armAlarmFor it delegates to, because an alarm set for one duty from a + * site that cannot see the other would silently drop the other's wake — a + * suspended run's deadline is then lost until something else happens to + * re-arm it, and nothing else has to. The unconditional watchdog arm + * (#armAlarmWatchdog, and the retry wake a failed reconcile leaves behind) + * is the deliberate exception: it is set before anything is read, when no + * duty's state is known yet, and the body's own final arm replaces it. + * + * `recoveryDueAt` is the recovery time a caller is arming right now; without + * it the stored recovery record supplies one. No duty pending at all is the + * only case that deletes the alarm. + */ + async #armNextAlarm(recoveryDueAt?: number): Promise { + await this.#armAlarmFor( + await this.#readSuspensionDeadlines(), + recoveryDueAt, + ); + } + + /** + * #armNextAlarm for a caller that already holds the record — the one it just + * read to derive entries, or the one it just wrote. Re-reading it here would + * charge every lifecycle boundary a second storage read for state the caller + * is holding. + */ + async #armAlarmFor( + stored: SuspensionDeadlineRecord | undefined, + recoveryDueAt?: number, + ): Promise { const storage = this.state?.storage; if (!storage) return; if (!storage.setAlarm) { - throw new Error('run owner recovery requires Durable Object alarms'); + throw new Error( + 'run owner recovery and suspension deadlines require Durable Object alarms', + ); + } + const suspensionDueAt = nextSuspensionDeadlineAt(stored); + const recoveryDue = + recoveryDueAt ?? + ((await storage.get(RUN_OWNER_RECOVERY_KEY)) !== undefined + ? Date.now() + RUN_OWNER_RECOVERY_DELAY_MS + : undefined); + // `now` is sampled AFTER the awaited recovery-key read above. Hoisting it + // before the read would shrink the floor by the read's latency, and the + // floor is the anti-spin guarantee for an entry that is already due. + const due = nextDutyAlarmAt(suspensionDueAt, recoveryDue, Date.now()); + if (due === undefined) { + await storage.deleteAlarm?.(); + return; + } + await storage.setAlarm(due); + } + + /** + * The unconditional watchdog wake every alarm() body opens with: set BEFORE + * any read, so a body that dies mid-flight still leaves this object woken, + * and never a delete, because no branch has converged yet. One recovery delay + * out because that is the cadence of the duty that can only be retried, and + * because it is the wake the object keeps if the body cannot get far enough + * to compute a better one; the body's own final #armNextAlarm() replaces it + * with the true next due time. + * + * A reconcile whose write failed leaves the same wake for the same reason: + * it too knows only that something must wake this object, and nothing about + * what is stored — which is precisely what it could not write. + */ + async #armAlarmWatchdog(): Promise { + const storage = this.state?.storage; + if (!storage?.setAlarm) { + throw new Error( + 'run owner recovery and suspension deadlines require Durable Object alarms', + ); } await storage.setAlarm(Date.now() + RUN_OWNER_RECOVERY_DELAY_MS); + } + + async #armRunOwnerRecovery(recovery: RunOwnerRecovery): Promise { + const storage = this.state?.storage; + if (!storage) return; + // Arm BEFORE journaling, never after: an isolate that dies between the two + // leaves a journal with no wake, and nothing else would recover the claim. + // An alarm with no journal yet is harmless — that wake finds nothing to do. + // One arm is enough, because a second one after the write would compute the + // same time from the same two inputs. + await this.#armNextAlarm(Date.now() + RUN_OWNER_RECOVERY_DELAY_MS); await storage.put(RUN_OWNER_RECOVERY_KEY, recovery); - await storage.setAlarm(Date.now() + RUN_OWNER_RECOVERY_DELAY_MS); } - async #clearRunOwnerRecovery(): Promise { + /** + * `keepWake` is set by a caller whose reconciliation failed with something + * to arm: the retry wake it left is the only thing that will re-derive that + * deadline, and re-arming from storage here would find no record and no + * journal and DELETE it. Keeping the recovery cadence instead costs one + * spurious wake and cannot lose a deadline. + */ + async #clearRunOwnerRecovery(keepWake: boolean): Promise { const storage = this.state?.storage; if (!storage) return; await storage.delete(RUN_OWNER_RECOVERY_KEY); - await storage.deleteAlarm?.(); + if (keepWake) { + await this.#armAlarmWatchdog(); + return; + } + await this.#armNextAlarm(); } - async #clearRunOwnerRecoveryBestEffort(): Promise { + async #clearRunOwnerRecoveryBestEffort(keepWake: boolean): Promise { try { - await this.#clearRunOwnerRecovery(); + await this.#clearRunOwnerRecovery(keepWake); } catch (error) { console.error('run owner recovery cleanup failed', error); } @@ -360,13 +528,14 @@ export abstract class DurableObjectRunner { async #settleRunOwnerBestEffort( recovery: RunOwnerRecovery, release: boolean, + keepWake = false, ): Promise { try { await this.runOwnership(this.env).settleReservation( recovery.token, release ? [{ kind: 'run', resourceId: recovery.runId }] : [], ); - await this.#clearRunOwnerRecoveryBestEffort(); + await this.#clearRunOwnerRecoveryBestEffort(keepWake); } catch (error) { console.error('run owner recovery settlement failed', error); try { @@ -380,9 +549,650 @@ export abstract class DurableObjectRunner { async #rearmRunOwnerRecovery(): Promise { const storage = this.state?.storage; if (!storage?.setAlarm) { - throw new Error('run owner recovery requires Durable Object alarms'); + throw new Error( + 'run owner recovery and suspension deadlines require Durable Object alarms', + ); + } + await this.#armNextAlarm(Date.now() + RUN_OWNER_RECOVERY_DELAY_MS); + } + + /** + * Hydrate this run's armed deadlines. A record this instance cannot parse is + * logged and deleted rather than thrown: the alarm has to converge, and a + * record that throws on every wake would strand the run's remaining + * deadlines forever. + */ + async #readSuspensionDeadlines(): Promise< + SuspensionDeadlineRecord | undefined + > { + const storage = this.state?.storage; + if (!storage) return undefined; + const stored = await storage.get(SUSPENSION_DEADLINE_STORAGE_KEY); + try { + return parseSuspensionDeadlineRecord(stored); + } catch (error) { + console.error('stored suspension deadline discarded', error); + await storage.delete(SUSPENSION_DEADLINE_STORAGE_KEY); + return undefined; + } + } + + /** + * Persist the run's armed deadlines and re-arm from what was just written. + * `stored` is the record the caller read to compute `entries`: a boundary + * with nothing stored that derives nothing writes nothing at all, because + * every consumer that never arms a deadline would otherwise pay a delete and + * an alarm write at each start, resume and termination. Nothing changed + * there, so the alarm the recovery duty owns is left exactly as it was. + */ + async #writeSuspensionDeadlines( + workflowId: string, + runId: string, + stored: SuspensionDeadlineRecord | undefined, + entries: readonly SuspensionDeadlineEntry[], + ): Promise { + const storage = this.state?.storage; + if (!storage) return; + if (stored === undefined && entries.length === 0) return; + let written: SuspensionDeadlineRecord | undefined; + if (entries.length === 0) { + await storage.delete(SUSPENSION_DEADLINE_STORAGE_KEY); + } else { + written = { version: 1, workflowId, runId, entries: [...entries] }; + await storage.put( + SUSPENSION_DEADLINE_STORAGE_KEY, + written, + ); + } + await this.#armAlarmFor(written); + } + + /** + * Re-derive this run's deadlines from an authoritative summary and re-arm. + * Called at every lifecycle boundary that produces one, and at every wake + * that ends without a resume: derivation is idempotent (an entry is keyed by + * the suspension that created it), so a boundary that changed nothing + * rewrites the same record, a run that is no longer suspended derives + * nothing and clears it, and a suspension nothing has armed yet is armed + * here. That last case is why this is the wake's own no-resume exit: a run + * whose only remaining event is its own alarm has no other boundary coming. + */ + async #reconcileSuspensionDeadlines( + workflowId: string, + runId: string, + summary: RunSummary, + stored?: SuspensionDeadlineRecord | null, + ): Promise { + if (!this.state?.storage) return; + const { entries: derived, rejected } = suspensionDeadlinesOf(summary); + for (const rejection of rejected) { + this.#logSuspensionDeadlineRejection(runId, rejection); + } + // `stored` is the record a wake already read this alarm; re-reading it + // here would be the second storage read #armAlarmFor's JSDoc refuses to + // pay. `null` is a wake that read and found NOTHING stored, which `??` + // could not tell from a lifecycle boundary holding no record at all + // (undefined) — the one that must read. + const previous = + stored === undefined + ? await this.#readSuspensionDeadlines() + : (stored ?? undefined); + // The ledger belongs to a suspension of THIS run: a record stamped for + // another run carries nothing this run may inherit. The merge matches on + // the suspension's own keys and not on the record's ids, so without this a + // foreign entry whose step, fence and deadline happened to coincide would + // hand its spent budget to this run's live deadline, which would then be + // born abandoned. `previous` still reaches the write below, whose + // write-or-skip decision is about the key that is stored, not about whose + // run it names. + const inherited = + previous?.workflowId === workflowId && previous.runId === runId + ? previous + : undefined; + await this.#writeSuspensionDeadlines( + workflowId, + runId, + previous, + mergeSuspensionDeadlines(inherited, derived), + ); + } + + /** + * Report a deadline the run asked for and did not get, once. Derivation runs + * at every boundary, so an unchanged suspension re-derives the identical + * rejection: logging each one every time would bury the rest of the log + * under a single author mistake. Bounded by the per-run entry cap so a + * long-lived object with a churning set of steps cannot grow this set. + */ + #logSuspensionDeadlineRejection( + runId: string, + rejection: RejectedSuspensionDeadline, + ): void { + const seen = `${rejection.step}\u0000${rejection.reason}`; + if (this.#reportedSuspensionRejections.has(seen)) return; + if ( + this.#reportedSuspensionRejections.size >= + MAX_SUSPENSION_DEADLINES_PER_RUN + ) { + this.#reportedSuspensionRejections.clear(); + } + this.#reportedSuspensionRejections.add(seen); + console.error( + `suspension deadline not armed for step '${rejection.step}' of run '${runId}': ${rejection.reason}`, + ); + } + + /** + * Reconciliation is bookkeeping ABOUT a transition that has already been + * persisted by the runtime. Failing the caller's start, resume, or + * termination over it would report a lifecycle failure that did not happen, + * so the failure is logged instead — and, when there was something to arm, + * answered with a retry wake, because a suspended run is guaranteed no other + * boundary. That wake re-derives from `authoritativeStatus()` whether or not + * a record exists (#reconcileFromAuthoritativeStatus); the record it holds is + * only the identity fallback, with its ledger carried by the merge. Returns + * whether the deadlines converged, so a caller that would otherwise re-arm + * from storage can leave the retry wake alone (#clearRunOwnerRecovery's + * `keepWake`). + * + * A failure to arm the retry wake itself loses the deadline. That is the + * lost-alarm failure mode the design already owns, now narrowed to it. + */ + async #reconcileSuspensionDeadlinesBestEffort( + workflowId: string, + runId: string, + summary: RunSummary, + ): Promise { + try { + await this.#reconcileSuspensionDeadlines(workflowId, runId, summary); + return true; + } catch (error) { + console.error('suspension deadline reconciliation failed', error); + // Derivation is pure and never throws, so asking it what was at stake + // costs nothing. Nothing derivable means nothing was lost: a stale + // record that could not be deleted is healed by its own alarm. + if (suspensionDeadlinesOf(summary).entries.length === 0) return true; + try { + await this.#armAlarmWatchdog(); + } catch (alarmError) { + console.error('suspension deadline retry wake failed', alarmError); + } + return false; + } + } + + // An entry is keyed by the DOT-JOINED suspended path, exactly as the summary + // keys the two fence fields read below, so the path is joined here too. The + // wake reads the rehydrated projection, which splits a stored key on every + // dot: comparing a single segment instead would fail to recognize a top-level + // step id containing a dot ([['a','b']] against 'a.b') and silently discard + // its deadline. Nested paths never reach this point (suspension-deadline.ts + // refuses them, having no fence of their own), so a joined match cannot be a + // nested suspension wearing a top-level entry's key. + #suspensionFenceMatches( + summary: RunSummary, + entry: SuspensionDeadlineEntry, + ): boolean { + return ( + summary.status === 'suspended' && + (summary.suspended ?? []).some((path) => path.join('.') === entry.step) && + summary.suspendedAt?.[entry.step] === entry.suspendedAt && + (summary.resumeCount?.[entry.step] ?? 0) === entry.resumeCount + ); + } + + /** + * Charge one failed wake to the retry ledger of the entry that wake was + * working on, and abandon that entry once the budget is spent. Every failure + * INSIDE the deadline duty that leaves its entry unconsumed comes through + * here, because an entry that stays due would otherwise re-arm the alarm at + * an already-past time, which Cloudflare fires immediately: the ledger is + * what turns a persistent failure into a backoff rather than a wake loop. + * A wake that fails before the duty runs at all cannot converge a ledger and + * does not try to — it keeps the watchdog's recovery cadence instead. + * + * Abandonment keeps the entry as a TOMBSTONE (`attempts` at the budget, no + * retry floor) rather than removing it: every reconcile re-derives the + * still-armed suspend payload, so a removed entry would come back with a + * clean ledger on the next boundary or wake — past due, armed at the floor — + * and the budget would bound one burst instead of the suspension. The merge + * carries the tombstone while the fence is unchanged and drops it when the + * suspension moves on, so a later suspension of the same step starts fresh. + */ + async #chargeSuspensionDeadlineAttempt( + stored: SuspensionDeadlineRecord, + entry: SuspensionDeadlineEntry, + now: number, + error: unknown, + ): Promise { + const { workflowId, runId } = stored; + const remaining = entriesWithoutStep(stored.entries, entry.step); + const attempts = (entry.attempts ?? 0) + 1; + if (attempts >= MAX_SUSPENSION_DEADLINE_ATTEMPTS) { + console.error( + `suspension deadline for step '${entry.step}' of run '${runId}' dropped after ${attempts} failed wakes`, + error, + ); + await this.#writeSuspensionDeadlines(workflowId, runId, stored, [ + ...remaining, + tombstoned(entry), + ]); + return; + } + console.error( + `suspension deadline wake for step '${entry.step}' of run '${runId}' failed (attempt ${attempts})`, + error, + ); + // Rebuilt field by field rather than spread from `entry`: the REBUILD is + // what drops the unreadable-state stamp. Two wakes reach a charge: one + // whose authoritative read SUCCEEDED — and a successful read is exactly + // what clears that stamp — and the identity assert, the one charged + // failure that read nothing. + // Clearing the stamp there costs nothing: an entry a foreign record put in + // front of this object's own name is charged every wake and tombstoned by + // the fifth, which is how that record goes quiet. + await this.#writeSuspensionDeadlines(workflowId, runId, stored, [ + ...remaining, + { + step: entry.step, + deadlineAt: entry.deadlineAt, + suspendedAt: entry.suspendedAt, + resumeCount: entry.resumeCount, + attempts, + // attempts caps at 4 here, so the backoff tops out at 480s and needs + // no ceiling of its own. + nextAttemptAt: now + SUSPENSION_DEADLINE_RETRY_MS * 2 ** (attempts - 1), + }, + ]); + } + + /** + * Record that this wake could not read the run's authoritative state, and + * abandon an entry once that has been true of it for a day. + * + * Stamps EVERY entry due at this wake, not just the one the wake selected: a + * read that did not succeed is one fact about the whole run, and one clock + * per due batch is what keeps an N-entry record's bound at a day rather than + * N days (a serial clock would only start the second entry's day once the + * first was abandoned). Entries not yet due are left byte-identical — their + * day begins when they fall due, not when a sibling's did. + * + * Never a charge: a read that did not succeed is no evidence about any entry, + * and the abandonment budget exists for failed resumes — spending it on a + * fifteen-minute storage incident would permanently abandon every deadline + * falling due inside one. The stamp is what still bounds the other side: a + * run whose state became permanently unreadable (a deploy that dropped the + * workflow registration for good) would otherwise keep a 60 s heartbeat + * forever. Any successful read clears it — the merge rebuilds the entry from + * the derived one, and a charge rebuilds it field by field. The bound is + * therefore conditional on this write LANDING: it is best effort, and a + * storage layer that cannot write this key converges nothing anyway, so the + * failure is logged and the wake keeps its cadence. + * + * Writes the record and NOTHING else, deliberately not through + * #writeSuspensionDeadlines: while the clock is merely running this wake has + * converged nothing, so the watchdog armed at the top of alarm() is the + * cadence it must keep. Arming from a still-due entry that no ledger backed + * off would land on the one-second floor and spin this object for the length + * of the incident. + * + * Answers whether anything was ABANDONED, which is a convergence: giving up + * on an entry and then waking every minute for it forever is incoherent, so + * the caller reports it as converged and the body's final arm re-reads what + * is left — for a record holding nothing else, the delete that finally ends + * the heartbeat this bound exists to end. + */ + async #markSuspensionDeadlineUnreadable( + stored: SuspensionDeadlineRecord, + now: number, + ): Promise { + const storage = this.state?.storage; + if (!storage) return false; + try { + let abandoned = false; + let changed = false; + const entries = stored.entries.map((entry) => { + if (!isSuspensionDeadlineDue(entry, now)) return entry; + // Clamped to this wake: a stamp in the FUTURE — a clock that stepped + // backwards between two wakes, or a hand-written record — would put + // `now - since` permanently below the limit and pin an uncharged + // heartbeat forever. Clamping alone would not end it either (the next + // wake would clamp the same untouched stamp again), so the corrected + // value is what gets written below: the day starts at this wake. + const since = Math.min(entry.unreadableSince ?? now, now); + if (now - since > SUSPENSION_DEADLINE_UNREADABLE_LIMIT_MS) { + console.error( + `suspension deadline for step '${entry.step}' of run '${stored.runId}' abandoned after 24 h of unreadable run state`, + ); + abandoned = true; + changed = true; + return tombstoned(entry); + } + // Stamped once: the first unsuccessful read is what starts the clock, + // and rewriting it every wake would push the bound out forever. A + // stamp already at its clamped value is that first one; anything else + // is the future stamp being corrected. + if (entry.unreadableSince === since) return entry; + changed = true; + return { ...entry, unreadableSince: since }; + }); + if (!changed) return false; + // The SECOND writer of a tombstone, and the one that deliberately does + // not arm: the charge writer goes through #writeSuspensionDeadlines and + // re-arms from what it wrote, while this one has converged nothing worth + // arming for and leaves the alarm to alarm()'s own final arm on the + // wakes that did converge. + await storage.put( + SUSPENSION_DEADLINE_STORAGE_KEY, + { ...stored, entries }, + ); + return abandoned; + } catch (error) { + console.error( + 'suspension deadline unreadable-state marker failed', + error, + ); + return false; + } + } + + /** + * The authoritative read every alarm path makes, carrying the one conclusion + * an alarm may draw from its failure: every way this read can fail — + * Mastra's in-memory fallback, an unregistered workflow, a storage fault — + * is a read that did not succeed, and nothing downstream of it may conclude + * anything else. One helper rather than the same try/catch at each site so + * that rule is structural: an alarm-path authoritative read that is NOT + * classified is one that did not come through here. + */ + async #authoritativeStatusOrUnreadable( + runtime: RunnerRuntime, + workflowId: string, + runId: string, + ): Promise { + try { + return await runtime.authoritativeStatus(workflowId, runId); + } catch (error) { + throw error instanceof RunStateUnreadableError + ? error + : new RunStateUnreadableError(workflowId, runId, { cause: error }); + } + } + + /** + * Resume the run for the ONE entry this wake selected. Bounded work per wake + * keeps a run with many armed steps from executing an unbounded number of + * workflow bodies in a single alarm; #armNextAlarm() schedules the next. + * + * Throws on every failure that leaves the entry unconsumed, so its caller can + * charge the ledger of the entry it selected and no other. The runtime comes + * from the caller because BUILDING one is not such a failure: a misconfigured + * binding throws on every wake and says nothing about this entry, so that + * throw must land before an entry is in hand to charge. + */ + async #resumeDueSuspensionDeadline( + runtime: RunnerRuntime, + stored: SuspensionDeadlineRecord, + entry: SuspensionDeadlineEntry, + now: number, + ): Promise { + const { workflowId, runId } = stored; + // INV-1 at the one boundary whose ids come from storage rather than from + // the trusted Worker: this wake is about to execute a workflow body, so it + // must be this object's own run (and its per-run serialization). Charged + // like any other duty failure, deliberately: it is the one charged failure + // that read nothing, and charging is what quiets a foreign record's entry + // (five wakes to a tombstone). It stays here rather than moving up with the + // runtime because the nothing-due path below deliberately prefers `id.name` + // over a foreign record, which asserting early would break. + this.#assertRunIdentity(workflowId, runId); + // The fence is re-read from authoritative state inside the operation lock: + // a real signal may have resumed this suspension since the entry was + // armed, and resuming again would run the step twice. + // + // ONLY A READ THAT SUCCEEDED MAY CONCLUDE ANYTHING, which is why this read + // goes through the classifier above rather than being taken at face value: + // the caller can then keep the watchdog instead of spending an abandonment + // budget on an incident, and nothing AFTER this fence can masquerade as + // one. + const summary = await this.#authoritativeStatusOrUnreadable( + runtime, + workflowId, + runId, + ); + // A read that succeeded and found NOTHING is not evidence that a signal got + // there first either — a read replica may not yet show a snapshot this + // object itself wrote — but it is evidence the store answered, and D1 + // staleness windows are sub-second against a 900 s budget. So it is charged + // as a failed wake and terminates as a tombstone, which is what quietly + // ends the record of a run whose state is genuinely gone. The predicate is + // the second line of defence, for a self-inconsistent 'suspended' + // projection from any cause; it must be applied BEFORE the fence check, + // which such a summary would fail, sending the moved-on reconcile below on + // to wipe every entry of a run that is still suspended. + if (!summary || !isReadableRunSummary(summary)) { + throw new Error( + `run '${runId}' of workflow '${workflowId}' read back absent or self-inconsistent`, + ); + } + if (!this.#suspensionFenceMatches(summary, entry)) { + // A discarded deadline must be visible: this is the branch a real signal + // reaches, and also the one a projection disagreement would reach. + console.error( + `suspension deadline for step '${entry.step}' of run '${runId}' dropped: the suspension it was armed against has moved on`, + ); + // Reconcile rather than merely drop the entry. The summary in hand is + // authoritative, and it may show a NEW suspension — one this run's own + // failed bookkeeping never armed. Dropping alone would leave a suspended + // run with a derivable deadline, no record, and no further wake. The + // record this wake read goes with it: re-reading it would be the second + // storage read #armAlarmFor's JSDoc refuses to pay. + await this.#reconcileSuspensionDeadlines( + workflowId, + runId, + summary, + stored, + ); + return; + } + const next = await runtime.resume(workflowId, runId, { + step: [entry.step], + resumeData: suspensionTimeoutResumeData(entry, now), + requestedBy: SUSPENSION_DEADLINE_PRINCIPAL_ID, + requestedByKind: 'system', + }); + // EVERYTHING after this point is best effort, because the resume has + // already run the step: charging a bookkeeping failure to the retry ledger + // would record a failed resume that in fact succeeded, and could resume the + // run a second time. The broadcast is one of those failures — its frame is + // built by JSON.stringify OUTSIDE safeSend's per-socket tolerance, so a + // step result JSON cannot encode (a bigint, a cycle, a throwing toJSON) + // throws here rather than in the send it was meant to survive. + try { + this.#broadcastRunSummary(next); + } catch (error) { + console.error('suspension deadline broadcast failed', error); + } + await this.#reconcileSuspensionDeadlinesBestEffort(workflowId, runId, next); + } + + /** + * The shared end of every wake that will not resume AND holds no + * authoritative summary of its own (the fence-moved branch above re-derives + * directly from the one it read): re-derive this run's deadlines from + * authoritative state. The set of wakes that land here with nothing due is + * ordinary, not anomalous — the watchdog survivor whose duty already ran, + * H1's retry wake after a failed reconcile, a recovery-armed wake whose + * start body outlived the 60 s recovery delay (a healthy slow workflow), a + * workerd alarm redelivery or at-least-once retry after a rethrown recovery + * error (up to 6 retries), and an entry still inside the 1 s arm floor. The + * `authoritativeStatus()` read lands off the lifecycle hot path on exactly + * those wakes — and it is the authoritative read, not `status()`, because + * this path DELETES a record it derives nothing from, so Mastra's in-memory + * fallback reaching it would silently wipe the wake state of a run that is + * still suspended. It cannot be skipped when a record exists: the boundary + * that should have armed this run's CURRENT suspension may have failed to + * write, and the record in hand is then stale — its own fence check only + * runs when its entry falls due, so re-arming to it would let a short + * re-suspension deadline hide behind a far-future stale one. Accepted cost: + * an interrupted-start recovery wake that DID arm pays a second + * `authoritativeStatus()` and record write here after #recoverRunOwner's own + * reconcile — a rare path. + * + * Identity, three ways: a valid `id.name` is the single source under + * workerd — the trusted Worker addressed this instance as + * idFromName(`${workflowId}:${runId}`), PATH_SAFE_ID_PATTERN excludes ':' + * from both ids, and stored ids are never asserted against it (a foreign + * record's entries are simply dropped by the merge). A name that is present + * but does not split into two path-safe ids skips the step entirely — a + * record written from an unvalidated name would discard itself on read-back, + * and a foreign record must not steer a status read either. Without a name + * (Node stubs) the stored record supplies the ids; without either, the wake + * converges to no alarm and a deadline whose record was never written is + * lost. Production is safe from that last case because the exported topology + * (host-kit/do-run-topology.ts) always addresses this object by idFromName. + */ + async #reconcileFromAuthoritativeStatus( + stored?: SuspensionDeadlineRecord | null, + ): Promise { + const name = this.state?.id?.name; + let workflowId: string; + let runId: string; + if (name !== undefined) { + const parts = name.split(':'); + if (parts.length !== 2) return; + const [namedWorkflowId, namedRunId] = parts; + // isPathSafeId rejects the empty string and narrows both halves, so a + // separate truthiness check would only restate it. + if (!isPathSafeId(namedWorkflowId) || !isPathSafeId(namedRunId)) return; + workflowId = namedWorkflowId; + runId = namedRunId; + } else if (stored) { + workflowId = stored.workflowId; + runId = stored.runId; + } else { + return; + } + // Built outside the wrap below on purpose: a runtime that cannot be BUILT + // is a binding fault, not an unreadable run, and its throw belongs to the + // caller's generic branch (which has no entry in hand here, so it keeps the + // watchdog either way). + const runtime = this.#ensureRuntime(); + // Through the same classifier the resume path uses, so a read that did not + // succeed reaches the caller as one fact with one log line instead of the + // charged-failure wording for a wake that charges nothing. + const summary = await this.#authoritativeStatusOrUnreadable( + runtime, + workflowId, + runId, + ); + if (!summary) return; + // The second line of defence, for a self-inconsistent 'suspended' + // projection the marker does not cover: it converges exactly like an + // absent read — without reconciling, so the record in hand stays and the + // caller's re-arm keeps its wake — but it is logged, because unlike a + // truly unknown run it means a suspended run's state could not be read. + if (!isReadableRunSummary(summary)) { + // Worded apart from the resume path's identical-looking refusal on + // purpose: that one is CHARGED to the entry's budget and this one never + // is, so an operator grepping the log can tell which consequence a line + // carried. + console.error( + `run '${runId}' of workflow '${workflowId}' state read back self-inconsistent; keeping the record`, + ); + return; + } + await this.#reconcileSuspensionDeadlines( + workflowId, + runId, + summary, + stored, + ); + } + + /** + * Never rethrows. workerd retries a thrown alarm(), which would re-run this + * whole wake — including a resume that may have executed workflow steps — so + * this branch owns its own retry ledger and reports failures instead. + * + * Answers whether the deadline duty CONVERGED, which is what tells the + * caller its final arm is safe. It did not converge when nothing could be + * charged or settled for a failure — the record unreadable, the runtime + * unbuildable, the ledger write itself failing, or an authoritative read + * that did not succeed (which is never charged at all) — because an arm + * computed from a still-due entry that no ledger backed off lands on the + * floor and wakes again a second later, forever. The one exception is the + * 24 h abandonment of a continuously unreadable entry: that settles the + * entry, so it converges. + */ + async #runDueSuspensionDeadline(now: number): Promise { + let stored: SuspensionDeadlineRecord | undefined; + let entry: SuspensionDeadlineEntry | undefined; + try { + stored = await this.#readSuspensionDeadlines(); + if (!stored) { + // `null`, not nothing: this wake HAS read the record key and found it + // empty, so the reconcile below must not read it a second time. + await this.#reconcileFromAuthoritativeStatus(null); + return true; + } + // Built BEFORE an entry is selected, and passed down rather than built + // where it is used. `build(env)` is the host's, and a misconfigured + // binding throws from it on every wake — a fault that says nothing about + // any entry, so charging it would tombstone a live deadline in five + // wakes. Landing the throw with no entry in hand keeps it uncharged, on + // the watchdog cadence, exactly like the identity failure one layer up. + const runtime = this.#ensureRuntime(); + entry = dueSuspensionDeadline(stored, now); + if (!entry) { + // Same end as the no-record wake: a wake that will not resume + // re-derives from authoritative state, because the boundary that + // should have armed this run's current suspension may have failed to + // write, and the record in hand is then stale (its own fence check + // only runs when it falls due). + await this.#reconcileFromAuthoritativeStatus(stored); + return true; + } + await this.#resumeDueSuspensionDeadline(runtime, stored, entry, now); + return true; + } catch (error) { + // Classified FIRST, or every degraded wake would also log the generic + // failure below. A read that did not succeed converges nothing and + // charges nothing: it keeps the watchdog cadence until it heals, and the + // entry it was working on carries the clock that bounds that. + if (error instanceof RunStateUnreadableError) { + console.error( + 'suspension deadline wake could not read authoritative state', + error, + ); + // Converged only where the marker ABANDONS an entry: that is the one + // outcome here that settles anything, and leaving the alarm armed for + // an entry just given up on would keep the heartbeat forever. + // + // `entry` is the gate, not the target: it is what proves this wake had + // duty work due at all (a wake whose nothing-due re-derivation could + // not read has no clock to run), and the marker then covers every entry + // due at the same moment, because one failed read is one fact about + // them all. + return stored && entry + ? await this.#markSuspensionDeadlineUnreadable(stored, now) + : false; + } + console.error('suspension deadline wake failed', error); + // The charge names the entry this wake had in hand, never one re-selected + // afterwards: with two entries due, re-selecting would push the second + // one out and spend a fifth of a budget it never used. A failure with no + // entry in hand — the record itself unreadable, or the re-derivation + // above — has nothing to charge, so it keeps the watchdog cadence. + if (!stored || !entry) return false; + try { + await this.#chargeSuspensionDeadlineAttempt(stored, entry, now, error); + return true; + } catch (ledgerError) { + console.error('suspension deadline ledger update failed', ledgerError); + return false; + } } - await storage.setAlarm(Date.now() + RUN_OWNER_RECOVERY_DELAY_MS); } async #recoverRunOwner(recovery: RunOwnerRecovery): Promise { @@ -395,7 +1205,25 @@ export abstract class DurableObjectRunner { recovery.token, summary ? [] : [{ kind: 'run', resourceId: recovery.runId }], ); - await this.#clearRunOwnerRecovery(); + // A start interrupted AFTER Mastra persisted a suspension is the one case + // where no other boundary is coming: the route that would have reconciled + // died with its isolate, and clearing the journal below re-arms the alarm + // from a record nothing ever wrote. This summary is the authoritative one + // recoverStartAttempt read back, so derive from it here or the run stays + // suspended with no wake at all. It needs no readability guard of its own: + // recoverStartAttempt now throws RunStateUnreadableError on Mastra's + // in-memory fallback BEFORE it concludes anything from it, so a degraded + // read never reaches this line. That throw leaves the journal and the + // reservation intact and propagates to alarm(), which classifies it and + // keeps the 60 s recovery cadence until the read heals. + const reconciled = summary + ? await this.#reconcileSuspensionDeadlinesBestEffort( + recovery.workflowId, + recovery.runId, + summary, + ) + : true; + await this.#clearRunOwnerRecovery(!reconciled); } #runOwnerRecovery(value: unknown): RunOwnerRecovery { @@ -430,21 +1258,49 @@ export abstract class DurableObjectRunner { async alarm(): Promise { await this.#withOperationLock(async () => { - await this.#rearmRunOwnerRecovery(); + await this.#armAlarmWatchdog(); + // The watchdog arm above is deliberately the ONLY arm on the path where + // this throws. A misbound namespace fails verification on every wake, so + // the deadline duty below never runs and its retry ledger can never + // converge; arming at an already-due deadline here (floored one second + // out) would then wake every suspended run's object every second for as + // long as the misbinding lasts. Failing with the watchdog in place keeps + // the recovery cadence this object woke at before it served deadlines. + await verifyDurableObjectDeploymentIdentity(this.state, this.env); + // The two duties are failure-isolated: a run-owner recovery that throws + // must not skip a due suspension deadline (and its re-arm), and neither + // must be able to leave the object with no wake at all. + let recoveryError: unknown; try { - await verifyDurableObjectDeploymentIdentity(this.state, this.env); - const stored = await this.state?.storage?.get( - RUN_OWNER_RECOVERY_KEY, - ); - if (!stored) { - await this.state?.storage?.deleteAlarm?.(); - return; - } - await this.#recoverRunOwner(this.#runOwnerRecovery(stored)); + await this.#recoverPendingRunOwner(); } catch (error) { - await this.#rearmRunOwnerRecovery(); - throw error; + recoveryError = error; } + const converged = await this.#runDueSuspensionDeadline(Date.now()); + // A duty that converged nothing has no arm to compute: the watchdog set + // at the top of this body IS the cadence it falls back to, and re-arming + // here would min it against an entry no ledger could back off — the same + // treatment, for the same reason, as the identity failure above. + if (recoveryError !== undefined) { + if (recoveryError instanceof RunStateUnreadableError) { + // Never rethrown: workerd retries a thrown alarm() up to six times, + // which would be a retry storm during the exact storage incident + // that caused it. Nothing was settled, so the journal survives and + // #armNextAlarm below reads it and arms the 60 s recovery cadence — + // the published rule for a wake that cannot read what it needs. The + // held reservation degrades owner() reads until the read heals or + // the isolate is evicted, which clears the in-memory run the marker + // comes from. + console.error( + 'run owner recovery could not read authoritative state', + recoveryError, + ); + } else { + if (converged) await this.#rearmRunOwnerRecovery(); + throw recoveryError; + } + } + if (converged) await this.#armNextAlarm(); }); } @@ -486,6 +1342,11 @@ export abstract class DurableObjectRunner { ); const runtime = this.#ensureRuntime(); await this.#recoverPendingRunOwner(); + // Stays on status(), and NOT because failing open would be safer: the + // dangerous shape here is a row miss with no in-memory Run, which + // carries no marker at all, so authoritativeStatus could not tell it + // from an absent run either. The recovery above is what covers the + // interrupted-start case, and it fails closed on an unreadable read. const existing = await runtime.status(workflowId, runId); if (existing) { const registered = await this.runOwnership(this.env).owner( @@ -540,18 +1401,47 @@ export abstract class DurableObjectRunner { runId, recovery.token, ); - } catch { + } catch (recoverError) { + // Logged, never swallowed silently: this read is the only thing + // that could tell an interrupted start apart from a failed one, + // and its own failure is the reason the journal is being left + // armed for the alarm to retry. + console.error( + 'interrupted start could not read authoritative state', + recoverError, + ); await this.#rearmRunOwnerRecovery(); throw error; } if (persisted) { - await this.#settleRunOwnerBestEffort(recovery, false); + const reconciled = + await this.#reconcileSuspensionDeadlinesBestEffort( + workflowId, + runId, + persisted, + ); + await this.#settleRunOwnerBestEffort( + recovery, + false, + !reconciled, + ); return json(persisted); } await this.#settleRunOwnerBestEffort(recovery, true); throw error; } - await this.#settleRunOwnerBestEffort(recovery, false); + // Reconcile BEFORE settling, never after: settling clears the + // recovery journal, and clearing it re-arms from storage — which, on + // a run whose FIRST deadline write has not happened yet, finds no + // record and no journal and deletes the alarm. With the journal + // still stored the write's own arm takes the min of the two due + // times, so no interleaving leaves this object without a wake. + const reconciled = await this.#reconcileSuspensionDeadlinesBestEffort( + workflowId, + runId, + summary, + ); + await this.#settleRunOwnerBestEffort(recovery, false, !reconciled); // DL-018: the authoritative RunSummary is the run-progress frame; push it // to any subscribed run-channel socket at this lifecycle boundary. this.#broadcastRunSummary(summary); @@ -647,17 +1537,23 @@ export abstract class DurableObjectRunner { const body = (await readJson(request)) ?? {}; const requestedBy = this.#requestedBy(body.requestedBy); const requestedByKind = this.#requestedByKind(body.requestedByKind); + const resumeData = this.#resumeData(body.resumeData); return this.#withOperationLock(async () => { const runtime = this.#ensureRuntime(); const summary = await runtime.resume(workflowId, runId, { step: body.step, - resumeData: body.resumeData, + resumeData, requestedBy, requestedByKind, ...(body.deadlineMs === undefined ? {} : { deadlineMs: body.deadlineMs as number }), }); + await this.#reconcileSuspensionDeadlinesBestEffort( + workflowId, + runId, + summary, + ); // DL-018: broadcast the post-resume authoritative summary. this.#broadcastRunSummary(summary); return json(summary); @@ -686,6 +1582,10 @@ export abstract class DurableObjectRunner { return this.#withOperationLock(async () => { const owner = await this.runOwnership(this.env).owner('run', runId); if (action === 'terminate-replay') { + // Degrades CLOSED: the in-memory fallback reports 'pending', which + // is neither terminal status, so a replay during a degraded read is + // refused with UnknownRunError rather than replayed against a run + // whose terminal state could not be read. const summary = await runtime.status(workflowId, runId); if ( summary?.status !== 'cancelled' && @@ -707,6 +1607,11 @@ export abstract class DurableObjectRunner { owner ?? principal, result, ); + await this.#reconcileSuspensionDeadlinesBestEffort( + workflowId, + runId, + summary, + ); this.#broadcastRunSummary(summary); return json(summary); }); @@ -742,6 +1647,12 @@ export abstract class DurableObjectRunner { return this.#withOperationLock(async () => { const owner = await this.runOwnership(this.env).owner('run', runId); if (!owner) { + // Degrades CLOSED, same as the replay guard: 'pending' is not + // 'timed_out', so the throw stands. It costs more here — this route + // is driven by the maintenance sweep, and the throw fails the whole + // sweep pass for that interval — but the next pass retries it, so a + // degraded read delays the sweep rather than timing out a run whose + // state nothing could read. const summary = await runtime.status(workflowId, runId); if (summary?.status !== 'timed_out') { throw new UnknownRunError(workflowId, runId); @@ -755,6 +1666,18 @@ export abstract class DurableObjectRunner { owner ?? principal, ); if (!result.casMatched || result.cleanup.cleanupCompleted) { + // The only terminal path that returns without finalizing, so it is + // also the only one that would leave a record armed for a run that + // can never suspend again. It reconciles from `result.summary` — a + // post-transition read of a run this call has just settled or found + // already settled or CAS-stale, taken after #loadSnapshot succeeded + // on the same store — so it is an ordinary re-derivation, not a + // conclusion drawn from a read that might not have reached storage. + await this.#reconcileSuspensionDeadlinesBestEffort( + workflowId, + runId, + result.summary, + ); return json(result.summary); } const summary = await this.#finalizeTerminal( @@ -764,6 +1687,11 @@ export abstract class DurableObjectRunner { owner ?? principal, result, ); + await this.#reconcileSuspensionDeadlinesBestEffort( + workflowId, + runId, + summary, + ); this.#broadcastRunSummary(summary); return json(summary); }); @@ -810,6 +1738,42 @@ export abstract class DurableObjectRunner { } } +/** + * The one decision behind every converged-state arm: the earlier of the two + * duties' due times, floored SUSPENSION_DEADLINE_ARM_FLOOR_MS out so an + * already-due entry cannot re-arm at its own past time (which Cloudflare + * fires immediately). `undefined` means no duty is pending and the caller + * deletes the alarm. Pure so it can be pinned in isolation; exported from + * this module only, never from the package barrel. Callers must sample `now` + * AFTER their last await — under workerd's I/O-frozen clock an earlier sample + * would shrink the floor by that await's latency and weaken the anti-spin + * guarantee. + */ +export function nextDutyAlarmAt( + suspensionDueAt: number | undefined, + recoveryDueAt: number | undefined, + now: number, +): number | undefined { + const due = + suspensionDueAt === undefined + ? recoveryDueAt + : recoveryDueAt === undefined + ? suspensionDueAt + : Math.min(suspensionDueAt, recoveryDueAt); + if (due === undefined) return undefined; + return Math.max(due, now + SUSPENSION_DEADLINE_ARM_FLOOR_MS); +} + +// One entry per suspended step is the stored record's key, so settling an entry +// is "drop this step". Matching on the step id rather than on object identity +// lets a caller settle an entry it read back separately from the record. +function entriesWithoutStep( + entries: readonly SuspensionDeadlineEntry[], + step: string, +): SuspensionDeadlineEntry[] { + return entries.filter((entry) => entry.step !== step); +} + /** The run-channel wire frame containing the authoritative RunSummary. */ function runFrame(summary: RunSummary): string { return JSON.stringify({ type: 'run', summary }); diff --git a/packages/flowsafe/src/do-runner/index.ts b/packages/flowsafe/src/do-runner/index.ts index 182a41b..9346cd3 100644 --- a/packages/flowsafe/src/do-runner/index.ts +++ b/packages/flowsafe/src/do-runner/index.ts @@ -125,6 +125,7 @@ export { RunLifecycleBlockedError, RunNotSuspendedError, RunnerRuntime, + RunStateUnreadableError, RunTerminalConflictError, UnknownRunError, UnknownWorkflowError, @@ -140,5 +141,28 @@ export type { ScheduleStartTarget, } from './schedule-source.js'; export { resolveScheduleStartOwner } from './schedule-source.js'; +// Per-suspension deadlines: the reserved suspend-payload key that arms one, the +// timeout envelope a resumed step branches on, and the bounds each is validated +// against (docs/do-runner-design.md, "Per-suspension deadlines"). The stored +// record, its parser, the envelope factory it feeds, and the wake arithmetic +// stay internal — they are the run object's own Durable-Object plumbing, and a +// consumer holding them could only misread state the alarm owns. +// MAX_SUSPENSION_DEADLINES_PER_RUN is the exception among the bounds: no single +// value is validated against it, and it ships as an operational figure — the +// per-run cap a host plans and documents against, which is how the README +// quotes it — not as something to check a deadline with before arming. +export type { + SuspensionTimeoutEnvelope, + SuspensionTimeoutResumeData, +} from './suspension-deadline.js'; +export { + isSuspensionTimeoutResumeData, + MAX_SUSPENSION_DEADLINE_MS, + MAX_SUSPENSION_DEADLINES_PER_RUN, + MIN_SUSPENSION_DEADLINE_MS, + SUSPENSION_DEADLINE_PAYLOAD_KEY, + SUSPENSION_DEADLINE_PRINCIPAL_ID, + SUSPENSION_TIMEOUT_RESUME_KEY, +} from './suspension-deadline.js'; export type { ThreadScope } from './thread-do.js'; export { ThreadDurableObject, ThreadIdentityError } from './thread-do.js'; diff --git a/packages/flowsafe/src/do-runner/runtime.test.ts b/packages/flowsafe/src/do-runner/runtime.test.ts index 8bab470..41220a2 100644 --- a/packages/flowsafe/src/do-runner/runtime.test.ts +++ b/packages/flowsafe/src/do-runner/runtime.test.ts @@ -4,6 +4,7 @@ import { InMemoryStore } from '@mastra/core/storage'; import { describe, expect, it, vi } from 'vitest'; import { z } from 'zod'; +import { RunStateUnreadableError as BarrelRunStateUnreadableError } from './index.js'; import { init } from './init.js'; import { createHostPubSub } from './pubsub.js'; import { @@ -13,8 +14,20 @@ import { type RunLeg, RunNotSuspendedError, type RunnerRuntime, + RunStateUnreadableError, + type RunSummary, UnknownWorkflowError, } from './runtime.js'; +import { + isReadableRunSummary, + isSuspensionTimeoutResumeData, + MASTRA_WORKFLOW_META_KEY, + SUSPENSION_DEADLINE_PAYLOAD_KEY, + SUSPENSION_DEADLINE_PRINCIPAL_ID, + type SuspensionDeadlineEntry, + suspensionDeadlinesOf, + suspensionTimeoutResumeData, +} from './suspension-deadline.js'; interface Counters { /** Times the approval step's post-approval body ran (the gated action). */ @@ -3178,3 +3191,426 @@ describe('RunnerRuntime snapshot provenance durability', () => { expect(loseAcknowledgement).toBe(false); }); }); + +// A step arms a per-suspension deadline through Mastra's own suspend payload, +// so these tests exercise the whole author-facing contract at the runtime level: +// what the summary carries back, and what a Mastra suspendSchema does to the +// reserved key on the way through. +describe('per-suspension deadline contract', () => { + const SUSPENSION_DEADLINE_MS = 900_000; + + function timedGateRuntime( + id: string, + suspendSchema?: z.ZodType, + resumeSchema?: z.ZodType, + ): { + runtime: RunnerRuntime; + storage: InMemoryStore; + workflow: { getWorkflowRunById: (runId: string) => Promise }; + start: () => Promise; + } { + const storage = new InMemoryStore(); + const { createWorkflow, createStep, runtime } = init({ storage }); + // Built as a value so the reserved key survives a suspendSchema that does + // not declare it — which is exactly what the stripping test measures. + const suspendPayload: Record = { + reason: 'awaiting signal', + [SUSPENSION_DEADLINE_PAYLOAD_KEY]: SUSPENSION_DEADLINE_MS, + }; + const gate = createStep({ + id: 'gate', + inputSchema: z.object({}), + outputSchema: z.object({ settledBy: z.string() }), + ...(suspendSchema ? { suspendSchema } : {}), + ...(resumeSchema ? { resumeSchema } : {}), + execute: async ({ resumeData, suspend }) => + resumeData + ? { + settledBy: isSuspensionTimeoutResumeData(resumeData) + ? 'timeout' + : 'signal', + } + : suspend(suspendPayload), + }); + const workflow = createWorkflow({ + id, + inputSchema: z.object({}), + outputSchema: z.object({ settledBy: z.string() }), + }) + .then(gate) + .commit(); + return { + runtime, + storage, + workflow, + start: () => + runtime.start(id, { runId: crypto.randomUUID(), inputData: {} }), + }; + } + + /** + * Blind the workflows store's row read — the exact seam Mastra falls back + * from — and hand back the restore. Deliberately NOT the Workflow method: + * stubbing that would FABRICATE the fallback, and what is under test is that + * Mastra produces it and stamps it. File-local rather than shared: the DO + * suite needs the same seam and keeping each copy beside the fixtures it + * serves is cheaper than a new shared module for eight lines. + */ + async function blindWorkflowRow(storage: InMemoryStore): Promise<() => void> { + const store = (await storage.getStore('workflows')) as unknown as { + getWorkflowRunById: (args: unknown) => Promise; + }; + const original = store.getWorkflowRunById; + store.getWorkflowRunById = async () => null; + return () => { + store.getWorkflowRunById = original; + }; + } + + it('lets the resumed step tell a timeout from a real signal', async () => { + const { runtime, start } = timedGateRuntime('timed-gate'); + const started = await start(); + + // #then — the summary carries everything the deadline is derived from + const { entries, rejected } = suspensionDeadlinesOf(started); + expect(rejected).toEqual([]); + expect(entries[0]).toEqual({ + step: 'gate', + deadlineAt: + (started.suspendedAt?.gate as number) + SUSPENSION_DEADLINE_MS, + suspendedAt: started.suspendedAt?.gate, + resumeCount: 0, + }); + + // #when — the deadline elapses and flowsafe resumes the run itself + const timedOut = await runtime.resume('timed-gate', started.runId, { + step: ['gate'], + resumeData: suspensionTimeoutResumeData( + entries[0] as SuspensionDeadlineEntry, + Date.now(), + ), + requestedBy: SUSPENSION_DEADLINE_PRINCIPAL_ID, + requestedByKind: 'system', + }); + + expect(timedOut).toMatchObject({ + status: 'success', + result: { settledBy: 'timeout' }, + requestedBy: SUSPENSION_DEADLINE_PRINCIPAL_ID, + requestedByKind: 'system', + }); + + // #when — the same step reached by a genuine signal instead + const signalled = await start(); + const resumed = await runtime.resume('timed-gate', signalled.runId, { + step: ['gate'], + resumeData: { approvedBy: 'bob' }, + requestedBy: 'reviewer-1', + requestedByKind: 'human', + }); + + expect(resumed).toMatchObject({ + status: 'success', + result: { settledBy: 'signal' }, + requestedBy: 'reviewer-1', + }); + }); + + it('pins Mastra stripping the reserved key from an undeclared suspendSchema', async () => { + // Tripwire on the documented authoring caveat: Mastra validates the suspend + // payload and SUBSTITUTES the parsed output, so a z.object() that does not + // declare the reserved field drops it and nothing arms. If a Mastra upgrade + // changes that, this fails instead of the caveat going silently stale. + const { start } = timedGateRuntime( + 'stripped-gate', + z.object({ reason: z.string() }), + ); + + const started = await start(); + + const payload = (started.suspendPayload as Record).gate; + expect(payload).toEqual({ reason: 'awaiting signal' }); + expect(suspensionDeadlinesOf(started).entries).toEqual([]); + }); + + it('pins the reserved key surviving a suspendSchema that declares it', async () => { + const { start } = timedGateRuntime( + 'declared-gate', + z.object({ + reason: z.string(), + [SUSPENSION_DEADLINE_PAYLOAD_KEY]: z.number(), + }), + ); + + const started = await start(); + + expect( + (started.suspendPayload as Record>) + .gate?.[SUSPENSION_DEADLINE_PAYLOAD_KEY], + ).toBe(SUSPENSION_DEADLINE_MS); + expect(suspensionDeadlinesOf(started).entries).toHaveLength(1); + }); + + it.each([ + ['loose-object-gate', z.looseObject({ reason: z.string() })], + [ + 'passthrough-gate', + z.object({ reason: z.string() }).passthrough() as unknown as z.ZodType, + ], + ])('arms through the documented loose-schema escape hatch (%s)', async (id, suspendSchema) => { + // The escape hatch the README and the design doc offer authors who do + // not want to name a flowsafe key in their own schema. If a Mastra or + // zod upgrade stops honouring it, that advice is wrong and this fails. + const { start } = timedGateRuntime(id, suspendSchema); + + const started = await start(); + + expect(suspensionDeadlinesOf(started).entries).toHaveLength(1); + }); + + it('fails the timeout resume of a step whose resumeSchema rejects the envelope', async () => { + // The likelier of the two authoring footguns: every realistic approval + // step declares a resumeSchema, and Mastra validates resume data before + // the engine is touched, so a schema that does not accept the envelope + // makes the timeout resume throw. The wake then charges its retry ledger + // and eventually drops the deadline — documented, and pinned here. + const { runtime, start } = timedGateRuntime( + 'resume-schema-gate', + undefined, + z.object({ approvedBy: z.string() }), + ); + const started = await start(); + const entry = suspensionDeadlinesOf(started) + .entries[0] as SuspensionDeadlineEntry; + + // The message matters: a bare rejection would also pass if the resume threw + // for an unrelated reason, and then this would stop pinning the footgun. + await expect( + runtime.resume('resume-schema-gate', started.runId, { + step: ['gate'], + resumeData: suspensionTimeoutResumeData(entry, Date.now()), + requestedBy: SUSPENSION_DEADLINE_PRINCIPAL_ID, + requestedByKind: 'system', + }), + ).rejects.toThrow('Invalid resume data'); + + // #then — the run is untouched: the step never took its timeout branch + const after = await runtime.status('resume-schema-gate', started.runId); + expect(after?.status).toBe('suspended'); + expect(after?.resumeCount?.gate).toBeUndefined(); + }); + + it('refuses to arm a nested suspension instead of arming nothing silently', async () => { + // Mastra reports a nested suspension as the nested path but keys the + // payload and the fence by the TOP-LEVEL step, so there is no fence for + // the step that actually suspended. v1 refuses it, loudly, rather than + // leaving an author to believe a deadline was accepted. + const { createWorkflow, createStep, runtime } = init({ + storage: new InMemoryStore(), + }); + const approval = createStep({ + id: 'approval', + inputSchema: z.object({}), + outputSchema: z.object({ settledBy: z.string() }), + execute: async ({ resumeData, suspend }) => + resumeData + ? { settledBy: 'signal' } + : suspend({ + reason: 'awaiting signal', + [SUSPENSION_DEADLINE_PAYLOAD_KEY]: SUSPENSION_DEADLINE_MS, + }), + }); + const inner = createWorkflow({ + id: 'nested', + inputSchema: z.object({}), + outputSchema: z.object({ settledBy: z.string() }), + }) + .then(approval) + .commit(); + createWorkflow({ + id: 'nested-outer', + inputSchema: z.object({}), + outputSchema: z.object({ settledBy: z.string() }), + }) + .then(inner) + .commit(); + + const started = await runtime.start('nested-outer', { + runId: crypto.randomUUID(), + inputData: {}, + }); + + expect(started.suspended).toEqual([['nested', 'approval']]); + expect(suspensionDeadlinesOf(started)).toEqual({ + entries: [], + rejected: [ + { + step: 'nested.approval', + reason: 'nested suspension paths are not supported', + }, + ], + }); + + // #then — and refused again on the projection the alarm and the recovered + // start read, which reports the SAME suspension as the enclosing step + // alone. A refusal on only one of the two projections is worse than none: + // the entry arms from the projection that misses it and then resumes a + // step whose fence describes a different suspension. + const rehydrated = await runtime.status('nested-outer', started.runId); + expect(rehydrated?.suspended).toEqual([['nested']]); + expect(suspensionDeadlinesOf(rehydrated as RunSummary)).toEqual({ + entries: [], + rejected: [ + { + step: 'nested', + reason: 'nested suspension paths are not supported', + }, + ], + }); + + // Tripwire on the marker that refusal depends on: Mastra stamps the inner + // path into the persisted payload, and that is the only thing telling this + // suspension apart from an ordinary top-level one. A rename would make + // nested deadlines arm again, so it fails here rather than there. + expect( + (rehydrated?.suspendPayload as Record>) + .nested?.[MASTRA_WORKFLOW_META_KEY], + ).toMatchObject({ path: ['approval'] }); + }); + + it.each([ + ['a plain step id', 'plain-gate'], + ['a step id containing a dot', 'dotted.gate'], + ])('derives the same deadline from the live summary and from status() for %s', async (_label, stepId) => { + // The contract the whole design rests on: a lifecycle boundary arms from + // the live summary while the alarm fences against the rehydrated one, so + // the two must derive identical entries. They key `suspended` + // differently for a dotted id — ['a.b'] live, ['a','b'] rehydrated — and + // an entry derived from one that the other cannot recognize is a deadline + // that arms and then silently disappears. + const { createWorkflow, createStep, runtime } = init({ + storage: new InMemoryStore(), + }); + const gate = createStep({ + id: stepId, + inputSchema: z.object({}), + outputSchema: z.object({ settledBy: z.string() }), + execute: async ({ resumeData, suspend }) => + resumeData + ? { settledBy: 'signal' } + : suspend({ + reason: 'awaiting signal', + [SUSPENSION_DEADLINE_PAYLOAD_KEY]: SUSPENSION_DEADLINE_MS, + }), + }); + createWorkflow({ + id: `projection-${stepId}`, + inputSchema: z.object({}), + outputSchema: z.object({ settledBy: z.string() }), + }) + .then(gate) + .commit(); + + const live = await runtime.start(`projection-${stepId}`, { + runId: crypto.randomUUID(), + inputData: {}, + }); + const rehydrated = await runtime.status(`projection-${stepId}`, live.runId); + + expect(suspensionDeadlinesOf(live).entries).toEqual([ + { + step: stepId, + deadlineAt: + (live.suspendedAt?.[stepId] as number) + SUSPENSION_DEADLINE_MS, + suspendedAt: live.suspendedAt?.[stepId], + resumeCount: 0, + }, + ]); + expect(suspensionDeadlinesOf(rehydrated as RunSummary)).toEqual( + suspensionDeadlinesOf(live), + ); + }); + + it('refuses to answer authoritatively from Mastra in-memory fallback state', async () => { + // Tripwire on the marker the whole wake discipline keys on, pinned the way + // __workflow_meta is: against the REAL producer. Mastra answers a state + // read from the in-memory Run it still holds whenever the row lookup comes + // back empty, and stamps `isFromInMemory` on exactly that answer. A rename + // or a dropped stamp would put the wake back to concluding things from a + // read that never reached storage — deleting a live record, spending an + // abandonment budget, deleting a real row in recoverStartAttempt — so it + // fails here rather than there. + const { runtime, storage, workflow, start } = + timedGateRuntime('marker-gate'); + const started = await start(); + expect(started.status).toBe('suspended'); + + const restore = await blindWorkflowRow(storage); + try { + // #then — with the row read blinded, the state Mastra hands back is the + // in-memory Run it still holds, carrying the marker + const state = (await workflow.getWorkflowRunById( + started.runId, + )) as Record; + expect(state.isFromInMemory).toBe(true); + expect(state.status).toBe('pending'); + expect(state.requestContext).toBeUndefined(); + const fallback = (await runtime.status( + 'marker-gate', + started.runId, + )) as RunSummary; + expect( + (fallback as unknown as { isFromInMemory?: boolean }).isFromInMemory, + ).toBeUndefined(); + + // #then — the authoritative read refuses it, naming both ids and NO + // cause: the run object mints this same class for any read that did not + // succeed, so a message claiming the in-memory fallback would name the + // wrong one during a storage incident. + await expect( + runtime.authoritativeStatus('marker-gate', started.runId), + ).rejects.toBeInstanceOf(RunStateUnreadableError); + await expect( + runtime.authoritativeStatus('marker-gate', started.runId), + ).rejects.toThrow( + new RegExp( + `^run '${started.runId}' of workflow 'marker-gate' state is not readable$`, + ), + ); + + // #then — while status() still serves the fabricated summary, which is + // 'pending' for a run that has never been resumed and so passes the + // self-consistency backstop: the marker is the only thing that catches + // this shape. + expect(fallback.status).toBe('pending'); + expect(fallback.suspended).toBeUndefined(); + expect(isReadableRunSummary(fallback)).toBe(true); + } finally { + restore(); + } + + // #then — and the run was suspended the whole time + const healed = await runtime.status('marker-gate', started.runId); + expect(healed?.status).toBe('suspended'); + expect(healed?.suspended).toEqual([['gate']]); + }); + + it('throws the class a consumer catches through the package barrel', async () => { + // #given — a host catching this failure imports it from the barrel, never + // from the module that throws it. Anything that turned the re-export into + // a second declaration would leave every consumer `instanceof` false while + // both files still compiled. + const { runtime, storage, start } = timedGateRuntime('barrel-gate'); + const started = await start(); + const restore = await blindWorkflowRow(storage); + + // #when / #then + try { + await expect( + runtime.authoritativeStatus('barrel-gate', started.runId), + ).rejects.toBeInstanceOf(BarrelRunStateUnreadableError); + } finally { + restore(); + } + }); +}); diff --git a/packages/flowsafe/src/do-runner/runtime.ts b/packages/flowsafe/src/do-runner/runtime.ts index 54e16a0..082ac4f 100644 --- a/packages/flowsafe/src/do-runner/runtime.ts +++ b/packages/flowsafe/src/do-runner/runtime.ts @@ -92,6 +92,27 @@ export class RunNotSuspendedError extends Error { } } +/** + * An authoritative run-state read did not succeed, so nothing that read + * returned is evidence about the run. One cause is Mastra answering from its + * in-memory fallback instead of from storage — what comes back then describes + * the Run object this isolate happens to hold rather than what is persisted — + * and the run object's alarm raises it for any other failed read too, carrying + * the underlying fault as `cause`. Distinct from UnknownRunError: the run may + * well exist and be suspended — nothing about it could be READ. The message + * therefore names no cause: it is minted where the read failed, not where the + * reason is known. + */ +export class RunStateUnreadableError extends Error { + constructor(workflowId: string, runId: string, options?: ErrorOptions) { + super( + `run '${runId}' of workflow '${workflowId}' state is not readable`, + options, + ); + this.name = 'RunStateUnreadableError'; + } +} + export class RunAlreadyExistsError extends Error { constructor(workflowId: string, runId: string, status: RunStatus) { super( @@ -1606,6 +1627,16 @@ export class RunnerRuntime { }); } + /** + * The PROJECTION read: the best answer available about a run, which is what + * an HTTP status route, a broadcast frame or an existence check wants. + * + * It can answer from Mastra's in-memory fallback, so a caller about to + * conclude something IRREVERSIBLE from what it reads — deleting wake state, + * spending an abandonment budget, resuming a run, deleting a row — must use + * {@link RunnerRuntime.authoritativeStatus} instead, which refuses a read + * that did not reach storage. + */ async status(workflowId: string, runId: string): Promise { const workflow = this.#getWorkflow(workflowId); const state = await this.#workflowState(workflow, runId); @@ -1613,6 +1644,37 @@ export class RunnerRuntime { return this.#summaryFromState(runId, state); } + /** + * status() for a caller about to CONCLUDE something irreversible from what it + * reads — delete a wake record, spend an abandonment budget, resume a run. + * + * Mastra answers a state read from an in-memory Run whenever storage is + * unavailable or the row lookup comes back empty while this isolate still + * holds the run, and that fallback reports the Run object's own status + * ('pending' until something updates it) with no suspended paths and no + * requestContext at all. Projecting it is indistinguishable from evidence + * about the run. Mastra stamps `isFromInMemory` on it at its single + * construction site and the persisted branch builds its result field by + * field without ever copying the marker, so this throws on exactly the reads + * that did not reach storage. `null` still means the read SUCCEEDED and + * found nothing. + * + * status() stays the projection read, unchanged for every existing caller: + * a summary is still the best answer an HTTP status route can give. + */ + async authoritativeStatus( + workflowId: string, + runId: string, + ): Promise { + const workflow = this.#getWorkflow(workflowId); + const state = await this.#workflowState(workflow, runId); + if (!state) return null; + if (state.isFromInMemory === true) { + throw new RunStateUnreadableError(workflowId, runId); + } + return this.#summaryFromState(runId, state); + } + /** * Reconcile an interrupted start against the token stored in the * authoritative workflow snapshot. Mastra's `createRun()` first persists a @@ -1631,6 +1693,18 @@ export class RunnerRuntime { return this.#withRunLock(workflowId, runId, async () => { const state = await this.#workflowState(workflow, runId); if (!state) return null; + // A read that did not reach storage cannot settle an interrupted start. + // The in-memory fallback carries no requestContext, so the token below + // can never match, and the 'pending' status it reports for a run that + // has not been resumed falls straight into the delete branch — which + // would destroy a live row and its snapshot behind a lagging read. The + // throw defers only a no-op: under the marker either the row exists, and + // deleting it destroys live state, or it does not, and the delete does + // nothing. A genuinely abandoned shell on an isolate that holds no Run + // carries no marker and converges here exactly as before. + if (state.isFromInMemory === true) { + throw new RunStateUnreadableError(workflowId, runId); + } const provenance = runProvenance(state); if (provenance?.startToken === attemptToken) { return summarizeState( diff --git a/packages/flowsafe/src/do-runner/suspension-deadline.test.ts b/packages/flowsafe/src/do-runner/suspension-deadline.test.ts new file mode 100644 index 0000000..16138f0 --- /dev/null +++ b/packages/flowsafe/src/do-runner/suspension-deadline.test.ts @@ -0,0 +1,1099 @@ +// SPDX-License-Identifier: Apache-2.0 +import { describe, expect, it } from 'vitest'; + +import type { RunSummary } from './runtime.js'; +import { + dueSuspensionDeadline, + isReadableRunSummary, + isSuspensionTimeoutResumeData, + MASTRA_WORKFLOW_META_KEY, + MAX_SUSPENSION_DEADLINE_ATTEMPTS, + MAX_SUSPENSION_DEADLINE_MS, + MAX_SUSPENSION_DEADLINES_PER_RUN, + MIN_SUSPENSION_DEADLINE_MS, + mergeSuspensionDeadlines, + nextSuspensionDeadlineAt, + parseSuspensionDeadlineRecord, + SUSPENSION_DEADLINE_PAYLOAD_KEY, + SUSPENSION_TIMEOUT_RESUME_KEY, + type SuspensionDeadlineEntry, + type SuspensionDeadlineRecord, + suspensionDeadlinesOf, + suspensionTimeoutResumeData, + tombstoned, +} from './suspension-deadline.js'; + +const SUSPENDED_AT = 1_751_882_400_000; + +function suspendedSummary(steps: { + [step: string]: { + payload?: unknown; + suspendedAt?: number; + resumeCount?: number; + }; +}): RunSummary { + const keys = Object.keys(steps); + const summary: RunSummary = { + runId: 'run-1', + status: 'suspended', + suspended: keys.map((key) => [key]), + suspendPayload: Object.fromEntries( + keys.map((key) => [key, steps[key]?.payload]), + ), + }; + const suspendedAt = keys.filter( + (key) => steps[key]?.suspendedAt !== undefined, + ); + if (suspendedAt.length > 0) { + summary.suspendedAt = Object.fromEntries( + suspendedAt.map((key) => [key, steps[key]?.suspendedAt as number]), + ); + } + const resumeCount = keys.filter( + (key) => steps[key]?.resumeCount !== undefined, + ); + if (resumeCount.length > 0) { + summary.resumeCount = Object.fromEntries( + resumeCount.map((key) => [key, steps[key]?.resumeCount as number]), + ); + } + return summary; +} + +function armedSummary(deadlineMs: unknown): RunSummary { + return suspendedSummary({ + gate: { + payload: { + reason: 'awaiting signal', + [SUSPENSION_DEADLINE_PAYLOAD_KEY]: deadlineMs, + }, + suspendedAt: SUSPENDED_AT, + }, + }); +} + +function storedRecord( + entries: SuspensionDeadlineEntry[], +): SuspensionDeadlineRecord { + return { version: 1, workflowId: 'gated', runId: 'run-1', entries }; +} + +describe('suspensionDeadlinesOf', () => { + it('arms nothing for a run that is not suspended', () => { + expect( + suspensionDeadlinesOf({ runId: 'run-1', status: 'success' }), + ).toEqual({ entries: [], rejected: [] }); + }); + + it('arms nothing when the suspension carries no reserved key', () => { + const summary = suspendedSummary({ + gate: { + payload: { reason: 'awaiting signal' }, + suspendedAt: SUSPENDED_AT, + }, + }); + + expect(suspensionDeadlinesOf(summary)).toEqual({ + entries: [], + rejected: [], + }); + }); + + it('arms nothing when the suspension carries no payload object at all', () => { + for (const payload of [undefined, null, 'awaiting', 7, [900_000]]) { + expect( + suspensionDeadlinesOf( + suspendedSummary({ gate: { payload, suspendedAt: SUSPENDED_AT } }), + ), + ).toEqual({ entries: [], rejected: [] }); + } + }); + + it('derives the deadline from the suspension time, never from now', () => { + const { entries, rejected } = suspensionDeadlinesOf(armedSummary(900_000)); + + expect(rejected).toEqual([]); + expect(entries).toEqual([ + { + step: 'gate', + deadlineAt: SUSPENDED_AT + 900_000, + suspendedAt: SUSPENDED_AT, + resumeCount: 0, + }, + ]); + }); + + it('derives byte-identical entries from the same summary twice', () => { + const summary = armedSummary(900_000); + + expect(suspensionDeadlinesOf(summary).entries).toEqual( + suspensionDeadlinesOf(summary).entries, + ); + }); + + it.each([ + ['below the minimum', MIN_SUSPENSION_DEADLINE_MS - 1], + ['above the maximum', MAX_SUSPENSION_DEADLINE_MS + 1], + ['zero', 0], + ['negative', -1_000], + ['fractional', 1_500.5], + ['NaN', Number.NaN], + ['a numeric string', '900000'], + ['a boolean', true], + ['null', null], + ['an object', { ms: 900_000 }], + ])('rejects a %s deadline without throwing', (_label, deadlineMs) => { + const { entries, rejected } = suspensionDeadlinesOf( + armedSummary(deadlineMs), + ); + + expect(entries).toEqual([]); + expect(rejected).toHaveLength(1); + expect(rejected[0]?.step).toBe('gate'); + expect(rejected[0]?.reason).toContain(SUSPENSION_DEADLINE_PAYLOAD_KEY); + }); + + it.each([ + MIN_SUSPENSION_DEADLINE_MS, + MAX_SUSPENSION_DEADLINE_MS, + ])('accepts the inclusive bound %i', (deadlineMs) => { + const { entries, rejected } = suspensionDeadlinesOf( + armedSummary(deadlineMs), + ); + + expect(rejected).toEqual([]); + expect(entries[0]?.deadlineAt).toBe(SUSPENDED_AT + deadlineMs); + }); + + it('refuses to arm a step with no suspendedAt fence', () => { + const summary = suspendedSummary({ + gate: { payload: { [SUSPENSION_DEADLINE_PAYLOAD_KEY]: 900_000 } }, + }); + + expect(suspensionDeadlinesOf(summary)).toEqual({ + entries: [], + rejected: [{ step: 'gate', reason: 'no suspendedAt fence' }], + }); + }); + + it('refuses to arm a step whose resumeCount fence is malformed', () => { + const summary = suspendedSummary({ + gate: { + payload: { [SUSPENSION_DEADLINE_PAYLOAD_KEY]: 900_000 }, + suspendedAt: SUSPENDED_AT, + resumeCount: -1, + }, + }); + + expect(suspensionDeadlinesOf(summary)).toEqual({ + entries: [], + rejected: [{ step: 'gate', reason: 'resumeCount fence is malformed' }], + }); + }); + + it('normalizes an absent resumeCount to 0 and carries a present one', () => { + const first = suspensionDeadlinesOf(armedSummary(900_000)); + const resumed = suspensionDeadlinesOf( + suspendedSummary({ + gate: { + payload: { [SUSPENSION_DEADLINE_PAYLOAD_KEY]: 900_000 }, + suspendedAt: SUSPENDED_AT, + resumeCount: 2, + }, + }), + ); + + expect(first.entries[0]?.resumeCount).toBe(0); + expect(resumed.entries[0]?.resumeCount).toBe(2); + }); + + it('arms one entry per suspended step, earliest deadline first', () => { + const summary = suspendedSummary({ + slow: { + payload: { [SUSPENSION_DEADLINE_PAYLOAD_KEY]: 900_000 }, + suspendedAt: SUSPENDED_AT, + }, + quick: { + payload: { [SUSPENSION_DEADLINE_PAYLOAD_KEY]: 60_000 }, + suspendedAt: SUSPENDED_AT, + }, + }); + + expect(suspensionDeadlinesOf(summary).entries.map((e) => e.step)).toEqual([ + 'quick', + 'slow', + ]); + }); + + it('derives one entry for a dotted step id from either projection', () => { + // The same suspension as both projections report it: the live result keeps + // a top-level step id whole, the rehydrated one splits the stored key on + // every dot. Both index the payload and the fence by the joined key, so + // both must derive the entry the wake will look for. + const armed = { + [SUSPENSION_DEADLINE_PAYLOAD_KEY]: 900_000, + reason: 'awaiting signal', + }; + const live: RunSummary = { + runId: 'run-1', + status: 'suspended', + suspended: [['a.b']], + suspendPayload: { 'a.b': armed }, + suspendedAt: { 'a.b': SUSPENDED_AT }, + }; + const rehydrated: RunSummary = { ...live, suspended: [['a', 'b']] }; + + expect(suspensionDeadlinesOf(live)).toEqual({ + entries: [ + { + step: 'a.b', + deadlineAt: SUSPENDED_AT + 900_000, + suspendedAt: SUSPENDED_AT, + resumeCount: 0, + }, + ], + rejected: [], + }); + expect(suspensionDeadlinesOf(rehydrated)).toEqual( + suspensionDeadlinesOf(live), + ); + }); + + it('refuses both suspensions when two suspended paths join to one key', () => { + // The live projection of the collision: a top-level step id 'a.b' and a + // nested workflow 'a' whose inner step 'b' suspended too. Mastra keys its + // snapshot namespace by the joined path, so both answer to 'a.b' and no + // fence can say which suspension an entry describes. + const summary: RunSummary = { + runId: 'run-1', + status: 'suspended', + suspended: [['a.b'], ['a', 'b']], + suspendPayload: { + 'a.b': { [SUSPENSION_DEADLINE_PAYLOAD_KEY]: 900_000 }, + a: { [SUSPENSION_DEADLINE_PAYLOAD_KEY]: 900_000 }, + }, + suspendedAt: { 'a.b': SUSPENDED_AT, a: SUSPENDED_AT }, + }; + + expect(suspensionDeadlinesOf(summary)).toEqual({ + entries: [], + rejected: [ + { step: 'a.b', reason: 'ambiguous suspended step path' }, + { step: 'a.b', reason: 'ambiguous suspended step path' }, + ], + }); + }); + + it('refuses a key implied by the nesting marker of another suspension', () => { + // The SAME collision on the rehydrated projection, where the two paths no + // longer coincide: the nested one collapses to 'a', and its marker names + // the inner step, so 'a.b' is a key two suspensions could have written. + const summary: RunSummary = { + runId: 'run-1', + status: 'suspended', + suspended: [['a', 'b'], ['a']], + suspendPayload: { + 'a.b': { [SUSPENSION_DEADLINE_PAYLOAD_KEY]: 900_000 }, + a: { + [SUSPENSION_DEADLINE_PAYLOAD_KEY]: 900_000, + [MASTRA_WORKFLOW_META_KEY]: { runId: 'run-1', path: ['b'] }, + }, + }, + suspendedAt: { 'a.b': SUSPENDED_AT, a: SUSPENDED_AT }, + }; + + expect(suspensionDeadlinesOf(summary)).toEqual({ + entries: [], + rejected: [ + { step: 'a.b', reason: 'ambiguous suspended step path' }, + { step: 'a', reason: 'nested suspension paths are not supported' }, + ], + }); + }); + + it('arms both when a plain step id and a dotted one suspend together', () => { + // The negative control for the rule above: 'a' and 'a.b' are two ordinary + // top-level steps, neither carries a nesting marker, and refusing them + // would strand deadlines the author is entitled to. + const summary = suspendedSummary({ + a: { + payload: { [SUSPENSION_DEADLINE_PAYLOAD_KEY]: 900_000 }, + suspendedAt: SUSPENDED_AT, + }, + 'a.b': { + payload: { [SUSPENSION_DEADLINE_PAYLOAD_KEY]: 900_000 }, + suspendedAt: SUSPENDED_AT, + }, + }); + + const { entries, rejected } = suspensionDeadlinesOf(summary); + + expect(rejected).toEqual([]); + expect(entries.map((entry) => entry.step)).toEqual(['a', 'a.b']); + }); + + it('rejects a nested suspension instead of silently arming nothing', () => { + // The shape Mastra's live result actually produces for a nested + // suspension: the nested path in `suspended`, but the payload and the + // fence keyed by the TOP-LEVEL step. There is no fence for the step that + // suspended, so the deadline is unarmable — and saying so is the point, + // because the author asked for one. + const summary: RunSummary = { + runId: 'run-1', + status: 'suspended', + suspended: [['nested', 'approval']], + suspendPayload: { + nested: { [SUSPENSION_DEADLINE_PAYLOAD_KEY]: 900_000 }, + }, + suspendedAt: { nested: SUSPENDED_AT }, + }; + + expect(suspensionDeadlinesOf(summary)).toEqual({ + entries: [], + rejected: [ + { + step: 'nested.approval', + reason: 'nested suspension paths are not supported', + }, + ], + }); + }); + + it('rejects a nested suspension on the rehydrated projection too', () => { + // The SAME suspension read back from the snapshot: it collapses to the + // enclosing step, so it is single-segment and does have a `suspendedAt` — + // it looks armable and is not, because that fence belongs to the enclosing + // step and not to the inner one that suspended. Mastra's own nesting + // marker in the persisted payload is the evidence that says so. + const summary: RunSummary = { + runId: 'run-1', + status: 'suspended', + suspended: [['nested']], + suspendPayload: { + nested: { + [SUSPENSION_DEADLINE_PAYLOAD_KEY]: 900_000, + [MASTRA_WORKFLOW_META_KEY]: { runId: 'run-1', path: ['approval'] }, + }, + }, + suspendedAt: { nested: SUSPENDED_AT }, + }; + + expect(suspensionDeadlinesOf(summary)).toEqual({ + entries: [], + rejected: [ + { + step: 'nested', + reason: 'nested suspension paths are not supported', + }, + ], + }); + }); + + it.each([ + ['an absent marker', undefined], + ['a renamed or reshaped marker', { runId: 'run-1' }], + ['a marker whose path is not an array', { path: 'approval' }], + ['a marker with an empty path', { path: [] }], + ])('arms a top-level step despite %s', (_label, meta) => { + // The marker is framework data inside an author payload, so a Mastra change + // must not start refusing ordinary top-level deadlines. The tripwire test + // in runtime.test.ts is what makes such a change loud. + const summary = suspendedSummary({ + gate: { + payload: { + [SUSPENSION_DEADLINE_PAYLOAD_KEY]: 900_000, + [MASTRA_WORKFLOW_META_KEY]: meta, + }, + suspendedAt: SUSPENDED_AT, + }, + }); + + expect(suspensionDeadlinesOf(summary).entries).toHaveLength(1); + }); + + it('stays silent for a nested suspension that arms no deadline', () => { + const summary: RunSummary = { + runId: 'run-1', + status: 'suspended', + suspended: [['nested', 'approval']], + suspendPayload: { nested: { reason: 'awaiting signal' } }, + suspendedAt: { nested: SUSPENDED_AT }, + }; + + expect(suspensionDeadlinesOf(summary)).toEqual({ + entries: [], + rejected: [], + }); + }); + + it('rejects a deadline whose fence plus offset leaves the safe range', () => { + const summary = suspendedSummary({ + gate: { + payload: { [SUSPENSION_DEADLINE_PAYLOAD_KEY]: 900_000 }, + suspendedAt: Number.MAX_SAFE_INTEGER, + }, + }); + + expect(suspensionDeadlinesOf(summary)).toEqual({ + entries: [], + rejected: [ + { + step: 'gate', + reason: `${SUSPENSION_DEADLINE_PAYLOAD_KEY} exceeds the supported range`, + }, + ], + }); + }); + + it('caps the armed entries per run and reports the overflow', () => { + const steps: Parameters[0] = {}; + for (let index = 0; index <= MAX_SUSPENSION_DEADLINES_PER_RUN; index += 1) { + steps[`gate-${index}`] = { + payload: { [SUSPENSION_DEADLINE_PAYLOAD_KEY]: 60_000 + index }, + suspendedAt: SUSPENDED_AT, + }; + } + + const { entries, rejected } = suspensionDeadlinesOf( + suspendedSummary(steps), + ); + + expect(entries).toHaveLength(MAX_SUSPENSION_DEADLINES_PER_RUN); + expect(rejected).toEqual([ + { + step: `gate-${MAX_SUSPENSION_DEADLINES_PER_RUN}`, + reason: `run already arms ${MAX_SUSPENSION_DEADLINES_PER_RUN} suspension deadlines`, + }, + ]); + }); +}); + +describe('parseSuspensionDeadlineRecord', () => { + it('round-trips a stored record, retry ledger included', () => { + const record = storedRecord([ + { + step: 'gate', + deadlineAt: SUSPENDED_AT + 900_000, + suspendedAt: SUSPENDED_AT, + resumeCount: 1, + attempts: 2, + nextAttemptAt: SUSPENDED_AT + 960_000, + }, + ]); + + expect( + parseSuspensionDeadlineRecord(JSON.parse(JSON.stringify(record))), + ).toEqual(record); + }); + + it('passes an absent record through', () => { + expect(parseSuspensionDeadlineRecord(undefined)).toBeUndefined(); + }); + + it.each([ + ['a non-object', 'record'], + ['null', null], + ['an array', []], + ['an unknown version', { ...storedRecord([]), version: 2 }], + [ + 'a path-unsafe workflowId', + { ...storedRecord([]), workflowId: 'gated/forged' }, + ], + ['a path-unsafe runId', { ...storedRecord([]), runId: 'run 1' }], + ['missing entries', { version: 1, workflowId: 'gated', runId: 'run-1' }], + ['non-array entries', { ...storedRecord([]), entries: {} }], + [ + 'more entries than the per-run cap', + { + ...storedRecord([]), + entries: Array.from( + { length: MAX_SUSPENSION_DEADLINES_PER_RUN + 1 }, + () => ({ + step: 'gate', + deadlineAt: SUSPENDED_AT, + suspendedAt: SUSPENDED_AT, + resumeCount: 0, + }), + ), + }, + ], + ['a non-object entry', { ...storedRecord([]), entries: ['gate'] }], + [ + 'an empty step', + { + ...storedRecord([]), + entries: [ + { + step: '', + deadlineAt: SUSPENDED_AT, + suspendedAt: SUSPENDED_AT, + resumeCount: 0, + }, + ], + }, + ], + [ + 'an unbounded step', + { + ...storedRecord([]), + entries: [ + { + step: 'a'.repeat(501), + deadlineAt: SUSPENDED_AT, + suspendedAt: SUSPENDED_AT, + resumeCount: 0, + }, + ], + }, + ], + [ + 'a fractional deadlineAt', + { + ...storedRecord([]), + entries: [ + { + step: 'gate', + deadlineAt: SUSPENDED_AT + 0.5, + suspendedAt: SUSPENDED_AT, + resumeCount: 0, + }, + ], + }, + ], + [ + 'a missing suspendedAt fence', + { + ...storedRecord([]), + entries: [{ step: 'gate', deadlineAt: SUSPENDED_AT, resumeCount: 0 }], + }, + ], + [ + 'a missing resumeCount fence', + { + ...storedRecord([]), + entries: [ + { step: 'gate', deadlineAt: SUSPENDED_AT, suspendedAt: SUSPENDED_AT }, + ], + }, + ], + [ + 'a negative resumeCount', + { + ...storedRecord([]), + entries: [ + { + step: 'gate', + deadlineAt: SUSPENDED_AT, + suspendedAt: SUSPENDED_AT, + resumeCount: -1, + }, + ], + }, + ], + [ + 'attempts without a retry floor', + { + ...storedRecord([]), + entries: [ + { + step: 'gate', + deadlineAt: SUSPENDED_AT, + suspendedAt: SUSPENDED_AT, + resumeCount: 0, + attempts: 1, + }, + ], + }, + ], + [ + 'a retry floor without attempts', + { + ...storedRecord([]), + entries: [ + { + step: 'gate', + deadlineAt: SUSPENDED_AT, + suspendedAt: SUSPENDED_AT, + resumeCount: 0, + nextAttemptAt: SUSPENDED_AT, + }, + ], + }, + ], + [ + 'a zero attempt count', + { + ...storedRecord([]), + entries: [ + { + step: 'gate', + deadlineAt: SUSPENDED_AT, + suspendedAt: SUSPENDED_AT, + resumeCount: 0, + attempts: 0, + nextAttemptAt: SUSPENDED_AT, + }, + ], + }, + ], + // A spent budget reads back as a tombstone, and a tombstone is never + // retried, so a retry floor on one is a shape this module never writes. + [ + 'a tombstone carrying a retry floor', + { + ...storedRecord([]), + entries: [ + { + step: 'gate', + deadlineAt: SUSPENDED_AT, + suspendedAt: SUSPENDED_AT, + resumeCount: 0, + attempts: MAX_SUSPENSION_DEADLINE_ATTEMPTS, + nextAttemptAt: SUSPENDED_AT, + }, + ], + }, + ], + [ + 'a fractional unreadable-state clock', + { + ...storedRecord([]), + entries: [ + { + step: 'gate', + deadlineAt: SUSPENDED_AT, + suspendedAt: SUSPENDED_AT, + resumeCount: 0, + unreadableSince: SUSPENDED_AT + 0.5, + }, + ], + }, + ], + // More than the budget cannot have been written here, and accepting it + // would let a foreign writer make this module's own drop log overcount. + [ + 'more attempts than the retry budget', + { + ...storedRecord([]), + entries: [ + { + step: 'gate', + deadlineAt: SUSPENDED_AT, + suspendedAt: SUSPENDED_AT, + resumeCount: 0, + attempts: MAX_SUSPENSION_DEADLINE_ATTEMPTS + 1, + nextAttemptAt: SUSPENDED_AT, + }, + ], + }, + ], + ])('rejects %s', (_label, stored) => { + expect(() => parseSuspensionDeadlineRecord(stored)).toThrow( + 'stored suspension deadline is malformed', + ); + }); + + it('round-trips the unreadable-state clock, unpaired with the retry ledger', () => { + // Unlike the retry floor, this one is not half of a pair: it counts how + // long reads have been failing, which is independent of how many resumes + // have — an entry can carry it with no ledger at all. + const record = storedRecord([ + { + step: 'gate', + deadlineAt: SUSPENDED_AT + 900_000, + suspendedAt: SUSPENDED_AT, + resumeCount: 0, + unreadableSince: SUSPENDED_AT + 60_000, + }, + ]); + + expect( + parseSuspensionDeadlineRecord(JSON.parse(JSON.stringify(record))), + ).toEqual(record); + }); + + it('round-trips a tombstone, whose spent budget carries no retry floor', () => { + const record = storedRecord([ + { + step: 'gate', + deadlineAt: SUSPENDED_AT, + suspendedAt: SUSPENDED_AT, + resumeCount: 0, + attempts: MAX_SUSPENSION_DEADLINE_ATTEMPTS, + }, + ]); + + expect( + parseSuspensionDeadlineRecord(JSON.parse(JSON.stringify(record))), + ).toEqual(record); + }); +}); + +describe('nextSuspensionDeadlineAt', () => { + it('has no wake for an absent or empty record', () => { + expect(nextSuspensionDeadlineAt(undefined)).toBeUndefined(); + expect(nextSuspensionDeadlineAt(storedRecord([]))).toBeUndefined(); + }); + + it('takes the earliest entry, honouring a retry floor over the deadline', () => { + const record = storedRecord([ + { + step: 'retrying', + deadlineAt: SUSPENDED_AT, + suspendedAt: SUSPENDED_AT, + resumeCount: 0, + attempts: 1, + nextAttemptAt: SUSPENDED_AT + 600_000, + }, + { + step: 'waiting', + deadlineAt: SUSPENDED_AT + 300_000, + suspendedAt: SUSPENDED_AT, + resumeCount: 0, + }, + ]); + + expect(nextSuspensionDeadlineAt(record)).toBe(SUSPENDED_AT + 300_000); + }); + + it('arms nothing for a tombstone, whose past deadline would be the floor', () => { + const tombstone: SuspensionDeadlineEntry = { + step: 'abandoned', + deadlineAt: SUSPENDED_AT, + suspendedAt: SUSPENDED_AT, + resumeCount: 0, + attempts: MAX_SUSPENSION_DEADLINE_ATTEMPTS, + }; + + expect(nextSuspensionDeadlineAt(storedRecord([tombstone]))).toBeUndefined(); + expect( + nextSuspensionDeadlineAt( + storedRecord([ + tombstone, + { + step: 'waiting', + deadlineAt: SUSPENDED_AT + 300_000, + suspendedAt: SUSPENDED_AT, + resumeCount: 0, + }, + ]), + ), + ).toBe(SUSPENDED_AT + 300_000); + }); +}); + +describe('dueSuspensionDeadline', () => { + const record = storedRecord([ + { + step: 'later', + deadlineAt: SUSPENDED_AT + 600_000, + suspendedAt: SUSPENDED_AT, + resumeCount: 0, + }, + { + step: 'earlier', + deadlineAt: SUSPENDED_AT + 300_000, + suspendedAt: SUSPENDED_AT, + resumeCount: 0, + }, + { + step: 'backing-off', + deadlineAt: SUSPENDED_AT, + suspendedAt: SUSPENDED_AT, + resumeCount: 0, + attempts: 1, + nextAttemptAt: SUSPENDED_AT + 900_000, + }, + ]); + + it('selects nothing before the earliest deadline elapses', () => { + expect( + dueSuspensionDeadline(record, SUSPENDED_AT + 299_999), + ).toBeUndefined(); + expect(dueSuspensionDeadline(undefined, SUSPENDED_AT)).toBeUndefined(); + }); + + it('selects the earliest due entry and skips one still backing off', () => { + expect(dueSuspensionDeadline(record, SUSPENDED_AT + 700_000)?.step).toBe( + 'earlier', + ); + }); + + it('selects a backing-off entry once its retry floor elapses', () => { + expect(dueSuspensionDeadline(record, SUSPENDED_AT + 900_000)?.step).toBe( + 'backing-off', + ); + }); + + it('never selects a tombstone, however long past due', () => { + const tombstoneRecord = storedRecord([ + { + step: 'abandoned', + deadlineAt: SUSPENDED_AT, + suspendedAt: SUSPENDED_AT, + resumeCount: 0, + attempts: MAX_SUSPENSION_DEADLINE_ATTEMPTS, + }, + ]); + + expect( + dueSuspensionDeadline(tombstoneRecord, SUSPENDED_AT + 31_536_000_000), + ).toBeUndefined(); + }); +}); + +describe('tombstoned', () => { + const spent: SuspensionDeadlineEntry = { + step: 'gate', + deadlineAt: SUSPENDED_AT + 900_000, + suspendedAt: SUSPENDED_AT, + resumeCount: 0, + attempts: MAX_SUSPENSION_DEADLINE_ATTEMPTS - 1, + nextAttemptAt: SUSPENDED_AT + 960_000, + unreadableSince: SUSPENDED_AT + 60_000, + }; + + it('keeps the suspension it was armed against and drops all that only a retry needs', () => { + const stone = tombstoned(spent); + + expect(stone).toEqual({ + step: 'gate', + deadlineAt: SUSPENDED_AT + 900_000, + suspendedAt: SUSPENDED_AT, + resumeCount: 0, + attempts: MAX_SUSPENSION_DEADLINE_ATTEMPTS, + }); + // Absent, not present-and-undefined: a stored property carrying no value + // is one the strict parser would have to special-case on read-back. + expect(Object.keys(stone)).not.toContain('nextAttemptAt'); + expect(Object.keys(stone)).not.toContain('unreadableSince'); + }); + + it('is read back as spent by the parser and refused by both selectors', () => { + const record = storedRecord([tombstoned(spent)]); + + // #then — the shape survives the storage round-trip it is written for ... + expect( + parseSuspensionDeadlineRecord(JSON.parse(JSON.stringify(record))), + ).toEqual(record); + // ... and neither selector will retry it or wake the object for it, + // however long its deadline has been past. + expect( + dueSuspensionDeadline(record, SUSPENDED_AT + 31_536_000_000), + ).toBeUndefined(); + expect(nextSuspensionDeadlineAt(record)).toBeUndefined(); + }); +}); + +describe('isReadableRunSummary', () => { + it('rejects the degraded in-memory fallback: suspended with no paths', () => { + expect(isReadableRunSummary({ runId: 'run-1', status: 'suspended' })).toBe( + false, + ); + expect( + isReadableRunSummary({ + runId: 'run-1', + status: 'suspended', + suspended: [], + }), + ).toBe(false); + }); + + it('accepts a genuinely suspended summary and every other status', () => { + expect( + isReadableRunSummary(suspendedSummary({ gate: { payload: {} } })), + ).toBe(true); + expect(isReadableRunSummary({ runId: 'run-1', status: 'success' })).toBe( + true, + ); + expect(isReadableRunSummary({ runId: 'run-1', status: 'running' })).toBe( + true, + ); + // Including the shape this predicate cannot catch: Mastra's in-memory + // fallback reports the Run object's own status, and that is 'pending' for + // a run which has never been resumed — the dominant degraded read. Only + // the isFromInMemory marker inside RunnerRuntime.authoritativeStatus tells + // it from a run that genuinely has not started executing yet. + expect(isReadableRunSummary({ runId: 'run-1', status: 'pending' })).toBe( + true, + ); + }); +}); + +describe('suspension timeout resume data', () => { + it('wraps the expired entry under the reserved key', () => { + const entry: SuspensionDeadlineEntry = { + step: 'gate', + deadlineAt: SUSPENDED_AT + 900_000, + suspendedAt: SUSPENDED_AT, + resumeCount: 0, + }; + + const resumeData = suspensionTimeoutResumeData( + entry, + SUSPENDED_AT + 900_123, + ); + + expect(resumeData).toEqual({ + [SUSPENSION_TIMEOUT_RESUME_KEY]: { + step: 'gate', + deadlineAt: SUSPENDED_AT + 900_000, + expiredAt: SUSPENDED_AT + 900_123, + }, + }); + expect(isSuspensionTimeoutResumeData(resumeData)).toBe(true); + }); + + it.each([ + ['a real signal payload', { approvedBy: 'bob' }], + ['no payload', undefined], + ['null', null], + ['a string', SUSPENSION_TIMEOUT_RESUME_KEY], + ['an array', [{ [SUSPENSION_TIMEOUT_RESUME_KEY]: {} }]], + ['an empty envelope', { [SUSPENSION_TIMEOUT_RESUME_KEY]: {} }], + [ + 'an envelope with no step', + { + [SUSPENSION_TIMEOUT_RESUME_KEY]: { + deadlineAt: SUSPENDED_AT, + expiredAt: SUSPENDED_AT, + }, + }, + ], + [ + 'an envelope with a fractional time', + { + [SUSPENSION_TIMEOUT_RESUME_KEY]: { + step: 'gate', + deadlineAt: SUSPENDED_AT + 0.5, + expiredAt: SUSPENDED_AT, + }, + }, + ], + ])('does not mistake %s for a timeout', (_label, value) => { + expect(isSuspensionTimeoutResumeData(value)).toBe(false); + }); +}); + +describe('mergeSuspensionDeadlines', () => { + const entry: SuspensionDeadlineEntry = { + step: 'gate', + deadlineAt: SUSPENDED_AT + 900_000, + suspendedAt: SUSPENDED_AT, + resumeCount: 0, + }; + const previous = storedRecord([ + { ...entry, attempts: 2, nextAttemptAt: SUSPENDED_AT + 960_000 }, + ]); + + it('carries the retry ledger across an unchanged suspension', () => { + expect(mergeSuspensionDeadlines(previous, [entry])).toEqual([ + { ...entry, attempts: 2, nextAttemptAt: SUSPENDED_AT + 960_000 }, + ]); + }); + + it.each([ + ['step', { ...entry, step: 'other' }], + ['suspendedAt', { ...entry, suspendedAt: SUSPENDED_AT + 1 }], + ['resumeCount', { ...entry, resumeCount: 1 }], + ['deadlineAt', { ...entry, deadlineAt: SUSPENDED_AT + 1 }], + ])('starts a clean ledger when %s moved', (_label, derived) => { + expect(mergeSuspensionDeadlines(previous, [derived])).toEqual([derived]); + }); + + it('drops a stored entry the summary no longer derives', () => { + expect(mergeSuspensionDeadlines(previous, [])).toEqual([]); + expect(mergeSuspensionDeadlines(undefined, [entry])).toEqual([entry]); + }); + + it('carries a tombstone across the unchanged suspension without a floor', () => { + const tombstoneRecord = storedRecord([ + { ...entry, attempts: MAX_SUSPENSION_DEADLINE_ATTEMPTS }, + ]); + + const merged = mergeSuspensionDeadlines(tombstoneRecord, [entry]); + + expect(merged).toEqual([ + { ...entry, attempts: MAX_SUSPENSION_DEADLINE_ATTEMPTS }, + ]); + // Not merely an equal shape: no `nextAttemptAt: undefined` property that + // the strict parser would have to special-case on read-back. + expect(Object.keys(merged[0] as object)).not.toContain('nextAttemptAt'); + }); + + it('clears the unreadable-state clock rather than carrying it', () => { + // A merge only ever follows an authoritative read that SUCCEEDED, and a + // successful read is exactly what ends an unreadable stretch — so not + // carrying the stamp IS the clear. + const unread = storedRecord([ + { ...entry, unreadableSince: SUSPENDED_AT + 60_000 }, + ]); + + const merged = mergeSuspensionDeadlines(unread, [entry]); + + expect(merged).toEqual([entry]); + expect(Object.keys(merged[0] as object)).not.toContain('unreadableSince'); + }); + + it('drops the tombstone when the fence moves, starting a fresh budget', () => { + const tombstoneRecord = storedRecord([ + { ...entry, attempts: MAX_SUSPENSION_DEADLINE_ATTEMPTS }, + ]); + const resuspended = { + ...entry, + suspendedAt: SUSPENDED_AT + 60_000, + deadlineAt: SUSPENDED_AT + 960_000, + resumeCount: 1, + }; + + expect(mergeSuspensionDeadlines(tombstoneRecord, [resuspended])).toEqual([ + resuspended, + ]); + }); + + it('lets a tombstone hold a capped slot against a live deadline', () => { + // The cap is applied to derived entries in deadline order, and a tombstone + // belongs to an OLD suspension, so its entry sorts first and keeps its + // slot for as long as that suspension lasts. Past the cap that is an + // abandoned deadline displacing a live one: the abandonment contract is + // cap-conditional, which the design doc now says out loud. + const steps: Record = {}; + for (let index = 0; index <= MAX_SUSPENSION_DEADLINES_PER_RUN; index += 1) { + steps[`step-${String(index).padStart(2, '0')}`] = { + payload: { + [SUSPENSION_DEADLINE_PAYLOAD_KEY]: MIN_SUSPENSION_DEADLINE_MS, + }, + suspendedAt: SUSPENDED_AT + index, + }; + } + const { entries, rejected } = suspensionDeadlinesOf( + suspendedSummary(steps), + ); + + expect(entries).toHaveLength(MAX_SUSPENSION_DEADLINES_PER_RUN); + expect(rejected).toEqual([ + { + step: `step-${MAX_SUSPENSION_DEADLINES_PER_RUN}`, + reason: `run already arms ${MAX_SUSPENSION_DEADLINES_PER_RUN} suspension deadlines`, + }, + ]); + + const abandoned: SuspensionDeadlineEntry = { + ...(entries[0] as SuspensionDeadlineEntry), + attempts: MAX_SUSPENSION_DEADLINE_ATTEMPTS, + }; + const merged = mergeSuspensionDeadlines(storedRecord([abandoned]), entries); + + // #then — the tombstone survived in the slot the newest live suspension + // could not have, and the alarm arms for the earliest entry that is not it + expect(merged[0]).toEqual(abandoned); + expect( + merged.some( + (entry) => entry.step === `step-${MAX_SUSPENSION_DEADLINES_PER_RUN}`, + ), + ).toBe(false); + expect(nextSuspensionDeadlineAt(storedRecord(merged))).toBe( + merged[1]?.deadlineAt, + ); + }); +}); diff --git a/packages/flowsafe/src/do-runner/suspension-deadline.ts b/packages/flowsafe/src/do-runner/suspension-deadline.ts new file mode 100644 index 0000000..ea67480 --- /dev/null +++ b/packages/flowsafe/src/do-runner/suspension-deadline.ts @@ -0,0 +1,584 @@ +// SPDX-License-Identifier: Apache-2.0 +// Per-suspension deadlines — the durable wake schedule that lets a run resume +// itself when the signal a suspended step waits for never arrives. +// +// `suspend()` is Mastra's, not flowsafe's, and a suspension is the one +// lifecycle transition the runtime persists nothing of its own for: the +// snapshot it writes is Mastra's workflow state. So the arming value cannot be +// a flowsafe option — it travels inside the payload the step hands `suspend()`, +// under a reserved dotted key (the `flowsafe.` convention the request-context +// keys already use), and this module reads it back out of the authoritative +// RunSummary that start, resume, and status already project. +// +// The derived record lives in Durable Object storage, NOT in the snapshot's +// requestContext beside RunLifecycleState (run-lifecycle.ts). A suspension +// deadline is one object's wake schedule: it has no meaning once that object's +// run leaves 'suspended', and the alarm that consumes it must read it before it +// has loaded any snapshot. Storage is also why the parse below is strict — +// stored state is untrusted input, so every field is validated and a malformed +// record throws instead of being coerced, exactly as parseRunLifecycle does. +// +// Everything here is pure; arming, wake scheduling, and the timeout resume live +// in durable-object.ts, so a non-DO host (tests, local runners) is unaffected. + +import { isPathSafeId } from './path-safe-id.js'; +import type { RunSummary } from './runtime.js'; + +/** Reserved key a step sets in its `suspend()` payload to arm a deadline. */ +export const SUSPENSION_DEADLINE_PAYLOAD_KEY = 'flowsafe.deadlineMs'; + +/** Reserved key wrapping the resume data an expired deadline resumes with. */ +export const SUSPENSION_TIMEOUT_RESUME_KEY = 'flowsafe.suspensionTimeout'; + +/** Durable Object storage key holding one run's armed deadlines. */ +export const SUSPENSION_DEADLINE_STORAGE_KEY = + 'flowsafe:suspension-deadline:v1'; + +/** + * Requester recorded on a timeout resume. A self-initiated resume must be + * distinguishable in run provenance from the human or automation that advanced + * the run to the suspension, so it files under a reserved system id of its own + * rather than borrowing the recorded requester's identity. + */ +export const SUSPENSION_DEADLINE_PRINCIPAL_ID = 'flowsafe-suspension-deadline'; + +/** Shortest armable deadline; anything smaller is a tight arm-then-wake loop. */ +export const MIN_SUSPENSION_DEADLINE_MS = 1_000; + +/** Longest armable deadline (365 days), so week-scale waits still fit. */ +export const MAX_SUSPENSION_DEADLINE_MS = 31_536_000_000; + +/** Per-run entry cap; a run cannot grow its object's storage without bound. */ +export const MAX_SUSPENSION_DEADLINES_PER_RUN = 32; + +/** Failed timeout resumes tolerated before an entry is abandoned. */ +export const MAX_SUSPENSION_DEADLINE_ATTEMPTS = 5; + +/** + * Mastra's marker for a suspension that happened inside a NESTED workflow: the + * persisted payload carries the inner step path under `__workflow_meta.path`. + * Verified in Mastra 1.50.0; `runtime.test.ts` pins it, because a rename would + * otherwise silently re-open the arming of unfenceable nested suspensions. + */ +export const MASTRA_WORKFLOW_META_KEY = '__workflow_meta'; + +// Step ids are bounded for the same reason every other stored free-text field +// is: a stored record must stay small enough to read back on every alarm wake. +const MAX_SUSPENSION_DEADLINE_STEP_LENGTH = 500; + +export interface SuspensionDeadlineEntry { + /** + * The suspended step's DOT-JOINED path, which is how both RunSummary + * projections key `suspendPayload`, `suspendedAt` and `resumeCount`. Nested + * paths are never armed, so this is a top-level step id — one that may + * itself contain a dot. + */ + step: string; + /** Epoch milliseconds: the suspension's own time plus its deadline. */ + deadlineAt: number; + /** Fence: the step's suspension time in the summary that armed this entry. */ + suspendedAt: number; + /** Fence: the step's resume ordinal, normalized to 0 on a first suspension. */ + resumeCount: number; + /** + * Failed timeout resumes so far; absent until the first failure. At the full + * budget the entry is a TOMBSTONE: abandoned for the suspension it was armed + * against, never selected and never armed again, and kept in the record only + * so that suspension cannot re-derive itself a fresh budget. + */ + attempts?: number; + /** + * Epoch milliseconds; paired with `attempts` as its backoff floor. A + * tombstone carries none, because it is never retried. + */ + nextAttemptAt?: number; + /** + * Epoch milliseconds of the FIRST wake whose authoritative read did not + * succeed, absent while reads are working. Such a wake charges nothing — an + * unread run is no evidence about the entry — so this is the only thing + * bounding an entry whose run became permanently unreadable; the wake that + * owns it abandons the entry once it has stood for a day. Any successful + * read clears it. + */ + unreadableSince?: number; +} + +export interface SuspensionDeadlineRecord { + version: 1; + workflowId: string; + runId: string; + entries: SuspensionDeadlineEntry[]; +} + +export interface SuspensionTimeoutEnvelope { + /** Id of the step whose deadline expired. */ + step: string; + /** Epoch milliseconds the deadline was armed for. */ + deadlineAt: number; + /** Epoch milliseconds the wake acted on it. */ + expiredAt: number; +} + +/** Resume data a timeout resume carries, distinguishable from a real signal. */ +export type SuspensionTimeoutResumeData = { + [SUSPENSION_TIMEOUT_RESUME_KEY]: SuspensionTimeoutEnvelope; +}; + +/** A derived deadline that will NOT be armed, with the reason to log. */ +export interface RejectedSuspensionDeadline { + step: string; + reason: string; +} + +const MALFORMED = 'stored suspension deadline is malformed'; + +function record(value: unknown): Record | undefined { + return value !== null && typeof value === 'object' && !Array.isArray(value) + ? (value as Record) + : undefined; +} + +// Every number this module persists — an epoch millisecond and the resume +// ordinal alike — has to clear the same bar, so the predicate is named for the +// bar rather than for one of its callers. run-lifecycle.ts names the identical +// check `validTime` because there it only ever guards timestamps. +function nonNegativeInteger(value: unknown): value is number { + return Number.isSafeInteger(value) && (value as number) >= 0; +} + +function validStep(value: unknown): value is string { + return ( + typeof value === 'string' && + value.length > 0 && + value.length <= MAX_SUSPENSION_DEADLINE_STEP_LENGTH + ); +} + +// Read Mastra's nesting marker defensively: the payload is author data carrying +// one framework-owned field, so a renamed, missing or reshaped marker must +// degrade to "not nested" here rather than throw on a suspension flowsafe has +// already been handed. The tripwire test is what makes such a rename loud. +function nestedPath( + payload: Record, +): readonly unknown[] | undefined { + const path = record(payload[MASTRA_WORKFLOW_META_KEY])?.path; + return Array.isArray(path) && path.length > 0 ? path : undefined; +} + +/** + * Detector for the "exclusively" half of the arming invariant (see + * suspensionDeadlinesOf): every joined key that more than one suspension of + * this run answers to. Each projection shows a collision differently, so both + * rules are needed: two suspended paths joining to one key can only appear on + * the live projection (the rehydrated one splits unique stored keys), and a + * key implied by another suspended key's nesting marker only on the rehydrated + * one (live nested payloads carry no marker). + * + * Both rules read only each suspended key's OWN payload, never the + * nested-payload fallback below, which deliberately reads a different step's + * payload. + */ +function ambiguousSuspendedKeys( + suspended: readonly (readonly string[])[], + payloads: Record | undefined, +): Set { + const seen = new Set(); + const ambiguous = new Set(); + for (const path of suspended) { + const key = path.join('.'); + if (seen.has(key)) ambiguous.add(key); + seen.add(key); + const payload = record(payloads?.[key]); + const nested = payload && nestedPath(payload); + if (nested) ambiguous.add([key, ...nested].join('.')); + } + return ambiguous; +} + +function storedEntry(value: unknown): SuspensionDeadlineEntry { + const stored = record(value); + if (!stored) throw new Error(MALFORMED); + const { attempts, nextAttemptAt, unreadableSince } = stored; + // A ledger below the budget is still retrying and needs its floor; one AT the + // budget is a tombstone, which is never selected again and so must carry no + // floor. The two fields are otherwise one fact — a count with no floor would + // retry immediately, a floor with no count would retry forever. + const retrying = + attempts !== undefined && attempts !== MAX_SUSPENSION_DEADLINE_ATTEMPTS; + if ( + !validStep(stored.step) || + !nonNegativeInteger(stored.deadlineAt) || + !nonNegativeInteger(stored.suspendedAt) || + !nonNegativeInteger(stored.resumeCount) || + // The budget itself reads back, because a spent one stays as a tombstone. + // More than the budget cannot have been written here, and accepting it + // would let a foreign writer make this module's own log say "after 6". + (attempts !== undefined && + (!Number.isSafeInteger(attempts) || + (attempts as number) < 1 || + (attempts as number) > MAX_SUSPENSION_DEADLINE_ATTEMPTS)) || + (nextAttemptAt !== undefined) !== retrying || + (nextAttemptAt !== undefined && !nonNegativeInteger(nextAttemptAt)) || + // Unpaired with the ledger, unlike the floor above: it records how long + // reads have been failing, which is independent of how many resumes have. + (unreadableSince !== undefined && !nonNegativeInteger(unreadableSince)) + ) { + throw new Error(MALFORMED); + } + return { + step: stored.step, + deadlineAt: stored.deadlineAt, + suspendedAt: stored.suspendedAt, + resumeCount: stored.resumeCount, + ...(attempts === undefined ? {} : { attempts: attempts as number }), + ...(nextAttemptAt === undefined + ? {} + : { nextAttemptAt: nextAttemptAt as number }), + ...(unreadableSince === undefined + ? {} + : { unreadableSince: unreadableSince as number }), + }; +} + +// Total order on (deadlineAt, step) so the entry a wake acts on is +// deterministic even for equal deadlines. It does not order the stored record: +// a retry charge appends the entry it re-armed. +function byDueThenStep( + left: SuspensionDeadlineEntry, + right: SuspensionDeadlineEntry, +): number { + if (left.deadlineAt !== right.deadlineAt) { + return left.deadlineAt - right.deadlineAt; + } + return left.step < right.step ? -1 : left.step > right.step ? 1 : 0; +} + +/** + * Read every armable deadline out of an authoritative suspended summary. + * + * NEVER throws on author data. By the time this runs Mastra has already + * persisted the suspension, so a malformed `flowsafe.deadlineMs` cannot be + * rejected by failing the caller: that would leave the run suspended AND the + * caller erroring. The entry is reported in `rejected` for the caller to log, + * and the suspension stays an ordinary one. + * + * Works on either projection of the same suspension — the live result and the + * snapshot-rehydrated `status()` — and must agree with itself across the two, + * because the boundary that arms an entry and the wake that fences it do not + * read the same one. + * + * The arming invariant: an entry may be armed only for a suspension that + * EXCLUSIVELY OWNS the dot-joined key its fence fields are read from. A + * nested suspension fails "owns" — the key's fence belongs to the enclosing + * step — and a key collision fails "exclusively" — two suspensions answer to + * one key, which Mastra's own snapshot namespace also collides. Refusing is + * all flowsafe can do there: arming would let a wake deliver one suspension's + * timeout envelope to an unrelated step. Each refusal below is a detector for + * one projection's way of breaking that invariant. + */ +export function suspensionDeadlinesOf(summary: RunSummary): { + entries: SuspensionDeadlineEntry[]; + rejected: RejectedSuspensionDeadline[]; +} { + const entries: SuspensionDeadlineEntry[] = []; + const rejected: RejectedSuspensionDeadline[] = []; + if (summary.status !== 'suspended') return { entries, rejected }; + const payloads = record(summary.suspendPayload); + const suspended = summary.suspended ?? []; + const ambiguous = ambiguousSuspendedKeys(suspended, payloads); + for (const path of suspended) { + // The DOT-JOINED path is the entry key, because it is what indexes + // `suspendPayload`, `suspendedAt` and `resumeCount` in BOTH projections + // (['a.b'] live, ['a','b'] rehydrated), so the wake's fence recognizes + // what a boundary armed. + const step = path.join('.'); + const payload = record(payloads?.[step]); + // A LIVE nested suspension is the one shape whose payload is NOT under its + // own path: Mastra reports the nested path but keys the payload by the + // TOP-LEVEL step. Reading it there is what turns a nested arming request + // into a loud rejection instead of silence. + const nestedPayload = + payload === undefined && path.length > 1 + ? record(payloads?.[path[0] as string]) + : undefined; + const armed = payload ?? nestedPayload; + // No payload, or no reserved key in it: an ordinary suspension, silently. + if (!armed) continue; + const deadlineMs = armed[SUSPENSION_DEADLINE_PAYLOAD_KEY]; + if (deadlineMs === undefined) continue; + // "Exclusively" is checked before "owns" so the log names the sharper + // reason, and after the silent exits above so a run that asked for nothing + // still says nothing. + if (ambiguous.has(step)) { + rejected.push({ step, reason: 'ambiguous suspended step path' }); + continue; + } + // Detector for the "owns" half of the invariant, and a LOUD one — silence + // here would look to the author exactly like a deadline that was accepted. + // Each projection presents a nested suspension its own way: the live one + // as a multi-segment path whose payload sits under its first segment + // (`nestedPayload` above), the rehydrated one as the top-level step alone, + // recognizable only by Mastra's nesting marker inside the payload. + if (nestedPayload !== undefined || nestedPath(armed) !== undefined) { + rejected.push({ + step, + reason: 'nested suspension paths are not supported', + }); + continue; + } + if (!validStep(step)) { + rejected.push({ step, reason: 'suspended step id is unusable' }); + continue; + } + if (!Number.isSafeInteger(deadlineMs)) { + rejected.push({ + step, + reason: `${SUSPENSION_DEADLINE_PAYLOAD_KEY} must be a safe integer`, + }); + continue; + } + if ( + (deadlineMs as number) < MIN_SUSPENSION_DEADLINE_MS || + (deadlineMs as number) > MAX_SUSPENSION_DEADLINE_MS + ) { + rejected.push({ + step, + reason: `${SUSPENSION_DEADLINE_PAYLOAD_KEY} must be between ${MIN_SUSPENSION_DEADLINE_MS} and ${MAX_SUSPENSION_DEADLINE_MS} ms`, + }); + continue; + } + const suspendedAt = summary.suspendedAt?.[step]; + // A deadline that cannot be fenced must not be armed: without suspendedAt + // the alarm could not tell this suspension from a later one at the same + // step, and would resume a run a real signal had already moved on. + if (!nonNegativeInteger(suspendedAt)) { + rejected.push({ step, reason: 'no suspendedAt fence' }); + continue; + } + // Absent resumeCount is normal on a step's FIRST suspension; the runtime + // only stamps the ordinal once the step has been resumed. + const resumeCount = summary.resumeCount?.[step]; + if (resumeCount !== undefined && !nonNegativeInteger(resumeCount)) { + rejected.push({ step, reason: 'resumeCount fence is malformed' }); + continue; + } + // suspendedAt + deadlineMs, never now + deadlineMs: re-deriving from an + // unchanged suspension must yield a byte-identical entry, or repeated + // reconciliation would walk the deadline forward and it would never fire. + const deadlineAt = suspendedAt + (deadlineMs as number); + if (!Number.isSafeInteger(deadlineAt)) { + rejected.push({ + step, + reason: `${SUSPENSION_DEADLINE_PAYLOAD_KEY} exceeds the supported range`, + }); + continue; + } + entries.push({ + step, + deadlineAt, + suspendedAt, + resumeCount: resumeCount ?? 0, + }); + } + entries.sort(byDueThenStep); + for (const overflow of entries.splice(MAX_SUSPENSION_DEADLINES_PER_RUN)) { + rejected.push({ + step: overflow.step, + reason: `run already arms ${MAX_SUSPENSION_DEADLINES_PER_RUN} suspension deadlines`, + }); + } + return { entries, rejected }; +} + +/** + * Whether a summary is self-consistent enough to reconcile from: a suspended + * run always has at least one suspended path, so a 'suspended' status with + * none describes no suspension anyone could act on. Reconciling from it would + * derive nothing and wipe the stored record of a run that is still suspended. + * + * The SECOND line of defence, not the primary one. It rejects that projection + * whatever produced it, but it does not cover Mastra's in-memory fallback in + * general: the fallback reports the Run object's own status, which is + * 'pending' for a run that has never been resumed — the dominant case, and one + * this predicate calls readable. The primary guard is the `isFromInMemory` + * marker check inside `RunnerRuntime.authoritativeStatus`, which is keyed on + * the read having reached storage rather than on the shape it returned. The + * two signals are independent, so both are kept. + */ +export function isReadableRunSummary(summary: RunSummary): boolean { + return !( + summary.status === 'suspended' && (summary.suspended ?? []).length === 0 + ); +} + +/** Strict hydration of the stored record; `undefined` in, `undefined` out. */ +export function parseSuspensionDeadlineRecord( + value: unknown, +): SuspensionDeadlineRecord | undefined { + if (value === undefined) return undefined; + const stored = record(value); + if ( + stored?.version !== 1 || + !isPathSafeId(stored.workflowId) || + !isPathSafeId(stored.runId) || + !Array.isArray(stored.entries) || + stored.entries.length > MAX_SUSPENSION_DEADLINES_PER_RUN + ) { + throw new Error(MALFORMED); + } + return { + version: 1, + workflowId: stored.workflowId, + runId: stored.runId, + entries: stored.entries.map(storedEntry), + }; +} + +/** + * A spent entry, kept for the suspension it was armed against: the ledger at + * the budget, and every field that only means something while it is still + * being retried — the backoff floor, the unreadable-state clock — dropped, + * because a tombstone is never selected, never armed and never read again. + * Lives beside its two recognizers, `abandoned()` and `storedEntry`'s ledger + * rule, so the shape they accept and the shape written here cannot drift. + * Exported from this module for the run object that writes it; deliberately + * NOT from the package barrel, which exposes no part of the stored record. + */ +export function tombstoned( + entry: SuspensionDeadlineEntry, +): SuspensionDeadlineEntry { + return { + step: entry.step, + deadlineAt: entry.deadlineAt, + suspendedAt: entry.suspendedAt, + resumeCount: entry.resumeCount, + attempts: MAX_SUSPENSION_DEADLINE_ATTEMPTS, + }; +} + +// A spent ledger is a tombstone: the entry stays stored so its suspension +// cannot re-derive itself a fresh budget, but both selectors below must skip +// it — selecting it would retry past the budget, and arming for it would wake +// the object at its long-past deadline forever. +function abandoned(entry: SuspensionDeadlineEntry): boolean { + return (entry.attempts ?? 0) >= MAX_SUSPENSION_DEADLINE_ATTEMPTS; +} + +/** When an entry needs its wake: the deadline, or the retry floor past it. */ +function dueAt(entry: SuspensionDeadlineEntry): number { + return Math.max(entry.deadlineAt, entry.nextAttemptAt ?? 0); +} + +/** Earliest wake the record needs, honouring a retry floor over a due entry. */ +export function nextSuspensionDeadlineAt( + stored: SuspensionDeadlineRecord | undefined, +): number | undefined { + let next: number | undefined; + for (const entry of stored?.entries ?? []) { + if (abandoned(entry)) continue; + const due = dueAt(entry); + if (next === undefined || due < next) next = due; + } + return next; +} + +/** + * Whether a wake at `now` owes this entry work: it has reached its deadline + * (or the retry floor past it) and its budget is not spent. One wake resumes + * ONE such entry, but a failed authoritative read is one fact about all of + * them, so the unreadable-state clock is stamped across the whole due batch. + */ +export function isSuspensionDeadlineDue( + entry: SuspensionDeadlineEntry, + now: number, +): boolean { + return !abandoned(entry) && dueAt(entry) <= now; +} + +/** The one entry a wake at `now` should act on, earliest deadline first. */ +export function dueSuspensionDeadline( + stored: SuspensionDeadlineRecord | undefined, + now: number, +): SuspensionDeadlineEntry | undefined { + let due: SuspensionDeadlineEntry | undefined; + for (const entry of stored?.entries ?? []) { + if (!isSuspensionDeadlineDue(entry, now)) continue; + if (!due || byDueThenStep(entry, due) < 0) due = entry; + } + return due; +} + +/** The resume data a timeout resume delivers to the expired step. */ +export function suspensionTimeoutResumeData( + entry: SuspensionDeadlineEntry, + expiredAt: number, +): SuspensionTimeoutResumeData { + return { + [SUSPENSION_TIMEOUT_RESUME_KEY]: { + step: entry.step, + deadlineAt: entry.deadlineAt, + expiredAt, + }, + }; +} + +/** + * Structural guard a step uses to tell a timeout resume from a real signal, + * so consumer code branches on this contract instead of a string literal. + */ +export function isSuspensionTimeoutResumeData( + value: unknown, +): value is SuspensionTimeoutResumeData { + const envelope = record(record(value)?.[SUSPENSION_TIMEOUT_RESUME_KEY]); + return ( + envelope !== undefined && + validStep(envelope.step) && + nonNegativeInteger(envelope.deadlineAt) && + nonNegativeInteger(envelope.expiredAt) + ); +} + +/** + * Carry the retry ledger of a stored entry onto the freshly derived one for + * the SAME suspension. An entry whose fence or deadline moved is a different + * suspension, so it starts with a clean ledger; a stored entry with no derived + * counterpart is gone and is dropped. This carry is also the tombstone's whole + * lifecycle: a spent ledger rides every reconcile of the unchanged suspension + * — which is what keeps the suspension from re-deriving a fresh budget — and + * is dropped the moment the suspension moves on, so a later suspension of the + * same step starts over. + * + * `unreadableSince` is deliberately NOT carried, and that is what clears it: a + * reconcile only ever follows evidence that the run is real and current — an + * authoritative read that succeeded, or a live summary handed back by a + * lifecycle boundary — and either ends the unreadable stretch the clock was + * counting. A host that drives boundaries oftener than daily therefore keeps + * the clock from maturing even while the row read keeps failing; in the case + * the bound exists for — a registration a deploy dropped for good — the + * boundaries throw too, so nothing clears it and the bound holds. + */ +export function mergeSuspensionDeadlines( + previous: SuspensionDeadlineRecord | undefined, + derived: readonly SuspensionDeadlineEntry[], +): SuspensionDeadlineEntry[] { + return derived.map((entry) => { + const carried = previous?.entries.find( + (stored) => + stored.step === entry.step && + stored.suspendedAt === entry.suspendedAt && + stored.resumeCount === entry.resumeCount && + stored.deadlineAt === entry.deadlineAt, + ); + return carried?.attempts === undefined + ? entry + : { + ...entry, + attempts: carried.attempts, + // A tombstone carries no floor; spreading `undefined` back in would + // store a property the strict parser then has to treat as absent. + ...(carried.nextAttemptAt === undefined + ? {} + : { nextAttemptAt: carried.nextAttemptAt }), + }; + }); +} diff --git a/packages/flowsafe/src/do-runner/thread-do.test.ts b/packages/flowsafe/src/do-runner/thread-do.test.ts index b8c9731..e03a0a9 100644 --- a/packages/flowsafe/src/do-runner/thread-do.test.ts +++ b/packages/flowsafe/src/do-runner/thread-do.test.ts @@ -1,7 +1,7 @@ // SPDX-License-Identifier: Apache-2.0 import type { DurableObjectState } from '@cloudflare/workers-types'; import { InMemoryStore } from '@mastra/core/storage'; -import { describe, expect, it } from 'vitest'; +import { describe, expect, it, vi } from 'vitest'; import { deploymentIdentityDatabase, @@ -11,6 +11,7 @@ import { import { encodeExecutionPrincipal } from '../approval-api/index.js'; import { EXECUTION_PRINCIPAL_HEADER } from './execution-principal-header.js'; import { type InitResult, init } from './init.js'; +import { RunStateUnreadableError } from './runtime.js'; import { ThreadDurableObject, type ThreadScope } from './thread-do.js'; class TestThread extends ThreadDurableObject { @@ -146,6 +147,39 @@ describe('ThreadDurableObject identity boundary', () => { expect(events.filter((event) => event === 'setAlarm')).toHaveLength(2); }); + it('keeps the wake and classifies an alarm whose authoritative read did not succeed', async () => { + // #given — the owner-recovery duty this alarm drives refuses to conclude + // anything from a read that did not reach storage, and raises it here. + const events: string[] = []; + const thread = threadWith('thread-1', { + events, + alarmError: new RunStateUnreadableError( + 'durable-agentic-loop', + 'acme_run', + ), + }); + const logged: string[] = []; + const log = vi + .spyOn(console, 'error') + .mockImplementation((...args: unknown[]) => { + logged.push(String(args[0])); + }); + + // #when + try { + await expect(thread.alarm()).resolves.toBeUndefined(); + } finally { + log.mockRestore(); + } + + // #then — never a rethrow: workerd retries a thrown alarm() up to six + // times, which would answer the storage incident that caused it with a + // retry storm. The successor is armed and the failure is named once — for + // this alarm, not for whichever subclass duty raised it. + expect(events.filter((event) => event === 'setAlarm')).toHaveLength(2); + expect(logged).toEqual(['thread alarm could not read authoritative state']); + }); + it('rejects a cross-deployment caller before building named-thread state', async () => { const thread = threadWith('thread-1'); const denied = request(true, 'different-deployment-identity-secret'); diff --git a/packages/flowsafe/src/do-runner/thread-do.ts b/packages/flowsafe/src/do-runner/thread-do.ts index cde9ca1..4693012 100644 --- a/packages/flowsafe/src/do-runner/thread-do.ts +++ b/packages/flowsafe/src/do-runner/thread-do.ts @@ -37,6 +37,7 @@ import { DoStatusError, doErrorResponse } from './do-error-response.js'; import { EXECUTION_PRINCIPAL_HEADER } from './execution-principal-header.js'; import type { InitResult } from './init.js'; import { isPathSafeId } from './path-safe-id.js'; +import { RunStateUnreadableError } from './runtime.js'; const THREAD_ALARM_RECOVERY_DELAY_MS = 60_000; @@ -113,7 +114,11 @@ export abstract class ThreadDurableObject { scope: ThreadScope, ): Promise; - /** Optional Durable Object alarm work after deployment identity is verified. */ + /** + * Optional Durable Object alarm work after deployment identity is verified. + * Every failure re-arms the wake; a `RunStateUnreadableError` is then + * classified and swallowed (see `alarm()`), anything else propagates. + */ protected async onAlarm( _env: TEnv, _threadId: string, @@ -176,6 +181,24 @@ export abstract class ThreadDurableObject { await this.state?.storage.setAlarm?.( Date.now() + THREAD_ALARM_RECOVERY_DELAY_MS, ); + // A read that did not reach storage is never rethrown out of an alarm: + // workerd retries a thrown alarm() up to six times, which would answer + // the storage incident that caused it with a retry storm — the same + // refusal the run object's alarm() makes for this class. The wake is + // already re-armed above, and the duty stays owed: the host concluded + // nothing from the failed read, so its journal survives for a wake that + // can read. Every other failure keeps the re-arm and the rethrow. + // + // Named for THIS alarm and not for any one subclass's duty: `onAlarm` is + // the extension point, and this base class cannot know whose read failed + // — only that its own wake could not conclude anything. Deliberately + // distinct from the run object's 'run owner recovery could not read + // authoritative state', so an operator grepping the log can tell the two + // objects apart. + if (error instanceof RunStateUnreadableError) { + console.error('thread alarm could not read authoritative state', error); + return; + } throw error; } } diff --git a/packages/flowsafe/src/signals/thread-do-routes.test.ts b/packages/flowsafe/src/signals/thread-do-routes.test.ts index 4e918e1..e9b58e1 100644 --- a/packages/flowsafe/src/signals/thread-do-routes.test.ts +++ b/packages/flowsafe/src/signals/thread-do-routes.test.ts @@ -11,7 +11,11 @@ import { import { describe, expect, it, vi } from 'vitest'; import { RUNTIME_DRIVEN_AGENT } from '../agent-runner/index.js'; -import { DoStatusError, type ThreadScope } from '../do-runner/index.js'; +import { + DoStatusError, + RunStateUnreadableError, + type ThreadScope, +} from '../do-runner/index.js'; import { createThreadSignalRoutes } from './thread-do-routes.js'; interface AgentCall { @@ -338,6 +342,60 @@ describe('createThreadSignalRoutes', () => { expect(settle).toHaveBeenCalledOnce(); }); + it('fails the schedule wake closed when the run status read did not reach storage', async () => { + // #given — the receipt is reconstructed through agent-host owner recovery, + // and that recovery now refuses to conclude anything from a read it could + // not make. + const { agent, calls } = mockAgent(); + const settle = vi.fn(async () => undefined); + const startIdleRun = vi.fn(); + const resolveScheduleRunStatus = vi.fn(async () => { + throw new RunStateUnreadableError('durable-agentic-loop', 'run_1'); + }); + const routes = createThreadSignalRoutes({ + resolveAgent: () => agent, + resolveResourceId: () => 'acme_t1', + resolveScheduleTarget: async () => scheduleTarget(), + resolveScheduleDispatchStore: () => ({ + begin: async () => ({ state: 'ready' as const }), + settle, + }), + resolveScheduleRunStatus, + startIdleRun, + }); + const log = vi.spyOn(console, 'error').mockImplementation(() => {}); + + // #when + let response: Response | null; + try { + response = await routes( + post('/signal/schedule', { + scheduleId: 'schedule_1', + agentId: 'agent', + prompt: 'scheduled instruction', + threadId: 'acme_t1', + resourceId: 'acme_t1', + dispatchId: 'dispatch_1', + runId: 'run_1', + }), + scopeWith(undefined), + ); + } finally { + log.mockRestore(); + } + + // #then — this router's own taxonomy, not the Durable Object shell's: it + // maps only DoStatusError 403/404/409 and sends everything else to a 502, + // so an unreadable read reads as an upstream fault here rather than as the + // 503 the run object answers with. No receipt is settled from it, and no + // run is started behind it. + expect(response?.status).toBe(502); + expect(await response?.json()).toEqual({ error: 'internal error' }); + expect(settle).not.toHaveBeenCalled(); + expect(startIdleRun).not.toHaveBeenCalled(); + expect(calls).toHaveLength(0); + }); + it('settles the canonical discard receipt when termination cancels a schedule wake', async () => { const { agent, calls } = mockAgent(); const settle = vi.fn(async () => undefined); From c3ff497ecbc9b16dc9df57ade6545111402a99e1 Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Thu, 20 Aug 2026 05:52:11 +0000 Subject: [PATCH 2/2] release: version packages --- .changeset/suspension-deadline.md | 13 ------------- packages/agent-starter/CHANGELOG.md | 7 +++++++ packages/agent-starter/package.json | 2 +- packages/fleet-control/CHANGELOG.md | 7 +++++++ packages/fleet-control/package.json | 2 +- packages/flowsafe/CHANGELOG.md | 14 ++++++++++++++ packages/flowsafe/package.json | 2 +- packages/showcase/CHANGELOG.md | 7 +++++++ packages/showcase/package.json | 2 +- 9 files changed, 39 insertions(+), 17 deletions(-) delete mode 100644 .changeset/suspension-deadline.md diff --git a/.changeset/suspension-deadline.md b/.changeset/suspension-deadline.md deleted file mode 100644 index 7e11368..0000000 --- a/.changeset/suspension-deadline.md +++ /dev/null @@ -1,13 +0,0 @@ ---- -"@proofoftech/flowsafe": minor ---- - -Time out one suspension. A workflow step can now arm a deadline on the suspension it creates by adding the reserved `flowsafe.deadlineMs` key to the payload it passes Mastra's `suspend()`. When that deadline elapses before the awaited signal arrives, the run's own Durable Object resumes the run itself and delivers the `flowsafe.suspensionTimeout` envelope to the expired step, which tells it apart from a real signal with the exported `isSuspensionTimeoutResumeData()` guard. The deadline is persisted in the run's Durable Object storage, so it survives eviction and hibernation, and the resume is fenced on the exact suspension it was armed against: a genuine signal that arrives first drops the deadline instead of resuming twice. - -This is backward compatible. Nothing arms unless the reserved key is present, so existing runs and existing suspensions behave exactly as before. A host that forwards a `timeoutMs` field today is unaffected: that field was ignored and remains ignored. - -Authoring notes. A step that declares a Zod `suspendSchema` must declare the reserved field, or use a loose object (`z.looseObject()`, or `.passthrough()` on a `z.object()`): Mastra validates the suspend payload and substitutes the parsed result, so a strict `z.object()` strips the key and no deadline arms. A step that declares a `resumeSchema` must accept the timeout envelope as well as its own signal shape, or every timeout resume fails that validation and the deadline is abandoned. Only a top-level suspended step can arm a deadline: a step suspended inside a nested workflow is reported under its nested path while its suspension time is recorded against the enclosing step, so there is no way to fence the resume, and the deadline is refused and logged instead of armed. A top-level step id containing a dot arms normally, but it must not coincide with a nested path: a step `'a.b'` suspended alongside a nested workflow `'a'` whose inner step `'b'` suspends gives one key for two suspensions, and every deadline on that key is refused and logged rather than armed against the wrong one. A `foreach` step arms one deadline for the step as a whole, because that is all the run summary reports for it: run sequentially, which is the default, each iteration suspends on its own and gets its own deadline, so clearing a whole loop by timeout takes one deadline per item; with `concurrency` above 1 the timeout resume delivers the envelope to every iteration suspended at that moment (at most `concurrency` of them), iterations not yet started suspend afresh with their own deadline, only the first suspended iteration's value is read per batch, and clearing the whole loop takes one deadline per batch of `concurrency` items. Values must be a whole number of milliseconds between `MIN_SUSPENSION_DEADLINE_MS` and `MAX_SUSPENSION_DEADLINE_MS` (365 days); a value outside that range is reported in the runner log and left unarmed rather than failing the suspension that Mastra has already persisted. - -Operational caveats. Wake precision is a Durable Object alarm's — near the requested time, not exact — and there is no maintenance-sweep backstop for suspension deadlines, so a lost alarm is a lost deadline. Run-level `deadlineMs` remains the swept mechanism when a hard expiry is required. One deadline fires per wake, a run arms at most `MAX_SUSPENSION_DEADLINES_PER_RUN` of them, and a failing timeout resume backs off before it is abandoned for that suspension — the spent entry is kept, never retried and never re-armed, so the same suspension cannot earn itself a fresh budget, while a later suspension of the same step starts one. Only a wake whose authoritative read succeeded spends that budget: a read that did not succeed keeps a 60 second retry cadence and charges nothing, so a storage incident cannot abandon a live deadline — unless an entry has been due for a full day with the state unreadable throughout, after which that entry is abandoned under its own log (`abandoned after 24 h of unreadable run state`). A run whose state has been readable and genuinely absent across five wakes is charged and abandoned the ordinary way, and either abandonment is permanent for the suspension it was armed against. The routes that must not answer from a fabricated read fail closed instead: while a pending owner recovery cannot read authoritative state, `POST /runs` and `GET /runs/:workflowId/:runId/dispatch-status` return 503 for that run. Each previously answered from the fabricated read in its own way: `dispatch-status` returned 200 with a fabricated summary after deleting the very row it could not see, and `POST /runs` refused the run with a 500 (`existing run … has no matching committed owner`) — the same recovery had deleted the row and released its claim, while the fabricated read still reported the run as existing. A start whose own body throws is the one path that answers otherwise: it logs the recovery read it could not make, leaves the journal armed for a later wake, and rethrows the original start failure, so the caller sees that error's own status — a 500 for an unclassified start fault, or the error's own code such as a 400 for a malformed `deadlineMs` — rather than the 503. The same guard covers agent-host owner recovery, which now retries fail-closed — keeping its journal, its reservation and its run record — instead of possibly deleting a row behind a lagging read. The agent-host routes that drive that recovery fail closed in the taxonomy of the surface they escape to rather than reporting a run they could not read. Those that escape to the thread Durable Object's own shell, such as the dispatch read (`GET /_flowsafe/agent-host/runs/:agentId/:runId?resourceId=…&dispatch=1`) and the start route (`POST /_flowsafe/agent-host/start`), answer 503 from that shell; the schedule-wake route (`POST /signal/schedule`, which recovers a lost dispatch receipt through that same recovery) answers 502 with an `internal error` body from the signals router instead of settling a dispatch receipt from a read it could not make. An armed deadline is not observable through `RunSummary` in this release: the record is the run object's own wake state, no route projects it, and the exported bounds exist so a consumer can validate its own `deadlineMs` before arming rather than to read back what is armed, while `MAX_SUSPENSION_DEADLINES_PER_RUN` is exported as the operational figure a host plans against. - -A timeout resume is not an approval decision. It records `requestedByKind: 'system'` with the reserved `flowsafe-suspension-deadline` principal id, mints no connector grant, and names no reviewer. Only the runner mints the envelope: a resume request whose resume data carries the reserved `flowsafe.suspensionTimeout` key is rejected with a 400 on both the workflow run route and the agent-host resume route, so no caller can present itself to a step as an expired deadline. A step that gates a privileged action must treat its timeout branch as a denial, an escalation, or a no-op — never as consent. Because the run object resumes itself, a timeout resume passes through no host route: `RunRouterOptions.beforeResume` does not vet it and approval reconciliation does not run at that moment, so an approval record filed for the expired suspension stays open until a later host status read supersedes it against the suspension it was bound to. Suspension deadlines cover workflow runs hosted by `DurableObjectRunner`; durable agents keep their own resume path. diff --git a/packages/agent-starter/CHANGELOG.md b/packages/agent-starter/CHANGELOG.md index 61b5d56..7fbc5fc 100644 --- a/packages/agent-starter/CHANGELOG.md +++ b/packages/agent-starter/CHANGELOG.md @@ -1,5 +1,12 @@ # anchorage-agent-starter +## 0.0.14 + +### Patch Changes + +- Updated dependencies [e7fb658] + - @proofoftech/flowsafe@0.18.0 + ## 0.0.13 ### Patch Changes diff --git a/packages/agent-starter/package.json b/packages/agent-starter/package.json index b955d79..86f0325 100644 --- a/packages/agent-starter/package.json +++ b/packages/agent-starter/package.json @@ -1,6 +1,6 @@ { "name": "anchorage-agent-starter", - "version": "0.0.13", + "version": "0.0.14", "private": true, "description": "Production-shaped Cloudflare Workers starter for durable, approval-gated Anchorage agents", "type": "module", diff --git a/packages/fleet-control/CHANGELOG.md b/packages/fleet-control/CHANGELOG.md index 4a83acc..9394e2e 100644 --- a/packages/fleet-control/CHANGELOG.md +++ b/packages/fleet-control/CHANGELOG.md @@ -1,5 +1,12 @@ # @proofoftech/fleet-control +## 0.3.3 + +### Patch Changes + +- Updated dependencies [e7fb658] + - @proofoftech/flowsafe@0.18.0 + ## 0.3.2 ### Patch Changes diff --git a/packages/fleet-control/package.json b/packages/fleet-control/package.json index e3e05ce..1f19e8e 100644 --- a/packages/fleet-control/package.json +++ b/packages/fleet-control/package.json @@ -1,6 +1,6 @@ { "name": "@proofoftech/fleet-control", - "version": "0.3.2", + "version": "0.3.3", "publishConfig": { "access": "public" }, diff --git a/packages/flowsafe/CHANGELOG.md b/packages/flowsafe/CHANGELOG.md index 34a6021..bf73071 100644 --- a/packages/flowsafe/CHANGELOG.md +++ b/packages/flowsafe/CHANGELOG.md @@ -1,5 +1,19 @@ # @proofoftech/flowsafe +## 0.18.0 + +### Minor Changes + +- e7fb658: Time out one suspension. A workflow step can now arm a deadline on the suspension it creates by adding the reserved `flowsafe.deadlineMs` key to the payload it passes Mastra's `suspend()`. When that deadline elapses before the awaited signal arrives, the run's own Durable Object resumes the run itself and delivers the `flowsafe.suspensionTimeout` envelope to the expired step, which tells it apart from a real signal with the exported `isSuspensionTimeoutResumeData()` guard. The deadline is persisted in the run's Durable Object storage, so it survives eviction and hibernation, and the resume is fenced on the exact suspension it was armed against: a genuine signal that arrives first drops the deadline instead of resuming twice. + + This is backward compatible. Nothing arms unless the reserved key is present, so existing runs and existing suspensions behave exactly as before. A host that forwards a `timeoutMs` field today is unaffected: that field was ignored and remains ignored. + + Authoring notes. A step that declares a Zod `suspendSchema` must declare the reserved field, or use a loose object (`z.looseObject()`, or `.passthrough()` on a `z.object()`): Mastra validates the suspend payload and substitutes the parsed result, so a strict `z.object()` strips the key and no deadline arms. A step that declares a `resumeSchema` must accept the timeout envelope as well as its own signal shape, or every timeout resume fails that validation and the deadline is abandoned. Only a top-level suspended step can arm a deadline: a step suspended inside a nested workflow is reported under its nested path while its suspension time is recorded against the enclosing step, so there is no way to fence the resume, and the deadline is refused and logged instead of armed. A top-level step id containing a dot arms normally, but it must not coincide with a nested path: a step `'a.b'` suspended alongside a nested workflow `'a'` whose inner step `'b'` suspends gives one key for two suspensions, and every deadline on that key is refused and logged rather than armed against the wrong one. A `foreach` step arms one deadline for the step as a whole, because that is all the run summary reports for it: run sequentially, which is the default, each iteration suspends on its own and gets its own deadline, so clearing a whole loop by timeout takes one deadline per item; with `concurrency` above 1 the timeout resume delivers the envelope to every iteration suspended at that moment (at most `concurrency` of them), iterations not yet started suspend afresh with their own deadline, only the first suspended iteration's value is read per batch, and clearing the whole loop takes one deadline per batch of `concurrency` items. Values must be a whole number of milliseconds between `MIN_SUSPENSION_DEADLINE_MS` and `MAX_SUSPENSION_DEADLINE_MS` (365 days); a value outside that range is reported in the runner log and left unarmed rather than failing the suspension that Mastra has already persisted. + + Operational caveats. Wake precision is a Durable Object alarm's — near the requested time, not exact — and there is no maintenance-sweep backstop for suspension deadlines, so a lost alarm is a lost deadline. Run-level `deadlineMs` remains the swept mechanism when a hard expiry is required. One deadline fires per wake, a run arms at most `MAX_SUSPENSION_DEADLINES_PER_RUN` of them, and a failing timeout resume backs off before it is abandoned for that suspension — the spent entry is kept, never retried and never re-armed, so the same suspension cannot earn itself a fresh budget, while a later suspension of the same step starts one. Only a wake whose authoritative read succeeded spends that budget: a read that did not succeed keeps a 60 second retry cadence and charges nothing, so a storage incident cannot abandon a live deadline — unless an entry has been due for a full day with the state unreadable throughout, after which that entry is abandoned under its own log (`abandoned after 24 h of unreadable run state`). A run whose state has been readable and genuinely absent across five wakes is charged and abandoned the ordinary way, and either abandonment is permanent for the suspension it was armed against. The routes that must not answer from a fabricated read fail closed instead: while a pending owner recovery cannot read authoritative state, `POST /runs` and `GET /runs/:workflowId/:runId/dispatch-status` return 503 for that run. Each previously answered from the fabricated read in its own way: `dispatch-status` returned 200 with a fabricated summary after deleting the very row it could not see, and `POST /runs` refused the run with a 500 (`existing run … has no matching committed owner`) — the same recovery had deleted the row and released its claim, while the fabricated read still reported the run as existing. A start whose own body throws is the one path that answers otherwise: it logs the recovery read it could not make, leaves the journal armed for a later wake, and rethrows the original start failure, so the caller sees that error's own status — a 500 for an unclassified start fault, or the error's own code such as a 400 for a malformed `deadlineMs` — rather than the 503. The same guard covers agent-host owner recovery, which now retries fail-closed — keeping its journal, its reservation and its run record — instead of possibly deleting a row behind a lagging read. The agent-host routes that drive that recovery fail closed in the taxonomy of the surface they escape to rather than reporting a run they could not read. Those that escape to the thread Durable Object's own shell, such as the dispatch read (`GET /_flowsafe/agent-host/runs/:agentId/:runId?resourceId=…&dispatch=1`) and the start route (`POST /_flowsafe/agent-host/start`), answer 503 from that shell; the schedule-wake route (`POST /signal/schedule`, which recovers a lost dispatch receipt through that same recovery) answers 502 with an `internal error` body from the signals router instead of settling a dispatch receipt from a read it could not make. An armed deadline is not observable through `RunSummary` in this release: the record is the run object's own wake state, no route projects it, and the exported bounds exist so a consumer can validate its own `deadlineMs` before arming rather than to read back what is armed, while `MAX_SUSPENSION_DEADLINES_PER_RUN` is exported as the operational figure a host plans against. + + A timeout resume is not an approval decision. It records `requestedByKind: 'system'` with the reserved `flowsafe-suspension-deadline` principal id, mints no connector grant, and names no reviewer. Only the runner mints the envelope: a resume request whose resume data carries the reserved `flowsafe.suspensionTimeout` key is rejected with a 400 on both the workflow run route and the agent-host resume route, so no caller can present itself to a step as an expired deadline. A step that gates a privileged action must treat its timeout branch as a denial, an escalation, or a no-op — never as consent. Because the run object resumes itself, a timeout resume passes through no host route: `RunRouterOptions.beforeResume` does not vet it and approval reconciliation does not run at that moment, so an approval record filed for the expired suspension stays open until a later host status read supersedes it against the suspension it was bound to. Suspension deadlines cover workflow runs hosted by `DurableObjectRunner`; durable agents keep their own resume path. + ## 0.17.0 ### Minor Changes diff --git a/packages/flowsafe/package.json b/packages/flowsafe/package.json index ce729ba..1e0bb5a 100644 --- a/packages/flowsafe/package.json +++ b/packages/flowsafe/package.json @@ -1,6 +1,6 @@ { "name": "@proofoftech/flowsafe", - "version": "0.17.0", + "version": "0.18.0", "publishConfig": { "access": "public" }, diff --git a/packages/showcase/CHANGELOG.md b/packages/showcase/CHANGELOG.md index 3b46801..2a700ba 100644 --- a/packages/showcase/CHANGELOG.md +++ b/packages/showcase/CHANGELOG.md @@ -1,5 +1,12 @@ # showcase +## 0.0.20 + +### Patch Changes + +- Updated dependencies [e7fb658] + - @proofoftech/flowsafe@0.18.0 + ## 0.0.19 ### Patch Changes diff --git a/packages/showcase/package.json b/packages/showcase/package.json index 3b43b84..042988a 100644 --- a/packages/showcase/package.json +++ b/packages/showcase/package.json @@ -1,6 +1,6 @@ { "name": "showcase", - "version": "0.0.19", + "version": "0.0.20", "private": true, "description": "Anchorage showcase — six runnable workflows and seven guardrail scenarios behind one React frontend", "type": "module",