Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -133,10 +133,10 @@ jobs:

# Mastra compat matrix: the libraries pin behavioral contracts to
# @mastra/core internals (resume-context merge-over semantics, snapshot
# shape, the reprocess-part key, the six-table inventory). The peer range
# (^1.50.0) admits any 1.x, so run the library suites against the newest
# published 1.x -- drift becomes a CI signal here instead of a consumer's
# runtime surprise. The tripwire tests (runtime.test.ts merge pin,
# shape, the reprocess-part key, the six-table inventory). The supported
# peer is exactly 1.50.0; run the library suites against the newest
# published 1.x as a signal for a deliberate future compatibility update.
# The tripwire tests (runtime.test.ts merge pin,
# ProcessorRunner reprocess-key probe, mastra-schema-guard) are the teeth.
mastra-compat:
runs-on: ubuntu-latest
Expand Down
8 changes: 4 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,16 +34,16 @@ Anchorage is not a model provider, identity provider, hosted SaaS, generic proce

Fleet control requires Node 22.22.0 or later. Flowsafe requires Node 22 or
later. Breakwater requires Node 22.3 or later for its built-in Agent CLI
executor. All three packages are ESM-only. Breakwater and Flowsafe declare
`@mastra/core` `^1.50.0` as their peer range. React 18 or 19 is needed only for
executor. All three packages are ESM-only. Breakwater and Flowsafe require
`@mastra/core` `1.50.0`. React 18 or 19 is needed only for
the optional Flowsafe approval UI. Hosts that run `flowsafe-provision` or the
Fleet Control Wrangler backend must provide Wrangler `>=4.118 <5`; Wrangler is
not a Flowsafe peer dependency.

## Start with breakwater

```bash
npm install @mastra/core@^1.50.0 @proofoftech/breakwater
npm install @mastra/core@1.50.0 @proofoftech/breakwater
```

Create a guarded agent and pass the authenticated actor through Mastra's `RequestContext`:
Expand Down Expand Up @@ -139,7 +139,7 @@ explains every permission, required store, and accepted limit:
## Add durable approvals

```bash
npm install @mastra/core@^1.50.0 @proofoftech/breakwater @proofoftech/flowsafe
npm install @mastra/core@1.50.0 @proofoftech/breakwater @proofoftech/flowsafe
```

Start from the copy-ready baseline Worker in [`packages/flowsafe/deploy/`](packages/flowsafe/deploy/README.md). It wires one Durable Object per run, D1 snapshots and approvals, authenticated run routes, server-derived connector grants, live-streaming opt-in, SLA sweep, retention, audit export, and a sample gated workflow.
Expand Down
6 changes: 4 additions & 2 deletions docs/breakwater-architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,7 @@ output: model/tools -> app output -> policy output

The durable resume line describes which `processInput` hook runs during rehydration. Before installing the registries, Flowsafe restores the complete input and LLM-request processor lists for later loop hooks. It also restores the same tools, memory, model, application output processors, and mandatory policy output processor as initial preparation. An RBAC denial stops rehydration before registry installation or resumed tool execution.

The call allowlist contains `requestContext`, `runId`, `memory`, and `abortSignal`. Unknown own properties fail even when their value is `undefined`. Construction fixes `maxSteps` and `toolChoice`, enables policy hold-back, and disables background continuations.
The call allowlist contains `requestContext`, `runId`, `memory`, and `abortSignal`. Calls are copied into frozen allowlisted snapshots, and unknown own properties fail even when their value is `undefined`. Construction fixes `maxSteps` and `toolChoice`, enables policy hold-back, and disables background continuations.

Application input processors may implement only `processInput`. Application output processors must implement both stream and final-result enforcement. Processor workflows and the reserved IDs `breakwater-rbac` and `breakwater-policy-engine` fail construction.

Expand All @@ -112,7 +112,9 @@ Application input processors may implement only `processInput`. Application outp
- structured-object stream snapshots;
- custom synchronous or asynchronous policy evaluators.

Structured-object values are available on Mastra's streaming path. Under the supported Mastra version, non-streaming `generate()` exposes structured output through answer text rather than a separate result object. A policy scoped only to `object` therefore has no non-streaming result coverage; the engine requires an audit sink for that configuration and records the limitation.
Under the supported Mastra version, structured objects parsed by `generate()` and the chunks core's `StructuredOutputProcessor` emits never pass through the agent's output processors. Mastra also copies a parsed value into messages and may send it to persistence and observability hooks before `generate()` returns. A post-generation wrapper gate is therefore not a containment boundary. The guarded agent rejects structured output before execution and rejects object-only policies at construction. A standalone `PolicyEngine` validates processor-visible object chunks as JSON, evaluates their canonical snapshots, forwards the same canonical clones, and aborts at the result boundary when an object-only policy saw no such chunk.

The guarded handle also carries a versioned host protocol. Flowsafe checks that protocol before durable wrapping and rejects guarded structured output on every durable entry point, preserving the narrow handle's refusal even though Mastra's durable runner invokes the raw agent through processor lists.

## Streaming behavior

Expand Down
2 changes: 1 addition & 1 deletion docs/breakwater-purpose-and-boundaries.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ write.

`createGuardedAgent()` is the supported in-process boundary for protected agent execution. It accepts construction-time agent configuration instead of an existing raw `Agent`, then returns a narrow handle with unstructured `generate()` and `stream()` methods.

The factory fixes the step budget and tool choice, requires an audit logger, forces streaming policy hold-back, and disables agent background continuations. A call must include trusted `requestContext`; the only other accepted keys are `runId`, `memory`, and `abortSignal`. Unknown keys fail even when their value is `undefined`.
The factory fixes the step budget and tool choice, requires an audit logger, forces streaming policy hold-back, and disables agent background continuations. A call must include trusted `requestContext`; the only other accepted top-level keys are `runId`, `memory`, and `abortSignal`. Unknown keys, including `structuredOutput`, fail even when their value is `undefined`.

The direct path authorizes before application processors run. Durable preparation lists the same gates in this order:

Expand Down
4 changes: 2 additions & 2 deletions docs/fleet-control.md
Original file line number Diff line number Diff line change
Expand Up @@ -185,7 +185,7 @@ await forceDecommissionDeployment({
});
```

The function always acquires `store.withDeploymentLease()` before it reads the ledger. An absent deployment succeeds without a provider mutation. A terminal `decommissioned` record also succeeds and removes the retained ledger row.
The function always acquires `store.withDeploymentLease()` before it reads the ledger. An absent deployment succeeds without a provider mutation. A terminal `decommissioned` record also succeeds, removes the retained ledger row, and emits no audit event.

A concurrent provision or decommission receives the same lease-acquisition error as `decommissionDeployment()`. A `database-reserved` row has not authorized provider creation, so force decommission removes that reservation without a provider call.

Expand All @@ -199,7 +199,7 @@ The force path uses persisted resource identities only. It never fetches an arti

Each provider mutation receives the active lease fence. `WranglerLoopBackend` requires route API implementations of both `getDatabase` and `deleteDatabase` before the first D1 lookup. The exact-ID lookup, deletion, and confirmation all run within that route fence; force deletion never falls back to Wrangler. The ledger persists `decommissioning`, `traffic-removed`, `credentials-revoked`, and `database-deleting`, so an interrupted call repeats the incomplete idempotent stage.

Provider 404 responses mean the resource is already absent. After D1 is absent, the function persists `decommissioned`, emits `DecommissionAuditEvent` with `forced: true`, and calls `lease.delete()`. If audit delivery fails, the terminal row remains and a retry emits the event again before deleting the row. Normal decommission uses the same optional event with `forced: false`.
Provider 404 responses mean the resource is already absent. After D1 is absent, the function persists `decommissioned`, emits `DecommissionAuditEvent` with `forced: true`, and calls `lease.delete()`. If initial audit delivery fails, the terminal row remains. A retry deletes it without repeating provider mutations or redelivering the event. Normal decommission uses the same optional event with `forced: false`.

Force decommission does not delete the ordinary Worker script, application R2 buckets, or control-plane retention data. It removes the deployment’s ingress, live Worker secrets, database, and fleet ownership record. After the call returns, the host deletes its separate retention row and revokes its gateway key. Workers for Platforms fails closed unless its backend implements equivalent spec-free primitives; its dispatch route and trusted-resource topology cannot use the ordinary Worker route API contract.

Expand Down
6 changes: 3 additions & 3 deletions docs/getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,15 +8,15 @@ This guide takes a Mastra application from package installation to one guarded c
complete repository and private fleet-control package require Node.js 22.22.0
or later
- An ESM TypeScript project using `moduleResolution: "NodeNext"`, `"Node16"`, or `"Bundler"`
- `@mastra/core` in the `^1.50.0` peer range
- `@mastra/core` `1.50.0`
- A Cloudflare account, D1 database, and Durable Objects only when deploying flowsafe

React is not required by flowsafe unless you import `@proofoftech/flowsafe/approval-ui`.

## Install breakwater

```bash
npm install @mastra/core@^1.50.0 @proofoftech/breakwater
npm install @mastra/core@1.50.0 @proofoftech/breakwater
```

### Add agent-boundary policy
Expand Down Expand Up @@ -66,7 +66,7 @@ export const agent = createGuardedAgent({
});
```

`createGuardedAgent()` forces policy hold-back, disables background continuations, and rejects per-call processor, tool, model, callback, hook, structured-output, and execution-limit overrides. Application input processors can enforce only initial input. Application output processors must enforce both streamed and final results.
`createGuardedAgent()` forces policy hold-back, disables background continuations, and rejects per-call processor, tool, model, callback, hook, structured-output, and execution-limit overrides. Under the supported Mastra version, parsed structured output reaches messages and persistence before a wrapper gate could inspect it, so guarded structured output remains unavailable. Object-only policies are rejected at construction. Application input processors can enforce only initial input. Application output processors must enforce both streamed and final results.

The narrow handle prevents accidental bypass through Mastra's larger `Agent` API. It is a trusted in-process API, not a sandbox against hostile code running with the same imports and credentials.

Expand Down
14 changes: 11 additions & 3 deletions docs/policy-engine-design.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,13 @@ interface PolicyEvaluator {

Each decision is either `{ allowed: true }` or `{ allowed: false, reason }`. An evaluator exception is a policy-engine failure, is audited as an error, and fails the request closed.

Policies run in array order. The first denial aborts the phase.
Policies run in array order. The engine snapshots the list and each evaluator's
name, phase/channel selectors, hold-back hint, and evaluator reference at
construction. Class-based evaluators keep their original receiver, so private
fields and helper methods continue to work. Later replacement of the evaluator
method or mutation of caller-owned selector arrays does not change enforcement;
mutable state owned by the evaluator instance or its closures remains the
application's responsibility. The first denial aborts the phase.

## Phases and channels

Expand All @@ -36,11 +42,11 @@ Input processing joins textual message parts and evaluates them under the `answe
| --- | --- |
| `answer` | Client-visible answer text |
| `reasoning` | Reasoning stream deltas |
| `object` | JSON-stringified structured-output snapshots |
| `object` | Canonical JSON structured-output snapshots |

A policy defaults to both phases and the `answer` channel. Set `phases` and `channels` when a policy applies more narrowly.

Mastra exposes structured objects to the processor on the streaming path. Under the supported core version, the non-streaming output result has no separate object field. JSON carried in answer text is still inspected by policies that include `answer`; an object-only policy has no final `generate()` coverage.
Under the supported core version, the engine sees the `object` channel only for object chunks that flow through the processor chain (model-native streaming). It requires those values to be JSON data, evaluates the canonical serialization, and forwards the same canonical clone. A `generate()` result's parsed object and core's `StructuredOutputProcessor` chunks never pass through the chain, and Mastra may expose the parsed value before `generate()` returns. `createGuardedAgent` therefore rejects structured output and rejects object-only policies at construction. JSON carried in answer text is still inspected by policies that include `answer`. A standalone engine with an object-only policy requires an audit sink and aborts at the result boundary unless a processor-visible object chunk provided coverage.

## Built-in content policies

Expand Down Expand Up @@ -105,6 +111,8 @@ Properties:
- A classifier that must see the full output should opt into `holdBackChars: Infinity`, accepting full buffering.
- Hold-back changes delta boundaries. Consumers must treat text deltas as chunks, not semantic tokens.

Measured cost (2026-08-15, Node 22.22.0, `@mastra/core` 1.50.0; opt-in evidence tests in `packages/breakwater/src/policy-engine/policy-engine.test.ts`, run with `BREAKWATER_PERF=1`): a 4 MB stream in 2 KB deltas against string-pattern policies processes at roughly 1 MB/s including per-chunk harness overhead, with peak held text of 17 characters — the pattern-bound window, not the stream length. Any RegExp policy forces the unbounded window: a 1 MB stream keeps all of it pending and releases nothing until the channel ends. Prefer string patterns, overriding `holdBackChars` when the match bound is known, for large-stream leak prevention.

## Tool policy

`ToolPolicyEvaluator` receives the connector manifest, input, request context, and connector identity before execution.
Expand Down
33 changes: 15 additions & 18 deletions docs/proposals/breakwater-improvement-roadmap.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ testable, and appropriately placed.
| **Shipped** | Optional connector invocation permissions | `PermissionManifest.requiredPermissions` is enforced against the trusted principal-permissions projection before dry-run and approval — authorization and approval answer different questions |
| **Shipped** | Secure single-deployment policy preset | `singleTenantConnectorPolicies()` validates durable stores, audit, egress, permission wiring, background policy, and deployment-wide isolation posture |
| **P1** | Add manifest-conformance tooling and stronger egress posture | Runtime safety depends on honest manifests and use of `runtime.fetch` |
| **P1** | Close known agent-output enforcement gaps | Structured output and override seams must not silently weaken mandatory policy |
| **Shipped** | Fail closed on known agent-output enforcement gaps | The guarded handle rejects structured output; processor-visible object chunks are canonicalized and missing required coverage aborts |
| **P1** | Introduce stable decision codes and richer audit correlation | Reason strings are not a durable API for alerts, metrics, or incident response |
| **P2** | Add policy bundles, versioning, and drift reporting | Useful once several agents/connectors share centrally managed policy |
| **P2** | Add stricter rate-limit algorithms and operational maintenance | Fixed windows and durable stores need production-scale alternatives and cleanup |
Expand Down Expand Up @@ -461,26 +461,23 @@ Infrastructure must still supply process/container/network controls for raw
sockets, DNS/IP restrictions, and compromised dependencies. Do not market
`runtime.fetch` as a complete sandbox.

### 13. Close Agent Output Coverage Gaps
### 13. Fail Closed on Agent Output Coverage Gaps (shipped)

Known processor/API limitations must remain loud:
#### Shipped implementation

- Under the pinned Mastra surface, non-streaming structured output does not
expose a parsed object to the output-result processor.
- Object-only policy therefore has weaker non-streaming coverage.
- The stream hold-back implementation depends on a Mastra reprocessing state
key guarded by compatibility tests.
Under the pinned core, parsed `generate()` objects and `StructuredOutputProcessor` chunks bypass output processors. Mastra also copies the parsed value into messages and may send it to memory, tracing, scorers, and logs before `generate()` returns. A post-generation wrapper gate is therefore too late. `createGuardedAgent()` rejects `structuredOutput` on both public methods before model execution and rejects object-only policies at construction. Guarded structured output remains planned until a verified pre-persistence seam exists.

The versioned host protocol lets Flowsafe preserve that refusal even though Mastra's durable runner uses the branded raw agent and processor lists instead of the narrow handle. Flowsafe rejects structured output on its guarded durable stream, generate, and prepare entry points.

Improvements:
For processor-visible object chunks, `PolicyEngine` rejects non-JSON values, evaluates the canonical serialization, and forwards the same canonical clone. An object-only standalone policy requires an audit sink and aborts at the result boundary when no object chunk reached the processor. Hold-back cost is measured by opt-in evidence tests (`BREAKWATER_PERF=1`): a bounded string-pattern window holds only the pattern-bound trailing window regardless of stream size, while any RegExp policy buffers the whole stream; numbers are recorded in [Policy engine](../policy-engine-design.md).

- Gate the final parsed object in the guarded-agent wrapper when Mastra returns
it outside the processor result.
- Refuse configurations whose chosen invocation mode cannot expose a required
channel.
- Keep an upgrade tripwire around Mastra's processor and stream semantics.
- Add leak tests proving no forbidden prefix is emitted before a late detector
fires.
- Measure hold-back memory and latency under large streams.
Historical limitations that motivated this section:

- Under the pinned Mastra surface, non-streaming structured output did not
expose a parsed object to the output-result processor.
- Object-only policy therefore had no enforceable guarded non-streaming path.
- The stream hold-back implementation depends on a Mastra reprocessing state
key guarded by compatibility tests (still true; unchanged).

### 14. Add Stable Decision Codes and Safe Error Surfaces

Expand Down Expand Up @@ -724,7 +721,7 @@ The shipped host deliberately omits a public raw-resume route. It accepts struct
1. ~~Publish secure policy presets.~~ Shipped as `singleTenantConnectorPolicies()`.
2. Add connector conformance tests and global-fetch linting.
3. Generate the manifest/policy coverage report.
4. Close structured-output policy coverage.
4. Close structured-output policy coverage. Shipped fail-closed refusal; gated support still requires a pre-persistence seam (see §13).
5. Add stable decision codes.

### Phase E: Operations and scale
Expand Down
Loading
Loading