Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
77 changes: 77 additions & 0 deletions .agents/plans/21-guidance-maintainability/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
# Flow guidance maintainability

## Intent

Make Flow guidance easier to follow and cheaper to change while preserving its
own product boundary: approved work, current-source evidence, independent review,
and explicit closure. The comparison with tstack motivates clarity, composition,
and bounded review; no skill text is copied and tstack is not a dependency.

## Phases

1. **Define and measure.** Record a Flow-specific writing standard. Add outcome
scenarios for ordinary read-only status, blocked detail, and status failure.
Test each grader with both conforming and adversarial traces. Reuse the
existing scenario runner, provenance, authorization and report pipeline.
2. **Simplify the guidance.** Put the ordinary plan/run path before recovery
details. Give goal alignment and delivery one authored source, embedded into
each existing guide by the catalog. Keep all four guide IDs and existing
commands. Make status routing ordered: compact, error or conditional detail,
render, stop. Keep approval, scope, evidence, reviewer, retry, and archive
replay contracts intact.
3. **Make verification routine.** Provide one offline contributor command for
prompt structure, scenario graders, native surface integration and cassette
replay. Document the optional live scenario command separately. Keep local
results, live model behavior and release qualification clearly distinguished.
Run normal CI and independent Sol review before merge.

## Writing standard

The maintained [Flow writing standard](../../../skills/README.md) owns
instruction order, shared contracts, authority, judgments, and eval discipline.

## Acceptance and boundaries

- Shared alignment and delivery contracts have one maintained source; packaged
guides remain self-contained and readable without filesystem access.
- Normal execution precedes exception detail in the authored run guide.
- Status never advances work, does not read detail on ordinary state, reads it
once when blocked/awaiting direction, and stops after any status error.
- Existing lifecycle, schema, source binding, permissions, review/retry rules,
release thresholds, frozen artifacts and historical receipts remain unchanged.
- New model scenarios are ordinary development cases, not silently added to the
sealed 10.0.1 release matrix or treated as qualification evidence.
- No publication or fresh paid campaign is part of this change. Live model runs
require their own current bounded dispatch authorization; offline tests and
replay cannot establish that rewritten instructions perform better on models.

## Follow-up measurement

Before claiming better model adherence, run the documented existing scenario
selection under a fresh bounded authorization. Compare retained outcomes and
instruction size with the base. Replace older phrase assertions only after the
matching behavioral coverage exists; do not remove them merely to permit a
rewrite. Full release qualification still applies to any future candidate.

## Evidence

- `bun run eval:guidance`: 84 focused tests pass; all 13 gated cassettes reproduce.
The new graders use actual native-plugin fixture responses and adversarial
variants, including a tool read after a status error.
- `bun run check`: 5,766 pass, 27 skip, zero failures before final review fixes;
the clean committed push preflight repeats the complete gate after those fixes.
- `bun run smoke:live`: two packed OpenCode tests, 128 assertions, pass using a
local scripted provider. No real provider dispatch or account use.
- Independent GPT-6.1 Sol review: three rounds, final APPROVE. Fixed all findings:
accepted-close archive retry remains exact and immediate; status error grading
rejects every following tool call, not just further status reads. Error handoffs
retain supplied delivery and the mutation statement, including archive collisions.
- Authored skill text including shared fragments falls from 36,220 to 29,853
bytes. The ordinary manager stack is effectively flat: 23,599 to 23,592 bytes;
compiled command/agent surfaces fall from 14,496 to 14,335. Existing prompt
ceilings ratchet down; documentation and source ceilings are unchanged.
- Contributor preflight now owns one mandatory offline guidance command. Live
model measurement is explicitly separate, rather than silently attempted by
an ordinary contribution. No release threshold or qualification gate changes.
- New scenario behavior on real models remains unmeasured. No release, model
comparison, new dispatch ledger, paid evaluation, or old-budget reuse occurred.
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ Use this after the scripted preflight when the touched boundary needs focused ev
| Session schema, domain transitions, application gates, persistence, or source identity | Narrow tests for the touched boundary; usually `bun test tests/domain-transitions.test.ts tests/runtime-gates.test.ts tests/workspace-persistence.test.ts tests/source-identity.test.ts` |
| `src/platform/opencode/**`, tool schemas, tool registration, command routing, or config projection | Push preflight runs `bun test tests/distribution-and-surface.test.ts tests/opencode-schema-contract.test.ts`; also run `bun run typecheck` when validating manually |
| Package files or OpenCode installation guidance | `bun run build`; `bun run package:smoke`; review the standard `opencode.json` plugin entry |
| `skills/**` | Push preflight runs the distribution and direct prompt-contract tests; review the relevant skill directly. When the diff also touches `src/prompt-surfaces.ts` or `src/guidance/**`, run `bun run replay` and `bun run eval:smoke -- --scenario plan-only-stops` before push (see `docs/release-qualification.md`). |
| `skills/**` | Run `bun run eval:guidance` for the offline contributor check; review the authored skills and compiled guides. Model behavior is a separate, explicitly authorized pass using the existing runner (see `evals/README.md#guidance-changes`); record a missing live pass as unmeasured, never as an offline success. Release qualification remains required before publication. |
| Package scripts, workflow files, or release process | Focused script/workflow review; `bun run check` before merge/release |
| Release notes, README install snippet, or package version | Verify `package.json`, `CHANGELOG.md`, standard OpenCode configuration, and tag name agree |
| Public surface change: command, tool, agent, `.flow/**` path, package export, or schema | Update `docs/maintainer-contract.md`, README, skills, tests, and release notes together; treat as release-sensitive |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -166,7 +166,7 @@ fi

if range_contains "$files" '^(src/platform/opencode/|src/config-shared\.ts|src/guidance/|src/prompt-surfaces\.ts|skills/)'; then
log "Run OpenCode surface focused checks"
bun test tests/distribution-and-surface.test.ts tests/opencode-schema-contract.test.ts tests/prompt-quality.test.ts
bun run eval:guidance
fi

if range_contains "$files" '^(package\.json|README\.md|src/index\.ts|src/platform/opencode/|scripts/lib/package-surface\.ts)'; then
Expand Down
8 changes: 8 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,14 @@

One short entry per release, written for users deciding whether to upgrade.

## Unreleased

- Put ordinary Flow execution before recovery details, share goal-alignment and
delivery instructions across the existing guides, and clarify read-only status
routing. Commands, tool inputs, Session v5 and runtime gates are unchanged.
- Add three development status evals and one offline guidance check. Offline
results do not claim model improvement or qualify a new release.

## [10.0.1]

- Simplify plugin tool response assembly and mutation routing while retaining
Expand Down
7 changes: 3 additions & 4 deletions docs/maintainer-contract.md
Original file line number Diff line number Diff line change
Expand Up @@ -295,10 +295,9 @@ user starting point.
| `flow_feature_reset` | Supersede a failed attempt and optionally atomically start the exact authorized retry or dependency-independent feature through `nextFeatureId`. |
| `flow_session_close` | Close and archive the session, returning the same concise derived delivery and tiered assurance on every durably accepted close path. |

The ten lifecycle tools accept a nested `request`, return state under
`workflowData`, and require the current revision plus a stable operation ID for
mutations. `flow_session_close` additionally returns derived delivery under
`workflowData`; `flow_guidance` instead accepts a guide ID and returns Markdown.
Tools accept nested `request` and return state under `workflowData`; mutations
require a current revision and stable operation ID. Closure adds derived delivery.
`flow_guidance` instead takes an ID and returns Markdown; see [authoring](../skills/README.md).

### Guides

Expand Down
33 changes: 33 additions & 0 deletions evals/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,39 @@ asserts durable Session v5 state and the observed tool-call sequence — never
prompt wording — so a prompt can be rewritten freely as long as the outcomes
hold.

## Guidance changes

Run `bun run eval:guidance` for the offline contributor check. It combines prompt
wiring/size checks, native-surface integration, adversarial tests of the guidance
scenario graders, and the existing cassette replay. It performs no model calls
or credential setup. This is regression evidence, not model-comprehension proof.
There is no second guidance runner, report format, budget, or qualification path.

The new `guidance-status-ready`, `guidance-status-blocked`, and
`guidance-status-error` cases measure compact-only status, one detail read for a
blocked state, and stopping after a status error. Each seeds synthetic Session
state and checks observed calls, unchanged persisted state/workspace, and the
actual returned status/error text. They are development report-only cases and do
not change the frozen 10.0.1 release matrix.

For model behavior, use the ordinary runner after obtaining a current bounded
[dispatch authorization](../.agents/plans/05-release-simplification/README.md#authorize-paid-work):

```sh
bun run eval -- --model openai/gpt-6.1-sol --concurrency 1 --repeat 1 \
--scenario plan-only-stops --scenario goal-change-refused \
--scenario continuation-accepted --scenario resumes-after-interruption \
--scenario guidance-status-ready --scenario guidance-status-blocked \
--scenario guidance-status-error
```

This selection covers normal planning, scope boundaries, recovery and status.
It is not release qualification. Count all scenario steps and route probes in
the authorization; failed starts remain consumed. Reuse neither an old campaign's
unused balance nor its partial outcomes. Native-account setup and exact-artifact
options below apply unchanged. Retain the ordinary report and original failures;
compare behavior and measured instruction size before claiming improvement.

## Running

Model runs require provider credentials and an approved
Expand Down
124 changes: 124 additions & 0 deletions evals/fixtures/guidance/blocked.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,124 @@
{
"version": 5,
"id": "guidance-session",
"revision": 6,
"goal": "Add a greeting to src/greet.mjs",
"approval": "approved",
"plan": {
"summary": "Add a greeting",
"overview": "Export greet",
"requirements": ["greet returns Hello"],
"decisions": [],
"features": [
{
"id": "greet",
"title": "Greeting",
"summary": "Export greet",
"targets": ["src/greet.mjs"],
"validation": ["node --test"],
"dependsOn": []
}
],
"evidence": [
{
"requirement": "Repository suite",
"environment": "this host",
"command": "node --test",
"scope": "gate",
"platform": "other",
"assertions": []
}
]
},
"runs": [
{
"id": "guidance-run",
"featureId": "greet",
"attempt": 1,
"state": "blocked",
"startedRevision": 3,
"summary": "The greeting has not been implemented.",
"artifactsChanged": [],
"validations": [
{
"id": "seed-validation",
"featureId": "greet",
"runId": "guidance-run",
"scope": "broad",
"command": "node --test",
"sourceDigest": "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
"exitCode": 0,
"outputDigest": "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
"outputComplete": true,
"recordedRevision": 4,
"hostPlatform": "linux"
}
],
"reviews": [
{
"id": "guidance-review",
"operationId": "seed-review",
"featureId": "greet",
"runId": "guidance-run",
"kind": "final",
"sourceDigest": "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
"validationIds": ["seed-validation"],
"packet": {
"summary": "Seeded review",
"riskLenses": []
},
"createdRevision": 5,
"result": {
"verdict": "failed",
"findings": [
{
"severity": "blocking",
"summary": "greet is missing",
"evidence": "src/greet.mjs lacks greet",
"findingId": "greet.R5-01"
}
],
"terminalDisposition": "submitted",
"recordedRevision": 6
}
}
]
}
],
"operations": [
{
"id": "seed-plan",
"kind": "plan-save",
"inputDigest": "sha256:80a6a3afe0c528f1d846950c07aad916eff125260b3c302dd773dc62e8aee8a0",
"committedRevision": 1
},
{
"id": "seed-approve",
"kind": "plan-approve",
"inputDigest": "sha256:950fbd0b2ef8fefab38f5604cd7c450e239f772769f72d7164dd77387a4033e0",
"committedRevision": 2
},
{
"id": "seed-run",
"kind": "run-start",
"inputDigest": "sha256:e2c4dd3799947abcb2ff76ad4e509325aeda359f3dbaf25b894e173b595a9cfd",
"committedRevision": 3,
"entityId": "guidance-run"
},
{
"id": "seed-review",
"kind": "review-start",
"inputDigest": "sha256:8bb3373a89d00b8b75625fe2c1966fcaf5c0f419045fbf5879ee0e8962d8c07a",
"committedRevision": 5,
"entityId": "guidance-review"
},
{
"id": "seed-failure",
"kind": "feature-complete",
"inputDigest": "sha256:02bc166fb687677b1dc549a67cd52f92ec58882095fc8cbba3d8b5df61c76d8f",
"committedRevision": 6,
"entityId": "guidance-run"
}
],
"closure": null
}
49 changes: 49 additions & 0 deletions evals/fixtures/guidance/ready.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
{
"version": 5,
"id": "guidance-session",
"revision": 2,
"goal": "Add a greeting to src/greet.mjs",
"approval": "approved",
"plan": {
"summary": "Add a greeting",
"overview": "Export greet",
"requirements": ["greet returns Hello"],
"decisions": [],
"features": [
{
"id": "greet",
"title": "Greeting",
"summary": "Export greet",
"targets": ["src/greet.mjs"],
"validation": ["node --test"],
"dependsOn": []
}
],
"evidence": [
{
"requirement": "Repository suite",
"environment": "this host",
"command": "node --test",
"scope": "gate",
"platform": "other",
"assertions": []
}
]
},
"runs": [],
"operations": [
{
"id": "seed-plan",
"kind": "plan-save",
"inputDigest": "sha256:80a6a3afe0c528f1d846950c07aad916eff125260b3c302dd773dc62e8aee8a0",
"committedRevision": 1
},
{
"id": "seed-approve",
"kind": "plan-approve",
"inputDigest": "sha256:950fbd0b2ef8fefab38f5604cd7c450e239f772769f72d7164dd77387a4033e0",
"committedRevision": 2
}
],
"closure": null
}
Loading
Loading