Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
68 changes: 68 additions & 0 deletions .agents/plans/22-flow10-guidance-release/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
# Flow 10.0.2 qualification

Ship PR #211's guidance consolidation and status-error reporting correction.
The package bytes changed, so the narrow offline patch path does not apply.
Qualify a new exact artifact; preserve 10.0.1's completed campaigns, canary,
publication records and consumed budgets without reuse or partial-result pooling.

## Policy

The operator selected OpenAI-only with review of an exact-version exception.
Only 10.0.2 gains that exception, pinned to direct `openai/gpt-6.1-sol` for manager
and reviewer. 10.0.0/10.0.1 retain their exact historical policies. Future versions
retain two distinct providers; aliases of the same provider do not create breadth.
No cross-provider reliability claim follows from a single-provider qualification.

Keep the existing 17-case prefix and all its pass/sample thresholds, zero false
completions, zero unsubmitted reviews, canonical Linux/OpenCode 2.1.4 host,
HTTP transport, recovery off, and environment-only reserves. Append the three
status cases at three primary attempts each, 100% pass rate, one declared reserve
each. This gives 20 cases, 81 primary cells, 20 reserves; 99 primary steps and 26
reserve steps, plus one model route probe: 100 planned or 126 maximum top-level
starts. Failed starts count; internal turns and tokens are not capped.

The status fixtures are synthetic starting state, not paid review observations.
In particular the blocked fixture carries a seeded failed review. Retain it as
fixture context; do not describe it as a review generated by this campaign.

## Gates and approvals

1. Review and merge version, policy, regression tests, installation pins and
changelog through normal CI. Pass frozen install, root check, replay, audit and
local-provider packed native smoke without importing an account.
2. Freeze the exact merged source, artifact bytes, manifest, grader closure,
canonical toolchain and Linux preparation image. Verify source before/after,
all packaged outputs and byte-exact repack. Prepare a sealed proposal and
supervisor using maintained entry points. Never run a stale 10.0.1 supervisor.
3. Ask for 126 fresh starts bound to the frozen candidate/model and a concrete
expiry. Create no ledger or model call before approval. Use the existing
OpenCode 2 OpenAI subscription, with no API-key fallback or other vendor.
4. After grant, run zero-generation native readiness, then the complete new
matrix once. Retain original failures. Only declared retryable environment
reserves activate; product/evaluator failures do not. At expiry/cap stop new
starts and drain authorized work. Close only owned worker/source/tunnel.
5. Inspect actual report rates, samples, false completion and reviewer gates.
Only after PASS prepare the exact-artifact canary offline and request its
separate one-start approval. The matrix budget cannot fund that canary.
6. Seal/regrade complete evidence, land it without changing product bytes,
recheck main CI and exact artifact identity, and publish through the maintained
release workflow under release authority. Verify npm and GitHub bytes match.

The earlier 224 qualification/canary starts stay consumed. Their unused balances
are not transferred or refunded. The protected native service is accessed only
through authorized APIs; no OS query or signal of the protected native account service.
No source guard removal, reauthentication or credential fallback is authorized.

## Preparation evidence

- Frozen install: no dependency changes. Audit: no vulnerabilities across 278
packages. Replay: all 13 gated cassettes reproduce.
- Fifty focused policy, sampling and documentation tests pass. The first full
check retained two stale current-version sampling assertions; both are fixed.
Clean committed push preflight repeats the complete gate before submission.
- Packed local-provider native smoke passes two tests and 128 assertions without
importing an account or generating real-model work.
- Independent GPT-6.1 Sol review: two rounds, final APPROVE. Exact-version scope,
historical policy preservation and 100/126 dispatch arithmetic were checked.
- This document and the count draft authorize no paid starts. The final proposal
still requires exact merged-source/artifact/evaluator bindings and approval.
12 changes: 8 additions & 4 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,17 @@

One short entry per release, written for users deciding whether to upgrade.

## Unreleased
## [10.0.2]

- Put ordinary Flow execution before recovery details, share goal-alignment and
delivery instructions across the existing guides, and clarify read-only status
routing. Commands, tool inputs, Session v5 and runtime gates are unchanged.
- Add three development status evals and one offline guidance check. Offline
results do not claim model improvement or qualify a new release.
routing. Commands, tool inputs, Session v5 schema and runtime gates are unchanged.
- Add three status evals and one offline guidance check. Qualification retains
all 17 earlier cases and adds the three status cases at 100% over three attempts.
Exact 10.0.2 pins manager and reviewer to OpenAI GPT-6.1 Sol; future releases
keep two providers. No cross-provider reliability claim follows.

Upgrade with `opencode2 plugin add opencode-plugin-flow@10.0.2`.

## [10.0.1]

Expand Down
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ Install OpenCode v2 and the exact Flow release:
```sh
npm install -g @opencode/cli@2.1.4
opencode2 --version
opencode2 plugin add opencode-plugin-flow@10.0.1
opencode2 plugin add opencode-plugin-flow@10.0.2
```

The version command must report `opencode v2.1.4` or a later tested 2.x version.
Expand All @@ -57,7 +57,7 @@ For project scope, add this entry to the project's `opencode.json`:

```json
{
"plugins": ["opencode-plugin-flow@10.0.1"]
"plugins": ["opencode-plugin-flow@10.0.2"]
}
```

Expand All @@ -74,7 +74,7 @@ Set plugin defaults:
```json
{
"plugins": [{
"package": "opencode-plugin-flow@10.0.1",
"package": "opencode-plugin-flow@10.0.2",
"options": {
"reviewer": { "model": "provider/model", "steps": 80 },
"typesafeApiKeyEnv": "MY_TYPESAFE_KEY"
Expand Down
2 changes: 1 addition & 1 deletion docs/development.md
Original file line number Diff line number Diff line change
Expand Up @@ -158,7 +158,7 @@ Follow the [frozen-candidate sequence](release-qualification.md#running-it):
finish fixes and dependency updates, pass deterministic checks, approve paid
evals.
`bun run qualify -- --campaign-dir <dir> --canary <record>` seals the
policy grid, canary and grader evidence. 10.0.0/10.0.1 pin `openai/gpt-6.1-sol`.
policy grid, canary and grader evidence. 10.0.0/10.0.1/10.0.2 pin `openai/gpt-6.1-sol`.
See [model policy](release-qualification.md#the-qualification-bar) for historical
exceptions and later defaults. Commit the bundle before tagging;
interrupted results cannot qualify.
Expand Down
29 changes: 14 additions & 15 deletions docs/release-qualification.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ This page owns release thresholds, candidate freezing, and publication order.

| Threshold | Value | Why |
| --- | --- | --- |
| Distinct providers | 1 for 9.1.0, 9.2.0, 9.3.0, 9.4.0, 9.5.0, 9.6.0, and 10.0.0/10.0.1; otherwise 2 | 9.5.0, 9.6.0, and 10.0.0/10.0.1 pin `openai/gpt-6.1-sol`; earlier exceptions pin GPT-6 Sol. No cross-provider reliability claim. |
| Distinct providers | 1 for 9.1.0, 9.2.0, 9.3.0, 9.4.0, 9.5.0, 9.6.0, and 10.0.0/10.0.1/10.0.2; otherwise 2 | Explicit version-scoped model pins below; no cross-provider reliability claim. |
| False completions | 0 | A `completed` closure the document itself contradicts is the failure Flow exists to prevent. |
| Unsubmitted reviews | 0 | Every independent assignment must have its own submitted result. |
| Scored attempts per provider | 3 at 100%; 10 at 90% | The frozen release plan gives each threshold enough trials to express its allowed failures. |
Expand All @@ -26,6 +26,7 @@ This page owns release thresholds, candidate freezing, and publication order.
| `auto-two-features-evidence` | 100% in 9.4.0 through 9.6.0 and the v2 profile | Two dependent features, native reviewer packet access and final gate. |
| `auto-prerequisite-repair` | 100% in 9.4.0 through 9.6.0 and the v2 profile | In-target repair or accepted failed-gate amendment; immutable canonical gate. |
| `auto-observe-with-required-pass` | 100% in 9.4.0 through 9.6.0 and the v2 profile | Honest nonzero audit plus separate required pass on the reviewed source. |
| Three status cases | 100% from 10.0.2 | Read-only compact/detail/error routes preserve state, supplied delivery and mutation disclosure. |
| Five delivery cases | 100% in 9.6.0 and the v2 profile | Completion, deferral, nonzero observations, full detail and idle status preserve truthful native facts. |

Ungated exploratory scenarios are listed in [evals](../evals/README.md#scenarios).
Expand All @@ -50,16 +51,14 @@ failures never activate reserves. Evaluator failure is `NOT VERIFIED`;
persistence failure stops without a finalized report.

Repository code owns catalog order; persisted `catalog.json` must match.
Primary/reserve cells on GPT-6 Sol are 38/8 in 9.1.0 and 9.2.0, 48/9 in
9.3.0, and 57/12 in 9.4.0. The 9.4.0 profile adds three autonomous cases.
Version 9.5.0 keeps those twelve cases on GPT-6.1 Sol, with 57/12 cells.
Version 9.6.0 adds five delivery cases, collects them first, and uses 72/17
cells. 10.0.0/10.0.1 retain all 17 cases on `openai/gpt-6.1-sol` with 72 primary
and 17 reserve cells. Other v2 releases retain two routes and 144/34 cells.
Manager and reviewer share the approved model per cell. Recovery is off.
Other historical generic profiles use 96/18. Cell counts are not dispatch limits.
Changed order requires a new freeze and approval. Existing campaigns stay immutable; narrowed or merged reports
cannot qualify.
Primary/reserve cells: 9.1.0/9.2.0 use 38/8; 9.3.0 uses 48/9;
9.4.0/9.5.0 use 57/12; 9.6.0 and 10.0.0/10.0.1 use 72/17.
10.0.2 appends three status cases to that 17-case prefix, each three attempts at
100%, giving 81/20 cells. Later v2 releases keep two providers and 20 cases,
162/40 cells. Other historical generic profiles use 96/18.
Manager and reviewer share the approved model per cell; recovery is off.
Cells are not dispatch limits. Changed order requires a new freeze and approval.
Existing campaigns stay immutable; narrowed or merged reports cannot qualify.

Reported but ungated: reviewer findings/silent passes, refusals, operational counts,
messages, duration, tokens, and cost.
Expand Down Expand Up @@ -115,7 +114,7 @@ Keep that ledger across retries. Budget-stopped campaigns cannot qualify.
For native accounts, follow the [source setup](../evals/README.md#native-account-authentication).
Prepare with `--native-auth`, launch with `--native-auth-source`, and record with
the matching `--launch-receipt`. Live connection data remains private.
For 9.1.0 through 9.4.0, pin GPT-6 Sol. Exact 9.5.0, 9.6.0, and 10.0.0/10.0.1 pin GPT-6.1 Sol.
For 9.1.0 through 9.4.0, pin GPT-6 Sol. Exact 9.5.0, 9.6.0, and 10.0.0/10.0.1/10.0.2 pin GPT-6.1 Sol.

```bash
env -u TYPESAFE_API_KEY -u OPENCODE_FLOW_REVIEWER_MODEL \
Expand All @@ -133,9 +132,9 @@ The canary launcher needs its separately approved ledger in `FLOW_EVAL_AUTHORIZA
and provider access through its environment. It prints the native export path,
collects it before stopping the isolated host, and charges failed launches.

Current steps derive 90 primary dispatches, 23 reserves, and one route
probe, giving 91 planned or 114 maximum top-level dispatches. Historical 9.6.0
has the same counts. Internal turns and reviewer generations are separate;
10.0.2 derives 99 primary dispatches, 26 reserves, and one route probe: 100 planned
or 126 maximum top-level starts. Historical 9.6.0 and 10.0.0/10.0.1 use 91/114.
The default two-provider 20-case profile uses 200 planned or 252 maximum starts. Internal turns and reviewer generations are separate;
no dollar cap follows. Canary approval is separate. Refresh the draft at final
freeze. Paid work stays HOLD until purpose, models, count, and expiry are approved.

Expand Down
7 changes: 4 additions & 3 deletions evals/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,8 @@ The new `guidance-status-ready`, `guidance-status-blocked`, and
`guidance-status-error` cases measure compact-only status, one detail read for a
blocked state, and stopping after a status error. Each seeds synthetic Session
state and checks observed calls, unchanged persisted state/workspace, and the
actual returned status/error text. They are development report-only cases and do
not change the frozen 10.0.1 release matrix.
actual returned status/error text. They remain development report-only in ordinary runs. From 10.0.2 they are also
required release cases; the frozen 10.0.1 matrix stays unchanged.

For model behavior, use the ordinary runner after obtaining a current bounded
[dispatch authorization](../.agents/plans/05-release-simplification/README.md#authorize-paid-work):
Expand Down Expand Up @@ -167,7 +167,8 @@ Release mode is strictly sequential (`--concurrency 1`). Versions 9.1.0 and
9.3.0 pins the same model with 48 primary targets and nine reserves. Version
9.4.0 uses 57 primary targets and 12 reserves on the same route. 10.0.0/10.0.1
pin `openai/gpt-6.1-sol` across all 17 cases with 72 primary cells and 17 reserves.
Other v2 releases retain two routes and 144/34 cells. These exceptions make
10.0.2 adds the three required status cases for 81/20 cells on the same route.
Later v2 releases require two providers and 162/40 cells. These exceptions make
no cross-provider reliability claim.
Cells are scheduling units; count scenario steps and probes separately for the
dispatch authorization. Only retained retryable
Expand Down
33 changes: 30 additions & 3 deletions evals/release-policy.ts
Original file line number Diff line number Diff line change
Expand Up @@ -165,6 +165,28 @@ const deliveryParsed = parseCaseCatalog([
if (!deliveryParsed.ok) throw new Error("Delivery release policy is invalid.");
const DELIVERY_RELEASE_CATALOG = deliveryParsed.value;

// 10.0.2 adds status coverage after the unchanged historical 17-case prefix.
const guidanceParsed = parseCaseCatalog([
...DELIVERY_RELEASE_CATALOG,
...[
"guidance-status-ready",
"guidance-status-blocked",
"guidance-status-error",
].map((caseId) => ({
caseId,
caseVersion: 1,
evidenceClass: "conformance",
oracle: "durable-state",
release: "required",
minProviders: 1,
minScoredAttempts: 3,
minPassRate: 1,
reviewerPromotionRecordSha256: null,
})),
]);
if (!guidanceParsed.ok) throw new Error("Guidance release policy is invalid.");
const GUIDANCE_RELEASE_CATALOG = guidanceParsed.value;

export type ReleaseProfile = {
readonly catalog: ValidatedCaseCatalog;
readonly requiredModels: readonly ModelIdentity[] | null;
Expand Down Expand Up @@ -197,18 +219,23 @@ export function targetsOpenCodeV2(packageVersion: string): boolean {
}

const V2_RELEASE: ReleaseProfile = {
catalog: DELIVERY_RELEASE_CATALOG.map((row) => ({
catalog: GUIDANCE_RELEASE_CATALOG.map((row) => ({
...row,
minProviders: 2,
})),
requiredModels: null,
};

const OPENAI_ONLY_V2_RELEASES = new Set(["10.0.0", "10.0.1"]);
const OPENAI_ONLY_V2_RELEASES = new Set(["10.0.0", "10.0.1", "10.0.2"]);

export function releaseProfile(packageVersion: string): ReleaseProfile {
if (OPENAI_ONLY_V2_RELEASES.has(packageVersion))
return openAiOnlyRelease(DELIVERY_RELEASE_CATALOG, "gpt-6.1-sol");
return openAiOnlyRelease(
packageVersion === "10.0.2"
? GUIDANCE_RELEASE_CATALOG
: DELIVERY_RELEASE_CATALOG,
"gpt-6.1-sol",
);
if (targetsOpenCodeV2(packageVersion)) return V2_RELEASE;
if (packageVersion === "9.6.0")
return openAiOnlyRelease(DELIVERY_RELEASE_CATALOG, "gpt-6.1-sol");
Expand Down
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "opencode-plugin-flow",
"version": "10.0.1",
"version": "10.0.2",
"description": "Durable planning, validation, and independent review for OpenCode v2",
"type": "module",
"repository": {
Expand Down
6 changes: 4 additions & 2 deletions tests/documentation-contract.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -599,8 +599,10 @@ describe("Flow documentation contract", () => {
changelog,
`[${packageVersion}]${releaseDate ? ` - ${releaseDate}` : ""}`,
);
expect(currentReleaseNotes).toContain(
`opencode plugin add opencode-plugin-flow@${packageVersion}`,
expect(currentReleaseNotes).toMatch(
new RegExp(
`opencode2? plugin add opencode-plugin-flow@${packageVersion.replaceAll(".", "\\.")}`,
),
);
// Release notes must state the schema impact explicitly rather than the
// uninformative "public surface is unchanged".
Expand Down
18 changes: 9 additions & 9 deletions tests/eval-release-sampling.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -381,24 +381,24 @@ describe("release eval sampling", () => {
).toThrow("openai/gpt-6.1-sol");
});

test("schedules the current OpenAI-only seventeen-case grid", () => {
test("schedules the current OpenAI-only twenty-case grid", () => {
const scenarios = releaseScenarios(packageJson.version);
expect(scenarios).toHaveLength(17);
expect(scenarios).toHaveLength(20);
const plan = campaignPlanFor({
models: ["openai/gpt-6.1-sol"],
scenarios,
sampling: { kind: "release", packageVersion: packageJson.version },
opencodeVersion: "2.1.4",
});
expect(plan.stoppingRule.count).toBe(72);
expect(plan.budget.maxAttempts).toBe(89);
expect(plan.abortPolicy.maxReplacementBlocks).toBe(17);
expect(plan.stoppingRule.count).toBe(81);
expect(plan.budget.maxAttempts).toBe(101);
expect(plan.abortPolicy.maxReplacementBlocks).toBe(20);
expect(
plan.cells.filter((cell) => cell.schedule === "primary"),
).toHaveLength(72);
).toHaveLength(81);
expect(
plan.cells.filter((cell) => cell.schedule === "environment-reserve"),
).toHaveLength(17);
).toHaveLength(20);
});

test("keeps ordinary scenario catalogs report-only", () => {
Expand Down Expand Up @@ -603,8 +603,8 @@ for (const version of ["10.0.0", "10.0.1"]) {
}
});
}
test("10.0.2 retains the default two-provider requirement", () => {
expect(releaseCatalog("10.0.2").every((row) => row.minProviders === 2)).toBe(
test("10.0.3 retains the default two-provider requirement", () => {
expect(releaseCatalog("10.0.3").every((row) => row.minProviders === 2)).toBe(
true,
);
});
10 changes: 5 additions & 5 deletions tests/qualification-cli.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -55,11 +55,11 @@ import { assertQualificationBundle } from "../scripts/release-metadata.js";

import { autoQualifiedOutcome } from "./fixtures/auto-qualified-outcome.js";

test("current release defaults retain 17 cases on OpenAI and three autonomous attempts", () => {
expect(packageJson.version).toBe("10.0.1");
test("current release defaults retain 20 cases on OpenAI and three autonomous attempts", () => {
expect(packageJson.version).toBe("10.0.2");
const scenarios = releaseScenarios(packageJson.version);
const catalog = releaseCatalog(packageJson.version);
expect(scenarios).toHaveLength(17);
expect(scenarios).toHaveLength(20);
expect([...new Set(catalog.map((row) => row.minProviders))]).toEqual([1]);
const plan = campaignPlanFor({
models: ["openai/gpt-6.1-sol"],
Expand All @@ -68,11 +68,11 @@ test("current release defaults retain 17 cases on OpenAI and three autonomous at
opencodeVersion: releaseHostPolicy(packageJson.version).opencodeVersion,
});
expect(plan.cells.filter((cell) => cell.schedule === "primary")).toHaveLength(
72,
81,
);
expect(
plan.cells.filter((cell) => cell.schedule === "environment-reserve"),
).toHaveLength(17);
).toHaveLength(20);
expect(
[
...new Set(plan.cells.map((cell) => cell.managerModel?.routeProvider)),
Expand Down
Loading
Loading