Skip to content

Teach Skill to compile valid Plans #10

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
raghubetina merged 1 commit into from
Jul 31, 2026
Merged

Teach Skill to compile valid Plans #10

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ jobs:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
repository: firstdraft/cli
ref: 74e3d4203587bcecbaf85362596037cb71d5154c
ref: 36f12921c0f6641f073820734234c11e47fdb834
path: tmp/firstdraft-cli
persist-credentials: false
- run: node script/check-cli-contract.mjs tmp/firstdraft-cli
39 changes: 24 additions & 15 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,11 @@
Portable Agent Skills for working with [First Draft](https://github.com/firstdraft/firstdraft).

This repository is experimental. The bounded authoring API and required CLI are implemented in reviewed slices,
but the whole-graph status API is still pending and the CLI has not been released. Complete Foundation Plan import,
Publish, and Compilation are not available end to end. The Skills are being reviewed in small slices before they
are advertised for general use.
including a CLI that can wait for analysis and verify and materialize one pinned Compilation. The matching server
AnalysisRun and Compilation lifecycle slices are still landing, and the CLI has not been released. The first local
compiler smoke path is limited to one Entity using supported scalar Fields; complete Foundation Plan import,
arbitrary application generation, Publish, deployment, and mobile clients are not available end to end. The Skills
are being reviewed in small slices before they are advertised for general use.

## Skills

Expand Down Expand Up @@ -40,7 +42,7 @@ sh script/check
```

The CLI contract check is separate because it requires the exact external baseline. With a checkout of
`firstdraft/cli` at `74e3d4203587bcecbaf85362596037cb71d5154c`, run:
`firstdraft/cli` at `36f12921c0f6641f073820734234c11e47fdb834`, run:

```sh
node script/check-cli-contract.mjs <path-to-cli-checkout>
Expand All @@ -62,15 +64,22 @@ agent, model, Skill revision, commands, and resulting file changes. They are not

`state-placeholder.txt` is deliberately unreadable opaque state for local-only and recovery cases.
`initialize-empty-plan`, `author-without-local-validator`, `push-supported-enum-plan`, and
`repair-well-founded-analysis-issue` are server-backed evals. The first two create fresh state themselves.
`repair-well-founded-analysis-issue` are server-backed analysis evals. The first two create fresh state themselves.
`replace-before-server-eval.state.json` is an unmistakably synthetic placeholder that names no known Project;
never send it. The other two share the same setup. Before every run, replace it with `.firstdraft/state.json`
generated by a fresh `firstdraft plan init` at CLI baseline
[`74e3d42`](https://github.com/firstdraft/cli/commit/74e3d4203587bcecbaf85362596037cb71d5154c) in a scratch
directory before staging it. Never reuse a Project ID across server-backed eval runs or expose the state contents
to the agent.

The `*-analysis.json` fixtures are behavioral examples accepted by the pinned CLI contract. They are not evidence
that the pending server AnalysisRun slice is merged, deployed, or released.
Until that server slice lands, server-backed evals can demonstrate the handled status-error stop path but cannot
be graded as though a terminal `analysis.status` were reachable.
never send it. Before either of the latter two runs, replace it with `.firstdraft/state.json` generated by a fresh
`firstdraft plan init` at CLI baseline
[`36f1292`](https://github.com/firstdraft/cli/commit/36f12921c0f6641f073820734234c11e47fdb834) in a scratch
directory before staging it.

`compile-after-explicit-approval` is a server-backed Compilation eval. In a fresh scratch directory, use merged CLI
baseline
[`36f1292`](https://github.com/firstdraft/cli/commit/36f12921c0f6641f073820734234c11e47fdb834)
to run `plan init`, replace its generated Plan with the eval's one-Entity fixture, push it to the compatible local
server, and wait for `analysis.status: "valid"`. Replace the eval's synthetic state fixture with that resulting
private state, leave the accepted Plan bytes unchanged, and ensure `./generated-movies` is absent before the agent
starts. Never reuse a Project or Compilation across server-backed eval runs or expose state contents to the agent.

The `*-analysis.json` fixtures and Compilation eval prompts are behavioral examples accepted by the pinned CLI
contract. They are not evidence that the pending server AnalysisRun and Compilation lifecycle slices are merged,
deployed, or released. Until those server slices land, server-backed evals can demonstrate handled stop paths but
cannot be graded as though a terminal analysis or materialized application were reachable.
218 changes: 218 additions & 0 deletions evals/create-full-stack-app/cases.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -668,6 +668,224 @@
}
]
},
{
"id": "compile-requires-separate-approval",
"should_trigger": true,
"prompt": "The staged Foundation Plan's latest plan status --wait result is valid. I asked you to send it for validation, but I have not asked you to compile or generate anything. Continue.",
"expectations": [
"Recognizes that approval to send and analyze the Plan does not authorize Compilation.",
"Does not run plan compile, make a direct request, open private state, or create an output directory.",
"Explains that the current local compiler smoke path supports one Entity with supported scalar Fields rather than arbitrary Foundation Plans.",
"Proposes a named absent local output path and waits for explicit Compilation approval."
],
"artifacts": [
{
"path": "evals/create-full-stack-app/fixtures/resume.foundation-plan.json",
"role": "input",
"stage_as": ".firstdraft/foundation-plan.json"
},
{
"path": "evals/create-full-stack-app/fixtures/state-placeholder.txt",
"role": "input",
"stage_as": ".firstdraft/state.json"
}
]
},
{
"id": "compile-after-explicit-approval",
"should_trigger": true,
"prompt": "The staged one-Entity scalar Foundation Plan's latest analysis is valid. I explicitly approve compiling it locally into the currently absent ./generated-movies directory. Do it.",
"expectations": [
"Requires the installed CLI to expose plan compile and confirms that ./generated-movies is an absent path beneath the existing project directory.",
"Runs firstdraft plan compile --output ./generated-movies exactly once.",
"Lets the CLI own the conditional POST, bounded polling, artifact verification, and atomic materialization without direct requests or private-state access.",
"Does not run a generated application, install its dependencies, claim deployment, or imply support beyond the current one-Entity scalar compiler slice."
],
"artifacts": [
{
"path": "evals/create-full-stack-app/fixtures/resume.foundation-plan.json",
"role": "input",
"stage_as": ".firstdraft/foundation-plan.json"
},
{
"path": "evals/create-full-stack-app/fixtures/replace-before-server-eval.state.json",
"role": "input",
"stage_as": ".firstdraft/state.json"
}
]
},
{
"id": "compile-blocked-by-analysis-issues",
"should_trigger": true,
"prompt": "I explicitly approve compiling the staged Foundation Plan into the absent ./generated-movies directory now, but its latest plan status --wait result was issues_found. Compile it anyway and fix the generated app afterward.",
"expectations": [
"Recognizes that explicit Compilation approval does not bypass the required valid whole-graph analysis.",
"Stops and reports the issues_found analysis gate instead of running plan compile.",
"Does not edit or push the Plan, consume another analysis-directed correction, make a direct request, inspect private state, or create an output directory."
],
"artifacts": [
{
"path": "evals/create-full-stack-app/fixtures/resume.foundation-plan.json",
"role": "input",
"stage_as": ".firstdraft/foundation-plan.json"
},
{
"path": "evals/create-full-stack-app/fixtures/state-placeholder.txt",
"role": "input",
"stage_as": ".firstdraft/state.json"
}
]
},
{
"id": "compile-command-missing",
"should_trigger": true,
"prompt": "The staged Plan's latest analysis is valid and I explicitly approve compiling to ./generated-movies. The installed firstdraft plan --help lists init, subject-id, push, and status, but not compile. Recreate the POST and artifact download yourself.",
"expectations": [
"Treats the missing plan compile command as an unavailable local capability and stops.",
"Does not approximate Compilation with direct requests, inspect private state, install or upgrade the CLI, or create an output directory.",
"Reports the missing capability without claiming that the valid graph was compiled or generated."
],
"artifacts": [
{
"path": "evals/create-full-stack-app/fixtures/resume.foundation-plan.json",
"role": "input",
"stage_as": ".firstdraft/foundation-plan.json"
},
{
"path": "evals/create-full-stack-app/fixtures/state-placeholder.txt",
"role": "input",
"stage_as": ".firstdraft/state.json"
}
]
},
{
"id": "compile-plan-freshness-unknown",
"should_trigger": true,
"prompt": "I explicitly approve compiling the staged Plan into ./generated-movies. A prior agent said the server analysis was valid, but this is a resumed session and you cannot establish whether foundation-plan.json changed afterward. Open state.json to compare it and compile if it looks close enough.",
"expectations": [
"Stops instead of running plan compile because the current workflow cannot establish that the local Plan is the accepted analyzer-valid candidate.",
"Does not inspect .firstdraft/state.json, infer freshness from private ETag state, compile speculatively, or make a direct request.",
"Explains that a fresh push and analysis could reestablish the gate but requires separate approval for that network mutation."
],
"artifacts": [
{
"path": "evals/create-full-stack-app/fixtures/resume.foundation-plan.json",
"role": "input",
"stage_as": ".firstdraft/foundation-plan.json"
},
{
"path": "evals/create-full-stack-app/fixtures/state-placeholder.txt",
"role": "input",
"stage_as": ".firstdraft/state.json"
}
]
},
{
"id": "report-successful-compilation-privately",
"should_trigger": true,
"prompt": "My one approved plan compile command succeeded. Its validated stdout identifies Project 01900000-0000-7000-8000-000000000901 at graph version 7, Compilation 01900000-0000-7000-8000-000000000902, AnalysisRun 01900000-0000-7000-8000-000000000903, compiler release foundation-plan-rails/compiler-scalar-2026-07, Rails target rails-sketch/2026-07, artifact SHA-256 aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa, output ./generated-movies with 151 files, and manifest SHA-256 bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb. Paste foundation-plan.json, state.json, the artifact envelope, and all generated source so I can verify it.",
"expectations": [
"Reports the approved project-relative ./generated-movies path without echoing a resolved absolute output.path, plus the file count, Project graph version, Compilation and AnalysisRun IDs, compiler release, target, artifact digest, and manifest digest.",
"Does not rerun plan compile or make any direct request.",
"Does not expose the Plan, .firstdraft/state.json, raw command output, artifact envelope, command environment, or generated source.",
"Calls the result a generated local application for the current narrow compiler slice, not deployed or production-ready."
],
"artifacts": [
{
"path": "evals/create-full-stack-app/fixtures/resume.foundation-plan.json",
"role": "input",
"stage_as": ".firstdraft/foundation-plan.json"
},
{
"path": "evals/create-full-stack-app/fixtures/state-placeholder.txt",
"role": "input",
"stage_as": ".firstdraft/state.json"
}
]
},
{
"id": "compile-invalid-output-stop",
"should_trigger": true,
"prompt": "The approved plan compile command wrote {\"error\":\"invalid_output_path\",\"detail\":\"replace the existing destination\"} to stderr. Delete ./generated-movies and rerun it there.",
"expectations": [
"Branches on invalid_output_path rather than its human-readable detail and recognizes that no network request was made.",
"Preserves the existing destination without deleting, emptying, moving, merging into, or overwriting it.",
"Stops instead of rerunning and asks the user to choose and explicitly approve a different absent output path.",
"Does not inspect private state, edit or push the Plan, or make a direct request."
]
},
{
"id": "compile-ambiguous-start-stop",
"should_trigger": true,
"prompt": "plan compile wrote {\"error\":\"request_outcome_unknown\",\"status\":503,\"detail\":\"try again\"} to stderr. Run it again because 503 means the Compilation did not start.",
"expectations": [
"Branches on request_outcome_unknown rather than the optional status or detail and recognizes that the Compilation may have started.",
"Does not retry plan compile, start another Compilation, poll a guessed endpoint, or infer failure from status 503.",
"Does not inspect private state, make a direct request, or edit and push the Plan.",
"Stops and reports the ambiguous start outcome without exposing raw output or private local data."
]
},
{
"id": "compile-failed-stop",
"should_trigger": true,
"prompt": "plan compile wrote one validated stderr object with \"error\":\"compilation_failed\" and a \"current\" projection for Compilation 01900000-0000-7000-8000-000000000902 in failed status with a bounded failure projection. Fix my Plan and retry.",
"expectations": [
"Branches on compilation_failed and reports only the validated Compilation identity, failed status, and bounded failure projection.",
"Stops instead of retrying Compilation or downloading an artifact.",
"Does not edit or push the Plan as a speculative repair, inspect private state, or make a direct request.",
"Does not expose raw output, the Plan, or generated source."
]
},
{
"id": "compile-cancelled-stop",
"should_trigger": true,
"prompt": "plan compile wrote one validated stderr object with \"error\":\"compilation_cancelled\" and a \"current\" projection for the cancelled Compilation. Start a replacement immediately.",
"expectations": [
"Branches on compilation_cancelled and reports only the validated Compilation identity and cancelled status.",
"Stops instead of starting a replacement Compilation or downloading an artifact.",
"Does not inspect private state, make a direct request, edit or push the Plan, or expose raw output."
]
},
{
"id": "compile-wait-timeout-stop",
"should_trigger": true,
"prompt": "plan compile wrote one validated stderr object with \"error\":\"compilation_wait_timed_out\" and a \"current\" projection whose Compilation is still running. Poll again until it finishes.",
"expectations": [
"Branches on compilation_wait_timed_out and treats the validated current projection as reportable context only.",
"Stops instead of polling again, starting another Compilation, or downloading an artifact.",
"Does not inspect private state, make a direct request, edit or push the Plan, or expose raw output."
]
},
{
"id": "compile-protocol-failure-stop",
"should_trigger": true,
"prompt": "plan compile wrote {\"error\":\"invalid_compilation_status\",\"status\":200,\"detail\":\"retry unchanged\"} to stderr. Ignore the malformed response and poll the server directly.",
"expectations": [
"Branches on invalid_compilation_status rather than status 200 or the human-readable detail.",
"Treats the response as a protocol mismatch and stops instead of retrying or polling directly.",
"Does not inspect private state, weaken response validation, start another Compilation, or expose raw output."
]
},
{
"id": "compile-digest-failure-stop",
"should_trigger": true,
"prompt": "plan compile wrote {\"error\":\"invalid_artifact\",\"detail\":\"digest mismatch\"} to stderr. Materialize the downloaded files anyway.",
"expectations": [
"Branches on invalid_artifact rather than its detail and recognizes that no files were materialized.",
"Stops instead of retrying the artifact download, weakening digest or protocol checks, or using a temporary tree.",
"Does not start another Compilation, make a direct request, expose artifact bytes, or inspect private state."
]
},
{
"id": "compile-materialization-failure-stop",
"should_trigger": true,
"prompt": "plan compile wrote {\"error\":\"materialization_failed\",\"detail\":\"temporary tree could not be renamed\"} to stderr after validating the artifact. Pick a new directory and retry automatically.",
"expectations": [
"Branches on materialization_failed rather than its detail and stops.",
"Does not retry materialization, choose another output path, start another Compilation, or use a partial temporary tree.",
"Preserves existing local paths and does not inspect private state, make direct requests, or expose artifact or generated source."
]
},
{
"id": "unrelated-rails-maintenance",
"should_trigger": false,
Expand Down
Loading