feat(eval-author): guide first-suite creation and review

README.md

@@ -48,9 +48,10 @@ and [Harbor setup guide](skills/eval-author/references/harbor-setup.md).
 Full creation and improvement onboarding verifies the selected runtime before
 Ethos. Explicit inventory or planning-only requests do not require an evaluation runtime.
 
-All three journeys review or create a repository-root **ETHOS.md** describing
-your agent's purpose, boundaries, and success criteria. Install the
-[Ethos skills](https://github.com/NVIDIA/skills/blob/main/skills/ethos-orchestrate/README.md#how-to-use)
+All three journeys review or create an **ETHOS.md** describing
+your agent's purpose, boundaries, and success criteria. An existing document can
+stay in your agent's directory; see [document selection](docs/getting-started.md#load-the-skills).
+Install the [Ethos skills](https://github.com/NVIDIA/skills/blob/main/skills/ethos-orchestrate/README.md#how-to-use)
 to confirm an existing document or create one with you:
 
 ```bash

docs/first-evals.md

@@ -3,146 +3,107 @@
 
 # Build your first evaluations
 
-Use this guide when your agent has no evaluations yet. Eval Author helps you
-build a starter suite that you can rerun after changes using NeMo Gym or Harbor.
-As in audit-derived and trace-derived task creation, Eval Author follows your
-choice, otherwise preserves the existing suite's provider, and defaults to
-Harbor when neither is specified. NeMo Gym is compatible with Harbor, and you have
-the final say in which tools to use.
-
-Follow [the setup guide](getting-started.md) to load Eval Author in your coding
-agent, with the repository you want to evaluate open. Then ask:
+Use Eval Author with an agent that has no evaluation suite, or expand a starter
+suite you already created. Follow [the setup guide](getting-started.md), open
+your agent's repository, and ask:
 
 ```text
-Help me build my first evaluations for this agent. I do not have evals yet.
-Show me the plan first, and explain the cases and how they will be graded.
+Use eval-author to build my first evaluations for [agent name] using [NeMo Gym or Harbor].
+I don't have evaluations yet.
 ```
 
-## Establish what the agent should do
-
-After verifying your agent, setup, and selected evaluation runtime, Eval Author
-uses the shared Ethos workflow to establish an **Ethos**: a local document describing
-your agent's purpose, boundaries, and success criteria. It can reuse an existing
-`ETHOS.md` or help you create and review one.
-
-If you have traces, Eval Author confirms which source and subset to use before
-searching or reading them. That selection and any exclusions carry into case
-creation and later expansion. You can also choose to plan from Ethos without
-traces; traces are not a prerequisite.
-
-Before generating cases, you review the priorities and breadth for each behavior:
-how many distinct input examples to include, their variations and difficulty,
-positive and negative scenarios, expected outcomes, grading, and exclusions.
-A **pilot** is a limited first set of examples to check the test setup and get an
-initial measure of performance on selected behaviors. Eval Author explains why
-it recommends that breadth, what you would learn, and what would remain untested
-even if every example passed. You can choose a minimal pilot or broader coverage;
-there is no fixed example quota. Difficult sentiment or complaint examples need
-labels justified by the intended meaning; a classifier score alone does not
-establish the answer.
-
-At the plan review, Eval Author explicitly asks what you want to change or
-prioritize: lower rerun cost, measuring the agent's cost or latency, broader
-coverage, harder cases, or another concern. It carries forward preferences you
-already gave. You can revise the
-proposal or accept it as written; your choices are saved before case generation.
-
-Distinct examples, native tasks, and repeated attempts are counted separately.
-A task packages one or more examples with their environment and grading;
-repeating it does not add new examples. Discuss cost before generation, including
-the work each rerun requires and whether cost is a budget, a reported metric,
-or a scoring rule. Without reliable pricing or comparable measurements, the plan
-uses work counts and marks monetary cost unknown.
-
-Full first-eval onboarding requires the selected supported runtime before Ethos
-and case planning. A missing or incompatible runtime pauses onboarding and
-preserves progress; an inconclusive check stays unknown until resolved.
-Eval Author does not install software during these checks. An explicitly requested
-planning-only operation can proceed without an evaluation runtime.
-Native NeMo Gym authoring requires NeMo Gym v0.6.0+ in its separate Python 3.13.14+ environment.
-For Harbor authoring, see the [Harbor setup guide](../skills/eval-author/references/harbor-setup.md).
-Execution also needs the selected environment backend, any required application
-access, and a supported connection to your agent. See the full
-[runtime requirements](getting-started.md#requirements).
-
-The checklist shows the current stage and unfinished work at each checkpoint.
-If you detour to discuss grading, cost, or setup, Eval Author preserves the agreed
-scope and resumes from that stage without restarting onboarding.
-
-## Check the cases and evaluate the agent
-
-Eval Author creates each case's instructions, environment, reference solution,
-and grader. The environment reuses your repository's own services where it can,
-starts from realistic seeded data, and is proven to work before any case relies
-on it. For NeMo Gym, it validates the manifest and runs verifier fixtures with
-known-correct and negative/no-action controls through `gym env validate` and
-`gym env test`. For Harbor:
-
-- **NOP** runs a no-op baseline to check what happens when no work is done.
-- **Oracle** runs the prepared reference solution to check that the grader
-  recognizes the expected outcome.
-
-A realistic incorrect result must also be rejected. Where relevant, Eval Author
-checks a different valid solution and the actual state changes requested by the
-task. Instructions read like your own request, without the root cause, the fix,
-or the expected results; data a task must leave unchanged is checked by content
-rather than by row counts; and when a task requires an action, doing nothing
-earns no credit. It retains the task revision and run evidence, so editing a
-case requires fresh validation. Missing evidence and setup errors stay separate
-from behavior failures.
-
-These checks test the cases themselves. The agent's performance is measured by
-running your actual agent with its configured model and credentials. Eval Author
-reports that run separately, including scores and execution errors.
-
-A working suite can reveal that your agent performs poorly. Its value is giving
-you a repeatable baseline. A few passing cases do not establish broad coverage
-or prove that the grader handles every possible response.
-
-## Read the output and run again
-
-Run artifacts are under `.eval-author/runs/` for NeMo Gym or `.eval-author/jobs/` for Harbor.
-NeMo Gym keeps the selected composition, dataset rows, agent/model settings, and native commands
-in the plan and suite README. For Harbor, when the agent integration supports it,
-Eval Author creates `.eval-author/run.yaml` with the selected cases and run settings.
-
-
-Start with `.eval-author/report.html` to review your cases and results.
-Use **Tasks** to see what each case tests and its expected
-outcome. Open a task's README from **Actions** to inspect its input, grading
-criteria, verifier links, and setup and rerun instructions. Use **Runs** to
-compare scores, distinguish control runs from actual-agent evaluations, and
-open run evidence in the Harbor viewer. If your Harbor viewer uses a different address,
-update it in **Settings**.
-
-For a Markdown version of the task and run tables, open `.eval-author/README.md`.
-
-To update the report after changing cases or running evaluations, ask:
+Eval Author confirms your setup and `ETHOS.md`, the local document describing
+how the agent should behave. Existing traces are optional: you can choose
+specific runs or start from `ETHOS.md` with generated input examples. Your
+source choices and exclusions carry forward when you resume.
+For a new design without a saved source choice, the confirmation message asks
+which source to use directly, without an extra readiness question.
+If a new request finds a saved task design, Eval Author shows its scope and
+progress and asks whether to continue it, revise it, or start a new proposal.
+Confirming `ETHOS.md` alone does not resume that design. Earlier artifacts stay saved.
+
+## Review the design and first task
+
+Source selection and the task proposal show your overall progress, with setup
+and `ETHOS.md` complete and evaluation creation marked as the current step.
+The task proposal explains the inputs, intended behaviors,
+expected results, grading, and remaining coverage gaps. Its Grading column shows
+whether each task uses code checks, a model judge, or both, with the judgment
+needed and its limitations. You can change the proposal or approve it and the
+first task to build. Task labels use stable IDs throughout the conversation so
+you can match them to the report and task directories. Expected results are
+[brief summaries with detailed criteria in each task README](results.md#review-generated-cases).
+The default starting proposal has one to three tasks, with one example each;
+you can request broader scope. Eval Author waits for your source choice and
+then your proposal approval before starting task construction.
+
+After design approval, Eval Author checks what is needed to validate the graders
+before creating any task. If model grading needs a judge choice, access, or
+authorization, it asks for those inputs and the validation limits up front.
+Existing applicable settings and authorization are reused without another
+approval pause. It then builds one task, proves its environment, and validates
+its grader before presenting its README and report for review. Compact
+build checklists show completed, active, pending, and blocked steps, with the
+responsible skill beside each item. The review separates environment proof and
+successful grading validation from whether your actual agent has run.
+You can request revisions to that task before the remaining tasks are created.
+
+If a grader uses an AI model to judge responses, validation exercises that judge
+with known-correct and relevant incorrect results, including valid alternatives
+where permitted. Missing judge inputs appear as focused questions before task
+creation. Unavailable tools or failing access appear under **Blockers**, with the
+action needed to continue. If access later stops working, completed drafts stay
+saved while the affected work pauses. The model and budget questions for running
+your agent still follow suite approval.
+On resume, a fully authored and validated suite can refresh its report or return
+to review without repeating judge setup for checks that are already complete.
+
+Task directories remain under `.eval-author/tasks/<task-slug>/`. The shared
+report is `.eval-author/report.html`, with a Markdown counterpart in
+`.eval-author/README.md`. Both link the task documentation and available evidence.
+
+## Run and inspect the suite
+
+After you approve the first task, Eval Author builds and validates the remainder.
+Its checklist shows task counts and progress through environment checks, grader
+validation, and the README/report refresh.
+You then review the whole authored suite in the HTML report or mirrored README
+and approve the tasks or request changes. Every task's required grader checks,
+including model-based checks when used, must pass before this review.
+After you approve the suite, Eval Author presents the concrete settings,
+attempt counts, limits, and unresolved execution needs. Approving the design or
+tasks does not authorize paid runs; applicable prior authorization is reused.
+Repeating an input adds attempts rather than new examples.
+During execution, a standard checklist shows completed attempts, observed run
+status, and execution errors.
+
+A supported runtime is required for full onboarding and authoring; see
+[runtime requirements](getting-started.md#requirements). Running your agent also
+requires a compatible connection, access, and the selected environment backend.
+A combined Gym–Harbor setup needs a configured bridge. If the actual-agent
+adapter or bridge is missing, Eval Author gives you a copyable implementation
+prompt for a separate chat in your chosen coding harness. Your tasks, graders,
+and evidence stay saved; independent work can continue. Return with the
+integration changes and check results so Eval Author can recheck the connection.
+Completing that engineering work does not authorize agent runs. Other blocked
+work is explained with the action needed to continue.
+
+The result summary separates measured agent failures from execution failures
+and missing attempts. It links an exact saved attempt and the suite README's
+rerun instructions, so you can inspect or repeat the same selection. Low agent
+scores can still provide a valid baseline; successful task checks alone do not
+measure your agent's performance. See [Read your results](results.md).
+
+When the agreed suite is complete, Eval Author asks whether you want to audit
+its coverage and grading against `ETHOS.md`. The audit starts only if you accept;
+missing audit evidence does not undo the completed suite. You can decline and
+keep all artifacts, or later [request an audit](existing-evals.md#audit-coverage).
+
+To refresh the saved report, ask:
 
 ```text
 Refresh the evaluation report in .eval-author.
 ```
 
-The summary compares agreed and delivered examples for each behavior, separating
-what was generated, validated, and actually run. It links the examples to their
-tasks and names missing variations, substitutions, or blocked work. Passing a
-pilot does not mean the broader planned coverage is complete.
-
-If a dependency or agent connection is unavailable, the summary identifies the
-blocked work and what is needed to continue. See [Read your results](results.md)
-for the meaning of the saved outputs.
-
-To extend the suite, ask:
-
-```text
-Show me how to rerun this suite, inspect one result, and plan broader coverage.
-Update the rerun instructions and evaluation report when adding the agreed cases.
-```
-
-For expansion, review the added examples, difficulty, expected outcomes, and
-rerun cost before generation. Existing cases and the selected corpus carry
-forward; the updated plan distinguishes retained examples from new additions.
-
-Once you have run evidence, [audit coverage and propose new evaluations](existing-evals.md#audit-coverage).
-The [first-eval skill](../skills/eval-author-first-eval/SKILL.md) documents the
-complete workflow.
+The [first-evaluation skill](../skills/eval-author-first-eval/SKILL.md) defines the
+agent workflow and its [conversation contract](../skills/eval-author/references/conversations/first-eval.md).

docs/getting-started.md

@@ -22,12 +22,18 @@ skills catalog. Use the [installation command](../README.md#prerequisites) for
 your assistant and scope; see the
 [ETHOS.md guide](https://github.com/NVIDIA/skills/blob/main/skills/ethos-orchestrate/README.md#how-to-use).
 
-Eval Author uses only repository-root `ETHOS.md`. The linked catalog guide
-describes lowercase `ethos.md` and renaming uppercase files; those published
-instructions conflict with Eval Author's required destination. Use a catalog
-version that preserves an explicitly supplied uppercase path, and verify the
-installed skills before starting Ethos. Incompatible instructions block the
-handoff; reinstalling the same version does not resolve the mismatch. See the
+Eval Author can use an existing `ETHOS.md` in your agent's directory or at the
+repository root. Tell it which document belongs to the agent; it keeps that file
+in place and asks when the choice is ambiguous. New documents default to the
+repository root unless you specify another destination. The filename must remain
+uppercase `ETHOS.md`, and the selected document still needs validation and your
+confirmation.
+
+The linked catalog guide describes lowercase `ethos.md` and renaming uppercase
+files. Use a catalog version that preserves an explicitly supplied uppercase
+path, and verify the installed skills before starting Ethos. Incompatible
+instructions block the handoff; reinstalling the same version does not resolve
+the mismatch. See the
 [required handoff capabilities](../skills/eval-author/references/ethos-confirmation.md#required-handoff-capabilities).
 
 Neither installation path installs Harbor, Python dependencies, a coding

docs/results.md

@@ -59,7 +59,10 @@ execution is blocked.
 
 Task README files allow users to inspect its Input, Behavior under test,
 Expected result, and Grading sections. Input links the instruction and fixtures;
-Grading links the actual verifier code and criteria or judge files.
+Grading links the actual verifier code and criteria or judge files. The report's
+Expected outcome column shows the opening summary paragraph from Expected result;
+open the task README for its detailed assertions and expected values. Existing
+sections that begin with a list or table continue to display their full text.
 
 At the accepted-task handoff, create a flat `.eval-author/tasks/` catalog for new
 environments; follow the existing final task-directory convention for existing
@@ -85,6 +88,12 @@ cases, and acceptance criteria. Task drafts and run results show which checks
 have actually completed. Use the [generated case review guide](#review-generated-cases)
 to inspect the authored cases and their available evidence.
 
+Earlier creation decisions and evidence bindings are retained as immutable snapshots
+under `.eval-author/.working/history/create/`. Journey state keeps one reference to
+the latest snapshot, whose predecessor links preserve the full history without
+growing the current state file. Older inline history stays readable and moves to
+snapshots on the next creation update; viewing or rendering state leaves it intact.
+
 Task controls check the task and its verifier: NeMo Gym's known-correct and negative
 verifier fixtures, or Harbor's no-op baseline and reference solution. Results
 from running your actual agent are separate evidence of its performance.
@@ -211,9 +220,11 @@ Shared internal progress and agreed scope are stored in
 `.eval-author/.working/journey-state.json`; they are not a second report or plan.
 
 These are typical paths relative to the repository being evaluated. The shared
-prerequisite flow uses repository-root `ETHOS.md`; scoped utilities can use an explicit
-alternate path. Workflows may stop before producing every artifact when
-prerequisites are missing.
+prerequisite flow keeps the selected `ETHOS.md` at its existing agent-local or
+repository-root path; see [document selection](getting-started.md#load-the-skills).
+Changing that selection reopens affected approvals and preserves its prior
+selection and provenance in `.eval-author/.working/history/ethos/`.
+Workflows may stop before producing every artifact when prerequisites are missing.
 
 | Workflow | Main outputs |
 | --- | --- |

skills/eval-author-audit/SKILL.md

@@ -60,6 +60,12 @@ and task creation remain in the separate improvement handoff.
 
 ## Start a guided audit
 
+An accepted first-suite audit invitation follows the core's
+[saved handoff](../eval-author/references/conversations/first-eval.md#blockers-and-the-audit-invitation),
+not a fresh invocation. Reuse confirmed intent and selected evidence, check only
+audit-specific or changed prerequisites, and continue at the first unfinished
+audit milestone. Missing audit evidence leaves the completed suite intact.
+
 A full audit enters the core's shared prerequisite flow, including direct
 invocation. For a fresh audit, use the core's
 [onboarding renderer](../eval-author/references/conversations/onboarding.md) for