diff --git a/README.md b/README.md index 82025bd4..d6187982 100644 --- a/README.md +++ b/README.md @@ -48,9 +48,10 @@ and [Harbor setup guide](skills/eval-author/references/harbor-setup.md). Full creation and improvement onboarding verifies the selected runtime before Ethos. Explicit inventory or planning-only requests do not require an evaluation runtime. -All three journeys review or create a repository-root **ETHOS.md** describing -your agent's purpose, boundaries, and success criteria. Install the -[Ethos skills](https://github.com/NVIDIA/skills/blob/main/skills/ethos-orchestrate/README.md#how-to-use) +All three journeys review or create an **ETHOS.md** describing +your agent's purpose, boundaries, and success criteria. An existing document can +stay in your agent's directory; see [document selection](docs/getting-started.md#load-the-skills). +Install the [Ethos skills](https://github.com/NVIDIA/skills/blob/main/skills/ethos-orchestrate/README.md#how-to-use) to confirm an existing document or create one with you: ```bash diff --git a/docs/first-evals.md b/docs/first-evals.md index 50220a53..4a7bf075 100644 --- a/docs/first-evals.md +++ b/docs/first-evals.md @@ -3,146 +3,107 @@ # Build your first evaluations -Use this guide when your agent has no evaluations yet. Eval Author helps you -build a starter suite that you can rerun after changes using NeMo Gym or Harbor. -As in audit-derived and trace-derived task creation, Eval Author follows your -choice, otherwise preserves the existing suite's provider, and defaults to -Harbor when neither is specified. NeMo Gym is compatible with Harbor, and you have -the final say in which tools to use. - -Follow [the setup guide](getting-started.md) to load Eval Author in your coding -agent, with the repository you want to evaluate open. Then ask: +Use Eval Author with an agent that has no evaluation suite, or expand a starter +suite you already created. Follow [the setup guide](getting-started.md), open +your agent's repository, and ask: ```text -Help me build my first evaluations for this agent. I do not have evals yet. -Show me the plan first, and explain the cases and how they will be graded. +Use eval-author to build my first evaluations for [agent name] using [NeMo Gym or Harbor]. +I don't have evaluations yet. ``` -## Establish what the agent should do - -After verifying your agent, setup, and selected evaluation runtime, Eval Author -uses the shared Ethos workflow to establish an **Ethos**: a local document describing -your agent's purpose, boundaries, and success criteria. It can reuse an existing -`ETHOS.md` or help you create and review one. - -If you have traces, Eval Author confirms which source and subset to use before -searching or reading them. That selection and any exclusions carry into case -creation and later expansion. You can also choose to plan from Ethos without -traces; traces are not a prerequisite. - -Before generating cases, you review the priorities and breadth for each behavior: -how many distinct input examples to include, their variations and difficulty, -positive and negative scenarios, expected outcomes, grading, and exclusions. -A **pilot** is a limited first set of examples to check the test setup and get an -initial measure of performance on selected behaviors. Eval Author explains why -it recommends that breadth, what you would learn, and what would remain untested -even if every example passed. You can choose a minimal pilot or broader coverage; -there is no fixed example quota. Difficult sentiment or complaint examples need -labels justified by the intended meaning; a classifier score alone does not -establish the answer. - -At the plan review, Eval Author explicitly asks what you want to change or -prioritize: lower rerun cost, measuring the agent's cost or latency, broader -coverage, harder cases, or another concern. It carries forward preferences you -already gave. You can revise the -proposal or accept it as written; your choices are saved before case generation. - -Distinct examples, native tasks, and repeated attempts are counted separately. -A task packages one or more examples with their environment and grading; -repeating it does not add new examples. Discuss cost before generation, including -the work each rerun requires and whether cost is a budget, a reported metric, -or a scoring rule. Without reliable pricing or comparable measurements, the plan -uses work counts and marks monetary cost unknown. - -Full first-eval onboarding requires the selected supported runtime before Ethos -and case planning. A missing or incompatible runtime pauses onboarding and -preserves progress; an inconclusive check stays unknown until resolved. -Eval Author does not install software during these checks. An explicitly requested -planning-only operation can proceed without an evaluation runtime. -Native NeMo Gym authoring requires NeMo Gym v0.6.0+ in its separate Python 3.13.14+ environment. -For Harbor authoring, see the [Harbor setup guide](../skills/eval-author/references/harbor-setup.md). -Execution also needs the selected environment backend, any required application -access, and a supported connection to your agent. See the full -[runtime requirements](getting-started.md#requirements). - -The checklist shows the current stage and unfinished work at each checkpoint. -If you detour to discuss grading, cost, or setup, Eval Author preserves the agreed -scope and resumes from that stage without restarting onboarding. - -## Check the cases and evaluate the agent - -Eval Author creates each case's instructions, environment, reference solution, -and grader. The environment reuses your repository's own services where it can, -starts from realistic seeded data, and is proven to work before any case relies -on it. For NeMo Gym, it validates the manifest and runs verifier fixtures with -known-correct and negative/no-action controls through `gym env validate` and -`gym env test`. For Harbor: - -- **NOP** runs a no-op baseline to check what happens when no work is done. -- **Oracle** runs the prepared reference solution to check that the grader - recognizes the expected outcome. - -A realistic incorrect result must also be rejected. Where relevant, Eval Author -checks a different valid solution and the actual state changes requested by the -task. Instructions read like your own request, without the root cause, the fix, -or the expected results; data a task must leave unchanged is checked by content -rather than by row counts; and when a task requires an action, doing nothing -earns no credit. It retains the task revision and run evidence, so editing a -case requires fresh validation. Missing evidence and setup errors stay separate -from behavior failures. - -These checks test the cases themselves. The agent's performance is measured by -running your actual agent with its configured model and credentials. Eval Author -reports that run separately, including scores and execution errors. - -A working suite can reveal that your agent performs poorly. Its value is giving -you a repeatable baseline. A few passing cases do not establish broad coverage -or prove that the grader handles every possible response. - -## Read the output and run again - -Run artifacts are under `.eval-author/runs/` for NeMo Gym or `.eval-author/jobs/` for Harbor. -NeMo Gym keeps the selected composition, dataset rows, agent/model settings, and native commands -in the plan and suite README. For Harbor, when the agent integration supports it, -Eval Author creates `.eval-author/run.yaml` with the selected cases and run settings. - - -Start with `.eval-author/report.html` to review your cases and results. -Use **Tasks** to see what each case tests and its expected -outcome. Open a task's README from **Actions** to inspect its input, grading -criteria, verifier links, and setup and rerun instructions. Use **Runs** to -compare scores, distinguish control runs from actual-agent evaluations, and -open run evidence in the Harbor viewer. If your Harbor viewer uses a different address, -update it in **Settings**. - -For a Markdown version of the task and run tables, open `.eval-author/README.md`. - -To update the report after changing cases or running evaluations, ask: +Eval Author confirms your setup and `ETHOS.md`, the local document describing +how the agent should behave. Existing traces are optional: you can choose +specific runs or start from `ETHOS.md` with generated input examples. Your +source choices and exclusions carry forward when you resume. +For a new design without a saved source choice, the confirmation message asks +which source to use directly, without an extra readiness question. +If a new request finds a saved task design, Eval Author shows its scope and +progress and asks whether to continue it, revise it, or start a new proposal. +Confirming `ETHOS.md` alone does not resume that design. Earlier artifacts stay saved. + +## Review the design and first task + +Source selection and the task proposal show your overall progress, with setup +and `ETHOS.md` complete and evaluation creation marked as the current step. +The task proposal explains the inputs, intended behaviors, +expected results, grading, and remaining coverage gaps. Its Grading column shows +whether each task uses code checks, a model judge, or both, with the judgment +needed and its limitations. You can change the proposal or approve it and the +first task to build. Task labels use stable IDs throughout the conversation so +you can match them to the report and task directories. Expected results are +[brief summaries with detailed criteria in each task README](results.md#review-generated-cases). +The default starting proposal has one to three tasks, with one example each; +you can request broader scope. Eval Author waits for your source choice and +then your proposal approval before starting task construction. + +After design approval, Eval Author checks what is needed to validate the graders +before creating any task. If model grading needs a judge choice, access, or +authorization, it asks for those inputs and the validation limits up front. +Existing applicable settings and authorization are reused without another +approval pause. It then builds one task, proves its environment, and validates +its grader before presenting its README and report for review. Compact +build checklists show completed, active, pending, and blocked steps, with the +responsible skill beside each item. The review separates environment proof and +successful grading validation from whether your actual agent has run. +You can request revisions to that task before the remaining tasks are created. + +If a grader uses an AI model to judge responses, validation exercises that judge +with known-correct and relevant incorrect results, including valid alternatives +where permitted. Missing judge inputs appear as focused questions before task +creation. Unavailable tools or failing access appear under **Blockers**, with the +action needed to continue. If access later stops working, completed drafts stay +saved while the affected work pauses. The model and budget questions for running +your agent still follow suite approval. +On resume, a fully authored and validated suite can refresh its report or return +to review without repeating judge setup for checks that are already complete. + +Task directories remain under `.eval-author/tasks//`. The shared +report is `.eval-author/report.html`, with a Markdown counterpart in +`.eval-author/README.md`. Both link the task documentation and available evidence. + +## Run and inspect the suite + +After you approve the first task, Eval Author builds and validates the remainder. +Its checklist shows task counts and progress through environment checks, grader +validation, and the README/report refresh. +You then review the whole authored suite in the HTML report or mirrored README +and approve the tasks or request changes. Every task's required grader checks, +including model-based checks when used, must pass before this review. +After you approve the suite, Eval Author presents the concrete settings, +attempt counts, limits, and unresolved execution needs. Approving the design or +tasks does not authorize paid runs; applicable prior authorization is reused. +Repeating an input adds attempts rather than new examples. +During execution, a standard checklist shows completed attempts, observed run +status, and execution errors. + +A supported runtime is required for full onboarding and authoring; see +[runtime requirements](getting-started.md#requirements). Running your agent also +requires a compatible connection, access, and the selected environment backend. +A combined Gym–Harbor setup needs a configured bridge. If the actual-agent +adapter or bridge is missing, Eval Author gives you a copyable implementation +prompt for a separate chat in your chosen coding harness. Your tasks, graders, +and evidence stay saved; independent work can continue. Return with the +integration changes and check results so Eval Author can recheck the connection. +Completing that engineering work does not authorize agent runs. Other blocked +work is explained with the action needed to continue. + +The result summary separates measured agent failures from execution failures +and missing attempts. It links an exact saved attempt and the suite README's +rerun instructions, so you can inspect or repeat the same selection. Low agent +scores can still provide a valid baseline; successful task checks alone do not +measure your agent's performance. See [Read your results](results.md). + +When the agreed suite is complete, Eval Author asks whether you want to audit +its coverage and grading against `ETHOS.md`. The audit starts only if you accept; +missing audit evidence does not undo the completed suite. You can decline and +keep all artifacts, or later [request an audit](existing-evals.md#audit-coverage). + +To refresh the saved report, ask: ```text Refresh the evaluation report in .eval-author. ``` -The summary compares agreed and delivered examples for each behavior, separating -what was generated, validated, and actually run. It links the examples to their -tasks and names missing variations, substitutions, or blocked work. Passing a -pilot does not mean the broader planned coverage is complete. - -If a dependency or agent connection is unavailable, the summary identifies the -blocked work and what is needed to continue. See [Read your results](results.md) -for the meaning of the saved outputs. - -To extend the suite, ask: - -```text -Show me how to rerun this suite, inspect one result, and plan broader coverage. -Update the rerun instructions and evaluation report when adding the agreed cases. -``` - -For expansion, review the added examples, difficulty, expected outcomes, and -rerun cost before generation. Existing cases and the selected corpus carry -forward; the updated plan distinguishes retained examples from new additions. - -Once you have run evidence, [audit coverage and propose new evaluations](existing-evals.md#audit-coverage). -The [first-eval skill](../skills/eval-author-first-eval/SKILL.md) documents the -complete workflow. +The [first-evaluation skill](../skills/eval-author-first-eval/SKILL.md) defines the +agent workflow and its [conversation contract](../skills/eval-author/references/conversations/first-eval.md). diff --git a/docs/getting-started.md b/docs/getting-started.md index d8465e29..d40ccf1d 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -22,12 +22,18 @@ skills catalog. Use the [installation command](../README.md#prerequisites) for your assistant and scope; see the [ETHOS.md guide](https://github.com/NVIDIA/skills/blob/main/skills/ethos-orchestrate/README.md#how-to-use). -Eval Author uses only repository-root `ETHOS.md`. The linked catalog guide -describes lowercase `ethos.md` and renaming uppercase files; those published -instructions conflict with Eval Author's required destination. Use a catalog -version that preserves an explicitly supplied uppercase path, and verify the -installed skills before starting Ethos. Incompatible instructions block the -handoff; reinstalling the same version does not resolve the mismatch. See the +Eval Author can use an existing `ETHOS.md` in your agent's directory or at the +repository root. Tell it which document belongs to the agent; it keeps that file +in place and asks when the choice is ambiguous. New documents default to the +repository root unless you specify another destination. The filename must remain +uppercase `ETHOS.md`, and the selected document still needs validation and your +confirmation. + +The linked catalog guide describes lowercase `ethos.md` and renaming uppercase +files. Use a catalog version that preserves an explicitly supplied uppercase +path, and verify the installed skills before starting Ethos. Incompatible +instructions block the handoff; reinstalling the same version does not resolve +the mismatch. See the [required handoff capabilities](../skills/eval-author/references/ethos-confirmation.md#required-handoff-capabilities). Neither installation path installs Harbor, Python dependencies, a coding diff --git a/docs/results.md b/docs/results.md index 60bd13fe..29d1d19b 100644 --- a/docs/results.md +++ b/docs/results.md @@ -59,7 +59,10 @@ execution is blocked. Task README files allow users to inspect its Input, Behavior under test, Expected result, and Grading sections. Input links the instruction and fixtures; -Grading links the actual verifier code and criteria or judge files. +Grading links the actual verifier code and criteria or judge files. The report's +Expected outcome column shows the opening summary paragraph from Expected result; +open the task README for its detailed assertions and expected values. Existing +sections that begin with a list or table continue to display their full text. At the accepted-task handoff, create a flat `.eval-author/tasks/` catalog for new environments; follow the existing final task-directory convention for existing @@ -85,6 +88,12 @@ cases, and acceptance criteria. Task drafts and run results show which checks have actually completed. Use the [generated case review guide](#review-generated-cases) to inspect the authored cases and their available evidence. +Earlier creation decisions and evidence bindings are retained as immutable snapshots +under `.eval-author/.working/history/create/`. Journey state keeps one reference to +the latest snapshot, whose predecessor links preserve the full history without +growing the current state file. Older inline history stays readable and moves to +snapshots on the next creation update; viewing or rendering state leaves it intact. + Task controls check the task and its verifier: NeMo Gym's known-correct and negative verifier fixtures, or Harbor's no-op baseline and reference solution. Results from running your actual agent are separate evidence of its performance. @@ -211,9 +220,11 @@ Shared internal progress and agreed scope are stored in `.eval-author/.working/journey-state.json`; they are not a second report or plan. These are typical paths relative to the repository being evaluated. The shared -prerequisite flow uses repository-root `ETHOS.md`; scoped utilities can use an explicit -alternate path. Workflows may stop before producing every artifact when -prerequisites are missing. +prerequisite flow keeps the selected `ETHOS.md` at its existing agent-local or +repository-root path; see [document selection](getting-started.md#load-the-skills). +Changing that selection reopens affected approvals and preserves its prior +selection and provenance in `.eval-author/.working/history/ethos/`. +Workflows may stop before producing every artifact when prerequisites are missing. | Workflow | Main outputs | | --- | --- | diff --git a/skills/eval-author-audit/SKILL.md b/skills/eval-author-audit/SKILL.md index ff1f589d..ae2ea115 100644 --- a/skills/eval-author-audit/SKILL.md +++ b/skills/eval-author-audit/SKILL.md @@ -60,6 +60,12 @@ and task creation remain in the separate improvement handoff. ## Start a guided audit +An accepted first-suite audit invitation follows the core's +[saved handoff](../eval-author/references/conversations/first-eval.md#blockers-and-the-audit-invitation), +not a fresh invocation. Reuse confirmed intent and selected evidence, check only +audit-specific or changed prerequisites, and continue at the first unfinished +audit milestone. Missing audit evidence leaves the completed suite intact. + A full audit enters the core's shared prerequisite flow, including direct invocation. For a fresh audit, use the core's [onboarding renderer](../eval-author/references/conversations/onboarding.md) for diff --git a/skills/eval-author-environment/SKILL.md b/skills/eval-author-environment/SKILL.md index 8c649074..8bd3349a 100644 --- a/skills/eval-author-environment/SKILL.md +++ b/skills/eval-author-environment/SKILL.md @@ -44,6 +44,9 @@ repeat agent, prerequisite, or Ethos onboarding. Real environment verification remains required here and is separate from prerequisite confirmation. Save workflow stage and pending decisions in shared journey state; the kit documents its dependency design and native evidence. +When called by first-eval creation, follow the caller's +[build-progress contract](../eval-author/references/conversations/first-eval.md#first-task-build-progress) +while preparing and proving the environment. ## Required outputs diff --git a/skills/eval-author-first-eval/SKILL.md b/skills/eval-author-first-eval/SKILL.md index 764e22e3..bbfd10c1 100644 --- a/skills/eval-author-first-eval/SKILL.md +++ b/skills/eval-author-first-eval/SKILL.md @@ -4,10 +4,10 @@ description: >- For a fresh start, read the eval-author core skill at ../eval-author/SKILL.md, use its pure conversation renderer for orientation and end the turn before repository inspection or prerequisite work. - Establish a required Ethos, plan evaluation cases, and build first evals or + Confirm ETHOS.md, plan evaluation tasks, and build first evals or expand a starter NeMo Gym or Harbor suite while explaining its parts. No prior traces or coverage reports are required. Missing selected - runtime blocks shared onboarding before Ethos and case planning. + runtime blocks shared onboarding before ETHOS.md and task planning. triggers: - help me build my first evals - my agent has no evals yet @@ -18,7 +18,7 @@ not-for: - eval-author-task-create (use for measured audit coverage gaps or ATIF-source tasks) - eval-author-discover (use to check an existing suite) compatibility: >- - Ethos is required before evaluation design and is saved and checked locally + ETHOS.md is required before evaluation design and is saved and checked locally in the user's repository. No NeMo service, account, platform CLI, or upload is needed. Shared onboarding requires the selected runtime. NeMo Gym and Harbor authoring need the Python/CLI pairs verified by eval-author’s shared locator and runtime-policy.json. @@ -36,397 +36,326 @@ metadata: -# Eval Author: first eval +# Eval Author: first evaluations ## ATIF as a starting source -When the user supplies an ATIF trace for a new suite, follow the shared -[ATIF source handoff](../eval-author/references/trace-input.md). Establish Ethos -and admit the source before constructing its case; keep the usual environment-kit -handoff. A rejected source is a retained outcome, not a reason to invent a case. - -## Purpose - -Read the [eval-author core skill](../eval-author/SKILL.md) for the shared standard -and boundaries. Direct invocations enter its shared prerequisite flow, consuming -an applicable saved result on return. A fresh start uses the core's -[onboarding renderer](../eval-author/references/conversations/onboarding.md) for -`orientation`, then ends the turn before repository reads, prerequisite checks, -or Ethos work. The core's narrow pure-renderer command exception applies; -rendering never supplies readiness. Continue through shared onboarding after the -actual reply. Complete and save the core's state-only prerequisite pass before -any target Ethos inspection, then reconcile artifacts and include the completed -findings in the final `results` response. -A statement that no evals exist settles the starting point; no discovery is -needed to prove absence. Reuse confirmed Ethos and prerequisite decisions. - -When the user asks to expand a starter suite created here, resume at scope -planning for the affected behaviors. Reuse the saved scope in journey state, corpus, Ethos, setup, -and accepted cases; recheck prerequisites only where the change affects them. -Do not restart onboarding or require an audit merely to add agreed examples. -Requested coverage measurement still belongs to `eval-author-audit`. - -No evals is a normal starting state. Do not require a previous run or manufacture -a coverage report to enter `eval-author-task-create`. The initial deliverable is -a working suite at the agreed breadth that the user understands and can rerun. -A minimal pilot is acceptable when explicitly chosen; comprehensive coverage, -repeated agent success, and trace-driven optimization are not prerequisites. -Include difficult cases when relevant to the agreed behavior and purpose. -A working suite and strong evaluation quality are separate claims. - -## 1. Reuse the confirmed Ethos - -Consume the path, confirmed digest, validation result, and unresolved intent from -the core's [Ethos confirmation](../eval-author/references/ethos-confirmation.md) -handoff after the user has confirmed the document and its current intent in the -shared flow. A new shared entry does not skip that review because saved state was -previously complete. After the handoff, reopen only changed or unresolved content; reuse the -current confirmation without repeating the interview or writing another Ethos. -When shared onboarding finishes, show the core's completion pause before source -selection or evaluation work. Continue the stages below only after the user replies. - -## 2. Reuse prerequisites and verify when needed - -Preserve the selected NeMo Gym or Harbor setup and recorded prerequisite evidence -and answers. Shared onboarding requires an available supported runtime before -Ethos and case planning. Before scaffolding or another -runtime-dependent operation, reuse applicable checks from shared prerequisites or -use the shared runtime locator as described in -[Runtime prerequisite checks](../eval-author/references/runtime-prerequisites.md#runtime-prerequisite-checks). Record -actual checks separately from user confirmation and reuse unchanged evidence. -If the selected runtime becomes unavailable, preserve completed work and return -to the shared prerequisite blocker before continuing the full flow. Other setup -needs pause the operations that depend on them. - -## 3. Understand the evaluation starting point - -Reuse the user's stated absence of evals or discovery's confirmed starting point, -along with available agent and documentation findings. Do not run discovery or -ask whether evals exist again solely to complete this milestone. Explain the -outcome concretely: a starter eval set of customer scenarios -with criteria for scoring responses, which the user can rerun after changes to -detect improvement or regression. - -Read [Corpus and coverage planning](../eval-author/references/coverage-planning.md#select-trace-evidence-before-using-it) -before trace search or inspection. Present already-known source candidates and -confirm the intended source and corpus, or reuse the user's explicit selection. -Support proceeding without traces. Knowing there are no evals does not settle -the trace choice. Carry the selected paths, bounds, exclusions, and inspected -counts into the plan; repository examples must not become the full corpus by -default. Save this selection and current progress in `.eval-author/.working/journey-state.json` -as soon as established, even before the case plan is complete. - -Use “starter eval set” for the collection and “eval case” for each scenario. -Introduce a case as a request with criteria for scoring the response. Reserve -“sanity checks” for validation of the cases themselves; use Harbor's technical -term “task” when discussing its files or CLI. - -## 4. Define the evaluation scope - -Read the agent entry point, tool definitions, and usage docs to understand how -the actual agent runs. Follow [Corpus and coverage planning](../eval-author/references/coverage-planning.md) -to propose breadth per behavior from Ethos and the selected evidence. Explain -a minimal pilot or broader coverage in terms of distinct examples, meaningful -variations and difficulty, positive and negative scenarios, and remaining gaps. -Include the request, initial fixture, expected observable outcome, a meaningful -failure example, and the Ethos requirement each case tests. Ask only for intent -or invocation details not already established. Do not make the user supply -Harbor YAML or choose a framework. - -When agent or setup evidence points to software or state outside the agent -process, use [Execution dependencies](../eval-author/references/execution-dependencies.md) -to establish the selected cases' runtime and result-collection requirements. -Record the agent's provider fit and dependency inventory using Steps 1 and 2 of -[the environment sub-flow](../eval-author-environment/SKILL.md); files and -embedded databases the agent works on count as dependencies. The kit is built -and proven at stage 5, before any test. Raise a poor fit for the provider selected -at stage 2 in the scope checkpoint; if the user changes provider, verify that -runtime as stage 2 describes before preparing cases. - -Save the selected journey’s `scopes` entry in `.eval-author/.working/journey-state.json` with the Ethos path and requirement references, -selected corpus (or no-trace choice), per-behavior scope and example counts, -outcomes and grading, priorities, exclusions, task/attempt mapping, cost -implications, agent invocation, prerequisites, and unresolved questions. -Separate proposed choices from the user's agreement and record subsequent feedback. -These are agreed planning decisions, not a coverage report or runnable evals. - -Use reproducible inputs and variations supported by Ethos and the agreed scope, -including semantically justified difficult negatives where relevant. Grade an -observable result rather than a specific tool call or exact prose. Simple -assertions are acceptable if their limits are clear. -Do not replace subjective quality with brittle string -matching or mock away the behavior under test. If the requested case cannot be -tested with available resources, retain the original requirement as blocked. -Explain the limitation; select a supported alternative only with the user, -and record it as a scope change rather than completion of the original case. - -If the selected runtime becomes unavailable at this stage, preserve the agreed -plan and return to the shared runtime blocker before continuing. - -Present the plan using the shared [scope review](../eval-author/references/coverage-planning.md#explain-ongoing-cost-and-settle-the-plan). -Settle unresolved scope choices and incorporate changes before preparing cases. -When expanding, record retained and added examples and the new unique totals; -the pilot's size does not determine the expansion's scope. - -## 5. Prepare cases and grading - -For each new case, use [private construction declarations](../eval-author/references/construction-plan.md) -to record grading authority, authored or copied inputs, and observable checks. -Validate and bind the private plan before recording native controls. Keep simple -cases simple; preserve correct inaction and nonbinary rewards where the agreed -requirements call for them. A healthy agent failure remains a valid baseline. - -After each native scaffold is created, record it as **Draft** using the shared -[Task status](../eval-author/references/shared-prerequisites.md#task-status) -procedure. Keep that status while completing and validating the task. - -Apply [Task validation and execution evidence](../eval-author/references/task-validation.md) -for requirement-to-check review, declared verifier controls, and revision-bound -run receipts. Include a realistic incorrect result and applicable valid-alternative -and side-effect cases. These are local task controls, not extra model repeats. - -Read the reference for each provider in the selected setup for the runtime-specific -work in stages 5–9: - -- [Native NeMo Gym first evals](references/gym-first-eval.md): native scaffolding, - component configuration, verifier fixtures, and NeMo Gym execution. -- [Harbor first evals](references/harbor-first-eval.md): native task files, - environment and agent setup, NOP/Oracle controls, and Harbor execution. - -Keep the shared scope, stage check-ins, explanations, suite README, and result -reconciliation here. A combined setup uses the relevant parts of both references; -reading a reference does not change the selected provider. - -Build and prove the kit before the tests. Before writing any case's verifier, -complete Steps 3 to 7 of [`eval-author-environment`](../eval-author-environment/SKILL.md): -choose how each dependency is provided, generate the kit's seeded starting data, -make its end state readable, set its isolation, and run its smoke task, recording -each job in `environment-plan.md`. Write each case's own records on top of that -data, and derive its expected values with an independent reference query over -the generated data rather than by hand. Hand-typed fixtures of a few rows are -not starting data. - -Create the agreed examples and map their stable IDs to native tasks or dataset -rows. Keep source provenance and expected-label rationale in shared scope or -verifier-only data, outside the agent's starting environment. Start with one -representative case and check its available parts before reusing the pattern. -If its live environment or agent connection is pending, continue independent -case and grading work for the other selected cases. - -Before running, document each case's reward format, metric names and ranges, -and expected control results. Derive these criteria from the intended outcome. -Add a task README with the Ethos requirement, fixtures, verifier, and run commands, -following the shared [Task documentation](../eval-author/references/suite-readme.md#task-documentation) -requirements for Harbor tasks. -Also create the suite's review entry point following -[Review and rerun guidance](../eval-author/references/suite-readme.md#review-and-rerun-guidance). -Present its case inventory and link the guide at this milestone, with input, -expected outcome, grading, and available source evidence; mark new cases as not -run until executed. Keep it current as the environment, agent connection, and run -configuration are completed and fresh control or agent results become available. - -Apply the core's **Explain the eval pieces as they become relevant** guidance to -the files being created. For each verifier, explain its actual assertion and a -concrete incorrect example it must reject. Fix demonstrated false acceptance -before claiming that check is validated; retain untested distinctions as explicit -limitations. Introduce the provider's reference solution or known-correct fixture -when creating it. Keep this teaching part of building the starter cases, without -a separate tutorial or exhaustive methodology exercise. - -Before the checkpoint question, introduce Harbor's case viewer in the reply and -walk the user to one created case's input and grading files, following -[Show the user how to review](../eval-author/references/suite-readme.md#show-the-user-how-to-review). -Include how to launch and open it; a link to the suite guide alone is insufficient. - -## 6. Prepare the execution environment - -Hand off the kit proven at stage 5 with Step 8 of -[`eval-author-environment`](../eval-author-environment/SKILL.md), and use the -selected provider reference to confirm each task's environment builds on that -kit; rerun the smoke task if the kit changed after it passed. Keep solutions and -verifier-only data outside the agent's initial environment. This milestone is complete only -when the kit's `environment-plan.md`, seeded data generator, and smoke results -exist under `.eval-author/.working/environments//`; the tasks' own controls -never count as environment proof, and without the smoke task's own jobs the -plan's status stays `unproven`. -Verify the selected backend and required application access before declaring -this milestone complete; identifying their requirements is only partial progress. -Record unavailable access and which cases or checks it blocks, naming credential -variables without recording secret values. - -## 7. Connect the agent - -Use the selected provider reference to establish a supported integration for the -actual agent from its entry point, installed interfaces, registry, and CLI help. -Do not substitute another agent or rewrite the application. If integration is -missing, identify the specific connection required. Verifier-control validation -remains available, but performance of the user's agent remains unmeasured. - -Record the selected cases, agent and model settings, agreed attempts, reset -procedure, and output paths using the provider's configuration format. Propose -one attempt for an initial baseline unless the purpose needs repeats; record -that choice and its cost during scope planning. Repeats do not add examples. -Do not include unrelated drafts merely because they share a parent directory. -Resolve paths from the repository root and reference credential environment -variables rather than embedding secrets. Configure request delivery, required -conversation state, and collection of outputs and actions through supported -interfaces. Record connection checks under **Validate the evals**; configuration -alone does not prove that the connection works. - -## 8. Validate the evals - -Finalize each task's files and README run instructions before preparing -[revision-bound evidence](../eval-author/references/task-validation.md#prepare-immutable-inputs) -for these checks. - -Follow [Execution recovery](../eval-author/references/execution-recovery.md) -for control and target-agent runs, compatibility repairs, and all result recaps. -Confirm the current environment kit's proof (Harbor's smoke task or NeMo Gym's -persistence and reset cases) passed first; a proof failure is an environment -defect to fix, not a task or agent result. That proof is recorded in the kit's -plan, not with the task evidence recorder below. - -Use the selected provider's native validation and controls, retaining fresh -artifacts on reruns. Require checks to complete without exceptions and meet the -case's predeclared reward criteria, including reward shape. When inaction is -correct, also exercise an explicitly incorrect response or action: a successful -no-action control alone does not show that incorrect behavior is rejected. - -Run the native commands through the shared evidence recorder, along with the -other declared controls. For each case, inspect trial results and rewards. -These are basic wiring and verifier sanity checks, not evidence of broad coverage -or a robust benchmark. Keep the declared controls proportional to the task; do -not require repeated proof or additional model runs as an onboarding gate. -Investigate obvious unconditional rewards or leaked answers. Fix broken cases -and rerun affected checks; do not weaken the intended assertion to force a pass. -Report blocked cases separately and deliver the working subset without silently -dropping planned cases or claiming the whole suite passed. Keep unfinished -configuration or connection validation visible separately from successful -verifier controls. - -## 9. Evaluate the agent - -Confirm that the resolved case set matches the selected starter cases. Explain -the task count, distinct examples, attempts, execution-cost assumptions, and -where rewards and errors will appear. Check that packaging still executes the -agreed examples. Once validated, show the cases, agent, model, and provider-native -command with the agreed attempt settings, then run with established execution -and spend authorization using the selected reference. - -Inspect results for every selected case; retain outputs, actions, rewards, and -exceptions as fresh run evidence. A completed evaluation can have low scores. -Recording this baseline with the evidence recorder is optional. - -## 10. Review results and explain reruns - -Confirm the retained controls, configuration validation, actual-agent baseline, -and completed result review describe the final task revision, including its README. -Then record **Accepted** through [Task status](../eval-author/references/shared-prerequisites.md#task-status). -Link the retained evidence; low agent -scores do not prevent acceptance of a functioning evaluation. Keep incomplete -tasks **Draft**. -For accepted tasks, follow [Keep task paths stable](../eval-author/SKILL.md#keep-task-paths-stable). -Acceptance updates status and suite membership without copying or moving task -files. Verify the native config resolves exactly the accepted set and report -links use those same paths. Keep unfinished tasks visible and exclude them from -runnable suite configs. Include the accepted task count and location in the handoff. - -Update `.eval-author/.working/journey-state.json` with artifact references, completion state, and blockers. Keep exact commands in the suite README and control -results, per-task agent rewards, and exceptions in native evidence. Show how -to rerun the suite, inspect one result, and add or modify a task. Maintain the -suite README's run instructions and results links, link the finalized task READMEs, and refresh -`.eval-author/report.html` and `.eval-author/README.md` following the shared report contract. -Preserve the full-suite command and concrete commands for individual coverage items, -following [Review and rerun guidance](../eval-author/references/suite-readme.md#review-and-rerun-guidance). - -Report refresh must not change the evaluation result or workflow state. Follow -the closing-sentence template in [Report generation](../eval-author/references/suite-readme.md#report-generation) -when linking the report and suite README. Do not link -the selected journey’s `scopes` entry in `.eval-author/.working/journey-state.json` in the run handoff unless asked. - -Continue saving agreed scope and current progress in `.eval-author/.working/journey-state.json` -through the existing workflow, including when HTML generation fails. -Refresh the case inventory with links to control and actual-agent results and -available execution traces, keeping source evidence and missing traces distinct. -Save any needed subset configs and verify their selected cases and retained -agent settings. Label rerun commands as task sanity checks or actual agent evaluation. - -When an actual-agent attempt exists, explain in the results reply how to open -and inspect its saved evidence, using the shared -[review walkthrough](../eval-author/references/suite-readme.md#show-the-user-how-to-review). -For Harbor, give the run viewer's launch command and browser URL or a verified -live link, and identify the saved job and case. For NeMo Gym, link the readable -evidence for the selected dataset rows and identify the saved attempt. -State what can be reviewed when the adapter emits no -ATIF; captured actions and replies do not imply a native trajectory exists. -If only controls ran or execution was blocked before any actual-agent run, -say the actual agent has not run and offer case review plus any clearly labeled -control evidence. - -Reconcile agreed versus delivered examples per behavior using -[Expand and reconcile](../eval-author/references/coverage-planning.md#expand-and-reconcile). -Separate generated, validated, and executed counts and name omissions, changed -fixtures, and remaining gaps. Repeated attempts never increase example totals. -Render the final three-item checklist, checking every completed step and leaving -blocked or partial steps open with their next action. - -The working-suite deliverable is reached when the selected starter tasks execute -and record rewards through a validated config, even if the agent scores poorly -or checks are basic. A functioning case that the agent fails provides a useful -baseline; do not weaken it to obtain success. - -Name each created eval case and explain the customer behavior it measures. -Translate the recorded controls into their meaning. For NeMo Gym, explain the actual -positive and negative fixtures and observed rewards. For a Harbor binary action case, -doing nothing failed and the prepared reference solution passed, so the grader -distinguishes those examples. Do not call NOP an empty answer unless that is what -it tested. For graded metrics or cases where inaction is correct, explain the -actual acceptance criteria and observed controls. These checks do not show how -the user's agent performs or prove that all correct answers receive credit. - -State whether the actual agent ran and explain the observed scores when it did. -If integration is missing, describe the concrete connection needed at the verified -entry point; use “Harbor adapter” only if that helps the user act. If execution is -blocked by a credential, name the required environment variable to configure in -the execution environment. Describe the next run as scoring actual agent responses, -distinguishing it from configuration or connection setup checks. - -Explain practical limitations, such as narrow fixtures, a wording assertion -rejecting a correct paraphrase, or an assertion checking only part of an outcome. -Link the eval cases, results, and established Ethos requirements; identify relevant -intent left outside the starter set. Keep detailed scores, interpreter details, -and full rerun commands in the saved report unless useful in the explanation; -include the viewer instructions in the reply as described above. - -Traces and improvement are optional follow-up work. Explain that traces reveal -steps and tool calls that can expose failure patterns, missing coverage, and weak -checks. Link actual trace artifacts when emitted; otherwise identify the adapter -or instrumentation needed to produce them. For requested coverage accounting, -hand the established Ethos and actual ATIF to `eval-author-audit`, then use -`eval-author-task-create` for measured actionable gaps. - -## Prerequisites - -Shared onboarding requires the selected NeMo Gym or Harbor runtime before Ethos. -Establish Ethos before case design; no prior traces or coverage reports are required. -Scaffolding and validation use that selected installation. Execution -needs the chosen backend, agent connection, and any configured provider access. -Real-agent and paid-judge runs require authorization for execution and spend. - -## Limitations - -A starter suite establishes a baseline for the selected cases, not comprehensive -coverage. NeMo Gym verifier controls or Harbor NOP and Oracle validate task wiring -and selected verifier behavior; they do not measure the real agent. If runtime -access is lost, retain existing plans and drafts while the flow is blocked. - -## Troubleshooting - -- Missing Ethos: resume the shared Ethos confirmation adapter before case design. -- Selected runtime unavailable: retain the case plan and identify the NeMo Gym setup - requirement or follow discovery's Harbor setup guidance. Install only when - authorized, following the user's selected setup. -- Oracle fails or NOP unexpectedly passes: inspect task state, reference solution, - and verifier against the intended outcome; repair and rerun the controls. -- Agent connection or credentials missing: record the verified entry point or - required variable name and the blocked run; never request secret values in chat. +For a user-selected ATIF trace, follow the shared [ATIF source handoff](../eval-author/references/trace-input.md). +Confirm `ETHOS.md` and admit the selected source before constructing its task; +preserve privacy decisions, provenance, and the environment-kit handoff. A rejected +source remains a recorded outcome, not a reason to invent a replacement task. + +## Purpose and conversation + +Read the [core skill](../eval-author/SKILL.md) for shared boundaries and onboarding. +Fresh invocations use its pure onboarding renderer and stop at its orientation +checkpoint. Complete the saved state-only prerequisite pass before target +`ETHOS.md` inspection, then reconcile artifacts normally. Reuse current +[ETHOS.md confirmation](../eval-author/references/ethos-confirmation.md) +and source choices through the shared completion pause. For fresh creation with +no saved design or source choice, confirmation includes source selection; wait +for that answer without adding a readiness pause. A stated absence of evaluations +does not require discovery. + +Introduce **evaluation tasks**, then use **tasks**. Name the behavior document +`ETHOS.md`; use **first task**, distinct input **examples**, and repeated +**attempts**. Do not use numbered journey labels or call the first task a pilot. +Keep provider-native identifiers when discussing their files or commands. + +Use the [first-evaluation conversation contract](../eval-author/references/conversations/first-eval.md) +with the shared renderer for every creation checkpoint. Its template module owns fixed +copy, table columns, source-choice labels, settings labels, and closing order. +Present the rendered output unchanged rather than copying those templates here +or adding routine policy explanations and duplicate checklists. Dynamic task +content and evidence interpretation still require judgment. + +The structured record and `first_eval_*` stages below apply to `create`. For an +active starter expansion, retain `expand` and follow [Starter expansion](#starter-expansion) +without routing its progress through the creation record. + +Save creation progress under `scopes.create.first_eval` through the existing revision-checked +[state interface](../eval-author/references/shared-prerequisites.md#save-and-resume). +The conversation contract defines the additive record and observed facts used +by its stages. Save actual user decisions before acting; a side question is not +approval. Reuse applicable answers and authorization without restarting onboarding. + +### Confirm a saved design before reusing it + +When a fresh request finds a saved creation design, first establish whether the +user wants that design. Follow the [saved-design checkpoint](../eval-author/references/conversations/first-eval.md#saved-design-checkpoint) +after the shared completion pause and continuation. It owns the separate user +decision, renderer pause, retained artifacts, and revision-checked updates. +Confirmation of `ETHOS.md` does not select a saved design. + +## 1. Select the task sources + +Use confirmed `ETHOS.md`, the agent entry point, tools, and usage documentation. +Existing traces are optional. Follow the +[source checkpoint inputs](../eval-author/references/conversations/first-eval.md#source-and-proposal-pauses): +fresh creation without a saved design combines this choice with the confirmed +`ETHOS.md` response. If that combined checkpoint already asked the question, +consume its answer rather than rendering `first_eval_traces` again. Otherwise +use `first_eval_traces` with its progress checklist and exact choices as the +final reply, then end the turn. Collect the reply to the rendered checkpoint +rather than a separate question tool. +Wait for the user's answer before trace inspection, task planning, +environment work, or integration investigation. +Reuse an already established choice. +Clarify locations, selected runs, bounds, +and exclusions before inspection, following [source selection](../eval-author/references/coverage-planning.md#select-trace-evidence-before-using-it). +Do not search unselected traces or silently treat repository fixtures as the +chosen corpus. Save the selected sources or explicit no-trace choice and mark +authored inputs as synthetic. + +Preserve the selected supported runtime and existing prerequisite evidence. A +configured Gym–Harbor path needs verified integration; conversion or two installed +CLIs is insufficient. Recheck only affected prerequisites through the [runtime locator](../eval-author/references/runtime-prerequisites.md#runtime-prerequisite-checks). +Do not switch providers or add a production adapter to bypass missing integration. +Missing run access does not prevent independent design or authoring. + +## 2. Propose and agree on the tasks + +After the source answer is saved, propose a small starting subset: one to three +tasks with one distinct input example per task by default. Choose fewer when +that is sufficient; do not turn `ETHOS.md` into an exhaustive suite. A broader +initial proposal requires the user's explicit request, retained as +`plan.breadth_request.source`, rather than an inferred preference or invented +approval. Keep additional ideas out of this initial agreement. + +Derive concrete requests, expected behavior, outcomes, and grading from `ETHOS.md` +and selected evidence. Include useful positive, negative, difficult, and boundary +examples at the agreed breadth. Explain why the tasks help, what remains untested, +and which will be built first. Do not use a model's current answer as grading +truth or replace subjective quality with brittle wording checks. +Follow [proposed grading](../eval-author/references/conversations/first-eval.md#proposed-grading) +when filling each task's Grading column, before resolving its validation setup. + +Reuse prerequisite findings and the selected source to make the proposal. +Read only the additional agent interface or selected evidence needed to describe +its input, observable result, and grading. Defer adapter investigation, hook +design, environment preparation, and runtime implementation until design approval; +unresolved execution details do not prevent this design review. Consult +[execution dependencies](../eval-author/references/execution-dependencies.md) and +[environment preparation](../eval-author-environment/SKILL.md) when that work is due. +Retain an untestable requirement as blocked; only the user can change its scope. + +Save the proposed design and render `first_eval_proposal`. The initial review +starts with the overall progress checklist, then the task table, scope explanation, +chosen first task, and final feedback question defined by the template. It contains +no execution-settings checklist, run-only blockers, or spending request. If a design +blocker prevents a meaningful proposal, use `first_eval_blockers` instead of offering +an approvable design. +The template explains that approval leads to building the named first task and +pausing for its review before the remainder. Emit this proposal as the final +reply and end the turn. Do not continue inspection, delegate implementation, +or generate task files while awaiting the user's response. + +Incorporate feedback until the design is approved. Record actual approval for the +current plan digest before any scaffolding, fixtures, instructions, graders, or +delegated authoring. Do not create a separate user-facing planning document. +An approved design is not paid-execution authorization. Expansion preserves +retained tasks and source choices and reviews only affected design changes. + +## 3. Build and review exactly the first task + +After design approval, resolve the [validation inputs before task creation](../eval-author/references/conversations/first-eval.md#validation-setup-before-task-creation). +Save the inspected setup and ask for any missing judge configuration, access, +or bounded validation authorization through `first_eval_validation_setup` before +creating task files or initializing build work. Reuse existing answers and +authorization; ready deterministic or configured model grading needs no extra +approval pause. Keep target-agent execution settings after suite approval. +Once validation setup is ready, render `first_eval_prepare_first`. +Initialize the saved build activity and use the [build-progress contract](../eval-author/references/conversations/first-eval.md#first-task-build-progress) +for coordinator-owned renderer updates throughout authoring and delegated work. +These progress updates continue the build; they are not additional approval pauses. + +Read the selected [Harbor](references/harbor-first-eval.md) or [native Gym](references/gym-first-eval.md) +reference. Create exactly the agreed first task directly in its permanent +`.eval-author/tasks//` directory. Record the native scaffold as **Draft** +through [Task status](../eval-author/references/shared-prerequisites.md#task-status). +Do not generate remaining tasks or Gym rows before first-task approval, including +through delegation. A reviewed native Gym task may contain several agreed rows. + +Complete and prove the [environment kit](../eval-author-environment/SKILL.md) +before writing the task's grader or reference solution. Its realistic starting +data, observable state, isolation, and smoke evidence establish environment +readiness; task controls never prove the kit. Build on that proven kit and keep reference +answers and verifier-only information outside the agent's initial environment. + +Use [private construction declarations](../eval-author/references/construction-plan.md) +for grading authority, authored or copied inputs, and observable checks. Validate +and bind the construction plan before recording native controls. Preserve +correct inaction and nonbinary rewards when required by the task. + +Finish the task README following [Task documentation](../eval-author/references/suite-readme.md#task-documentation), +including input, behavior, expected result, grading, a meaningful limitation, +and setup/rerun instructions. Finalize it before revision-bound execution evidence; +put later run results and evidence links in the outer suite README. + +Satisfy [grader validation before authored-task review](../eval-author/references/conversations/first-eval.md#grader-validation-before-authored-task-review) +for this task, including its actual model-based judge when used. Generate the +report and Markdown mirror using [Report generation](../eval-author/references/suite-readme.md#report-generation) +before `first_eval_review_first`. Link the actual Tasks view and task README, +and report environment proof, grading validation, and actual-agent execution +separately. Missing or stale required validation or artifacts block review. +Reports may be refreshed as drafts while validation remains blocked, without +presenting an approval question or claiming review readiness. + +The report's Runs table currently reads native Harbor jobs; native Gym evidence +remains linked from the suite README. Keep planned-but-uncreated tasks, separate +review/validation/execution findings, exact attempt navigation, model identity, +and evidence limitations explicit in the saved summary. Do not claim missing +report capabilities exist; record required report-owner work as a blocker or +limitation instead of inventing UI behavior. + +Request approval of the current first-task revision. Apply requested changes in +the same directory, retain historical evidence, invalidate affected observations, +refresh artifacts, and request review again. First-task approval covers the +reviewed approach; it does not claim individual review of every later task. + +## 4. Build and validate the remainder + +Only after current first-task approval, render `first_eval_prepare_remaining` +and build the remaining agreed tasks with the approved pattern. Reuse applicable +proof and controls, refreshing the report after authoring, revisions, and validation. +Keep task paths stable; a blocked first task never unlocks remaining authoring. +Follow the [remaining-task build checklist](../eval-author/references/conversations/first-eval.md#remaining-task-build-progress) +for saved activity and coordinator-owned renderer updates throughout this work. + +Follow [task validation](../eval-author/references/task-validation.md) and +[execution recovery](../eval-author/references/execution-recovery.md). Finalize +task files and README instructions before preparing immutable evidence. Use +provider-native checks with known correct results and relevant incorrect-result controls, including +valid alternatives and state preservation where applicable. Keep controls separate +from target-agent attempts. Confirming that graders accept the correct results +and reject the incorrect results tested validates grading; it does not measure +the agent's quality. Do not import audit gap closure or two successful agent attempts. + +Preserve failed and superseded evidence under fresh output names. Task changes +reopen affected approval and validation; configuration changes reopen affected +execution claims. Unknown or missing evidence never establishes completion. + +Once all agreed tasks are authored, their environments are proven, and every +task satisfies the [pre-review validation gate](../eval-author/references/conversations/first-eval.md#grader-validation-before-authored-task-review), +refresh the report and Markdown mirror and use the [suite review checkpoint](../eval-author/references/conversations/first-eval.md#review-the-authored-suite-before-execution-settings). +Render `first_eval_review_suite` as the final reply and end the turn. Wait for +approval of the whole suite before discussing target-agent execution models, +budgets, credentials, attempt settings, or run authorization. Grader-validation +inputs were resolved before authoring. First-task approval does not approve the +remaining authored tasks. Revisions reopen the affected review. + +Do not offer review while required model-based grading or calibration is +pending. If a previously ready validation dependency changes, resolve only that +dependency before more authoring or validation work. Successful grader controls +do not establish agent performance. + +## 5. Agree on and execute concrete runs + +Use `first_eval_execution_settings` only after approval of the authored suite when +concrete agent runs are ready. The conversation contract owns its +settings checklist; record the actual target/model, task/example mapping, repeats, +validation work, time/cost bounds, and unresolved questions. Mark costs unknown +without supported estimates. Settings are separate from design and never gate +task generation or either task review. Required grading validation has already +passed before either authored-task review; it does not wait for this checkpoint. +Reuse applicable previously displayed settings and prior authorization without +another generic approval gate. + +Paid work requires explicit authorization within its concrete displayed bounds. +Design approval, task approval, or positive feedback alone grants no spending +consent. Ask only about the missing authorization or execution setting, preserving +independent work. Do not ask again when current authorization already covers it. + +Use the actual repository agent through its supported integration. Save and +inspect explicit provider-native selection, agent/model settings, attempts, reset, +and fresh output paths. Harbor uses `.eval-author/run.yaml`; directory membership +never selects tasks. Native Gym retains native composition, complete selected +dataset rows, and repeats. A Gym–Harbor run must use its configured bridge. +Never substitute another agent or direct Harbor execution for that requirement. +When the supported connection is missing, follow the +[external integration handoff](../eval-author/references/conversations/first-eval.md#missing-actual-agent-integration-handoff) +instead of requesting permission to implement it inside Eval Author. + +After successful checks, render `first_eval_ready_to_run`. Announce execution only +when current authorization, selected configuration, and actual access are ready; +otherwise resolve the one missing decision. Run through the selected provider and +retain native outputs, rewards, errors, commands, and exact job/attempt identities. +A low score is a valid baseline; absent or diagnostic-only execution is not. + +Once native launch or queuing is observed, follow the [agent-run progress contract](../eval-author/references/conversations/first-eval.md#agent-run-progress) +and render `first_eval_run_progress` throughout execution. Keep the standard +checklist visible without skill labels through concise commentary updates based on +actual native status and attempt evidence. On interruption, inspect the existing +job before describing its activity or starting another run. Do not turn a +progress update into another approval gate or authorization for retries. + +## 6. Review evidence and hand off + +Inspect an informative actual-agent attempt. Explain its input, observed actions +or response, grade, available agent/model identity, evidence navigation, and +material limitations. Save the exact reviewed run and attempt; later runs must +not silently change that identity. Separate completed attempts, agent-quality +failures, infrastructure failures, and missing attempts in the reply and saved +summary. Do not invent traces or model-health information. + +Maintain [review and rerun guidance](../eval-author/references/suite-readme.md#review-and-rerun-guidance) +in the suite README. Link the verified rerun heading as the template requires, +with actual full-suite and task-specific instructions. Verify commands and +navigation against the installed provider and retained evidence. Keep later +results outside finalized task directories so they do not change task digests. + +Reconcile agreed, created, validated, and executed tasks and examples. Record +**Accepted** only when final-revision controls, configuration, baseline, and result +review support that decision through [Task status](../eval-author/references/shared-prerequisites.md#task-status). +Keep incomplete tasks **Draft**; low agent scores do not prevent acceptance of +functioning tasks. Report refresh observes evidence and never accepts tasks, +changes results, or advances workflow state. + +Only after all agreed work and required artifacts are complete, render +`first_eval_results` with its completed checklist and saved pending audit invitation. +Follow [completed run review](../eval-author/references/conversations/first-eval.md#completed-run-review) +for artifact navigation and invitation order. When creation is incomplete, lead +with unfinished work and actual **Blockers**, preserving progress, and retain +the shared report closing sentence. + +Wait for the user's audit decision. Acceptance enters the existing audit flow +with confirmed `ETHOS.md`, task paths, provider, exact reviewed run/attempt, +selected sources, evidence limits, and preferences. Reuse unchanged prerequisites +and resolve only audit-specific or changed needs. Missing audit evidence blocks +only the audit; never silently run evaluations to create traces. Declining keeps +the completed suite and does not start or repeatedly offer an audit. Audit consent +does not authorize more tasks, agent edits, or additional paid runs. + +## Starter expansion + +Keep the existing `expand` goal and `scopes.expand`; do not reinitialize creation +or require a new `scopes.create.first_eval` record to continue. Follow the core's +[expansion handoff](../eval-author/SKILL.md#expansion-handoff) and [coverage scope](../eval-author/references/coverage-planning.md#expand-and-reconcile). +Reuse the confirmed `ETHOS.md`, selected corpus, provider, reviewed task pattern, +accepted task paths, and applicable approvals and evidence. Recheck only changed +prerequisites; an audit is optional rather than an entry gate. + +Review retained and added tasks, distinct examples, grading, breadth, exclusions, +and meaningful difficulty changes before generating additions. Record the actual +scope agreement in `scopes.expand`. Apply the authoring, environment, construction, +first-task review, validation, explicit selection, and evidence standards above +without invoking the create-only `first_eval_*` stages. Reuse an already-approved +first-task pattern where it remains applicable; reopen only affected review and +validation claims rather than requiring new approval of unchanged tasks. + +Use the provider references for native authoring and execution, preserving main's +Draft/Accepted lifecycle and finalized task documentation. Show concrete execution +settings when paid work is ready and reuse existing bounded authorization. Update +the suite README/report outside validated task trees, reconcile retained versus +added work, and retain the existing expansion progress and handoff. Do not claim +that the create-only renderer or record establishes expansion completion. + +## Blocked work and resume + +Use the flow's blocker stage for actual current-step blockers, with the cause, +blocked action, and specific needed answer or action. Retain run-only limitations +without presenting them during independent first-task preparation and review. +Pause only dependent work. Do not request secret values or fabricate approval, +delivery, validation, or execution observations. Resume from current saved progress +with applicable source choices, task revisions, and authorization intact after +the saved-design choice above when entering from a fresh request. diff --git a/skills/eval-author-first-eval/references/gym-first-eval.md b/skills/eval-author-first-eval/references/gym-first-eval.md index 9f32fe97..c70ec324 100644 --- a/skills/eval-author-first-eval/references/gym-first-eval.md +++ b/skills/eval-author-first-eval/references/gym-first-eval.md @@ -3,22 +3,28 @@ # Native NeMo Gym first evals -Use this reference for the NeMo Gym-specific work in first-eval stages 5–9. Keep the -parent skill's stage boundaries and shared deliverables. NeMo Gym is compatible with -Harbor; follow the user's selected setup. No audit report, actionable tool gap, -or measured gap closure is required. +Use this reference for native NeMo Gym authoring, validation, and execution. +The [parent skill](../SKILL.md) and [conversation contract](../../eval-author/references/conversations/first-eval.md) +own design approval, first-task and suite review, execution authorization, and handoff. +A selected Gym–Harbor setup uses its verified configured bridge; this native +path is not a substitute. Keep missing integration as an execution blocker +without adding a production adapter. No audit gap or coverage closure applies. -## 5. Prepare cases and grading +## 5. Prepare tasks and grading Use the NeMo Gym Python/CLI pair selected by the [shared runtime locator](../../eval-author/references/runtime-prerequisites.md#runtime-prerequisite-checks) with `--provider gym --flow authoring`. Its central policy owns version and -Python requirements; keep NeMo Gym in its separate environment. Inspect its CLI help and native scaffolder before use. Create each -planned draft in a fresh `.eval-author/tasks//` directory using +Python requirements; keep NeMo Gym in its separate environment. Inspect its CLI +help and native scaffolder before use. Create only the +approved first task directly in `.eval-author/tasks//` using NeMo Gym's `nemo_gym.environment.scaffold.scaffold_environment(kind="environment", name=, root=)`. Use a valid Python module name for the NeMo Gym environment. Do not call the audit-driven `task_pipeline.py scaffold` route: -first-eval has no coverage report or uncovered-tool selector. +first evaluations require no coverage report or uncovered-tool selector. +Do not create remaining task directories or rows before first-task review approval. +One reviewed task may map to several explicitly agreed complete dataset rows; +retain verifier metadata and count distinct examples separately from repeats. Follow [Complete the task without changing its intended capability](../../eval-author-task-create/references/gym-tasks.md#complete-the-task-without-changing-its-intended-capability) for the generated manifest, component configuration, resources server, dataset @@ -26,9 +32,10 @@ rows, and verifier fixtures. Replace template tools, rewards, and fixtures with the agreed cases. Seed the resources server from the environment kit's generated data and pass the kit's persistence and reset cases before writing verifier fixtures, then derive expected rewards and end states with an independent -reference over that data. Map stable case IDs to selected +reference over that data. Map stable task/example IDs to selected dataset rows; keep expected answers and provenance outside model inputs. Start -with one representative case before extending the pattern to the agreed breadth. +with exactly the approved first task and wait for its review before extending +the pattern to the agreed breadth. Document each case's reward contract and expected positive and negative control results before testing. Include a known-correct response or interaction and a @@ -45,14 +52,15 @@ Prepare the generated resources, agent, and model components, dependency inputs, fixtures, and state reset. Build the backends the resources server wraps, their seeded session state, and its reset from the environment kit that [`eval-author-environment`](../../eval-author-environment/SKILL.md) built and -proved at stage 5, following its [NeMo Gym environments](../../eval-author-environment/references/gym.md) +proved before task grading, following its [NeMo Gym environments](../../eval-author-environment/references/gym.md) reference, and add each case's own records to its dataset row or session seed. Apply the [NeMo Gym version and rerun plan](../../eval-author-task-create/references/gym-tasks.md#version-and-rerun-plan) to the selected suite, recording it in the selected journey’s `scopes` entry in `.eval-author/.working/journey-state.json` and the drafts' `reproducibility.md`; no dataset recommendation or audit proposal is required. Use the component packaging checks linked there for the selected runtime. Verify actual application access and component setup. Record missing credential -variable names and blocked cases without recording secret values. +variable names and blocked work without recording secret values. Defer run-only +settings and blockers until execution is ready while independent authoring proceeds. ## 7. Connect the agent @@ -60,8 +68,10 @@ Use the actual repository agent through a supported NeMo Gym integration. Inspec its entry point and the installed NeMo Gym interfaces; do not substitute NeMo Gym's example agent to claim the user's agent ran. Record the resolved composition, configured agent instance, model settings, selected dataset rows, reset procedure, and -output paths. If integration is missing, retain the drafts and verifier work and -name the specific connection required. +output paths. If integration is missing when execution is ready, use the +[missing-integration handoff](../../eval-author/references/conversations/first-eval.md#missing-actual-agent-integration-handoff), +retaining task and verifier work. Grader setup remains a separate dependency; +verifier controls cannot prove the user's agent is connected. Propose one attempt per selected example for the initial baseline unless the agreed purpose requires repeats. Record the count and cost in shared scope. A Harbor @@ -107,8 +117,10 @@ managed execution using prepared splits. Run only with the established execution and spend authorization. Retain fresh outputs, rewards, errors, resolved settings, and exact commands for every selected -case under `.eval-author/runs/`. Link these from the plan and suite README using +task under `.eval-author/runs/`. Keep exact native run and attempt identities +and readable selected-row evidence in the suite README, outside finalized task +directories, using paths resolved from the documented working directory. A working suite requires native execution of the actual agent and recorded rewards, even when scores are -low. Return to stage 10 for reconciliation and rerun instructions. Report NeMo Gym +low. Return to the parent skill for reconciliation and linked rerun instructions. Report NeMo Gym control results in their own terms, without labeling them Harbor NOP/Oracle jobs. diff --git a/skills/eval-author-first-eval/references/harbor-first-eval.md b/skills/eval-author-first-eval/references/harbor-first-eval.md index 07699269..34ca6c05 100644 --- a/skills/eval-author-first-eval/references/harbor-first-eval.md +++ b/skills/eval-author-first-eval/references/harbor-first-eval.md @@ -3,17 +3,21 @@ # Harbor first evals -Use this reference for the Harbor-specific work in first-eval stages 5–9. Keep -the parent skill's stage boundaries, scope, check-ins, and shared deliverables. -Harbor is compatible with NeMo Gym; follow the user's selected setup. No audit report -or measured gap closure is required. +Use this reference for native Harbor authoring, validation, and execution. +The [parent skill](../SKILL.md) and [conversation contract](../../eval-author/references/conversations/first-eval.md) +own design approval, first-task and suite review, execution authorization, and handoff. +A configured Gym–Harbor path must execute through its verified bridge; converted +tasks run directly by Harbor do not prove that path. Keep missing integration +as an execution blocker without substituting an agent or adding a new adapter. +No audit gap or coverage-closure requirement applies. -## 5. Prepare cases and grading +## 5. Prepare tasks and grading Use the verified Harbor CLI and Python environment. Read `harbor task init --help` and `harbor trial start --help` before using options. Use an unused descriptive slug for -each planned task under `.eval-author/tasks/`. One task may contain several -examples; preserve the agreed mapping of stable example IDs to tasks. +the approved first task directly in `.eval-author/tasks//`. Do not +scaffold remaining tasks before first-task review approval. One task may contain +several agreed examples; preserve the agreed mapping of stable example IDs to tasks. ```bash harbor task init / --tasks-dir .eval-author/tasks \ @@ -21,15 +25,17 @@ harbor task init / --tasks-dir .eval-author/tasks \ ``` Verify that the expected directory was created. Build its `environment/` first, -from the environment kit proven at stage 5: copy the kit's build files and its +from the proven environment kit: copy the kit's build files and its generated data, which passed the kit's smoke task, into the task and add the -case's own records, so the task builds on its own rather than `FROM` a local +task's own records, so the task builds on its own rather than `FROM` a local kit image. Then complete the generated `instruction.md`, `task.toml`, `tests/test.sh`, and `solution/solve.sh` using the installed schema, deriving expected values with an independent reference query over that data. Set executable permissions, realistic timeouts, and deterministic rewards; leave no scaffold placeholders. -Introduce the Oracle as the prepared reference solution. +Use Harbor’s [reference solution](https://docs.harborframework.com/tasks/solution) +and [verifier](https://docs.harborframework.com/tasks/verifier) documentation +when explaining the reference and grader. Environment proof precedes both. Write the task README following the shared [Task documentation](../../eval-author/references/suite-readme.md#task-documentation) @@ -46,25 +52,31 @@ explicit incorrect-behavior control when inaction itself is correct. Confirm each task's `environment/` builds on the kit that [`eval-author-environment`](../../eval-author-environment/SKILL.md) built and -proved at stage 5, following its [Harbor environments](../../eval-author-environment/references/harbor.md) +proved before task grading, following its [Harbor environments](../../eval-author-environment/references/harbor.md) reference, and rerun its smoke task if the kit changed after it passed. Check documented access requirements, credential variable names, and reset behavior. Keep solutions and verifier-only data outside the agent's initial environment. For Docker-backed execution, check `docker info` first. Verify the selected backend and required application access; record unavailable -access and the cases it blocks. +access and the work it blocks. Show only blockers affecting current work; +defer run-only settings and blockers during independent authoring and review. ## 7. Connect the agent Establish a supported Harbor integration for the actual repository agent from its entry point, installed adapter code, registry, and CLI help. If the adapter -is missing, identify the specific integration requirement. NOP/Oracle validation -remains available, but performance of the user's agent remains unmeasured. +is missing when execution is ready, follow the +[missing-integration handoff](../../eval-author/references/conversations/first-eval.md#missing-actual-agent-integration-handoff). +Keep the target-agent adapter separate from grader setup: NOP/Oracle controls +can validate tasks when their own dependencies are ready, but they do not +measure the user's agent. Preserve that distinction when rechecking the adapter +after the external engineering work. When supported, write `.eval-author/run.yaml` using the installed Harbor -JobConfig schema. Explicitly select the working starter tasks, actual agent and +JobConfig schema. Explicitly select exactly the agreed tasks, actual agent and model settings, agreed attempts per task, and jobs under `.eval-author/jobs/`. -Use the parent skill's agreed attempt count and cost; do not include unrelated +Use the parent skill's authorized attempt count and limits; directory membership +never selects tasks. Do not include unrelated drafts merely because they share a parent directory. Resolve paths from the repository root and reference credential environment variables rather than embedding secrets. Configure request delivery, conversation state, and output @@ -116,6 +128,6 @@ harbor job start -c .eval-author/run.yaml Retain outputs, actions, rewards, exceptions, and exact commands for every selected task under fresh job paths. Low agent scores do not invalidate a working -suite. Return to stage 10 to reconcile delivered cases and explain full-suite and -single-case reruns. Label NOP/Oracle commands as task validation and the actual +suite. Return to the parent skill for exact-run review, reconciliation, and +linked full-suite and single-task rerun instructions. Label NOP/Oracle commands as task validation and the actual agent command as evaluation; use new trial and job names so reruns preserve prior evidence. diff --git a/skills/eval-author/SKILL.md b/skills/eval-author/SKILL.md index cd443ec8..f7c3e7f3 100644 --- a/skills/eval-author/SKILL.md +++ b/skills/eval-author/SKILL.md @@ -113,12 +113,20 @@ instead of repeating orientation. | Request | Route after shared prerequisites and Ethos | |---|---| -| Build first evals | `eval-author-first-eval`, continuing at the first unfinished authoring stage | +| Build first evals | `eval-author-first-eval`; confirm reuse of any saved design on a fresh request before continuing its unfinished stage | | Audit coverage, gaps, and improvement opportunities | `eval-author-audit`, continuing with eval and trace selection | | Improve or expand evals | Preserve the expansion goal and use the expansion handoff below; an audit is optional | | Create a task from a recorded interaction | [Task-create ATIF source mode](../eval-author-task-create/SKILL.md#atif-source-mode), or [first-eval](../eval-author-first-eval/SKILL.md#atif-as-a-starting-source) for a new suite; use the [shared ATIF handoff](references/trace-input.md), without requiring an audit gap | | Explicit trace-environment request or existing workspace continuation | [Trace environment](../eval-author-trace-environment/SKILL.md); retain existing artifacts and the compatibility workflow | +Acceptance of the first-evaluation flow's audit invitation is a goal switch +within the existing workflow. Follow its [saved handoff](references/conversations/first-eval.md) +and reuse confirmed `ETHOS.md`, provider, task paths, exact run/attempt evidence, +source choices, limitations, and preferences. Check only audit-specific or changed +prerequisites and source needs. Missing audit evidence blocks only the audit; +it does not reopen completed creation or authorize new runs. Offering the +invitation or receiving a decline starts no audit work. + When the goal is unclear, use `orientation` without a journey. Its shorter welcome and canonical goal choices ask only for the unresolved goal; reuse known context. @@ -163,8 +171,11 @@ An explicitly selected addition or proposal-only request keeps that scope. ### Downstream progress -Once prerequisites and Ethos are confirmed, present the completion pause and wait -for the user to continue before following the selected skill's stages. +Once prerequisites and Ethos are confirmed, follow the +[completion pause](references/shared-prerequisites.md#pause-after-ethos-confirmation). +For fresh creation without a saved design or source choice, that pause asks for +task sources directly; do not add a separate readiness question before it. +Other flows retain their selected continuation and saved decisions. Keep the selected third checklist item current. Reuse completed shared steps. Keep scope and evidence reviews, source selection, and execution authorization. For creation and improvement, that item covers building the agreed cases or diff --git a/skills/eval-author/references/conversation-rendering.md b/skills/eval-author/references/conversation-rendering.md index 533175ad..595849fb 100644 --- a/skills/eval-author/references/conversation-rendering.md +++ b/skills/eval-author/references/conversation-rendering.md @@ -10,8 +10,19 @@ produce byte-identical messages without clocks, randomness, repository reads, or writes. This guarantee concerns rendered messages. It does not make workflow execution, model decisions, tool results, or evaluations deterministic. +Use this entry point for complete checkpoint responses. Flow-specific helpers +render fragments; calling them directly can omit state gates and the overall +progress checklist. Do not use those fragments or a hand-written question as a +fallback when the entry point rejects the supplied state. + Present rendered Markdown unchanged, with no additional opening, duplicate -checklist, or appended next-step invitation. Canonical shared journey and +checklist, or appended next-step invitation. Copy the renderer's output verbatim +into the workflow response, preserving heading markers, order, links, and +escaping. Its headings are required workflow structure, not optional prose +formatting: Recommendations remains a heading even when there are no active +Blockers, and Next Steps remains a heading above its checklist. Before sending, +compare the response body with the final renderer output; do not remove headings +while shortening or reformatting the reply. Canonical shared journey and checklist labels, status labels, and generic headings live in [conversation_templates.py](../scripts/conversation_templates.py). Flow-specific copy belongs to that flow's template module, not a second prose template. @@ -74,11 +85,17 @@ preview permissions, row facts, blocker selection, and final-response ordering. Use it alongside [Shared prerequisites and Ethos](shared-prerequisites.md). Shared rendering guidance does not change those workflow gates. +[First evaluation rendering](conversations/first-eval.md) continues creation +with optional traces, design and first-task review, validation, execution +settings, results, and the optional audit handoff. Its stages require reconciled +saved state and use the `first_eval_` prefix; they have no stateless preview. + [conversation_renderer.py](../scripts/conversation_renderer.py) dispatches through an explicit [journey/stage registry](../scripts/conversation_flows/__init__.py) -to implemented flow renderers. Duplicate registrations are rejected. The current -flow is [onboarding.py](../scripts/conversation_flows/onboarding.py), with its -copy in [onboarding_templates.py](../scripts/conversation_flows/onboarding_templates.py). +to implemented flow renderers. Duplicate registrations are rejected. Implemented +flows are [onboarding.py](../scripts/conversation_flows/onboarding.py) and +[first_eval.py](../scripts/conversation_flows/first_eval.py), with copy in their +respective `_templates.py` modules. Shared formatting and escaping live in [conversation_formatting.py](../scripts/conversation_formatting.py). diff --git a/skills/eval-author/references/conversations/first-eval.md b/skills/eval-author/references/conversations/first-eval.md new file mode 100644 index 00000000..a08bcc7d --- /dev/null +++ b/skills/eval-author/references/conversations/first-eval.md @@ -0,0 +1,757 @@ + + + +# First evaluation conversations + +Use the [shared renderer](../../scripts/conversation_renderer.py) for creation +responses after the shared prerequisites and confirmed `ETHOS.md`. The fixed +copy lives only in +[first_eval_templates.py](../../scripts/conversation_flows/first_eval_templates.py). +Present its output unchanged, following the shared contract for response order +and any explanations separately required by the host. + +## Saved-design checkpoint + +A fresh creation request may find an earlier design without asking to resume it. +After the shared prerequisites, `ETHOS.md` confirmation, and completion +continuation, register this unresolved choice before using its source selection +or proceeding with authoring. Use normal revision-checked updates, preserving the +existing record and artifacts. Set its optional `saved_design_review` to: + +```json +{ + "digest": "", + "decision": "pending", + "request_source": "", + "source": null +} +``` + +Render `first_eval_saved_design` as the final response and end the turn. The +renderer summarizes the saved tasks, distinct examples, source selection, +historical design approval, and evidence-backed progress before its fixed +choice. It does not inspect trace contents or require execution settings. +Neither `ETHOS.md` confirmation nor the shared continuation answers this choice. +Do not turn a historical continuation reference into a reply in the current chat. + +Save the actual response in `source` and change `decision` to `continue`, +`revise`, or `new`. Continuing must refer to the same current `plan_digest`; +if it changes before the answer, present the updated saved design again. +Continuing reuses applicable approvals, evidence, and authorization; it grants +no new spending permission. Revising reopens proposal approval while retaining +the source choice. A new proposal also reopens source selection. Both retain +prior task and run records and historical evidence; choose stable new task IDs +and directories when building new tasks rather than overwriting earlier work. +Retire any old audit invitation and handoff into history when revising or +starting a new proposal, so it cannot route or suppress the new completion handoff. +Source changes and a replacement plan still require their ordinary checkpoints. + +Skip this choice when the current user request explicitly identifies the saved +design to resume, or when it was already answered during ongoing work. Do not +re-prompt on every turn or treat every legacy record as unapproved. Missing +`saved_design_review` remains readable, but it does not establish the user's +intent on a fresh request. Once pending, the state helper and renderer block +dependent stages until the saved-design choice is resolved. + +## Source and proposal pauses + +Source selection, saved-design confirmation, and task proposals start with the +shared overall progress checklist. It shows the completed prerequisite and +`ETHOS.md` steps and marks creation as current before the decision content. +Keep its heading and current-step marker when presenting the rendered response. +This context does not introduce build steps, execution settings, or a new pause; +run-only blockers do not prevent independent design review. + +For fresh creation without a structured saved design or source choice, the shared +[confirmed handoff](onboarding.md#stateful-responses) includes the canonical source +introduction, question, and choices from `first_eval_templates.py`, immediately +after document confirmation and the overall checklist. Save that canonical +question as `pending_question` and `eval-author-first-eval` as `next_handoff` +before rendering, then end the turn and wait for the source answer. Do not +insert a readiness question before it or render `first_eval_traces` again after +the user has answered it. Legacy continuation questions remain readable and +render as that same combined checkpoint when the fresh condition applies. + +Otherwise, after the applicable shared continuation or saved-design decision, +render `first_eval_traces` as the final response when the source choice remains +unresolved, then end the turn. For a standalone checkpoint without a saved +design, supply only `stage: first_eval_traces`, the fresh reconciled `state`, and +its `revision` to the shared renderer. No proposal record, task planning, +observations, or execution settings are needed first. +When a saved design remains after choosing a new proposal, also supply its fresh +`facts.observations` as usual. The canonical question and choices come from the +template module. + +For either checkpoint, present the complete output, including `Next Steps`, both +completed shared steps, and the current creation step before the source question. +Collect the user's next reply to this rendered checkpoint; do not move its source question +into a separate question tool. Do not continue planning or investigating +integrations while the answer is pending. Reuse a saved choice instead of asking +again. +Use the shared entry point for the proposal too; its internal `proposal` helper +omits the checklist and state gates. Do not add free-form previews of task scope +or adapter implementation in commentary. + +Save the actual source answer and its provenance, then clear the pending source +question. For existing traces, resolve the selected location, runs, and exclusions +before inspection. Then prepare only +the bounded design needed for `first_eval_proposal`: by default one to three +tasks with one example per task. Larger task or example counts require the +user's explicit broader request in `plan.breadth_request.source`. This optional +field binds the proposal to that request; it does not grant approval or spending +authorization. Existing records remain readable and previously approved scope +is retained. + +The proposal states the counts, shows the task table and coverage limitations, +names the first task, and explains the one-task build/review pause before the +remainder. Its fixed feedback question is the final content. End the turn and +wait for actual design approval before implementation or further integration +work. Source selection and readiness to continue do not approve the design. +No additional planning document or generic approval checkpoint is needed. + +### Proposed grading + +Each task's existing `plan.tasks[].grading` supplies the proposal table's +**Grading** cell. State whether the proposed method uses code checks, an actual +model judge, or both, and what each part evaluates. For a model-based portion, +describe the judgment needed and its material limitations; include the proposed +judge configuration when it is already established, otherwise identify it as +unresolved. Do not invent a model choice, replace semantic judgment with brittle +wording checks, or hide model grading until after authoring. + +The proposal presents the grading design without requiring a ready judge setup. +After design approval, resolve any required judge inputs through +[validation setup](#validation-setup-before-task-creation) before task creation. +Reuse answers already established during design review without another pause. + +## Validation setup before task creation + +After saving design approval and before any task scaffolding, fixtures, +instructions, reference solutions, graders, or delegated authoring, inspect what +the agreed graders need for validation. Resolve missing judge configuration, +compatible runtime, access, and explicit authorization for the concrete control +work now. Ask for the missing inputs before starting task creation; do not leave +a foreseeable setup question until authored-task review. Target-agent execution +models, budgets, and run settings remain after whole-suite approval. + +Save the optional `validation_setup` sibling of `plan` under +`scopes.create.first_eval`, through the existing revision-checked interface: + +```json +{ + "plan_digest": "", + "mode": "model", + "requirements": [""], + "open_questions": [""], + "source": "" +} +``` + +Use `mode: "deterministic"` when no grader needs a model judge. Record all known +requirements as nonempty plain-text items. For model grading, identify the +supported judge configuration and the controls and bounds covered by existing +or requested authorization. State unknown costs honestly; do not infer spending +consent from design approval or access alone. `source` records actual inspected +findings or replies, not an invented approval. The setup is separate from the +plan so answering these questions does not reopen an unchanged task design. + +When required user inputs are missing, render `first_eval_validation_setup` with +the overall progress checklist, known requirements, and focused questions; its +template owns the fixed wording and final call to action. Emit it as the final +reply and end the turn before any dependent work. Ask only what is unresolved, +including applicable validation authorization. Do not request secret values; +ask the user to make credentials available in the execution environment. A +pending user choice does not itself require a blocker. For unavailable tooling +or failing access, include the specific dependency and needed action through the +usual **Blockers** section. + +When deterministic controls or an existing model setup are already ready, save +the inspected setup with an empty `open_questions` list and continue without a +new approval pause. After a reply, verify the supplied configuration and access, +save its provenance and resolve the answered questions before beginning work. +An empty list is readiness only when inspected findings and actual applicable +authorization support it. Do not guess a model, add an integration to bypass +missing support, or substitute a stub judge to make setup appear ready. + +Readiness must already be saved before new task creation, changed task bytes, +authoring activity or receipts, remaining-task authoring, or new validation +evidence can be recorded. Missing setup in an older record is unknown; preserve +completed files, approvals, and evidence, and resolve only missing inputs before +resuming unfinished authoring or validation. When all agreed task files, +environment proofs, and full grading-validation evidence remain current, +report-only refresh and authored-task review can continue without reconstructing +judge configuration or prompting for unused validation inputs. This exception +does not authorize new task bytes, authoring receipts, or grading checks. Changed +grading requirements reopen affected setup questions before further authoring. +A completed suite does not receive another pre-authoring setup prompt. + +## Grader validation before authored-task review + +Before `first_eval_review_first`, validate the first task's complete grader. Before +`first_eval_review_suite`, every agreed task must have current grading-validation +evidence. Follow [task validation](../task-validation.md): known-correct results +and permitted valid alternatives must pass, while relevant incorrect results +must reach grading and receive the expected rejection. Bind retained native +control evidence to the current task revision and declared checks. + +When grading uses a model to judge reply meaning or another semantic requirement, +the controls must exercise that actual judge as configured. Writing its rubric, +checking output structure, passing environment smoke checks, or running only the +deterministic portion cannot establish full grading validation. A substitute +judge, hardcoded verdict, skipped semantic check, or target-agent run does not +satisfy this gate. Task revisions reopen affected validation before review. + +Use the configuration, access, and bounded authorization resolved through +[validation setup](#validation-setup-before-task-creation). If a previously ready +dependency stops working, show **Blockers** with the specific needed action and +hold dependent work and authored-task review. Preserve completed work and allow +draft report refreshes with validation clearly incomplete; do not use review-ready wording +or an approval question. The initial task proposal remains a design review +before building and does not require grader execution. + +## First-task build progress + +After saving design approval and ready validation setup, initialize +`first_task_build` in the existing creation record through a normal revision-checked update. Bind it to the derived +`plan_digest` and agreed `first_task`, set `active_step` to `environment`, and +record the actual started activity in `source`: + +```json +{ + "plan_digest": "", + "task_id": "", + "active_step": "environment", + "source": "", + "receipts": {} +} +``` + +Render `first_eval_prepare_first` once before work. It includes the provider's initial +explanation and fixed build checklist. During the build, use +`first_eval_build_progress` for subsequent commentary snapshots instead of +free-form build narration. Keep the same rows and ordering; the template owns +their labels, responsible skills in parentheses, and short statuses. Refresh +only when a displayed status, task count, or blocker changes. Keep the latest +snapshot visible during long-running checks; tool calls, polling, and delegation +do not require another unchanged checklist. + +The coordinating agent alone emits visible build checklists. Delegated workers, +including environment work, return milestone results and evidence as they finish +so the coordinator can update the shared progress. Yield during long checks so +changed progress can be shown. Do not append deliberation, speculative next +steps, command transcripts, or another checklist. Retain verbose command output +in evidence logs and capture state-helper output instead of printing full state +and history. A focused question remains appropriate when work needs a user decision. + +For `first_eval_build_progress` and `first_eval_remaining_progress`, include +`facts.previous_progress` when a snapshot has already been shown for this build +in the current chat. Supply that exact last visible rendered build message, +including its initial explanation and **Blockers** when present. Do not reconstruct that message or store it in +workflow state. After validating current state and evidence, the renderer returns +empty Markdown when the complete visible checklist and blockers are unchanged. +Do not post an empty result or replace it with a handwritten duplicate. If a +commentary update is needed while no displayed progress has changed, use one +brief factual sentence about the current operation without claiming completion. + +Omit `previous_progress` for a new chat, an explicit status request, plan or task +rebinding, or a return from review, so the full current snapshot is shown. This +field is accepted only on the two explicit build-progress stages, not on +`first_eval_resume` or review stages. It affects presentation only; current +approvals, evidence, and workflow gates remain required. + +The ordered step keys are `environment`, `environment_check`, `grading`, and +`artifacts`. Update `active_step` when the corresponding work starts. Save a +receipt under `receipts.` only after that checkpoint succeeds, with +`{source, evidence: [{path, digest}]}` identifying inspected artifacts or command +results. Preparation retains the actual environment files and starting data; +environment checking retains successful smoke/reset evidence; grading retains +the authored reference and grader files; artifacts retains the finalized task +README, HTML report, and Markdown mirror. Use scoped file digests, so adding a +grader does not invalidate environment preparation. Failed or merely started +checks never receive a completion receipt. Preserve their logs and keep the +step in progress or explain its blocker. + +Reload normal state and observations before each snapshot. The helper derives +completed rows from current receipts; a changed or missing receipt reopens its +row and dependent work. A changed plan or first task requires rebinding build +progress with empty receipts, then re-recording any still-applicable evidence. +When replacing an earlier receipt, clear its dependent later receipts before +recording their renewed checkpoints; earlier evidence remains in history. +Absent legacy receipts remain pending. Pass only current build blockers +in `facts.blockers`; the active row shows Blocked and the usual **Blockers** +section explains the needed action. Missing agent credentials or later run +integration do not block independent authoring. + +Authored reference and grader files establish construction, not successful +grading validation. Checklist receipts never replace the +[pre-review validation gate](#grader-validation-before-authored-task-review), +agent-performance evidence, or human approval. Keep those claims separate in +`first_eval_review_first`. After the first task is approved, later report +refreshes do not restart this build. + +## Remaining-task build progress + +After saving approval of the current first-task revision, initialize +`remaining_build` through the existing revision-checked update interface: + +```json +{ + "plan_digest": "", + "first_task_sha256": "", + "active_step": "tasks", + "source": "", + "artifacts": null +} +``` + +Choose the actual current step from `tasks`, `environment_check`, `validation`, +and `artifacts`; completed prior work may make a later step the right starting +point. Render `first_eval_prepare_remaining` initially, then +`first_eval_remaining_progress` for subsequent snapshots. Follow the same compact +commentary and evidence-output rules as the first-task checklist. Both checklists +include the responsible skill names beside each item, owned by the template. +Do not add individual approval pauses for later tasks. + +Record task creation, environment proof, and grading validation through the +existing task/evidence records. The renderer derives their counts from current +evidence. Creation and environment rows count only remaining agreed tasks; +validation counts all agreed tasks, including the validated first task. +Tasks can progress individually, so the active row need not follow a global +sequence. An empty remaining set creates no new tasks. Reuse compatible +environment proof; only changed or additional environment work needs rebuilding. + +After authoring all tasks, proving their environments, and satisfying the +[pre-review validation gate](#grader-validation-before-authored-task-review), +refresh both report artifacts and save `artifacts` as +`{digest, source, evidence}`. Its `digest` is the current derived +`remaining_build_digest`; its evidence names the actual `.eval-author/report.html` +and `.eval-author/README.md` with their byte digests. This binds the refresh to +current task revisions and successful grading controls. Earlier draft refreshes +do not establish review readiness. Missing or stale artifacts stay pending; +later refreshes during execution do not restart completed authoring. + +Pass only blockers affecting current build or validation work. Run-only access +does not block independent authoring. Changed judge prerequisites reopen only +the affected [validation setup](#validation-setup-before-task-creation) and block +dependent work; task or first-task approval does not authorize spending. Keep +target-agent execution questions after suite approval. +The checklist never supplies validation evidence or agent performance claims. + +## Review the authored suite before execution settings + +When every agreed task is authored, its environment is proven, the +[pre-review validation gate](#grader-validation-before-authored-task-review) +passes, and the refreshed HTML report and Markdown mirror are current, render +`first_eval_review_suite`. +Use its grounding in the approved design and `ETHOS.md`, leading review links, +task summaries, honest validation/execution status, limitations, and final +approval question. Required model-based grading and calibration must already +have passed. Task review does not require choosing the target-agent execution +model, credentials, limits, or how to execute it. +Keep those questions and run-only blockers out of this review. Do not append a +generic report footer or routine pause explanation. + +Emit the rendering as the final response and end the turn. A nonblocking +question-tool return does not answer it. Do not proceed with execution settings +or agent runs while the suite decision is pending. Revise tasks in place, +refresh their evidence and artifacts, and request review again when +needed. A side question or first-task approval does not approve the whole suite. + +Save `suite_review_presented` as `{digest, source}`, using the derived +`suite_digest` and the actual delivered review message reference. After the +user explicitly approves, save `suite_approval` with that digest and the actual +approval reference in a separate revision-checked update. If the host ends the +turn when delivering the review, record its presentation from the prior message +at the start of the response turn, before recording the user's decision. + +The digest binds the approved plan and exact task revisions. Changed task files +or design reopen approval; later validation results or report refreshes alone +do not. Missing legacy fields remain unknown, rather than implied approval. +Historical evidence stays readable. Only after current suite approval, present +the execution-settings checklist and resolve missing run decisions. Suite approval +authorizes the task design, not spending or extra evaluations. + +## Agent-run progress + +After the provider confirms that the approved job has launched or entered its +queue, save an optional `run_progress` under `scopes.create.first_eval` through +the revision-checked state interface: + +```json +{ + "selection_digest": "", + "run_id": "", + "status": "running", + "active_task_ids": [], + "source": "", + "evidence": [{"path": "", "digest": ""}] +} +``` + +Use `running` or `waiting` only when observed in native job status. Name an active +task only when that status identifies it, using a subset of the explicitly +selected tasks; otherwise leave `active_task_ids` empty. Retain an immutable +status snapshot with its exact run identity rather than binding progress to a +changing log. A launch announcement, elapsed time, or unfinished attempt is not +evidence that a job is running. + +Render `first_eval_run_progress` with fresh observations. Its applicability +requires current suite approval, explicit authorized selection, environment and +grading evidence, matching selection digest, and intact snapshot evidence. A +matching current, explicitly selected actual-agent run with status `running` +preserves the live snapshot; a terminal run or control record with that identity +retires it. The renderer's four fixed rows come from the template module and +omit skill labels; they cover grading, execution, +inspection of an actual attempt, and the final artifact refresh. The checklist +does not create evidence or grant permission for retries or additional runs. + +Completed counts come only from current terminal attempt evidence in explicitly +selected actual-agent runs, including a job whose status remains `running`. +Retain completed attempts in that same native run record and explicitly select +its exact ID; do not invent separate jobs or wait for job exit to record progress. +Attempt statuses stay terminal, and the suite remains incomplete until the +selected running job becomes terminal. A completed attempt with a zero grade +still counts as complete; execution errors remain explicit. Controls do not count as agent +attempts. Never infer active attempts by subtracting completed from planned +attempts or imply that every unfinished attempt is running. A retained exact +review may complete its row only while its evidence remains current. + +Show the checklist after confirmed launch or queuing and refresh it when a task, +attempt, or native status changes. During a quiet run, inspect native status +about once a minute and refresh only when there is a new observation or meaningful +change; do not repeat unchanged updates. Use concise commentary snapshots, +without command logs, deliberation, or additional approval questions. Pass +current blockers through the usual **Blockers** section and pause only dependent +work. Use the affected template row's label as `blocker.step` to mark that row +blocked; an artifact blocker must not mark the running job blocked. +On interruption or resume, inspect the native job before describing it as +running or launching it again. When the job finishes, retain its terminal native +evidence and continue to result review and artifact refresh; the live checklist +does not establish completion of that handoff. + +## State and artifact authority + +All creation stages require the complete normal, artifact-reconciled state from +the existing state helper and its exact revision. They have no stateless preview. +The saved journey is `create`; `state.ethos.stage` must be `complete`. Normal +state inspection retains completed work and reconciles changed `ETHOS.md`, task +files, and evidence. A state-only prerequisite snapshot is insufficient. + +The caller obtains fresh `first_eval_observations` from the state helper and +passes them as `facts.observations`. They contain actual task digests under +`tasks` and actual retained-file digests under `files`; missing artifacts are +`null`. The renderer derives applicability from these observations and the saved +`scopes.create.first_eval` record. It never accepts supplied approval, validation, +completion, or progress booleans. Save decisions and actual results before +presenting them. Rendering is read-only and cannot establish permission to build, +run an agent, or spend money. + +Before a design record exists, `first_eval_resume` renders the source question +when no choice is saved. Once the source is known, save the small concrete +proposal before resuming; the renderer does not invent its task rows or insert +another readiness question. Bind the saved choice in `plan.trace_choice` before +presenting that design. An unanswered source choice resumes at `traces`, and an +unapproved design resumes at `proposal`. Once design approval is current, +unresolved pre-authoring validation inputs resume at `validation_setup` before +new authoring or validation work. + +```json +{ + "stage": "first_eval_resume", + "state": "", + "revision": 7, + "facts": { + "observations": { + "tasks": {"": ""}, + "files": {"": ""} + } + } +} +``` + +Submit the complete payload through the shared entry point: + +```bash +python3 -B /scripts/conversation_renderer.py --input - +``` + +The caller owns all file inspection, provider checks, report refreshes, and +evidence provenance. No renderer stage performs those actions or changes saved +state. Identical supplied inputs produce identical Markdown. + +## Saving the agreed design and evidence + +Use the existing `journey_state.py --repo update --input +--expected-revision ` interface. A patch places `first_eval` inside the +`scopes.create`; preserve the rest of the saved record. +The minimal initial record has this shape (replace descriptions with actual +values): + +```json +{ + "schema_version": 1, + "provider": "harbor", + "ethos_digest": "", + "plan": { + "trace_choice": "none", + "tasks": [{ + "id": "", + "example_ids": [""], + "title": "", + "input": "", + "behavior": "", + "expected_result": "", + "grading": "" + }], + "presentation": {"coverage_note": ""} + }, + "plan_approval": null, + "first_task": "", + "tasks": {}, + "selection": null, + "runs": [], + "run_review": null, + "audit_invitation": null +} +``` + +Select only the user's agreed provider. For selected traces, replace `none` with +the bounded source selection already agreed with the user. Copy a previously +saved `scopes.create.trace_choice` into `plan.trace_choice` before obtaining +design approval so that its scope digest binds the reused selection. Omitted decisions and +evidence stay unknown. The schema and pure applicability implementation are in +[first_eval_state.py](../../scripts/first_eval_state.py); use its `derive(state, +observations)` return values instead of reimplementing digest formulas. Normal +`journey_state show` exposes both `first_eval` progress and +`first_eval_observations` for this purpose. + +After the actual design approval, save `plan_approval` with the current derived +`plan_digest` as `digest` and the user's reply reference as `source`. Save it, +then resolve and save [validation setup](#validation-setup-before-task-creation), +before recording generated tasks. Each task record contains its stable `path`, +observed task-tree `sha256`, `review` (initially `null`), `environment`, +`validation` (initially `null`), and `limitations`. Environment and validation +observations use `{sha256, status, source, evidence}`; each retained evidence +reference uses `{path, digest}`. A first-task approval uses +`{sha256, decision: "approved", source}` bound to the actual reviewed revision. +Obtain task bytes from `journey_state.task_digest(repo, task_path)` and retained +file digests from SHA-256 of their actual bytes. Never invent a digest from a +filename, intended result, or approval. The state helper enforces previously saved checkpoints, so approval cannot be +combined with the dependent generation in one update. + +After current suite approval, when preparing execution, save `execution_settings`; +after presenting it, record +`execution_settings_presented` with the current derived `settings_digest` and +the actual delivered message reference. `selection` retains the explicit config +reference, selected `cases`, agent/model identity, source, and applicable +`settings_digest`. Each case names `task_id`, agreed `example_ids`, and positive +`attempts`; native Gym also names the corresponding one-based dataset `rows` and +retains complete input references under `selection.inputs`. Actual bounded +permission is recorded separately in `selection.authorization` using the derived +`selection_digest` and the authorizing reply reference. + +Retain native `runs` with their stable IDs, selected task revisions, native +attempt identities, statuses, rewards, and evidence. Explicitly save +`selected_run_ids`; directory membership, append order, and a reviewed attempt do +not select a run. Save `run_review` with `run_id`, `attempt_id`, the run's derived +`run_digests[run_id]` as `digest`, inspection `source`, and material `limitations`. +Write an immutable per-run Markdown review outside the task tree, such as +`.eval-author/task-measurements/-review.md`, identifying the same run and +attempt and its observed findings. Bind it under `run_review.summary` as +`{path, digest, run_id, attempt_id}` after inspecting the actual saved text. +Do not bind this immutable review to the report-generated suite README, which +changes on refresh. Missing or changed summary evidence reopens the review. + +After completion, `audit_invitation` records `{status: "pending", source: null}`; +the user's actual response changes its status to `accepted` or `declined` with +that reply's source. The existing handoff state controls entering the audit. + +## Stages and review boundaries + +| Stage | Required context and rendered response | +|---|---| +| `first_eval_saved_design` | A pending saved-design choice on a fresh request; compact saved scope, historical approval, and current progress, then a fixed continue/revise/new-proposal question. End the turn before any dependent work. | +| `first_eval_traces` | Confirmed `ETHOS.md`; fixed optional-source question and choices unless the source decision is already saved in the creation scope or plan. Inspect only selected traces after resolving their location, runs, and exclusions. | +| `first_eval_proposal` | Saved proposed task designs; five-column task table, concise coverage and limitations, first-task choice, and fixed design feedback question. The question is the last content. Execution settings and run-only blockers remain deferred. | +| `first_eval_validation_setup` | Saved design approval and inspected grading requirements. Show known validation setup and ask only missing judge configuration, access, or bounded validation authorization before any task creation. Ready setup continues without an extra approval pause. | +| `first_eval_prepare_first` | Previously saved design approval and ready validation setup; initial provider explanation and compact build checklist. Author exactly one task in its permanent directory, prove its environment before writing its grader, and satisfy the pre-review validation gate before returning for review. No target-agent execution checklist is required for local authoring. | +| `first_eval_build_progress` | Saved approved first-task build activity and fresh observations; the same fixed checklist with short completed, active, pending, or blocked statuses. Use commentary during work, without adding narration or requesting approval. | +| `first_eval_review_first` | Current first-task files, saved environment proof, complete current grading validation, and refreshed HTML and Markdown artifacts. Lead with the approved task's stable ID and grounding in `ETHOS.md`, then direct the user to review the HTML report's Tasks view or mirrored README. Follow with input, expected result, grading with a meaningful limitation, and separate environment, grading-validation, and execution status. Link the task README and end with the first-task approval question. | +| `first_eval_prepare_remaining` | Saved approval of the current first-task revision and environment proof. Initial explanation and skill-labeled checklist for building the remainder, checking environments, validating all graders, and refreshing artifacts. | +| `first_eval_remaining_progress` | Current remaining-build activity after first-task approval; compact checklist with evidence-derived task counts, responsible skills, short statuses, and current build blockers. | +| `first_eval_review_suite` | Every authored task's current environment and complete grading-validation evidence plus current report artifacts. Ground the summary in the approved design and `ETHOS.md`, link the HTML Tasks view or mirrored README, summarize each task and current validation/execution evidence, and end with the explicit suite-approval question. No run settings or run-only blockers. | +| `first_eval_execution_settings` | Current authored-suite approval, when actual agent execution is being prepared. Display attempts, validation work, time/cost limits, and open questions, plus relevant run blockers. Do not repeat agent/model or task/example inventory bullets. This display never supplies spending authorization. | +| `first_eval_ready_to_run` | Current suite approval, validation of every selected task, and a saved applicable settings display. Explain that graders accepted the known correct results and rejected the incorrect results tested; this validates grading rather than measuring agent performance. Announce execution only with applicable explicit authorization, access, and no blocking dependency; otherwise ask one focused question about the missing setting or authorization. | +| `first_eval_run_progress` | Observed native launch or queued status with current authorized selection and immutable status evidence. Render the standard checklist without skill labels, evidence-derived completed/planned attempts, recorded execution errors, and only actually observed activity. No run-only approval pause or inferred activity. | +| `first_eval_results` | Review an exact retained current actual-agent attempt when its evidence is available; otherwise present the incomplete inventory and missing-evidence blocker. Separate completed, quality-failing, infrastructure-failing, missing, and unavailable-evidence outcomes. Link verified evidence and rerun instructions, reconcile delivered work, and show completion only when justified. | +| `first_eval_resume` | Derive the pending creation stage from saved decisions and fresh artifact observations, including an unresolved saved-design choice. Reuse applicable answers and authorization after that choice. It emits no welcome or initial readiness question. | +| `first_eval_blockers` | One or more saved blockers preventing the current action, with specific reasons and needed actions. Preserve independent progress. | + +The proposal uses the saved plan's `presentation.coverage_note` for its usefulness, +untested behavior, and grading explanation. Task rows provide `id`, `input`, +`behavior`, `expected_result`, and `grading`. Use the saved stable `id` for task +labels in proposals, saved designs, build and run progress, and reviews so users +can locate the same task in the report and filesystem. A descriptive `title` +does not replace that ID. Follow the shared +[expected-result summary contract](../suite-readme.md#expected-result-summaries) +for the proposal and review fields. Do not create a separate user-facing planning +document. + +Late execution settings come from the saved record's `execution_settings`, with +existing plan-presentation fields retained as a compatibility fallback. They +contain `agent_model`, `attempts`, `validation`, `limits`, and `open_questions`. +Retain target/model and selection details for reproducibility without repeating +them as settings bullets. Ask a focused question only when a target-agent model choice is +still missing. Record the actual display +through the state helper; viewing rendered output cannot count as delivery or +human approval. Local controls remain distinct from actual-agent runs. +An affirmative reply to an explicit request to run within displayed limits is +execution authorization, not another task-review decision. Record that reply +and proceed when its authorization is current; do not request a second approval. +The ready-to-run recap reuses current saved grading results and does not imply +that calibration was performed again. + +## Facts and verified navigation + +Prose facts are plain text and escaped as literal content. First-task review +requires `input`, `expected_result`, `grading`, and `grading_limitation`. Results +require `input`, `observed`, `grade_explanation`, `quality_failures`, +`identity_limitations`, and `reconciliation`; retain unknown model health and +other material limitations. If no current selected actual-agent review exists, +the incomplete results response requires only the honest `reconciliation` and +verified report links; optional rerun navigation may be supplied when available. +It reports saved identities as historical, never fabricates a reviewed response +or agent performance, and shows a missing-evidence blocker. `access_ready` is an observed boolean for run +readiness, never an authorization flag. Supply `missing_execution_question` only +when a focused question is needed. + +Pass destinations separately under `facts.links`. Each destination is an object +with an absolute `target` and optional `fragment`: + +```json +{ + "links": { + "report": {"target": "/absolute/repository/.eval-author/report.html"}, + "readme": {"target": "/absolute/repository/.eval-author/README.md"}, + "tasks_view": { + "target": "/absolute/repository/.eval-author/report.html", + "fragment": "" + }, + "task_readme": {"target": "/absolute/repository/.eval-author/tasks//README.md"}, + "rerun": { + "target": "/absolute/repository/.eval-author/README.md", + "fragment": "" + } + } +} +``` + +Before calling the renderer, verify that every local target exists, is within +the allowed repository without symlink traversal, and identifies the intended +artifact. Verify HTML fragments against real element IDs and Markdown fragments +against actual heading anchors, including duplicate-heading suffixes. A plain +local target containing `#` names that literal file; supply a navigation fragment +in the separate field. The renderer formats these verified values without +reading files or proving that navigation exists. + +Both task reviews require `readme` and `tasks_view` with its actual `cases` +fragment; first-task review also requires `task_readme`. Results require +`report`, `readme`, and `rerun` with a verified +fragment. Optional `task_rerun` also requires a verified fragment. Do not hard-code +a heading slug without checking the generated README. + +Results additionally require `links.run_evidence` and `links.attempt_evidence`. +Each is a mapping from the exact saved evidence path to its verified destination. +Their key sets must match the reviewed run and attempt evidence respectively. +Later runs do not replace that saved review identity. Inspect the actual attempt +before supplying its input, actions, and grading explanation; configured identity +alone does not establish the observed backend model or model health. + +First-task and full-suite review put their report and mirrored README links in the leading +review action, before the item summaries. Do not append the generic report footer +or routine skill-pause explanation to that rendering; separately required host +attribution still applies. Incomplete results use the single canonical closing +sentence in [conversation_templates.py](../../scripts/conversation_templates.py). +Follow [suite review guidance](../suite-readme.md); do not duplicate artifact links. + +## Completed run review + +Completed creation results use the flow template's linked run-review guidance +after the substantive results and completed checklist, immediately before any +pending audit invitation. Link the verified HTML report and mirrored README +once each in that guidance; do not append the generic report footer. Keep this +guidance when an accepted or declined invitation is no longer shown. + +Explain how to inspect the saved run's tasks, attempts, grades, and available +evidence using the provider's supported navigation. For Harbor, direct the user +to the report's Runs tab and the selected saved run. Native NeMo Gym run evidence +is linked from the README; do not imply that its attempts appear in the report's +Harbor-only Runs table. Preserve verified evidence links and limitations, and +follow [review walkthrough requirements](../suite-readme.md#show-the-user-how-to-review) +without duplicating the report or README links. + +## Missing actual-agent integration handoff + +When actual-agent execution is being prepared and inspection establishes that +its supported adapter, entry point, or configured Gym–Harbor bridge is missing, +use the renderer's external engineering handoff. Do not ask for an Eval Author +scope override to implement the production integration or offer to build it in +this workflow. Preserve independent task authoring, grader validation, and review; +a target-agent connection is not a judge dependency, and a working judge does +not prove the target-agent connection. + +Save the missing integration through the existing state helper as a blocker +with `id: "agent-integration"`, `phase: "create"`, and a concrete `message` +identifying the actual agent, selected provider path, missing interface, and +inspected evidence. Normalize an older custom blocker for this same dependency +to this ID and phase through a revision-checked update, retaining its findings +and unrelated blockers rather than duplicating it. Execution, results, and the +dedicated blocker stage include this saved blocker automatically; an optional +matching `facts.blockers` entry can supply the observed reason. The renderer +owns its step label and needed action regardless of caller wording, and adds a +copyable coding-agent prompt after **Blockers**; fixed handoff copy lives only in +[first_eval_templates.py](../../scripts/conversation_flows/first_eval_templates.py). +Do not replace that output with a permission question. Defer this run-only +blocker during independent proposals, task preparation, validation, and authored +task review rather than appending the handoff to those checkpoints. + +The prompt takes its context from the saved repository, actual agent, confirmed +`ETHOS.md` path, selected provider or configured Gym–Harbor path, and agreed task +and example IDs. Include the actual retained paths for already created selected +tasks; planned tasks have no invented paths. Unrelated task directories are not +selected by membership. Keep existing tasks, graders, source selections, +construction declarations, and evidence intact. The user can copy the prompt +into a separate chat in their chosen coding harness; displaying it does not +authorize dispatching that chat, changing the agent, or starting a model run. + +On return, inspect the changed integration and its reported checks. Recheck the +installed adapter or entry point, native configuration, selected task/row and +example mapping, reset behavior, and output collection against the actual agent. +A configured Gym–Harbor path must use that bridge; direct Harbor execution of +converted tasks does not satisfy it. Do not substitute controls, a fixture/demo +target, or another agent for the user's agent. Preserve limitations and failed +checks, and clear the saved blocker only when the missing supported connection +is established. Engineering completion alone is not native execution evidence +or run authorization: reuse applicable authorization and resolve missing run +settings through their existing checkpoint before any target-agent attempt. + +## Blockers and the audit invitation + +Save blockers through the existing state helper. Supply `facts.blockers` as +objects containing the matching saved `id`, `step`, `reason`, and `needed`. +The renderer uses the shared Blockers heading and escapes these explanations. +Proposal approval remains focused on design; use the dedicated blocker stage +when design itself cannot proceed. During local authoring, show only supplied +blockers for the current action and defer run-only dependencies. At execution +and results checkpoints, explain all applicable saved creation blockers. + +When creation is complete, save a pending audit invitation before rendering the +handoff. The fixed invitation follows the linked completed-run review guidance +and is the final substantive content. Acceptance enters the existing audit flow with +confirmed `ETHOS.md`, task locations, provider, selected evidence, exact reviewed +run and attempt, limitations, and preferences. Recheck only changed or +audit-specific prerequisites. Missing audit traces block the audit while leaving +creation complete; do not silently create them with extra evaluations. Declining +starts nothing and suppresses repeat invitations. Acceptance alone authorizes +neither task expansion nor paid execution. Incomplete creation instead leads +with unfinished work and its blockers. diff --git a/skills/eval-author/references/conversations/onboarding.md b/skills/eval-author/references/conversations/onboarding.md index aaefa8a3..a1b26cb1 100644 --- a/skills/eval-author/references/conversations/onboarding.md +++ b/skills/eval-author/references/conversations/onboarding.md @@ -168,12 +168,21 @@ are described below. | `prerequisite_results` | Fresh state-only snapshot and recorded checks; canonical completed-results sentence, table, optional Recommendations, and Blockers directly in final when blocked; no checklist or target-artifact facts | | `results` | Routine final after a completed, recorded prerequisite pass and normal artifact reconciliation; canonical completed-results sentence, table, optional Recommendations, then Blockers or Next Steps with one checklist and the current Ethos action; also available for standalone full reports | | `ethos` | Later Ethos-only action; Next Steps, state-derived checklist, and inspected Ethos/draft/question without rows or Recommendations | -| `confirmed` | Saved complete Ethos and applicable completed prerequisites, before downstream work has started; saved document confirmation and the selected continuation pause, unless a relevant blocker prevents continuing | +| `confirmed` | Saved complete Ethos and applicable completed prerequisites, before downstream work has started; saved document confirmation and the selected pending decision. Fresh creation without a structured saved design or source choice shows the canonical first-eval source choice directly; other cases retain their continuation. A relevant blocker takes precedence. | | `resume` | Saved pending decision or unfinished work; current progress and question, without a welcome, initial readiness question, or invented continuation approval | -Known continuation prompts -from earlier wording retain the same gate and render with current canonical copy; -rendering does not rewrite the saved question. +Known continuation prompts from earlier wording remain readable. For fresh +creation without a structured saved design or source choice, they render as the +combined source checkpoint instead of an extra readiness gate; other cases keep +their continuation and current canonical copy. Rendering does not rewrite the +saved question or record an answer. Save the canonical source question and +`eval-author-first-eval` handoff before presenting a new combined checkpoint, +following [the shared pause](../shared-prerequisites.md#pause-after-ethos-confirmation). +Its question and choices use +[first_eval_templates.py](../../scripts/conversation_flows/first_eval_templates.py), +not a second copy of that text. The same fresh, pre-start condition applies when +`results` or `resume` reaches the completed-Ethos handoff; it does not replace a +saved-design decision or restart a journey that has already begun. ## Prerequisite row facts @@ -205,8 +214,9 @@ Supply links separately through optional row `links` entries with `label` and ## Ethos and draft facts For Ethos facts, use `existing`, `absent`, or `unknown`. An existing Ethos link must -point to the exact repository-root `ETHOS.md` from state. Supply a grounded -summary for an existing-document review. File presence or successful structural +point to the exact selected document, formed from `state.repository` and +`state.ethos.path`. Supply a grounded summary for an existing-document review. +File presence or successful structural validation is not current intent confirmation. When the saved Ethos stage is `draft` or `write` and `state.ethos.draft` exists, supply `facts.draft` with the exact recorded @@ -308,6 +318,12 @@ After the user answers, persist the real reply and follow the existing handoff. Rendering `confirmed` alone cannot start downstream work, and rendering `resume` cannot answer a saved question. An already-started downstream journey resumes its pending work without repeating the completion pause. +For the combined creation checkpoint, wait for the actual source choice before +planning or trace inspection, then save its provenance and clear the pending +question. Continue directly to the bounded task proposal under the +[first-eval contract](first-eval.md#source-and-proposal-pauses); do not render +another readiness or source-choice pause. Saved-design, known-source, audit, and +expansion handoffs keep their applicable decisions. After a goal switch, retained started or completed journey progress also prevents an inferred new continuation pause when the saved question is empty. An explicitly saved continuation question from a fresh review is still presented; rendering diff --git a/skills/eval-author/references/ethos-confirmation.md b/skills/eval-author/references/ethos-confirmation.md index c31388db..263feefe 100644 --- a/skills/eval-author/references/ethos-confirmation.md +++ b/skills/eval-author/references/ethos-confirmation.md @@ -10,13 +10,13 @@ Do not implement another interview or local authoring fallback in Eval Author. ## Required handoff capabilities -Pass the absolute repository-root `ETHOS.md` path as the explicit destination -to `ethos-orchestrate`. The installed skills must preserve that exact path and -uppercase filename, including in repositories with several agents. They must -not rename it to `ethos.md` or choose the agent's package root instead. +Pass the absolute path selected in `ethos.path` as the explicit destination to +`ethos-orchestrate`. The installed skills must preserve that exact path and +uppercase `ETHOS.md` filename, including in repositories with several agents. +They must not rename it to `ethos.md` or choose a different directory. Check the installed skills' location and writing instructions before delegation. -If they rename the file, choose a package-level path, or otherwise override the +If they rename the file, choose another path, or otherwise override the supplied destination, that version is incompatible with this contract. Report the actual conflicting skill and instructions under **Blockers**, preserving existing documents and progress. Resolve the dependency @@ -42,15 +42,23 @@ artifact reconciliation, and final response selection. This adapter owns the Ethos work and confirmation rules below. Existing evaluations describe coverage; they do not establish intended behavior. -The shared prerequisite flow confirms repository-root `ETHOS.md`, which must be a -regular file, not a symlink. Preserve existing alternate-path documents; do not -silently copy, delete, or replace them. If the intended document lives elsewhere, -offer to move it to the repository root and, if its old path must remain usable, -leave a symlink there pointing to the root document. Confirm the move and any -symlink with the user before changing files. Explicitly scoped `--ethos` -utilities can use their supplied path without claiming the shared prerequisite -flow confirmed that document. Resolve ambiguous identity or conflicting intent -before accepting it. +The shared prerequisite flow confirms the selected `ETHOS.md`, which must be a +regular file, not a symlink, inside the repository. Store its canonical +repository-relative path in `ethos.path`; repository-root `ETHOS.md` remains the +default for older records and new documents without an explicit destination. +An existing agent-local document stays in place. Do not require a move, copy, or +root symlink. Resolve ambiguous identity or conflicting intent with the user +before selecting a document; never silently choose another agent's document. + +After entering the Ethos step, select a different destination with the normal +revision-checked `update` interface, setting `ethos.path` and `ethos.stage: review`. +During prerequisites, `--state-only` can mark review pending but cannot select a +different document or inspect its contents. A path change +reopens affected approvals even when its bytes match the previous document; +the prior selection and provenance remain under +`.eval-author/.working/history/ethos/.json`. If the selected file changes +or disappears, preserve that selection and reopen review without falling back to +another file. ## Review an existing document @@ -105,7 +113,7 @@ do not treat the former agent's document or draft approval as current permission Follow the [shared procedure](shared-prerequisites.md#present-results-and-the-next-step) for state loading before each call. Eval Author owns the selected journey, checklist, state layout, and return routing; Ethos needs none of those details. -Pass the selected `repository`, `agent`, root `ethos_path` as the explicit +Pass the selected `repository`, `agent`, `ethos_path` as the explicit destination, the requested operation, and only the Ethos context needed to continue: - `source_findings` from `ethos.source_findings`: records with `path` and `summary`. @@ -154,7 +162,7 @@ user replies with provenance, and draft contents from that conversation. Eval Author maps these facts into its own `stage`, `validation`, `remaining_questions`, and confirmation fields; Ethos does not implement that state schema. -Check that the document path resolves to this repository's root `ETHOS.md`. +Check that the document path matches `state.ethos.path` inside this repository. Eval Author computes `confirmed_digest` as SHA-256 over the exact saved bytes, encoded as 64 lowercase hexadecimal characters without a prefix. Before reusing the current flow's confirmation, compare the current bytes with that @@ -162,18 +170,25 @@ digest and retain its user-reply provenance. Changed contents reopen the affecte review or intent step. Persist unfinished stages and answers through `journey_state.py update`. For completion, require successful returned validation and explicit confirmation of the exact saved content, reusing approval of an -unchanged draft when applicable, then use `confirm-ethos` with the +unchanged draft when applicable, then use `confirm-ethos` with `--ethos` set to +the selected repository-relative path, the latest `--expected-revision`, the computed digest, the user's confirmation reference as `--source`, and the validator result's provenance as `--validation`. Preserve the validation details in a linked working artifact or the conversation; do not pass a structured validation object as if it were the helper's provenance -string. Failed validation remains an unfinished review, even if the file has -a digest. Use only the +string. An explicit `--ethos` must match the saved selection; omitting it retains +the legacy repository-root `ETHOS.md` default. Failed validation remains an +unfinished review, even if the file has a digest. A supported path does not waive +schema or heading validation. Use only the confirmed exact content for downstream cases or audit items. Validation alone is not intent confirmation. Present the returned path and unresolved gaps, then follow the shared [completion pause](shared-prerequisites.md#pause-after-ethos-confirmation). -End the turn after the rendered `confirmed` response with its continuation question. -Do not begin source selection or delegate evaluation work until the user continues. +End the turn after the rendered `confirmed` response with its pending decision. +For fresh creation without a saved design or source choice, the shared pause +includes the source question instead of a readiness question. Wait for its +answer before planning or inspecting traces; do not ask the same choice again +in a later turn. Other flows retain their continuation. Do not delegate +evaluation work before the applicable reply. If Blockers remain, use the blocked response instead. ## Recovery diff --git a/skills/eval-author/references/shared-prerequisites.md b/skills/eval-author/references/shared-prerequisites.md index 7e4c6897..0ed15f40 100644 --- a/skills/eval-author/references/shared-prerequisites.md +++ b/skills/eval-author/references/shared-prerequisites.md @@ -94,7 +94,7 @@ Select checks by the requested work. The common checks are: interview, or other Ethos preflight steps during prerequisite checks - Verify that the installed Ethos skills meet the [required handoff capabilities](ethos-confirmation.md#required-handoff-capabilities), - including preserving the explicit repository-root `ETHOS.md` destination. A + including preserving the selected explicit `ETHOS.md` destination. A version that renames it to lowercase or overrides that path is incompatible; record the conflicting resource as an `ethos_skills` blocker before delegation - Establish file and command access from host permissions and actual bounded @@ -208,7 +208,8 @@ the pending question, then reload normal state and render `results` in the final reply. Supply the completed prerequisite findings, optional Recommendations, and applicable Ethos, draft, question, and blocker facts. The final owns the canonical grounding sentence, completed table, Next Steps, one state-derived -checklist, and the inspected Ethos action. +checklist, and the inspected Ethos action. Preserve the renderer's headings and +Markdown in that final response under the [shared output contract](conversation-rendering.md). Each prerequisite turn shows an initial status table and one completed table in the final reply. An unblocked final includes one checklist; a blocked final @@ -267,7 +268,7 @@ Ethos work. ## Confirm Ethos After completing and saving the unblocked prerequisite pass, invoke -[Ethos confirmation](ethos-confirmation.md) for repository-root `ETHOS.md`. +[Ethos confirmation](ethos-confirmation.md) for the selected `ETHOS.md`. Reuse content and prior answers, but have the user confirm an existing document is the right one and still accurate, even when saved state records an earlier confirmation. Before target-artifact inspection, use `--state-only update` to mark @@ -283,28 +284,56 @@ document before its content is confirmed. After the saved `ETHOS.md` has been validated and its content reviewed and confirmed, persist completion using `confirm-ethos`. A draft approval or file -write alone does not complete this step. Save the selected journey's canonical -continuation question as `pending_question`, with the selected downstream skill -as `next_handoff`; obtain that question from -[onboarding_templates.py](../scripts/conversation_flows/onboarding_templates.py), not a parallel prose -template. Preserve confirmed Ethos and prerequisite findings. +write alone does not complete this step. Preserve confirmed Ethos and prerequisite +findings, then save the pending decision and downstream skill before rendering: + +- **Fresh, unstarted creation, no structured saved design, no saved source choice:** Save + the canonical source question as `pending_question` and + `eval-author-first-eval` as `next_handoff`. Follow the + [source checkpoint](conversations/first-eval.md#source-and-proposal-pauses) for + the template-owned question and choices. This replaces the separate readiness + question; do not ask whether to continue first. +- **Other unstarted cases:** Retain the selected journey's canonical continuation + from [onboarding_templates.py](../scripts/conversation_flows/onboarding_templates.py) + and the selected downstream skill. Preserve applicable source choices and + progress; a saved design still requires its own reuse decision. Already-started + work resumes its saved pending decision without another completion pause. Reload reconciled state and render `confirmed`, including the real saved Ethos link in `facts.ethos`. Use its Markdown as the final visible response, then **end the turn**. The renderer derives the complete shared steps and current journey step from state. A relevant blocker produces only the blocked action -instead of a continuation invitation. The invitation indicates readiness to -begin the selected flow, not proof that execution is configured or successful. - -Do not ask source-selection or case-planning questions, inspect new suite or -trace sources, probe runtimes, create evaluations, or delegate downstream work in -this turn. The pause must remain visible after tools finish; commentary alone is -not sufficient. Rendering the pause does not supply the user's continuation. - -After the user continues, clear the pending continuation question and hand off -with the saved context. Do not repeat this pause when resuming downstream work -that has already started. Keep later execution requirements explicit and pause -only the work they actually block. +instead of the pending decision. The pause does not establish design approval or +prove that execution is configured or successful. + +Only the fresh-creation exception above includes source selection in this turn. +Do not ask task-planning questions, inspect new suite or trace sources, probe +runtimes, create evaluations, or delegate downstream work while the reply is +pending. The pause must remain visible after tools finish; commentary alone is +not sufficient. Rendering it does not answer its decision. + +After the actual source answer to the combined checkpoint, save the choice and +its provenance, clear the pending question, and hand off directly to the small +task proposal. Resolve selected trace locations, runs, and exclusions before +inspection. Do not insert another readiness pause or repeat the source question. +For other cases, wait for and save the actual continuation reply before handing +off with the retained context. Do not repeat this pause when resuming downstream +work that has already started. Keep later execution requirements explicit and +pause only the work they actually block. + +For creation, this continuation and `ETHOS.md` confirmation do not choose a +previously saved task design. On a fresh request, hand off to the first-eval +[saved-design checkpoint](../../eval-author-first-eval/SKILL.md#confirm-a-saved-design-before-reusing-it) +before resuming that design. Existing metadata can identify this pending choice +during the state-only pass; save its creation record only through normal updates +after the shared prerequisites and `ETHOS.md` confirmation are complete. + +For creation whose source decision was not eligible for the combined checkpoint, +continue at the first-eval +[source checkpoint](conversations/first-eval.md#source-and-proposal-pauses) only +when that choice remains unresolved and no saved-design decision is pending. +Keep its overall checklist and canonical choices; do not replace either checkpoint +with a free-form source or task-preparation question. ## Save and resume @@ -329,6 +358,11 @@ into the existing scope. These observations persist even when prerequisites are blocked; case, corpus, and approval updates remain forbidden in this mode. Save agreed decisions in the selected journey's `scopes` entry. +Creation uses the additive `scopes.create.first_eval` record described in the +[first-evaluation conversation contract](conversations/first-eval.md). Normal +`show` returns its reconciled progress and artifact observations; creation +updates require the latest `--expected-revision`. These fields do not change +the top-level schema or grant approvals to older records. Put final task behavior in task READMEs and audit checks in the audit report. Do not create a separate first-eval or expansion plan file. Retire `first-eval.md`, `audit-progress.md`, and intent notes as active writers. @@ -382,10 +416,12 @@ until the pass can be saved. Preserve existing state rather than replacing it. ## Hand off -After the user continues from the completion pause, pass the selected journey, +After the user answers the completion pause, pass the selected journey, repository/agent, setup and prerequisite evidence and answers, -confirmed Ethos result, and unresolved dependencies. First-eval continues with -its source/scope decision; audit continues with selected evals and traces; +confirmed Ethos result, saved source choice when applicable, and unresolved +dependencies. First-eval continues with its next unresolved source/design/scope +decision; do not repeat a source question already answered in the combined +checkpoint. Audit continues with selected evals and traces; expansion follows the core's explicit expansion handoff. Never repeat shared onboarding or convert confirmation into a claim of successful runtime execution. @@ -550,6 +586,7 @@ During Ethos work, the `ethos` update accepts: | Field | Value | |---|---| +| `path` | Canonical repository-relative path ending in uppercase `ETHOS.md`; defaults to `ETHOS.md`. Select it through a normal update with the latest `--expected-revision` and `stage: review`; path selection is unavailable with `--state-only`. | | `stage` | `explore`, `interview`, `draft`, `write`, or `review`; completion uses `confirm-ethos` | | `source_findings` | List of records with nonempty `path` and `summary` | | `prior_answers` | List of records with nonempty `topic`, `question`, `answer`, and provenance `source` | @@ -569,19 +606,24 @@ content retains them. Changed content or an intent-revision stage invalidates those agreements while preserving their source material. The `write` stage requires confirmed draft content. `show` invalidates approval of a changed or missing draft while retaining prior answers. Changed canonical -Ethos similarly reopens its review without discarding the intent history. +Ethos similarly reopens its review without discarding the intent history. A +selected path change reopens affected approvals even when the bytes match; +follow the [selection and provenance rules](ethos-confirmation.md#enter-the-ethos-step). Only after the canonical workflow returns exact confirmed saved content and successful validation may the adapter call: ```bash python3 /scripts/journey_state.py --repo /absolute/repository confirm-ethos \ + --ethos --expected-revision \ --digest <64-character-sha256> --source \ --validation python3 /scripts/journey_state.py --repo /absolute/repository import-legacy ``` -The digest is for the exact saved bytes of root `ETHOS.md`. It cannot be set via -an arbitrary state update. `import-legacy` retains old notes as unreviewed -snapshots; review them before recording current decisions. Errors leave the +The digest is for the exact saved bytes at `ethos.path`. An explicit `--ethos` +must match that selection; the omitted flag retains the legacy `ETHOS.md` +default. The digest cannot be set via an arbitrary state update. +`import-legacy` retains old notes as unreviewed snapshots; review them before +recording current decisions. Errors leave the prior valid record intact and never imply that prerequisites or Ethos passed. diff --git a/skills/eval-author/references/suite-readme.md b/skills/eval-author/references/suite-readme.md index df20ccc1..03f498e0 100644 --- a/skills/eval-author/references/suite-readme.md +++ b/skills/eval-author/references/suite-readme.md @@ -102,15 +102,16 @@ explicitly selected. Refresh after cases are authored, after additions or edits, and after validation or actual-agent runs, including failed runs. Refresh again at the final handoff -and link the report. When mentioning the report, end the response with one -sentence containing each artifact link once: - -```markdown -Review your evaluations in [the report]() or [the suite README](). -``` - -Use the actual paths to `.eval-author/report.html` and `.eval-author/README.md`, -and keep both links in that closing sentence. +and link the report. For [first-task and full-suite review](conversations/first-eval.md#stages-and-review-boundaries), +use the renderer's leading review action and artifact links before its summaries, +without a repeated report footer. Completed creation results follow +[completed run review](conversations/first-eval.md#completed-run-review) instead. +For other report handoffs, including incomplete creation results, end the response with the +canonical `REPORTS` sentence in [conversation_templates.py](../scripts/conversation_templates.py), +containing each artifact link once. Supply the actual paths to +`.eval-author/report.html` and `.eval-author/README.md`, using the report and suite +README link labels from the selected conversation flow. Keep both links in +that closing sentence. Present evaluation findings and report links to the user; describe generation internals only when asked. Reuse the settled scope and authorization: a missing report is not @@ -354,12 +355,33 @@ section into these headings while preserving its substantive content. Write the real behavior, result, and grading details, not the example prose. Preserve the README's other required sections. Behavior under test and Expected result are the canonical sources for the report's Behavior and Expected outcome -columns. Missing, empty, duplicate, or oversized sections show `Not documented`; +columns. Missing, empty, duplicate, or oversized report values show `Not documented`; rendering never asks a model to invent them. For existing tasks, reuse explicit documentation or inspect the instruction, fixtures, verifier, and rubric once to supply source-grounded metadata. Routine refresh must not rewrite existing task documentation. +### Expected-result summaries + +Start `Expected result` with a short prose paragraph, usually one sentence, +summarizing observable success and its essential conditions or boundaries. +Keep the behavior's motivation in `Behavior under test`. Put detailed expected +values, individual assertions, and supporting explanation after a blank line +in the task README. + +The report's Expected outcome column and mirrored table use the complete opening +paragraph; the task README preview retains the full section. Wrapped lines in +the summary become spaces. Existing sections that begin with a list or table +retain their full text. Refresh neither paraphrases nor cuts off sentences to +meet a word limit. + +Use this concise style for conversation `expected_result` fields. Proposal and +suite review use the saved plan's summary; first-task review supplies the README's +opening summary as `facts.expected_result`. Keep the authored summary consistent +with the approved result, without rewriting an approved plan just to shorten +its prose. Preserve conditions that affect correctness; brevity does not change +the agreed behavior or verifier criteria. + ## Review and rerun guidance Use this guidance for setup, run commands, and evidence review with either diff --git a/skills/eval-author/scripts/conversation_flows/first_eval.py b/skills/eval-author/scripts/conversation_flows/first_eval.py index 5e77e635..f7b859c5 100644 --- a/skills/eval-author/scripts/conversation_flows/first_eval.py +++ b/skills/eval-author/scripts/conversation_flows/first_eval.py @@ -214,18 +214,17 @@ def proposal(record: dict) -> str: "| " + " | ".join( literal(task.get(key), key, table=True) - for key in ("title", "input", "behavior", "expected_result", "grading") + for key in ("id", "input", "behavior", "expected_result", "grading") ) + " |" ) - first = next(task for task in tasks if task["id"] == record["first_task"]) examples = len({example for task in tasks for example in task["example_ids"]}) return "\n\n".join( [ copy.PROPOSAL_INTRO, "\n".join(rows), literal(presentation.get("coverage_note"), "coverage_note"), - copy.FIRST_CHOICE.format(title=literal(first.get("title"), "first task title")), + copy.FIRST_CHOICE.format(task_id=literal(record["first_task"], "first task ID")), copy.PROPOSAL_PROCESS.format(tasks=len(tasks), examples=examples), copy.PROPOSAL_QUESTION, ] @@ -243,7 +242,7 @@ def saved_design(state: dict, record: dict, observed: dict) -> str: rows = [copy.SAVED_DESIGN_TABLE] rows.extend( "| " - + literal(task.get("title") or task["id"], "saved task title", table=True) + + literal(task["id"], "saved task ID", table=True) + " | " + literal(task.get("behavior") or copy.SAVED_BEHAVIOR_UNKNOWN, "saved task behavior", table=True) + " |" @@ -352,12 +351,11 @@ def build_rows( def build_checklist(record: dict, progress: dict, facts: dict) -> str: - first = next(task for task in record["plan"]["tasks"] if task["id"] == record["first_task"]) labels = dict(copy.BUILD_STEPS) if record["provider"] == "gym": labels["grading"] = copy.GYM_BUILD_GRADING return ( - copy.BUILD_HEADING.format(title=literal(first.get("title") or first["id"], "first task title")) + copy.BUILD_HEADING.format(task_id=literal(record["first_task"], "first task ID")) + "\n\n" + build_rows(labels, progress["first_task_build"]["steps"], facts) ) @@ -422,10 +420,9 @@ def run_checklist(record: dict, progress: dict, facts: dict) -> str: ), copy.RUN_IDENTITY.format(run=literal(activity["run_id"], "native run identity")), ] - titles = {task["id"]: task.get("title") or task["id"] for task in record["plan"]["tasks"]} parts.append( copy.RUN_CURRENT_TASKS.format( - tasks=", ".join(literal(titles[key], "current task") for key in activity["active_task_ids"]) + tasks=", ".join(literal(key, "current task ID") for key in activity["active_task_ids"]) ) if activity["active_task_ids"] else copy.RUN_TASK_UNKNOWN @@ -458,10 +455,9 @@ def first_review(record: dict, progress: dict, facts: dict) -> str: validation=copy.VALIDATION_PASSED if status["validated"] else copy.VALIDATION_PENDING, execution=copy.EXECUTION_COMPLETED if status["executed"] else copy.EXECUTION_PENDING, ) - first = next(task for task in record["plan"]["tasks"] if task["id"] == record["first_task"]) return "\n\n".join( [ - copy.FIRST_REVIEW_INTRO.format(title=literal(first.get("title") or first["id"], "first task title")), + copy.FIRST_REVIEW_INTRO.format(task_id=literal(record["first_task"], "first task ID")), copy.FIRST_REVIEW_ACTION.format( report=named_link(facts, "tasks_view", fragment=True, label=copy.FIRST_REVIEW_REPORT_LABEL), readme=named_link(facts, "readme", label=copy.FIRST_REVIEW_README_LABEL), @@ -487,7 +483,7 @@ def suite_review(record: dict, progress: dict, facts: dict) -> str: rows.append( "| " + " | ".join( - literal(task.get(key), key, table=True) for key in ("title", "behavior", "expected_result", "grading") + literal(task.get(key), key, table=True) for key in ("id", "behavior", "expected_result", "grading") ) + " |" ) diff --git a/skills/eval-author/scripts/conversation_flows/first_eval_templates.py b/skills/eval-author/scripts/conversation_flows/first_eval_templates.py index 4720d93b..c39fb87d 100644 --- a/skills/eval-author/scripts/conversation_flows/first_eval_templates.py +++ b/skills/eval-author/scripts/conversation_flows/first_eval_templates.py @@ -11,7 +11,7 @@ TRACE_CHOICES = ("Start from ETHOS.md", "Use existing traces") SAVED_DESIGN_INTRO = "I found a saved design for your evaluation tasks." SAVED_DESIGN_COUNTS = "Saved design: {tasks} tasks; {examples} distinct input examples." -SAVED_DESIGN_TABLE = "| Task | Behavior under test |\n|---|---|" +SAVED_DESIGN_TABLE = "| Task ID | Behavior under test |\n|---|---|" SAVED_DESIGN_SOURCE = "**Source:** {source}" SAVED_SOURCE_ETHOS = "`ETHOS.md`, with no selected traces." SAVED_SOURCE_TRACES = "`ETHOS.md` with previously selected traces." @@ -32,8 +32,8 @@ ) SAVED_DESIGN_QUESTION = "Would you like to continue this saved design, revise it, or start a new proposal?" PROPOSAL_INTRO = "Here are the proposed evaluation tasks." -PROPOSAL_TABLE = "| Task | Input | Behavior under test | Expected result | Verifier |\n|---|---|---|---|---|" -FIRST_CHOICE = "I’ll build **{title}** first." +PROPOSAL_TABLE = "| Task ID | Input | Behavior under test | Expected result | Verifier |\n|---|---|---|---|---|" +FIRST_CHOICE = "I’ll build **{task_id}** first." PROPOSAL_PROCESS = ( "Proposed tasks: {tasks}; distinct input examples: {examples}. " "After you approve the proposal, I’ll build only that first task and pause for your review " @@ -56,7 +56,7 @@ "I’m preparing the first task’s starting data and checking that its environment works. " "Then I’ll add the reference fixtures and native Gym verifier." ) -BUILD_HEADING = "**First task: {title}**" +BUILD_HEADING = "**First task: {task_id}**" BUILD_STEPS = { "environment": "Prepare starting data and environment", "environment_check": "Check the environment", @@ -93,7 +93,7 @@ } NO_REMAINING_TASKS = "No additional tasks agreed." FIRST_REVIEW_INTRO = ( - "Your first task, **{title}**, is ready for review. It follows your approved design and confirmed `ETHOS.md`." + "Your first task, **{task_id}**, is ready for review. It follows your approved design and confirmed `ETHOS.md`." ) FIRST_REVIEW_ACTION = "Review your first task in the {report} or {readme}." FIRST_REVIEW_REPORT_LABEL = "HTML report’s Tasks view" @@ -112,7 +112,7 @@ "It follows your approved design and confirmed `ETHOS.md`." ) SUITE_REVIEW_ACTION = "Review your tasks in the {report} or {readme}." -SUITE_REVIEW_TABLE = "| Task | Behavior under test | Expected result | Verifier |\n|---|---|---|---|" +SUITE_REVIEW_TABLE = "| Task ID | Behavior under test | Expected result | Verifier |\n|---|---|---|---|" SUITE_REVIEW_STATUS = ( "**Status:** Environment proof: {environments}/{tasks} passed. Verifier validation: {validated}/{tasks} passed. " "Agent execution: {executed}/{tasks} tasks complete." diff --git a/skills/eval-author/scripts/report.py b/skills/eval-author/scripts/report.py index e56cf1a6..a79f0bc6 100644 --- a/skills/eval-author/scripts/report.py +++ b/skills/eval-author/scripts/report.py @@ -323,12 +323,15 @@ def previous_snapshot(path: Path, output: Path) -> tuple[dict | None, bytes | No def readme_sections(text: str) -> dict[str, str | None]: """Read exact H2 sections; ignore headings in fenced code and reject duplicates.""" names = ("Behavior under test", "Expected result") - sections: dict[str, list[list[str]]] = {name: [] for name in names} + sections: dict[str, list[tuple[list[str], list[str]]]] = {name: [] for name in names} current: list[str] | None = None + source: list[str] | None = None fence = "" for line in text.splitlines(): match = re.match(r"^ {0,3}(`{3,}|~{3,})", line) if match: + if source is not None: + source.append(line) token = match[1] if not fence: fence = token @@ -336,24 +339,44 @@ def readme_sections(text: str) -> dict[str, str | None]: fence = "" continue if fence: + if source is not None: + source.append(line) continue heading = re.match(r"^ {0,3}(#{1,6})\s+(.+?)\s*#*\s*$", line) if heading and len(heading[1]) <= 2: current = None + source = None if len(heading[1]) == 2 and heading[2] in sections: current = [] - sections[heading[2]].append(current) + source = [] + sections[heading[2]].append((current, source)) elif current is not None: current.append(line) + assert source is not None + source.append(line) result = {} for name, values in sections.items(): if not values: continue - text = "\n".join(line.rstrip() for line in values[0]).strip() if len(values) == 1 else "" + text = "" + if len(values) == 1: + lines, source = values[0] + text = "\n".join(line.rstrip() for line in lines).strip() + if name == "Expected result": + text = opening_readme_paragraph("\n".join(source)) or text result[name] = text if text and len(text) <= 2000 else None return result +def opening_readme_paragraph(source: str) -> str | None: + from markdown_it import MarkdownIt + + tokens = MarkdownIt("commonmark", {"html": False}).enable(["table", "strikethrough"]).parse(source) + if tokens and tokens[0].type == "paragraph_open": + return tokens[1].content.replace("\n", " ") or None + return None + + def nested_checkout(repo: Path, path: Path) -> bool: for parent in [path, *path.parents]: if parent == repo: diff --git a/tests/fixtures/workflow-scope/contracts.json b/tests/fixtures/workflow-scope/contracts.json index 413e2045..2fb48503 100644 --- a/tests/fixtures/workflow-scope/contracts.json +++ b/tests/fixtures/workflow-scope/contracts.json @@ -331,7 +331,7 @@ "The saved ETHOS.md matches my intent.", "Yes, I confirm the saved Ethos." ], - "context": "The selected goal is first evaluations. Shared prerequisites are complete; the canonical Ethos skills have saved and validated root ETHOS.md and presented it for review. The user now confirms its saved content. Evaluation source selection has not started.", + "context": "The selected goal is fresh first evaluations without a saved task design or source choice. Shared prerequisites are complete; the canonical Ethos skills have saved and validated root ETHOS.md and presented it for review. The user now confirms its saved content. The confirmation response asks about task sources after its overall progress checklist, without an extra readiness question.", "flow": "eval-author", "allowed_actions": [ "read_skill", @@ -348,10 +348,10 @@ }, { "prompts": [ - "Continue.", - "Yes, let\u2019s build the evaluation suite." + "Start from ETHOS.md.", + "Use ETHOS.md without existing traces." ], - "context": "The previous turn confirmed saved Ethos, showed Next Steps with the first two checklist items complete, and asked Ready to continue? Shared state retains the continuation question and first-eval handoff. Source selection has not started.", + "context": "The previous turn confirmed saved ETHOS.md, showed Next Steps with the first two checklist items complete, and asked whether to use existing traces or start from ETHOS.md. Shared state retains that source question and first-eval handoff, with no saved task design. The user now selects ETHOS.md; save that answer and propose the small task design without asking readiness or sources again. This does not approve authoring or execution.", "flow": "eval-author-first-eval", "allowed_actions": [ "read_skill", @@ -359,13 +359,14 @@ "read_repository", "validate", "write_artifact", - "delegate" + "delegate", + "propose" ], "required_actions": [], "write_paths": [ ".eval-author/.working/" ], - "handoff": "source_scope_selection" + "handoff": "review_proposals" } ], "clarify-goal-before-setup": [ diff --git a/tests/scenarios/creation-guidance.md b/tests/scenarios/creation-guidance.md index 4e05760d..b2f4efe8 100644 --- a/tests/scenarios/creation-guidance.md +++ b/tests/scenarios/creation-guidance.md @@ -58,54 +58,180 @@ Repeat with the same selection supplied in the opening request: no duplicate source question is needed. A newly discovered source requires a changed choice before use, not an automatic fallback. -## 2. Review priorities, breadth, and cost before generation - -**Start:** Resume at scope planning with scenario 1's confirmed corpus. Have the -agent propose four complaint examples and two policy-answer examples, with -variations, difficulty, expected outcomes, grading, and exclusions. - -**Inspect before answering:** In the plan-review reply itself, the agent explains -that a pilot is a limited first set of eval cases, what it is intended to help -learn, what passing would and would not establish, and why the proposed breadth -fits that purpose. Linking to a saved explanation is insufficient. The reply -explicitly invites actionable feedback on priorities such as lower ongoing cost, -broader coverage, difficult cases, or another user concern; a generic “does this -look right?” alone is insufficient. Judge the meaning, not exact wording. - -**Reply at the scope checkpoint:** “Make rerun cost a priority. Start with one -example per behavior, one task per example and one attempt per task. Cost is a -budget consideration, not a passing score. Keep the rest as future expansion.” - -**Inspect:** The agreement changes to two distinct examples, two tasks, and two -agent trials per rerun, with controls and any judge work accounted for separately. -The agent explains what the pilot omits and asks only about unresolved choices. -The saved plan retains the user's priorities and feedback before generation. -Inspect the saved file immediately before the first fixture/scaffold write: -accepted counts and attempt settings must already be current, not left as -"pending" until handoff or recorded only in a conversation log. -No monetary estimate is invented when pricing or comparable measurements are -missing. Increasing attempts does not increase distinct-example counts; neither -renaming an example nor reusing it under another behavior increases the unique -total. A broad plan is not silently reduced to the pilot without this answer. -Repeat with the cost preference and two-example scope already agreed in the -conversation: the agent applies them to the preparation work. - -## 3. Pilot, then meaningful expansion - -**Start:** Continue scenario 2 through the stages that can actually be exercised. -Record the resulting tasks and available validation or execution evidence. A -failed agent response remains a useful baseline when its task works. - -**Reply:** “Retain both pilot examples. Add three distinct complaint inputs, +## 2. Review task design before generation + +**Saved-design variant:** Start a fresh first-evaluations request in a workspace +with a previously approved design, selected source, and one drafted task. Follow +the shared prerequisites and reconfirm `ETHOS.md`. Inspect the next downstream +checkpoint: the agent summarizes the saved scope and progress and asks whether +to continue, revise, or start a new proposal using `first_eval_saved_design`. +No source inspection, environment work, authoring, controls, or execution may +resume while that choice is pending. The earlier approval and continuation +reference remain historical; `ETHOS.md` approval does not answer this question. +Interrupt and resume here: the same choice remains pending. + +Answer “Continue this saved design”: applicable approvals and evidence are +reused without a duplicate proposal approval or new spending permission. Repeat +with “Revise it”: retain the selected source and review the changed proposal +before authoring. Repeat with “Start a new proposal”: ask for source selection +before a small new proposal; keep old task files and evidence. Change the design +after the question but before a continue answer: review the updated design +instead of applying an answer to a different revision. An opening request that +explicitly chooses the saved design to resume skips this question; an ordinary +turn within confirmed ongoing work does not trigger it again. + +**Default-size variant:** Complete `ETHOS.md` confirmation without selecting +traces. Both source selection and the proposal begin with the overall three-item +checklist: prerequisites and `ETHOS.md` checked, creation current. Repeat with +missing run credentials; they must not remove the current marker or introduce +execution settings at these checkpoints. The first source question ends the turn; +inspect tool logs to ensure no adapter investigation, task planning, or environment +preparation continues while waiting. Answer “Start from ETHOS.md.” Inspect the next +final reply: it proposes one to three tasks with one example each, states that one first task will be +built and reviewed before the rest, and ends with the rendered design feedback +question. No task files or integration implementation exist. Interrupt and +resume at both unanswered questions; neither readiness nor the source answer +counts as task-design approval. + +**Broader-request variant:** Explicitly ask for four tasks with two examples +each before the proposal. The saved proposal retains that user request as +breadth provenance and still waits for design approval; it does not silently +shrink or treat the breadth request as approval. + +**Start:** Resume with scenario 1's confirmed source selection and `ETHOS.md`. +Explicitly request four complaint examples and two policy-answer examples, with concrete +inputs, expected outcomes, grading, exclusions, and a chosen first task. + +**Inspect:** Use the [first-evaluation conversation contract](../../skills/eval-author/references/conversations/first-eval.md) +and shared renderer's `first_eval_proposal` stage. Compare fixed structure and +question copy against the template source rather than paraphrased expectations. +The proposal ends with actionable design feedback/approval, with no execution +settings, run-only blockers, or spending footer. No task files exist yet. + +**Reply:** “Make rerun cost a priority. Start with one example per behavior and +one task per example. Keep the rest as future expansion. I approve this design +but have not authorized paid grading or actual-agent runs.” + +**Inspect:** Save actual approval for the current two-task design before any +scaffolding or fixture write. The design explains remaining gaps and retains +source exclusions. With ready deterministic controls or an already configured +and authorized judge, resume goes directly to first-task preparation without +execution bullets or a redundant setup pause. If the proposed grading requires +missing judge configuration, access, or authorization, use +`first_eval_validation_setup` before creating task files. The user's design +approval does not authorize those paid checks. Unknown target-agent run +credentials remain saved and block only dependent execution. Repeats never +increase the number of distinct examples; no monetary estimate is invented. + +## 3. Review the first task, then build the remainder + +**Start:** Continue the approved design once required validation setup is ready. +Create exactly one permanent `.eval-author/tasks//` directory and record Draft. +Prove its environment before writing its reference solution or grader. Retain +construction declarations, source provenance, and native smoke evidence. + +**Build-progress variant:** Inspect commentary during a slow environment build +and smoke check. The initial provider explanation includes the renderer's compact +build checklist; subsequent snapshots keep the same four rows and short statuses +with the responsible skill in parentheses beside each item, without paragraphs +of implementation deliberation. Completed preparation remains +visible while checks run. A failed smoke check leaves environment checking +unfinished, with **Blockers** only when dependent work is blocked. Missing +run-only credentials do not mark the build blocked. Interrupt and resume: +completed checkpoints survive, missing or changed evidence reopens affected +rows, and adding the grader does not erase completed environment preparation. +No build receipt establishes formal grading validation or agent execution. + +**Inspect:** Satisfy the [pre-review validation gate](../../skills/eval-author/references/conversations/first-eval.md#grader-validation-before-authored-task-review), +then generate HTML and Markdown review artifacts before +`first_eval_review_first`. Lead with which task is ready and its approved design +and `ETHOS.md` grounding. The next paragraph directs the user to the linked HTML +Tasks view or mirrored README before the item summaries. Each artifact is linked +once, without a duplicate report footer. Link the task README, with input, +expected result, grading, a meaningful limitation, and distinct environment, +validation, and actual-agent execution findings. All required grading controls, +including the configured model judge when used, have passed for this revision; +target-agent execution may remain pending. The second task is still planned and +uncreated. Missing required report capabilities stay explicit; native Gym run +evidence remains in the suite README. + +**Pending first-task validation variant:** Withhold required judge setup before +creation: expect `first_eval_validation_setup` and no task files. Separately, +make a previously ready judge unavailable or fail a required hosted control +after authoring. Expect build progress and **Blockers** describing the affected +validation and the specific action needed. Draft reports may be refreshed, but +`first_eval_review_first` must reject this state; no review-ready claim, +approval question, or remaining-task authoring is allowed. Interrupt and resume: +the validation stays unfinished. Do not substitute a judge or invent successful +controls to unblock this offline exercise. Only after the required controls pass +with current retained evidence, refresh the artifacts and request first-task review. + +**Reply:** “Change the first task's input to include an ambiguous complaint. I +haven't approved this revision yet.” + +**Inspect:** Edit the same directory, retain prior evidence, invalidate affected +observations, rerun affected controls, refresh the report, and request first-task +review again only after validation passes. An interruption resumes that pending +review without repeated onboarding or a +settings checklist. No remaining task is generated before current approval. + +**Reply:** “I approve the revised first task.” + +**Inspect:** Save approval of that revision, then build and validate the remainder +without individual human review gates. Reference controls remain separate from +actual-agent performance. Once all required controls pass and both report +artifacts are current, show `first_eval_review_suite` and wait for approval of +all tasks. Do not present execution settings or questions yet. +Applicable prior settings and authorization are reused only after this review. +Task approval alone does not authorize spending. + +**Remaining-build variant:** The initial explanation includes a new renderer +checklist; later snapshots show counts and current status for creation, +environments, all-task grader validation, and artifact refresh. Each row includes +the responsible skill. A control failure or stale evidence stays unfinished; +missing run-only credentials do not block authoring. Interrupt and resume without +losing completed work or repeating approval. The validation count includes the +already validated first task; only affected evidence needs renewing. Refreshing +a report after execution starts does not send the conversation back to task authoring. +Repeat with one agreed +task: no additional tasks are created, and current first-task validation is reused. + +**Pending suite-validation variant:** All tasks are authored and their environments +work, but hosted grading/calibration remains pending after a previously ready +dependency changes. Refreshed reports remain drafts. Expect **Blockers** with the +missing dependency or failed controls and the specific action needed; resolve +only affected validation setup before continuing. `first_eval_review_suite` +must reject this state. There is no suite-approval question or target-agent +execution-settings discussion. Interrupt and resume: preserve completed work, +keep validation unfinished, and do not ask for suite approval until all required +controls pass with current evidence. + +**Suite-review variant:** All tasks are authored, their environments work, and +every task has passed its required grading controls, including hosted grading or +calibration when used. Refresh the report and mirror. Inspect the final response: +it leads with the suite scope and `ETHOS.md` grounding, directs the user to review +the linked HTML Tasks view or +mirrored README, summarizes the tasks and honest check status, and asks whether +the tasks look good. There are no model, budget, credential, attempt-setting, +or run-authorization questions. No work proceeds while that approval is pending, +including after a nonblocking question tool returns. + +Ask a side question, then interrupt and resume: the suite decision stays pending. +Request a task revision: preserve its directory, refresh artifacts, and review +the changed suite only after affected controls pass, before execution settings. +Reply “These tasks look good; I approve the suite.” Save the displayed revision +and actual approval separately, then present the target-agent run settings and +resolve any missing run authorization. +Changing only validation results later does not ask for task approval again. + +**Expansion reply:** “Retain both tasks. Add three distinct complaint inputs, including confusable negatives, and one new policy question. Review the added -coverage and ongoing cost with me before generating them.” +coverage with me before generating them.” -**Inspect:** The new plan maps two retained and four added examples to six unique -inputs: four complaint and two policy-answer examples. It explains the new -variation, difficulty, expected outcomes, and cost rather than copying the pilot -size or counting repeats as expansion. Source exclusions and existing failing -cases remain intact. Generation waits for the affected scope choices, then the -handoff reconciles each agreed example against what was actually delivered. +**Inspect:** Two retained and four added inputs total six examples. Preserve +accepted tasks, source choices, and user priorities; reopen only affected design +and task claims. Save the agreement before generating additions and reconcile +agreed work against created, validated, and executed evidence. ## 4. Confusable negatives need semantic evidence @@ -337,3 +463,34 @@ requested NeMo Gym rollouts. This scenario tests recovery and reporting, not an allegation that the author fabricated application outputs. Static checks of this file do not establish that a live author passed the scenario. + +## Deterministic continuation, results, and audit handoff + +Exercise the first-evaluation flow through the shared renderer, using normally +reconciled state and its actual observations. Retain rendered input/output, +user decisions, and evidence references. A side question never grants approval. +Test interruption after design approval, first-task creation, requested changes, +first-task approval, validation, and a partial run; resume the first unfinished +step without repeating source choices or completed runs. + +Exercise passing, agent-quality-failing, infrastructure-failing, partial, +controls-only, and missing-evidence outcomes. Keep prior evidence under fresh +output names. Task edits reopen affected review and validation; configuration +edits reopen affected execution claims. Explicitly selected recovery jobs may +complete only missing authorized work; directory membership never selects tasks. +Native Gym keeps complete agreed rows and repeats; a configured Gym–Harbor +bridge is exercised as that path rather than direct Harbor conversion. + +For results, inspect one informative actual-agent attempt and name its exact run +and attempt, observed response/actions, grade, verified evidence navigation, +available model identity, and limitations. Create a later run and verify the +reviewed identity remains stable. The suite README and reply link the real +full-suite rerun section by its verified heading anchor. Task README bytes stay +unchanged after their evidence is bound; later results live in the suite README. + +Only completed agreed creation receives the template's audit invitation, followed +by the shared single closing report/README sentence. Accepting reuses confirmed +`ETHOS.md`, provider, sources, exact runs, and preferences; audit-specific missing +evidence produces Blockers for the audit alone. Declining starts nothing and is +not repeatedly offered. Unfinished creation never receives a completed-stage +audit handoff, and accepting an audit never authorizes new tasks or paid runs. diff --git a/tests/scenarios/entry-routing.md b/tests/scenarios/entry-routing.md index 75c97f30..d06f4444 100644 --- a/tests/scenarios/entry-routing.md +++ b/tests/scenarios/entry-routing.md @@ -206,7 +206,11 @@ direct flow invocations. Subsequent Ethos-only replies can use `ethos` without repeating prerequisite findings. - Optional assistant skills are absent: put optional Recommendations after the - completed table in final, followed by the Ethos checklist and question + completed table in final, followed by Next Steps with the Ethos checklist and + question. Inspect the actual final reply against renderer stdout: both section + headings and the optional-suggestions introduction must survive even when + there are no active Blockers. A reply that drops the headings fails this check + even when the renderer output was correct - The selected authoring runtime is missing, incompatible, or unresolved: show the completed table and **Blockers** once in final, preserve progress, and omit Next Steps, the checklist, and Ethos guidance. Existing or historically @@ -358,7 +362,11 @@ document for the selected agent and still accurate. It must not reuse historical approval, mark Ethos complete, or show the evaluation readiness question yet. - Confirm it is correct and current: validate and record the exact saved content - with this reply's provenance, then show the completion pause and wait + with this reply's provenance, then show the completion pause and wait. For + fresh creation without a saved design or source choice, this response includes + the source question after its checklist; no separate readiness pause precedes + it. Save the source answer and proceed to the small task proposal without + repeating that question; design approval still precedes authoring - Say it belongs to another agent or needs changes: keep review pending and let `ethos-orchestrate` resolve the identity or revise intent before confirmation - Ask a side question or say you are unsure: keep Ethos incomplete; preserve the diff --git a/tests/scenarios/suite-reruns.md b/tests/scenarios/suite-reruns.md index c615f5a8..10d1b467 100644 --- a/tests/scenarios/suite-reruns.md +++ b/tests/scenarios/suite-reruns.md @@ -142,3 +142,31 @@ renaming it. Retain the generated guides, configs, resolved selections, and any observed failure as evidence. One successful exercise is a bounded check, not a claim that all future authoring runs will produce correct documentation. + +## First-task review and saved run identity + +When this suite is authored as first evaluations, use the shared +[first-evaluation conversation contract](../../skills/eval-author/references/conversations/first-eval.md). +Before task generation, save actual design approval and resolve required +grader-validation setup, including any judge access and bounded authorization. +Build exactly one permanent task, pass its required grading controls, and prepare +its report and Markdown review before authoring the remainder. Pending or failed +controls require setup or **Blockers**, not an authored-task approval question. +Request a revision, interrupt, and verify resume retains the task path and +outstanding approval; affected controls must pass again before review. +Target-agent execution settings appear after whole-suite approval. Required +grading validation passes before either authored-task review. + +For each provider, retain original controls and use explicit selected tasks or +complete Gym rows with agreed repeats. Add an unrelated task in the parent +directory and confirm it is excluded. Keep controls separate from target-agent +runs and give later retries fresh output names. A low score with healthy native +execution is completed baseline evidence; controls-only or absent output is not. + +Save an informative exact run/attempt review, then add another run. Both the +conversation and suite README must still navigate to the original reviewed +attempt and identify actual agent/model information or its absence. Link the +verified full-suite rerun heading and any separate single-task instructions. +Finalize task READMEs before binding evidence and keep later result summaries +outside those directories. Use main's Draft/Accepted lifecycle without treating +scope approval or report refresh as task acceptance. diff --git a/tests/test_first_eval_rendering.py b/tests/test_first_eval_rendering.py index 9f07320b..2425c562 100644 --- a/tests/test_first_eval_rendering.py +++ b/tests/test_first_eval_rendering.py @@ -48,7 +48,7 @@ def creation(monkeypatch, tmp_path): { "id": task_id, "example_ids": [task_id + "-example"], - "title": task_id, + "title": "Allow eligible refund" if task_id == "refund" else "Reject ineligible request", "input": "Request | with newline\ncontinued", "behavior": "Respect policy", "expected_result": "Correct disposition", @@ -336,7 +336,7 @@ def test_proposal_is_concise_deterministic_and_read_only(creation): output = renderer.render(payload(creation, "proposal")) assert output == PLANNING_PROGRESS + "\n\n" + ( "Here are the proposed evaluation tasks.\n\n" - "| Task | Input | Behavior under test | Expected result | Verifier |\n|---|---|---|---|---|\n" + "| Task ID | Input | Behavior under test | Expected result | Verifier |\n|---|---|---|---|---|\n" "| refund | Request | with newline
continued | Respect policy | Correct disposition | Code checks verify state; model judge assesses response meaning |\n" "| reject | Request | with newline
continued | Respect policy | Correct disposition | Code checks verify rejected action and unchanged state |\n\n" "These check refunds and rejection. Other regions remain untested.\n\n" @@ -372,6 +372,24 @@ def test_default_proposal_stays_small_and_explains_the_next_review(creation, tas assert "Time/cost limits" not in output +@pytest.mark.parametrize("stage", ["proposal", "saved_design", "prepare_first", "build_progress"]) +def test_task_ids_are_literal_in_tables_and_build_labels(creation, stage): + _, record, _, renderer, _ = creation + task_id = "refund_*[choice]|\nnext" + record["plan"]["tasks"][0]["id"] = record["first_task"] = task_id + if stage in {"prepare_first", "build_progress"}: + build_started(creation) + output = renderer.render(payload(creation, stage)) + escaped = r"refund\_\*\[choice\]|<b>" + if stage in {"proposal", "saved_design"}: + assert f"| {escaped}
next |" in output + if stage == "proposal": + assert f"I’ll build **{escaped} next** first." in output + elif stage in {"prepare_first", "build_progress"}: + assert f"**First task: {escaped} next**" in output + assert "Allow eligible refund" not in output and "" not in output + + @pytest.mark.parametrize("broader_scope", ["more_tasks", "more_examples"]) def test_broader_proposal_requires_an_explicit_user_request(creation, broader_scope): state, record, helper, renderer, copy = creation @@ -937,7 +955,7 @@ def test_suite_review_is_a_linked_deterministic_checkpoint_before_execution_choi "It follows your approved design and confirmed `ETHOS.md`.\n\n" f"Review your tasks in the [HTML report’s Tasks view](<{root}/.eval-author/report.html#cases>) " f"or [mirrored README](<{root}/.eval-author/README.md>).\n\n" - "| Task | Behavior under test | Expected result | Verifier |\n|---|---|---|---|\n" + "| Task ID | Behavior under test | Expected result | Verifier |\n|---|---|---|---|\n" "| refund | Respect policy | Correct disposition | Code checks verify state; model judge assesses response meaning |\n" "| reject | Respect policy | Correct disposition | Code checks verify rejected action and unchanged state |\n\n" "**Status:** Environment proof: 2/2 passed. Verifier validation: 2/2 passed. " @@ -1100,7 +1118,7 @@ def test_saved_design_overview_is_a_deterministic_decision_before_resume(creatio assert output == PLANNING_PROGRESS + "\n\n" + ( "I found a saved design for your evaluation tasks.\n\n" "Saved design: 2 tasks; 2 distinct input examples.\n\n" - "| Task | Behavior under test |\n|---|---|\n" + "| Task ID | Behavior under test |\n|---|---|\n" "| refund | Respect policy |\n| reject | Respect policy |\n\n" "**Source:** `ETHOS.md`, with no selected traces.\n\n" "**Saved design approval:** Previously recorded for this design.\n\n" @@ -1183,7 +1201,7 @@ def test_saved_design_accepts_a_legacy_broad_plan_and_unknown_approval(creation) assert "`ETHOS.md` with previously selected traces." in output assert "selected.json" not in output and "known-run" not in output and "private" not in output assert record["plan"]["trace_choice"]["runs"] == ["known-run"] - assert "| Task 6 | Keep the saved behavior |" in output + assert "| task-6 | Keep the saved behavior |" in output def test_saved_design_missing_source_and_fields_remain_unknown(creation): @@ -1314,10 +1332,14 @@ def test_review_leads_with_grounding_and_action_before_item_summaries(creation, renderer.render(payload(creation, "review_first", **facts)) -def test_review_intro_uses_the_approved_first_task_title_as_literal_text(creation): +def test_review_intro_uses_the_approved_first_task_id_as_literal_text(creation): _, record, _, renderer, _ = creation record["plan"]["tasks"][0]["title"] = "Preserve *explicit* choices" built(creation, validated=True) + task_id = "preserve_*explicit*_[choices]" + record["plan"]["tasks"][0]["id"] = record["first_task"] = task_id + record["tasks"][task_id] = record["tasks"].pop("refund") + approved(creation) output = renderer.render( payload( creation, @@ -1329,7 +1351,8 @@ def test_review_intro_uses_the_approved_first_task_title_as_literal_text(creatio grading_limitation="Live tool effects excluded", ) ) - assert output.startswith("Your first task, **Preserve \\*explicit\\* choices**, is ready for review.") + assert output.startswith("Your first task, **preserve\\_\\*explicit\\*\\_\\[choices\\]**, is ready for review.") + assert "Preserve" not in output.splitlines()[0] def test_late_settings_and_explicit_execution_authorization(creation): @@ -2287,7 +2310,7 @@ def test_proposal_explains_each_grading_method_before_validation_setup_questions output = renderer.render(payload(creation, "proposal")) assert copy.PROPOSAL_TABLE in output for index, method in enumerate(methods): - row = next(line for line in output.splitlines() if line.startswith(f"| Task {index} |")) + row = next(line for line in output.splitlines() if line.startswith(f"| task-{index} |")) assert row.endswith(f" | {method} |") assert output.endswith(copy.PROPOSAL_QUESTION + "\n") assert copy.VALIDATION_SETUP_INTRO not in output and state == before diff --git a/tests/test_first_eval_selection.py b/tests/test_first_eval_selection.py new file mode 100644 index 00000000..9b7042d2 --- /dev/null +++ b/tests/test_first_eval_selection.py @@ -0,0 +1,43 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Resolve a saved starter configuration without starting provider execution.""" + +import asyncio +from pathlib import Path + +import yaml +from harbor.job import Job +from harbor.models.job.config import JobConfig + + +def test_run_yaml_selects_only_agreed_tasks_and_preserves_attempts(tmp_path): + selected = ["coding-request", "missing-repository"] + for name in [*selected, "unrelated-draft"]: + (tmp_path / ".eval-author/tasks" / name).mkdir(parents=True) + run_config = tmp_path / ".eval-author/run.yaml" + run_config.write_text( + yaml.safe_dump( + { + "tasks": [{"path": str(tmp_path / ".eval-author/tasks" / name)} for name in selected], + "datasets": [], + "agents": [{"name": "terminus-2", "model_name": "configured-target", "override_timeout_sec": 180}], + "n_attempts": 2, + "jobs_dir": str(tmp_path / ".eval-author/jobs"), + } + ) + ) + config = JobConfig.model_validate(yaml.safe_load(run_config.read_text())) + resolved = asyncio.run(Job._resolve_task_configs(config)) + assert [task.get_local_path().name for task in resolved] == selected + assert config.n_attempts == 2 + assert config.agents[0].name == "terminus-2" + assert config.agents[0].model_name == "configured-target" + assert config.agents[0].override_timeout_sec == 180 + assert not (tmp_path / ".eval-author/jobs").exists() + + reference = ( + Path(__file__).resolve().parents[1] / "skills/eval-author-first-eval/references/harbor-first-eval.md" + ).read_text() + assert ".eval-author/run.yaml" in reference + assert "first-eval.yaml" not in reference diff --git a/tests/test_html_report.py b/tests/test_html_report.py index e7c29419..dd0a348e 100644 --- a/tests/test_html_report.py +++ b/tests/test_html_report.py @@ -341,6 +341,59 @@ def test_readme_section_boundaries(body, expected): assert report.readme_sections(body).get("Behavior under test") == expected +@pytest.mark.parametrize( + "body,expected", + [ + ( + "Save the source once;\nleave other records untouched.\n\nDetailed checks.", + "Save the source once; leave other records untouched.", + ), + ( + "Return `a b` unchanged;\npreserve the two spaces.\n\nDetailed checks.", + "Return `a b` unchanged; preserve the two spaces.", + ), + ("Saved.\n```text\nSample output.\n```\nDetailed checks.", "Saved."), + ("Saved.\n\n| Check | Result |\n| --- | --- |\n| State | Exact match |", "Saved."), + ("Saved.\n\n" + "Detailed checks. " * 200, "Saved."), + ("x" * 2001 + "\n\nShorter detail.", None), + ("```text\nExample only.\n```", None), + ], +) +def test_expected_result_uses_complete_opening_paragraph(body, expected): + assert report.readme_sections("## Expected result\n" + body)["Expected result"] == expected + + +@pytest.mark.parametrize( + "body", + [ + "- Save the source once.\n\n Keep every source line.\n\n- Leave other records untouched.\n\nConfirm the destination.", + "| Check | Result |\n| --- | --- |\n| State | Exact match |\n\nConfirm the destination.", + ], +) +def test_expected_result_preserves_list_and_table_first_sections(body): + assert report.readme_sections("## Expected result\n" + body)["Expected result"] == body + + +@pytest.mark.parametrize( + "source", + [ + "## Expected result\n\n## Grading\nChecks stored state.", + "## Expected result\nSaved.\n## Expected result\nDifferent result.", + "## Expected result\n\n## Expected result\nSaved.", + "## Expected result\n```text\nExample only.\n```", + "## Expected result\n" + "x" * 2001, + ], +) +def test_invalid_canonical_expected_result_does_not_fall_back_to_import(tmp_path, source): + path = task(tmp_path, readme=source) + metadata = {"tasks": {path: {"expected_outcome": "Imported result.", "source_refs": [path + "/README.md"]}}} + data = report.refresh(tmp_path, patch=metadata) + assert data["tasks"][0]["expected_outcome"] is None + markdown = (tmp_path / ".eval-author/README.md").read_text() + assert "Not documented" in markdown + assert "Imported result." not in markdown + + def test_import_fields_and_canonical_readme(tmp_path): path = task(tmp_path, readme="# Existing documentation\nOther sections.") metadata = { @@ -1496,6 +1549,50 @@ def test_readme_sections_preserve_paragraphs_and_lists_in_both_reports(tmp_path) assert "First paragraph.

- Check one.
- Check two." in raw +def test_expected_result_summary_is_shared_by_reports_and_preserves_readme_details(tmp_path): + behavior = "Save the selected source.\n\n- Exercise project selection.\n- Inspect final state." + summary = ( + "Save the transcript once, verbatim, in the selected project;\n" + "leave other data untouched; confirm its location without summarizing it." + ) + details = ( + "Compare the saved bytes against the original transcript.\n\n" + "- Require exactly one write.\n- Reject changes to unrelated records.\n\n" + "```text\n## Expected result\nExample output must not become the summary.\n```\n\n" + "| Check | Evidence |\n| --- | --- |\n| Content | Original bytes |\n\n" + "### Additional assertions\n\n" + "Keep the complete verification procedure here. " * 60 + ) + source = f"# Save source\n\n## Behavior under test\n{behavior}\n\n## Expected result\n{summary}\n\n{details}\n" + path = task(tmp_path) + readme = tmp_path / path / "README.md" + original = source.replace("\n", "\r\n").encode() + readme.write_bytes(original) + data = report.refresh(tmp_path) + embedded, _ = report.previous_report(tmp_path / ".eval-author/report.html") + expected = " ".join(summary.split()) + assert embedded["tasks"][0]["expected_outcome"] == data["tasks"][0]["expected_outcome"] == expected + assert data["tasks"][0]["behavior"] == behavior + preview = data["tasks"][0]["readme_html"] + for text in ( + "Compare the saved bytes", + "Require exactly one write.", + "Example output must not become the summary.", + "Original bytes", + "Additional assertions", + "Keep the complete verification procedure here.", + ): + assert text in preview + assert text not in data["tasks"][0]["expected_outcome"] + markdown = (tmp_path / ".eval-author/README.md").read_text() + assert report.markdown_text(expected) in markdown + assert "Compare the saved bytes" not in markdown + assert "Example output" not in markdown + assert "Original bytes" not in markdown + assert "Additional assertions" not in markdown + assert "complete verification procedure" not in markdown + assert readme.read_bytes() == original + + def test_markdown_text_cannot_inject_tables_headings_links_or_html(tmp_path): dangerous = ( "| row\n## forged\n [bad](javascript:alert(1)) ![remote](https://example.test/pixel)" diff --git a/tests/test_skill_contract.py b/tests/test_skill_contract.py index f2a340a1..700a477f 100644 --- a/tests/test_skill_contract.py +++ b/tests/test_skill_contract.py @@ -955,16 +955,45 @@ def test_environment_kit_records_digests_for_preservation_checks() -> None: assert "per-table content digests" in body +def test_first_eval_fresh_request_confirms_saved_design_before_reuse() -> None: + _, body = _frontmatter_and_body(_FIRST_EVAL_DIR) + checkpoint = body.split("### Confirm a saved design before reusing it", 1)[1].split("\n## ", 1)[0] + checkpoint = " ".join(checkpoint.split()) + assert "fresh request" in checkpoint and "#saved-design-checkpoint" in checkpoint + assert "Confirmation of `ETHOS.md` does not select a saved design" in checkpoint + contract = (_CORE_DIR / "references" / "conversations" / "first-eval.md").read_text(encoding="utf-8") + saved = " ".join(contract.split("## Saved-design checkpoint", 1)[1].split("\n## ", 1)[0].split()) + assert '"decision": "pending"' in saved + assert "Render `first_eval_saved_design` as the final response and end the turn" in saved + assert "Neither `ETHOS.md` confirmation nor the shared continuation answers this choice" in saved + assert "current user request explicitly identifies the saved design to resume" in saved + assert "block dependent stages until the saved-design choice is resolved" in saved + + +def test_first_eval_waits_for_suite_review_before_execution_questions() -> None: + _, body = _frontmatter_and_body(_FIRST_EVAL_DIR) + remainder = " ".join(body.split("## 4. Build and validate the remainder", 1)[1].split("\n## ", 1)[0].split()) + assert "Render `first_eval_review_suite` as the final reply and end the turn" in remainder + assert "Wait for approval of the whole suite before discussing target-agent execution models" in remainder + assert "Do not offer review while required model-based grading or calibration is pending" in remainder + first_task = " ".join( + body.split("## 3. Build and review exactly the first task", 1)[1].split("\n## ", 1)[0].split() + ) + assert "#validation-setup-before-task-creation" in first_task + assert "through `first_eval_validation_setup` before creating task files" in first_task + execution = " ".join(body.split("## 5. Agree on and execute concrete runs", 1)[1].split("\n## ", 1)[0].split()) + assert "Use `first_eval_execution_settings` only after approval of the authored suite" in execution + + def test_first_eval_builds_and_proves_the_kit_before_tests() -> None: - """Bench sessions did the stage 5 work but skipped stage 6, so the smoke proof moved into stage 5.""" + """First-task grading cannot precede environment smoke proof.""" _, body = _frontmatter_and_body(_FIRST_EVAL_DIR) - stage5 = body.split("## 5. Prepare cases and grading", 1)[1].split("\n## ", 1)[0] - stage6 = body.split("## 6. Prepare the execution environment", 1)[1].split("\n## ", 1)[0] - assert "Build and prove the kit before the tests" in stage5 - assert "smoke task" in stage5 - assert "../eval-author-environment/SKILL.md" in stage5 - assert "environment-plan.md" in stage6 - assert "never count as environment proof" in stage6 + first_task = body.split("## 3. Build and review exactly the first task", 1)[1].split("\n## ", 1)[0] + first_task = " ".join(first_task.split()) + assert "../eval-author-environment/SKILL.md" in first_task + assert "before writing the task's grader or reference solution" in first_task + assert "smoke evidence" in first_task + assert "task controls never prove the kit" in first_task harbor = (_FIRST_EVAL_DIR / "references" / "harbor-first-eval.md").read_text(encoding="utf-8") assert "in the next milestone" not in harbor, "Harbor first-eval must not defer environment/ until after the tests" diff --git a/tests/test_workflow_scope.py b/tests/test_workflow_scope.py index 9d6d34ef..af31ed04 100644 --- a/tests/test_workflow_scope.py +++ b/tests/test_workflow_scope.py @@ -169,12 +169,21 @@ def test_ethos_completion_can_save_state_but_cannot_create_evaluations(): assert "turn[0]: forbidden write attempt .eval-author/tasks/new-case/task.toml" in report["violations"] -def test_ethos_completion_continuation_allows_source_scope_handoff(): +def test_ethos_completion_source_answer_allows_design_proposal(): evidence, raw = capture("ethos-completion-pause") raw = add_call(evidence, raw, action="delegate", turn=1) + raw = add_call(evidence, raw, action="propose", turn=1) assert scope.check(CONTRACTS, evidence, raw)["status"] == "pass" +@pytest.mark.parametrize("action", ["scaffold", "execute"]) +def test_ethos_source_answer_does_not_authorize_building_or_execution(action): + evidence, raw = capture("ethos-completion-pause") + raw = add_call(evidence, raw, action=action, turn=1) + report = scope.check(CONTRACTS, evidence, raw) + assert f"turn[1]: forbidden action {action}" in report["violations"] + + def test_transient_write_is_rejected_even_if_snapshots_match(): evidence, raw = capture("recommendations-only") raw = add_call(evidence, raw, action="write_artifact", writes=[".eval-author/tasks/unauthorized/task.toml"])