Skip to content

feat(eval-author): add deterministic first-suite creation checkpoints - #126

Draft
jessbuhl wants to merge 9 commits into
mainfrom
jbuhl/ase-1172-bootstrap
Draft

jessbuhl wants to merge 9 commits into
mainfrom
jbuhl/ase-1172-bootstrap

Conversation

@jessbuhl

@jessbuhl jessbuhl commented Oct 23, 2026 •

Copy link
Copy Markdown
Collaborator

Implements first-suite creation through the shared deterministic conversation renderer. Users choose existing traces or ETHOS.md, then approve a small proposal before construction: one to three tasks with one input example each by default, with broader scope requiring an explicit request. A fresh request that finds a saved design separately asks whether to continue, revise, or start a new proposal.

The first task is built and reviewed before the remainder. Both build phases show compact checklists with responsible skill names, evidence-backed statuses, and saved progress; the remaining-task checklist includes completion counts and any pending first-task validation. The first-task handoff leads with the approved design and ETHOS.md grounding, then links the HTML Tasks view or mirrored README before the summaries and approval question. Once all tasks are authored, a separate suite review leads with ETHOS.md grounding and report/README links, summarizes the tasks and evidence, and asks for approval before any run-settings questions. Paid grading can remain pending. Execution settings unlock only after approval of the exact suite revisions; task approval remains separate from spending authorization.

Creation progress extends the existing revision-checked state interface. Approvals and validation bind to task revisions; explicit task/row/run selections, repeats, exact reviewed attempts, and immutable review summaries survive resume. Changed evidence reopens affected claims. Controls stay separate from agent results. The completed flow explicitly offers an audit and retains context only when accepted.

No files are renamed and no demo assets are added. Existing starter expansion keeps its own saved scope.

Validation:

  • 525 focused state, conversation, selection, formatting, and instruction checks passed, including both build checklists, skill labels, saved-design decisions, interruption/resume, stale evidence, blockers, and suite approval before execution settings.
  • Earlier full repository run: 2,025 passed and 11 environment-dependent skips. The run collected an earlier renderer fixture during editing and found a copied-installation fixture missing the new helper; all 21 failing tests passed after those fixture fixes (pytest --lf).
  • Ruff lint/format, Linux type checks for Python 3.12 and 3.13, runtime policy, copyright, third-party licenses, whitespace, and DCO checks passed.

This delivers the workflow/state changes for ASE-1172. Integrated acceptance remains pending the report dependency's separate review/validation/execution status, planned-task and exact-attempt/native-Gym presentation, plus execution through a verified configured Gym–Harbor bridge. No live target-agent or paid model runs were performed, and report visual changes remain outside this PR.

Refs ASE-1172. Preserve task and run evidence across review, execution, and the optional audit handoff.

Signed-off-by: Jessica Buhl <jbuhl@nvidia.com>
Keep the renderer's Recommendations and Next Steps headings in final replies, including when no active blockers remain. Add focused renderer coverage and a final-reply acceptance check.

Signed-off-by: Jessica Buhl <jbuhl@nvidia.com>
Make source selection and design review separate turn-ending checkpoints. Default to one to three tasks with one example each, preserve explicit broader requests, and explain the first-task review pause in rendered copy.

Signed-off-by: Jessica Buhl <jbuhl@nvidia.com>
Signed-off-by: Jessica Buhl <jbuhl@nvidia.com>
Signed-off-by: Jessica Buhl <jbuhl@nvidia.com>
Signed-off-by: Jessica Buhl <jbuhl@nvidia.com>
Signed-off-by: Jessica Buhl <jbuhl@nvidia.com>
Signed-off-by: Jessica Buhl <jbuhl@nvidia.com>
Signed-off-by: Jessica Buhl <jbuhl@nvidia.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant