Skip to content

Successful recipe E2E runs omit audit commissioning evidence #5301

Closed

Description

@Trecek

Successful recipe E2E runs omit audit commissioning evidence

Fix target and integration order

Repository: TalonT-Org/AutoSkillit. Fix PR base: develop, starting at 1fa72475dc7bd311fbbb038d95120a6d4da5e807 or a descendant. The affected harness and tests already exist there. Merge the remediation into develop, then rebase PR #5255. Do not target its feature branch. The implementation catalog entry and its pipeline selector are still unmerged; use the existing recipe kind and a synthetic ordinary recipe fixture on develop rather than depending on that absent selector or publishing the entry early.

Confirmed behavior and impact

run_test calls collect_failure_evidence only when not result["passed"]. The collector copies generated plans and raw Claude root/worker/native transcripts. A successful recipe run therefore uploads normal session summaries, process/usage telemetry, lifecycle evidence and harness API/command captures, but omits the primary plan/transcript artifacts needed to independently verify its internal audit verdict and stage results.

This is now observable on the first strict-passing implementation E2E run for #5255. The original plan at .autoskillit/temp/make-plan/issue_5233_implementation_e2e_plan_2026-10-08_015622.md:127 explicitly requires inspection of saved logs for planning through audit GO and open-PR completion. Its exact wording includes:

inspect saved logs for completed planning, dry-walkthrough, implementation, tests, run-branch integration, audit GO, and open-PR completion.

The archived invocation and tool-success metadata prove that an attested audit worker and writer ran; they omit the verdict-bearing semantic result and the writer input. The writer accepts a verdict as input and returns a result path/digest, not the verdict itself (writer source). The GO verdict and final stage-discharge result cannot be independently checked from the retained bytes. Do not substitute a green workflow, worker exit0 or tool-success metadata for those payloads.

Source at the tested feature revision:

Live reproduction

Workflow: 39861583843, attempt1. Tested AutoSkillit head: c492383fd9d72dfb76387164311675df640bcaee, containing develop1fa and fixes #5300/#5298/#5299. Retained model identifiers: MiniMax-M3.1-Flash, observed 1M context rows. This was the unchanged fixed sandbox issue1, baseline d6c143d2494488deeb6680d7d5cf2e55e3e01732.

gh run download 39861583843 --repo TalonT-Org/AutoSkillit \
  --name e2e-implementation-39861583843-1 \
  --dir .autoskillit/temp/investigate/success-evidence-39861583843

Read the downloaded files directly:

  1. out/result.json is passed:true, outcome:"passed", failures/expected_findings empty. out/transcripts/ and out/pipeline-artifacts/ are absent. There is no saved audit semantic result file elsewhere in the uploaded artifact.
  2. logs/sessions.jsonl, one-based record4, binds /audit-impl with audit_output_mode:"attested", an admitted sealed plan set and an audit_semantic_submission containing the plan reference/reservation. Its outcome_fields retains audit_semantic_result_path ending audit-admission/sha256:30b7639bc911f1003534f88640f626a2838dc9237e5d73fe7d7fa2d137b8ac49/843349d7b4f6ee21a2b771a98fa6c9c1/semantic.json and semantic_digest:sha256:e40597fdd6eb5e54241971290e6a4b08cc0e305b193d61a29dbfe8d141e55ac5. The referenced bytes and GO verdict are absent; a path/digest is not the payload.
  3. logs/sessions/459b6a53-b95d-479f-9ccd-e9acf4103aba/summary.json:467-470 names write_audit_semantic_result in turn_tool_calls. logs/otlp.jsonl, record243 (record_id:379e99c5e4c041bfacc430aabe655bbb), event sequences470–471 records the accepted MCP call and success:true, with input/result byte lengths26414/376. The verdict/assessment payload and submitted ArtifactRef fields are omitted; invocation metadata and the retained output path/digest do not establish the verdict.
  4. Root session is 6ef571d9-91ab-4edd-b5f9-dc8ec5759c6e. Its summary lists verify_review_receipt, but that call's returned result is not archived. Records7–8 bind review-pr/resolve-review in local mode and do retain compact outcomes: approved_with_comments, review_post_state:LOCAL, review_receipt_path:none, and a processed resolution with one fix. No GitHub review or remote publication receipt is required by those local invocations. Their referenced local findings/report files and raw root routing/gate results are absent; this does not establish a review publisher regression or failed final-stage discharge.

The workflow/harness success is real: its saved responses validate sandbox PR26 at final head 358b4a8de7aeed128c10f81cb6ecfe03766f2e6d, detached-head tests and successful sandbox CI39869340829 with explicit PR/repository/head association under the harness token. Lifecycle coverage is complete, peak6, no violations. The seed was observed OPEN/unclaimed and the owned branch absent between runs; a subsequent run legitimately claimed/recreated it. No seed-leak, concurrency or CI-association bug is alleged. The missing archive prevents completing the separate manual audit/receipt inspection.

Test gap and bounded reproduction

tests/infra/test_e2e_harness.py:74 (test_run_test_retains_evidence_only_when_it_does_not_pass) explicitly asserts no evidence key, no plan folder and no transcript folder for its passing case at lines91–95. Existing error handling, source containment/symlink and redaction checks cover failure collection. They do not prove that a passing pipeline recipe retains the evidence required for commissioning.

Using the existing fake runner/fixture, simulate a successful recipe result with generated plans, root/worker/native transcripts and a verdict-bearing semantic result plus applicable local findings/verification artifacts. Assert that the primary evidence survives the existing artifact/redaction route, including secret removal on the success path. Update the existing passed-case absence assertions where the new recipe retention rule applies, rather than leaving contradictory tests. Keep the underlying recipe outcome distinct from evidence collection; never relabel this observed pass as a recipe failure to force the old collector path. The investigator should select the necessary audit/report/gate evidence and error policy without adding a general evidence subsystem, uploading signing credentials or blindly copying a credential-bearing home/workspace.

Requirements

Successful pipeline recipe runs must retain primary evidence sufficient to verify plan binding, attested audit verdict and relevant review/publication-stage results after container teardown. Preserve existing failure evidence, source containment, redaction, strict harness checks and cleanup behavior; add a success-path regression through the established harness fixtures and configured test wrapper. Recommission #5255 against the same fixed seed after integration, and only claim the internal audit/publication facts supported by the newly retained evidence.

Scope and classification

One concern: success-path evidence preservation. bug, recipe:remediation. This is not proof that the first audit failed, a reason to reopen the three repaired issues, or permission to weaken commissioning acceptance. No source fix is included in this report.

Failure-path control and final run status — 2026-10-22

The second same-head/same-seed attempt39861593447 strictly failed on its malformed final fleet-result delimiter, not on evidence retention. Its result.json reports plan_files1/transcripts32/errors[] and contains out/transcripts plus out/pipeline-artifacts. This is the matching failure-path control for the first passing39861583843, whose corresponding folders and verdict-bearing semantic bytes are absent. Both use c492383.

The failed run's audit transcript46ee0a11-2312-4eb2-8c91-0b6c082cf64c records123/125 retains the GO writer input and success/path/digest return; root76d9f59f-6723-41fb-93ae-9a5d7a58d4df record754 returns server-authored PUBLISHED/GO,759 acknowledges its receipt and998/999 verifies the PR stage obligations. These live checks validate the repaired audit path in that failed attempt; they do not reconstruct the first successful run's missing primary bytes or turn the second result into a pass. The semantic file itself is still not among the collector's plan-only file copies, reinforcing the need to select sufficient primary audit/report evidence rather than only preserving metadata paths.

Both runs cleaned up their owned PRs/branches and left the unchanged seed OPEN/unclaimed. There was no manual repair. This fix continues to target develop; the generic harness and recipe fixtures already exist there. The malformed sentinel is a separate observed caller-output error; no new parser bug is claimed, and this retention fix alone does not guarantee a future model follows every output contract.

Activity

  1. added
    bugExisting behavior is broken
    recipe:remediationRoute: investigate/decompose before implementation
    on Oct 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugExisting behavior is brokenrecipe:remediationRoute: investigate/decompose before implementation

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions