Repository navigation
Description
Successful recipe E2E runs omit audit commissioning evidence
Fix target and integration order
Repository: TalonT-Org/AutoSkillit. Fix PR base: develop, starting at 1fa72475dc7bd311fbbb038d95120a6d4da5e807 or a descendant. The affected harness and tests already exist there. Merge the remediation into develop, then rebase PR #5255. Do not target its feature branch. The implementation catalog entry and its pipeline selector are still unmerged; use the existing recipe kind and a synthetic ordinary recipe fixture on develop rather than depending on that absent selector or publishing the entry early.
Confirmed behavior and impact
run_test calls collect_failure_evidence only when not result["passed"]. The collector copies generated plans and raw Claude root/worker/native transcripts. A successful recipe run therefore uploads normal session summaries, process/usage telemetry, lifecycle evidence and harness API/command captures, but omits the primary plan/transcript artifacts needed to independently verify its internal audit verdict and stage results.
This is now observable on the first strict-passing implementation E2E run for #5255. The original plan at .autoskillit/temp/make-plan/issue_5233_implementation_e2e_plan_2026-10-08_015622.md:127 explicitly requires inspection of saved logs for planning through audit GO and open-PR completion. Its exact wording includes:
inspect saved logs for completed planning, dry-walkthrough, implementation, tests, run-branch integration, audit GO, and open-PR completion.
The archived invocation and tool-success metadata prove that an attested audit worker and writer ran; they omit the verdict-bearing semantic result and the writer input. The writer accepts a verdict as input and returns a result path/digest, not the verdict itself (writer source). The GO verdict and final stage-discharge result cannot be independently checked from the retained bytes. Do not substitute a green workflow, worker exit0 or tool-success metadata for those payloads.
Source at the tested feature revision:
e2e_harness.py:1861: failure collector copies plans and~/.claude/projects/**/*.jsonl, excluding symlinks.e2e_harness.py:1914: collection is conditional on a non-passing result.- The same policy is present on develop1fa, lines1147 and1200.
Live reproduction
Workflow: 39861583843, attempt1. Tested AutoSkillit head: c492383fd9d72dfb76387164311675df640bcaee, containing develop1fa and fixes #5300/#5298/#5299. Retained model identifiers: MiniMax-M3.1-Flash, observed 1M context rows. This was the unchanged fixed sandbox issue1, baseline d6c143d2494488deeb6680d7d5cf2e55e3e01732.
gh run download 39861583843 --repo TalonT-Org/AutoSkillit \
--name e2e-implementation-39861583843-1 \
--dir .autoskillit/temp/investigate/success-evidence-39861583843Read the downloaded files directly:
out/result.jsonispassed:true,outcome:"passed", failures/expected_findings empty.out/transcripts/andout/pipeline-artifacts/are absent. There is no saved audit semantic result file elsewhere in the uploaded artifact.logs/sessions.jsonl, one-based record4, binds/audit-implwithaudit_output_mode:"attested", an admitted sealed plan set and anaudit_semantic_submissioncontaining the plan reference/reservation. Itsoutcome_fieldsretainsaudit_semantic_result_pathendingaudit-admission/sha256:30b7639bc911f1003534f88640f626a2838dc9237e5d73fe7d7fa2d137b8ac49/843349d7b4f6ee21a2b771a98fa6c9c1/semantic.jsonandsemantic_digest:sha256:e40597fdd6eb5e54241971290e6a4b08cc0e305b193d61a29dbfe8d141e55ac5. The referenced bytes and GO verdict are absent; a path/digest is not the payload.logs/sessions/459b6a53-b95d-479f-9ccd-e9acf4103aba/summary.json:467-470nameswrite_audit_semantic_resultinturn_tool_calls.logs/otlp.jsonl, record243 (record_id:379e99c5e4c041bfacc430aabe655bbb), event sequences470–471 records the accepted MCP call andsuccess:true, with input/result byte lengths26414/376. The verdict/assessment payload and submitted ArtifactRef fields are omitted; invocation metadata and the retained output path/digest do not establish the verdict.- Root session is
6ef571d9-91ab-4edd-b5f9-dc8ec5759c6e. Its summary listsverify_review_receipt, but that call's returned result is not archived. Records7–8 bind review-pr/resolve-review in local mode and do retain compact outcomes:approved_with_comments,review_post_state:LOCAL,review_receipt_path:none, and a processed resolution with one fix. No GitHub review or remote publication receipt is required by those local invocations. Their referenced local findings/report files and raw root routing/gate results are absent; this does not establish a review publisher regression or failed final-stage discharge.
The workflow/harness success is real: its saved responses validate sandbox PR26 at final head 358b4a8de7aeed128c10f81cb6ecfe03766f2e6d, detached-head tests and successful sandbox CI39869340829 with explicit PR/repository/head association under the harness token. Lifecycle coverage is complete, peak6, no violations. The seed was observed OPEN/unclaimed and the owned branch absent between runs; a subsequent run legitimately claimed/recreated it. No seed-leak, concurrency or CI-association bug is alleged. The missing archive prevents completing the separate manual audit/receipt inspection.
Test gap and bounded reproduction
tests/infra/test_e2e_harness.py:74 (test_run_test_retains_evidence_only_when_it_does_not_pass) explicitly asserts no evidence key, no plan folder and no transcript folder for its passing case at lines91–95. Existing error handling, source containment/symlink and redaction checks cover failure collection. They do not prove that a passing pipeline recipe retains the evidence required for commissioning.
Using the existing fake runner/fixture, simulate a successful recipe result with generated plans, root/worker/native transcripts and a verdict-bearing semantic result plus applicable local findings/verification artifacts. Assert that the primary evidence survives the existing artifact/redaction route, including secret removal on the success path. Update the existing passed-case absence assertions where the new recipe retention rule applies, rather than leaving contradictory tests. Keep the underlying recipe outcome distinct from evidence collection; never relabel this observed pass as a recipe failure to force the old collector path. The investigator should select the necessary audit/report/gate evidence and error policy without adding a general evidence subsystem, uploading signing credentials or blindly copying a credential-bearing home/workspace.
Requirements
Successful pipeline recipe runs must retain primary evidence sufficient to verify plan binding, attested audit verdict and relevant review/publication-stage results after container teardown. Preserve existing failure evidence, source containment, redaction, strict harness checks and cleanup behavior; add a success-path regression through the established harness fixtures and configured test wrapper. Recommission #5255 against the same fixed seed after integration, and only claim the internal audit/publication facts supported by the newly retained evidence.
Scope and classification
One concern: success-path evidence preservation. bug, recipe:remediation. This is not proof that the first audit failed, a reason to reopen the three repaired issues, or permission to weaken commissioning acceptance. No source fix is included in this report.
Failure-path control and final run status — 2026-10-22
The second same-head/same-seed attempt39861593447 strictly failed on its malformed final fleet-result delimiter, not on evidence retention. Its result.json reports plan_files1/transcripts32/errors[] and contains out/transcripts plus out/pipeline-artifacts. This is the matching failure-path control for the first passing39861583843, whose corresponding folders and verdict-bearing semantic bytes are absent. Both use c492383.
The failed run's audit transcript46ee0a11-2312-4eb2-8c91-0b6c082cf64c records123/125 retains the GO writer input and success/path/digest return; root76d9f59f-6723-41fb-93ae-9a5d7a58d4df record754 returns server-authored PUBLISHED/GO,759 acknowledges its receipt and998/999 verifies the PR stage obligations. These live checks validate the repaired audit path in that failed attempt; they do not reconstruct the first successful run's missing primary bytes or turn the second result into a pass. The semantic file itself is still not among the collector's plan-only file copies, reinforcing the need to select sufficient primary audit/report evidence rather than only preserving metadata paths.
Both runs cleaned up their owned PRs/branches and left the unchanged seed OPEN/unclaimed. There was no manual repair. This fix continues to target develop; the generic harness and recipe fixtures already exist there. The malformed sentinel is a separate observed caller-output error; no new parser bug is claimed, and this retention fix alone does not guarantee a future model follows every output contract.
Activity