Skip to content

E2E test: implementation recipe on a seeded sandbox issue #5233

Closed

Description

@Trecek

Maintainer setup

  • Seed an open issue in TalonT-Org/autoskillit-e2e-sandbox describing a small, self-contained feature with an obvious test.

Everything else comes from #5230 (MINIMAX_API_KEY, E2E_SANDBOX_TOKEN). No Anthropic credential.

Depends on: #5230; #5232 (smoke) should pass first.

What it proves

The bundled implementation recipe completes on MiniMax from issue to open PR: make-plan, dry-walkthrough, implement, test, merge into the run's branch, audit, PR.

Cost and limits

  • Steps run one at a time, so the peak is the orchestrator, one step's session and that step's children: 6 during implement (implement-worktree-no-merge names four exploration responsibilities) and up to 8 during resolve-review in the local review loop (three to six groups on a typical PR). make-plan's exploration fan-out depends on the task.
  • audit-impl starts one deviation evaluator per plan deviation, so its fan-out depends on the plan; the ceiling below bounds it.
  • Architecture lenses stay off (arch_lenses defaults to false); they would run several sessions in parallel.
  • The harness's CLAUDE_CODE_MAX_CONCURRENT_SUBAGENTS=6 must not go lower, because skills require their declared children to run concurrently.
  • For the timeout: a MiniMax implement session took a median 7.0 minutes in the local session index (n = 175).

Scope

  • Catalog entry: selected only for larger PRs touching recipes, skills, the server or execution code; never more than one pipeline-class test per PR.
  • Run with open_pr=true, auto_merge=false; the harness closes the PR and deletes the branch afterwards.
  • Assertions: envelope success; a PR open on the sandbox referencing the seeded issue; the diff touches the sandbox's source and tests; sandbox tests green on the PR branch.

Each defect found gets its own bug issue; the catalog entry is marked expected-failure with the link until fixed. The first candidate to watch for: the fleet run orchestrator ignores providers.default_provider, which the settings-file route is meant to make irrelevant.

Acceptance criteria

  • Completes in CI on MiniMax with no Anthropic credential.
  • Peak concurrent sessions measured from the session logs and recorded in the catalog entry.

Live commissioning status: 2026-10-09

The first implementation attempt was run 37854525992, using PR #5255 commit 017b3f1 and reusable sandbox issue 1. E2E_IMPLEMENTATION_ISSUE_URL is configured.

Docker setup and the sandbox baseline tests passed. The bundled implementation recipe explicitly supplies output_dir: . to make-plan, whose declared scope is {{AUTOSKILLIT_TEMP}}/make-plan/. The run failed before implementation with: run_skill output_dir is outside the skill's declared write scope.

A resource-limited, network-isolated Docker probe of the exact published source reproduced the mismatch: the root path is rejected, and .autoskillit/temp/make-plan/ is admitted. The probe used about 55 MB peak Python RSS and ran no model or test suite. This is a deterministic planner-dispatch contract mismatch, not an observed memory failure. The separate reporter crash remains tracked in #5249.

No implementation PR or sandbox PR CI success was produced. The seed stays OPEN with a fail label and no claim label. A matching branch remains at the same commit as main, while ownership.json records no owned branch; teardown is not fully verified.

Commissioning remains incomplete. Recommended gate: repair planner admission independently, verify it with the Docker reproducer, then obtain two genuine successful MiniMax implementation runs and complete ownership/teardown evidence before recording full-pipeline peak_sessions_evidence. Expected-failure routing, if explicitly adopted for a provisional harness merge, must link to the defect and must not be described as successful commissioning.

Planner blocker: #5256

The deterministic planner output-path mismatch is now filed separately as bug #5256, labeled bug and recipe:remediation. It includes the exact tested revision, source contract mismatch, live failure evidence, complete self-contained Docker probe/build/run commands, resource limits, observed results, and artifact retrieval commands.

The reporter defect remains #5249. PR #5255 remains open while commissioning is incomplete; the planning-only Docker reproduction is not a passing full implementation run. Next: fix planner admission independently, verify in isolated Docker, and rerun MiniMax twice with complete PR CI and teardown evidence.

Recommissioning results — 2026-10-22

PR5255 is rebased onto develop1fa72475dc7bd311fbbb038d95120a6d4da5e807 (0.10.1286), integrating the merged fixes5300/5298/5299 for closed5295/5296/5297. Published head c492383. All-files pre-commit and the focused configured gate passed (1,019 tests); full Tests39861563609 and automatic E2E39861563578, headless-smoke plus canary, succeeded.

The first implementation 39861583843 is a strict harness pass: true/passed/no expected findings. Sandbox PR26 retained the seed URL, source/tests and final head358b4a8de7aeed128c10f81cb6ecfe03766f2e6d. Detached-head install/tests passed (10 tests); saved sandbox CI39869340829 under the harness token includes the exact PR/head/repository association and successful test job. Its attested audit invocation/writer metadata exist, but success-path collection omits the verdict-bearing bytes and raw transcripts: new bug5301, fix base develop, blocks the independent audit GO inspection.

The second implementation 39861593447 failed strictly: fleet_envelope: resumable: fleet_l3_no_result_block. Its raw root76d9f59f-6723-41fb-93ae-9a5d7a58d4df record1010 contains success JSON/reason implementation_complete but closes the sentinel with %% instead of ---. Startup record3 and fleet/_prompts.py specify the correct delimiter; fleet/result_parser.py correctly requires it. No compaction occurred. This is not proof of a new parser defect or a recurrence of5272; no forgiving parser/PR-existence fallback or expected-failure annotation was added. The failure occurred before the harness's detached-head/CI acceptance checks.

The second failure archive does positively verify the live audit fixes: audit46ee0a11-2312-4eb2-8c91-0b6c082cf64c record123 submits GO with typed refs,125 returns success/path/digest; root754 is server-authored PUBLISHED/GO,759 acknowledges the receipt, and998/999 verifies the PR stage obligations. PR27/headf578b9db8398fb5428d3aa7527fb9c2f4a8aa270 had successful sandbox CI39872806996; that observation does not replace its failed harness acceptance. Reviews were LOCAL, with no GitHub approval/comment provenance claimed.

Both lifecycle reports have complete coverage, no violations, observed peak6; parent independently swept the intervals. Retained model is MiniMax-M3.1-Flash, with all389/435 observed turn rows at1M. Seed title/body and main d6c143d2494488deeb6680d7d5cf2e55e3e01732 are unchanged. Final PR26/27 CLOSED/unmerged, owned /1-3 ref absent, fixed seed OPEN/unclaimed, no open sandbox PRs; reuse between runs was observed. All workflows are finished. Only one strict harness pass and zero fully evidenced commissioning pairs are established; peak_sessions8 remains provisional. Integrate5301 from develop before the next full verification pair. Existing5249 recurred as a secondary reporter error.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions