From 908733a02af16b879a50be2ef222f44b429e32ef Mon Sep 17 00:00:00 2001 From: Trecek Date: Fri, 30 Oct 2026 09:25:42 -0700 Subject: [PATCH 1/5] docs: record independent release readiness experiments --- .../developer/release-readiness-2026-10-30.md | 51 +++++++++++++++++++ 1 file changed, 51 insertions(+) create mode 100644 docs/developer/release-readiness-2026-10-30.md diff --git a/docs/developer/release-readiness-2026-10-30.md b/docs/developer/release-readiness-2026-10-30.md new file mode 100644 index 0000000000..8f94b88577 --- /dev/null +++ b/docs/developer/release-readiness-2026-10-30.md @@ -0,0 +1,51 @@ +# Develop release readiness experiment — 2026-10-30 + +This is an experimental evidence branch. Keep its PR in draft and do not merge or enable auto-merge. Product fixes for #4751, #5249, and #4711 are being handled separately. This branch records independent release testing; it does not certify release readiness from a green selection-only gate. + +## Candidate and isolation + +- Baseline product commit: `f84bae324c3497ec0851d5ac887770d0e895fbad` (`0.10.1306`). +- Test branch: `test/release-readiness-20261030`. +- Playground: [TalonT-Org/autoskillit-e2e-sandbox](https://github.com/TalonT-Org/autoskillit-e2e-sandbox). +- Work takes place in an isolated linked worktree. The primary checkout and active AutoSkillit installation are preserved. +- Workflow dispatches record their actual `headSha`; results from another candidate are listed separately. A documentation commit on this branch does not replace the recorded product baseline. + +## Campaign results + +| Check | Candidate | Result | Evidence | +|---|---|---|---| +| Canary | `f84bae324` | Passed | [E2E run 41088605318](https://github.com/TalonT-Org/AutoSkillit/actions/runs/41088605318) — actual canary job completed successfully | +| Clean installation | `f84bae324` | Passed | Same explicitly selected workflow run; clean-install execution and aggregate gate both completed successfully | +| Headless sandbox smoke | `f84bae324` | Running | [E2E run 41089434602](https://github.com/TalonT-Org/AutoSkillit/actions/runs/41089434602); execution and exact teardown still to be checked | +| Seeded implementation | `f84bae324` | Queued | [E2E run 41089453034](https://github.com/TalonT-Org/AutoSkillit/actions/runs/41089453034); same candidate, fixed seed, PR/CI evidence and teardown still to be checked | +| Real package upgrade | Old source to candidate | Running independently | Scratch home and uv tool/bin directories; actual resulting source and install invariance still to be checked | +| Configured local gate | `f84bae324` | Incomplete | Lint and Linux/macOS type checks passed; test process timed out after 1,200 seconds around 80%, with failure markers and no final failure summary | + +## Reproduction + +The existing workflow accepts at most two catalog test names in one dispatch. Run against the intended branch, and confirm each resulting run's full `headSha` before attributing a pass to the candidate: + +```bash +gh workflow run e2e.yml --repo TalonT-Org/AutoSkillit --ref develop -f tests=canary,clean-install +gh workflow run e2e.yml --repo TalonT-Org/AutoSkillit --ref develop -f tests=headless-smoke +gh workflow run e2e.yml --repo TalonT-Org/AutoSkillit --ref develop -f tests=implementation +``` + +Wait for each scenario's actual execution and preserve first-attempt failures. The existing repository-wide E2E queue and session limits remain in effect. The implementation scenario needs the configured canonical sandbox seed issue, OPEN and unclaimed, and a passing sandbox baseline. + +Run the repository's local checks with `task test-local-gate`. Use `task install-worktree` for linked-worktree setup; do not run `autoskillit init` inside a linked worktree. + +## Evidence required before marking a row passed + +- A selected test job actually executed, rather than only the selector and aggregate gate passing. +- Installed source/version and provider/backend identity match the recorded case. +- A recipe produced its required artifacts and a PR with the expected diff; sandbox CI passed for that exact PR head. +- Implementation's PR is OPEN and unmerged when validated. After teardown, owned PRs are CLOSED, owned remote branches are gone, the claim is released, and the seed remains OPEN. +- Failure, cancellation and cleanup results remain recorded even if a subsequent retry succeeds. +- Known clean-install doctor warnings #5242/#5243 are distinguished from unexpected warnings and from actual plugin/MCP failures. + +## Additional probes and limits + +The current catalog has no seeded remediation scenario; #5234 tracks that gap. A separate feasibility check is investigating the smallest additional sandbox probe without introducing a new framework. A real upgrade rehearsal uses pristine old source commit `5c4924749b3b52fabc487233c3747e511ab35512` (`0.10.1013`) in a scratch installation; installed local hot fixes are outside this experiment. + +Current live E2E uses Claude Code through MiniMax. Its success does not establish every native-provider or Codex workflow. Fixed-join Codex skill admission remains a separate adoption requirement under #4711. Release acceptance must be repeated against the final candidate after the separate fixes land. From e4f03e72df8ce919ccc6f2f12ce1a603a76e9fa5 Mon Sep 17 00:00:00 2001 From: Trecek Date: Fri, 30 Oct 2026 09:52:22 -0700 Subject: [PATCH 2/5] docs: record verified smoke teardown and install warnings --- docs/developer/release-readiness-2026-10-30.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/developer/release-readiness-2026-10-30.md b/docs/developer/release-readiness-2026-10-30.md index 8f94b88577..13baf1b18b 100644 --- a/docs/developer/release-readiness-2026-10-30.md +++ b/docs/developer/release-readiness-2026-10-30.md @@ -15,9 +15,9 @@ This is an experimental evidence branch. Keep its PR in draft and do not merge o | Check | Candidate | Result | Evidence | |---|---|---|---| | Canary | `f84bae324` | Passed | [E2E run 41088605318](https://github.com/TalonT-Org/AutoSkillit/actions/runs/41088605318) — actual canary job completed successfully | -| Clean installation | `f84bae324` | Passed | Same explicitly selected workflow run; clean-install execution and aggregate gate both completed successfully | -| Headless sandbox smoke | `f84bae324` | Running | [E2E run 41089434602](https://github.com/TalonT-Org/AutoSkillit/actions/runs/41089434602); execution and exact teardown still to be checked | -| Seeded implementation | `f84bae324` | Queued | [E2E run 41089453034](https://github.com/TalonT-Org/AutoSkillit/actions/runs/41089453034); same candidate, fixed seed, PR/CI evidence and teardown still to be checked | +| Clean installation | `f84bae324` | Expected warnings accepted | Workflow/gate succeeded, while the downloaded harness result is `passed: false`, `outcome: expected_failure`, with no unexpected failures; allowlisted doctor findings are #5242/#5243 | +| Headless sandbox smoke | `f84bae324` | Passed, cleanup independently verified | [E2E run 41089434602](https://github.com/TalonT-Org/AutoSkillit/actions/runs/41089434602); downloaded harness result has `passed: true`, no failures; scenario ran 16:23:38–16:27:17 UTC | +| Seeded implementation | `f84bae324` | Running | [E2E run 41089453034](https://github.com/TalonT-Org/AutoSkillit/actions/runs/41089453034); same candidate, fixed seed, PR/CI evidence and teardown still to be checked | | Real package upgrade | Old source to candidate | Running independently | Scratch home and uv tool/bin directories; actual resulting source and install invariance still to be checked | | Configured local gate | `f84bae324` | Incomplete | Lint and Linux/macOS type checks passed; test process timed out after 1,200 seconds around 80%, with failure markers and no final failure summary | @@ -44,6 +44,8 @@ Run the repository's local checks with `task test-local-gate`. Use `task install - Failure, cancellation and cleanup results remain recorded even if a subsequent retry succeeds. - Known clean-install doctor warnings #5242/#5243 are distinguished from unexpected warnings and from actual plugin/MCP failures. +The smoke artifacts identify [sandbox PR #59](https://github.com/TalonT-Org/autoskillit-e2e-sandbox/pull/59), head `b83fc3ae4ceb05cb5eb8d69e3a1323c3bc1ff497`, and branch `e2e-smoke-071e3714980e4da1b92be7f495614abf`. A separate GitHub query verified the PR is CLOSED and unmerged and the branch is absent. Unrelated fixture PR #50 remains OPEN and the two pre-existing fixture branches still point at `d6c143d2494488deeb6680d7d5cf2e55e3e01732`. The seed was OPEN and claimed by the subsequently running implementation test at that observation; claim release must be checked after implementation ends. + ## Additional probes and limits The current catalog has no seeded remediation scenario; #5234 tracks that gap. A separate feasibility check is investigating the smallest additional sandbox probe without introducing a new framework. A real upgrade rehearsal uses pristine old source commit `5c4924749b3b52fabc487233c3747e511ab35512` (`0.10.1013`) in a scratch installation; installed local hot fixes are outside this experiment. From f12e4872065e8bd02aec3ee0f40e7bf43de1ba23 Mon Sep 17 00:00:00 2001 From: Trecek Date: Fri, 30 Oct 2026 09:53:55 -0700 Subject: [PATCH 3/5] fix(docs): index the readiness report and follow filename rules --- docs/developer/README.md | 1 + .../{release-readiness-2026-10-30.md => release-readiness.md} | 0 2 files changed, 1 insertion(+) rename docs/developer/{release-readiness-2026-10-30.md => release-readiness.md} (100%) diff --git a/docs/developer/README.md b/docs/developer/README.md index 54ac99f264..008becb9fa 100644 --- a/docs/developer/README.md +++ b/docs/developer/README.md @@ -5,3 +5,4 @@ For people working on AutoSkillit itself. - [contributing.md](contributing.md) — branch-targeted CI runner/filter policy, pre-commit, version bumps, code-index setup - [diagnostics.md](diagnostics.md) — session diagnostic logs and `sessions.jsonl` - [end-turn-hazards.md](end-turn-hazards.md) — recipe authoring hazards at session end +- [release-readiness.md](release-readiness.md) — experimental release test evidence; do not merge the experiment PR diff --git a/docs/developer/release-readiness-2026-10-30.md b/docs/developer/release-readiness.md similarity index 100% rename from docs/developer/release-readiness-2026-10-30.md rename to docs/developer/release-readiness.md From 4fb145b261fe3ce028760e16a5ac4f88710b3985 Mon Sep 17 00:00:00 2001 From: Trecek Date: Fri, 30 Oct 2026 10:15:27 -0700 Subject: [PATCH 4/5] docs: record successful E2E and reproduced updater exit failure --- docs/developer/release-readiness.md | 31 ++++++++++++++++++++++++----- 1 file changed, 26 insertions(+), 5 deletions(-) diff --git a/docs/developer/release-readiness.md b/docs/developer/release-readiness.md index 13baf1b18b..dd361f0033 100644 --- a/docs/developer/release-readiness.md +++ b/docs/developer/release-readiness.md @@ -17,9 +17,11 @@ This is an experimental evidence branch. Keep its PR in draft and do not merge o | Canary | `f84bae324` | Passed | [E2E run 41088605318](https://github.com/TalonT-Org/AutoSkillit/actions/runs/41088605318) — actual canary job completed successfully | | Clean installation | `f84bae324` | Expected warnings accepted | Workflow/gate succeeded, while the downloaded harness result is `passed: false`, `outcome: expected_failure`, with no unexpected failures; allowlisted doctor findings are #5242/#5243 | | Headless sandbox smoke | `f84bae324` | Passed, cleanup independently verified | [E2E run 41089434602](https://github.com/TalonT-Org/AutoSkillit/actions/runs/41089434602); downloaded harness result has `passed: true`, no failures; scenario ran 16:23:38–16:27:17 UTC | -| Seeded implementation | `f84bae324` | Running | [E2E run 41089453034](https://github.com/TalonT-Org/AutoSkillit/actions/runs/41089453034); same candidate, fixed seed, PR/CI evidence and teardown still to be checked | -| Real package upgrade | Old source to candidate | Running independently | Scratch home and uv tool/bin directories; actual resulting source and install invariance still to be checked | -| Configured local gate | `f84bae324` | Incomplete | Lint and Linux/macOS type checks passed; test process timed out after 1,200 seconds around 80%, with failure markers and no final failure summary | +| Seeded implementation | `f84bae324` | Passed, cleanup independently verified | [E2E run 41089453034](https://github.com/TalonT-Org/AutoSkillit/actions/runs/41089453034); scenario passed 16:27:21–17:07:21 UTC, followed by a passing gate | +| Real package upgrade | `5c4924749` to `f84bae324` | Install passed; CLI exit status failed | Real packages in scratch Docker; candidate version/doctor passed and old worker survived, but update re-exec returned status 20 | +| Configured local gate | Product baseline plus report fix `f12e4872` | Passed | 13,123 passed, 187 skipped, 12 xfailed; conservative filter selected 13,321 and deselected 40,860; lint and both platform type checks passed | +| PR test shards | `f12e4872` | Passed | [Tests run 41094167109](https://github.com/TalonT-Org/AutoSkillit/actions/runs/41094167109); Channel B is unfiltered, other shards retain configured path filtering | +| Unfiltered local test task | Same product baseline | Pending | Existing `AUTOSKILLIT_TEST_FILTER=none task test-check` route; smoke/canary markers and feature gates still apply | ## Reproduction @@ -44,10 +46,29 @@ Run the repository's local checks with `task test-local-gate`. Use `task install - Failure, cancellation and cleanup results remain recorded even if a subsequent retry succeeds. - Known clean-install doctor warnings #5242/#5243 are distinguished from unexpected warnings and from actual plugin/MCP failures. -The smoke artifacts identify [sandbox PR #59](https://github.com/TalonT-Org/autoskillit-e2e-sandbox/pull/59), head `b83fc3ae4ceb05cb5eb8d69e3a1323c3bc1ff497`, and branch `e2e-smoke-071e3714980e4da1b92be7f495614abf`. A separate GitHub query verified the PR is CLOSED and unmerged and the branch is absent. Unrelated fixture PR #50 remains OPEN and the two pre-existing fixture branches still point at `d6c143d2494488deeb6680d7d5cf2e55e3e01732`. The seed was OPEN and claimed by the subsequently running implementation test at that observation; claim release must be checked after implementation ends. +The smoke artifacts identify [sandbox PR #59](https://github.com/TalonT-Org/autoskillit-e2e-sandbox/pull/59), head `b83fc3ae4ceb05cb5eb8d69e3a1323c3bc1ff497`, and branch `e2e-smoke-071e3714980e4da1b92be7f495614abf`. A separate GitHub query verified the PR is CLOSED and unmerged and the branch is absent. + +Implementation produced [sandbox PR #60](https://github.com/TalonT-Org/autoskillit-e2e-sandbox/pull/60), head `1942f885c49652b61e778cf290f53ca217c113a3`, on `e2e-implementation-fixture-normalize-whitespace/1-3`. Downloaded harness results report no failures, complete lifecycle coverage, and peak concurrency of six sessions without violations. A separate GitHub query verified matching-head CI SUCCESS, the PR CLOSED and unmerged, its owned branch absent, and seed issue #1 OPEN with no claim label. Unrelated fixture PR #50 remains OPEN at `a7e6d89767a9659d0d8854306defbe5e675c8b55`; both pre-existing fixture branches remain at `d6c143d2494488deeb6680d7d5cf2e55e3e01732`. + +## Reproduced upgrade failure + +The real-package upgrade from pristine develop-track `0.10.1013` (`5c4924749b3b52fabc487233c3747e511ab35512`) to `0.10.1306` (`f84bae324c3497ec0851d5ac887770d0e895fbad`) installed and pivoted successfully, then returned `failed-upgrade`, exit **20**. A scratch-only Git mirror pinned the source commits, and the supported uv version ran in a container with only the owned probe directory mounted. No fake package or active host installation was used. + +Reproduction sequence: + +1. Start with the pristine older develop-track install in an isolated home outside any checkout; start an old-version worker. +2. Run `autoskillit update` with the candidate as the remote develop target. It installs the candidate and prints success. +3. Restart re-executes the original `update` arguments. That second invocation observes previous, observed and target commits all equal and exits 20. +4. Confirm the candidate's `--version` and `doctor` exit 0; a standalone already-current `update` also exits 20. The existing worker remains alive on the old version. + +Current code dispatches restart at `src/autoskillit/cli/update/_update.py:63`, re-executes original arguments in `_restart.py:10`, and rejects unchanged advance verdicts in `_transaction.py:304`. Existing updater tests mock restart, leaving this real exec boundary uncovered. The transaction's no-advance guard is intentional; the user-visible problem is a second update replacing the successful transaction's result. No matching ticket was found in the saved issue snapshot; this PR includes evidence, not a product fix. + +Host executable path, hash, version, direct-URL provenance and uv tool inventory matched the pre-test snapshot afterward. Complete reproduction commands and logs are retained in the local campaign's `upgrade/` directory. ## Additional probes and limits -The current catalog has no seeded remediation scenario; #5234 tracks that gap. A separate feasibility check is investigating the smallest additional sandbox probe without introducing a new framework. A real upgrade rehearsal uses pristine old source commit `5c4924749b3b52fabc487233c3747e511ab35512` (`0.10.1013`) in a scratch installation; installed local hot fixes are outside this experiment. +The current catalog has no seeded remediation scenario; #5234 tracks that gap. The fixed seed is an implementation fixture, so it was preserved rather than rewritten into a bug fixture. The upgrade rehearsal used pristine old source; installed local hot fixes were outside this experiment. + +An earlier main-checkout local gate timed out after 1,200 seconds around 80% completion without a final failure summary. The isolated worktree's configured gate later completed after correcting the experiment report's filename/index mistakes. Its conservative filter is explicitly recorded above; an unfiltered run is tracked separately. The current Tests workflow exposes no manual dispatch or PR-label override for full path-unfiltered CI. Current live E2E uses Claude Code through MiniMax. Its success does not establish every native-provider or Codex workflow. Fixed-join Codex skill admission remains a separate adoption requirement under #4711. Release acceptance must be repeated against the final candidate after the separate fixes land. From 813004724044bbdbdfcd0972f90a180ae3cdbcff Mon Sep 17 00:00:00 2001 From: Trecek Date: Fri, 30 Oct 2026 10:46:36 -0700 Subject: [PATCH 5/5] docs: finalize release readiness results --- docs/developer/release-readiness.md | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/docs/developer/release-readiness.md b/docs/developer/release-readiness.md index dd361f0033..c6553139ce 100644 --- a/docs/developer/release-readiness.md +++ b/docs/developer/release-readiness.md @@ -21,7 +21,7 @@ This is an experimental evidence branch. Keep its PR in draft and do not merge o | Real package upgrade | `5c4924749` to `f84bae324` | Install passed; CLI exit status failed | Real packages in scratch Docker; candidate version/doctor passed and old worker survived, but update re-exec returned status 20 | | Configured local gate | Product baseline plus report fix `f12e4872` | Passed | 13,123 passed, 187 skipped, 12 xfailed; conservative filter selected 13,321 and deselected 40,860; lint and both platform type checks passed | | PR test shards | `f12e4872` | Passed | [Tests run 41094167109](https://github.com/TalonT-Org/AutoSkillit/actions/runs/41094167109); Channel B is unfiltered, other shards retain configured path filtering | -| Unfiltered local test task | Same product baseline | Pending | Existing `AUTOSKILLIT_TEST_FILTER=none task test-check` route; smoke/canary markers and feature gates still apply | +| Unfiltered local test task | Evidence commit `4fb145b2`, product baseline `f84bae324` | Passed | 56,876 passed, 809 skipped, 26 xfailed in 25m 54s; exit 0, with path filtering disabled; smoke/canary markers and feature gates still apply | ## Reproduction @@ -37,6 +37,14 @@ Wait for each scenario's actual execution and preserve first-attempt failures. T Run the repository's local checks with `task test-local-gate`. Use `task install-worktree` for linked-worktree setup; do not run `autoskillit init` inside a linked worktree. +The additional path-unfiltered run used the existing task, explicitly selecting the entire test tree and clearing inherited path ignores: + +```bash +AUTOSKILLIT_TEST_FILTER=none PYTEST_TEST_PATHS=tests/ PYTEST_IGNORE_PATHS= task test-check +``` + +It ran on Python 3.11.14. Reported feature scope had Codex backend disabled and exploration, fleet, headless fleet execution, planner, and providers enabled. The active host entrypoint hash and installed source commit matched the pre-test snapshot after this full run. + ## Evidence required before marking a row passed - A selected test job actually executed, rather than only the selector and aggregate gate passing. @@ -69,6 +77,6 @@ Host executable path, hash, version, direct-URL provenance and uv tool inventory The current catalog has no seeded remediation scenario; #5234 tracks that gap. The fixed seed is an implementation fixture, so it was preserved rather than rewritten into a bug fixture. The upgrade rehearsal used pristine old source; installed local hot fixes were outside this experiment. -An earlier main-checkout local gate timed out after 1,200 seconds around 80% completion without a final failure summary. The isolated worktree's configured gate later completed after correcting the experiment report's filename/index mistakes. Its conservative filter is explicitly recorded above; an unfiltered run is tracked separately. The current Tests workflow exposes no manual dispatch or PR-label override for full path-unfiltered CI. +An earlier main-checkout local gate timed out after 1,200 seconds around 80% completion without a final failure summary. The isolated worktree's configured gate later passed after correcting the experiment report's filename/index mistakes; the separate unfiltered task also passed. The current Tests workflow exposes no manual dispatch or PR-label override for full path-unfiltered CI, so the broader run was performed locally. Current live E2E uses Claude Code through MiniMax. Its success does not establish every native-provider or Codex workflow. Fixed-join Codex skill admission remains a separate adoption requirement under #4711. Release acceptance must be repeated against the final candidate after the separate fixes land.