Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
129 commits
Select commit Hold shift + click to select a range
844510d
feat: add seeded sandbox implementation E2E validation
Trecek Oct 8, 2026
9efed3c
fix(review): centralize E2E session log root construction
Trecek Oct 8, 2026
4c41bd9
fix(review): name the E2E session trace artifact
Trecek Oct 8, 2026
84db95f
fix(review): name sandbox CI polling settings
Trecek Oct 8, 2026
54d56bf
fix(review): clarify claim label predicate naming
Trecek Oct 8, 2026
a8acd0d
fix(review): ignore non-string transcript tool names
Trecek Oct 8, 2026
ea59abd
fix(review): reject dot session directory components
Trecek Oct 8, 2026
3ba1d69
test(review): isolate ownership proof scenarios
Trecek Oct 8, 2026
021bead
test(review): assert both unpaired branch diagnostics
Trecek Oct 8, 2026
2e3a5ed
feat: add seeded sandbox implementation E2E validation
Trecek Oct 8, 2026
e9866f5
fix(review): centralize E2E session log root construction
Trecek Oct 8, 2026
59813e3
fix(review): name the E2E session trace artifact
Trecek Oct 8, 2026
6feef27
fix(review): name sandbox CI polling settings
Trecek Oct 8, 2026
4313956
fix(review): clarify claim label predicate naming
Trecek Oct 8, 2026
5b69f5e
fix(review): ignore non-string transcript tool names
Trecek Oct 8, 2026
99531c1
fix(review): reject dot session directory components
Trecek Oct 8, 2026
d8a776e
test(review): isolate ownership proof scenarios
Trecek Oct 8, 2026
c995db4
test(review): assert both unpaired branch diagnostics
Trecek Oct 8, 2026
4adccdb
fix(e2e): recover ownership evidence and reconcile sandbox branches (…
Trecek Oct 13, 2026
2412116
Merge published PR 5255 ownership fix into verification history
Trecek Oct 13, 2026
93457f1
Merge develop tracker admission fix into PR 5255
Trecek Oct 13, 2026
d1eaf42
fix(e2e): scope native lifecycle capture to named agents (#5277)
Trecek Oct 14, 2026
96a18e7
Merge develop commissioning fixes into PR 5255
Trecek Oct 15, 2026
20ee392
Merge develop runner continuation fix into PR 5255
Trecek Oct 16, 2026
2844880
Merge develop with commissioning fixes for #5285 and #5286
Trecek Oct 18, 2026
6e92c10
Merge develop with delegated prep-output write fix for #5289
Trecek Oct 18, 2026
7cfcf6d
Merge develop commissioning repairs into PR 5255
Trecek Oct 27, 2026
cba74f6
test: assert shipped implementation trigger paths
Trecek Oct 27, 2026
c02c451
test: assert effective E2E selection capacity
Trecek Oct 27, 2026
dbe4b63
test: align pipeline fixture with sandbox changed paths
Trecek Oct 27, 2026
2cf2e56
fix(review): require pipeline validation fields
Trecek Oct 27, 2026
8687d11
fix(review): reject pipeline metadata on ordinary recipes
Trecek Oct 27, 2026
a8e4340
fix(review): include scan roots in MCP literal diagnostics
Trecek Oct 27, 2026
f66589d
fix(review): require resolvable Claude agent names
Trecek Oct 27, 2026
5c37d35
test(review): pin the unfiltered session hook events
Trecek Oct 27, 2026
24a593c
test(review): retain ownership-critical E2E tools
Trecek Oct 27, 2026
f47c6f8
test(review): bind ownership parsing to actual server tool envelopes
Trecek Oct 27, 2026
4c360b9
test(review): verify collector lineage from production session record…
Trecek Oct 27, 2026
00e619a
fix(review): JSON-escape materialized fixture paths
Trecek Oct 27, 2026
c46e0e7
fix(review): document captured model field distinction
Trecek Oct 27, 2026
aa4d7b9
fix(review): share the captured fixture root helper
Trecek Oct 27, 2026
ec2c6e5
fix(review): narrow the fixture sanitization claim
Trecek Oct 27, 2026
8f5975d
fix(review): remove repeated ownership assertions
Trecek Oct 27, 2026
bb6723f
fix(review): document the captured orphan lifecycle stop
Trecek Oct 27, 2026
e9e598d
fix(review): retain pipeline harness exception traceback
Trecek Oct 27, 2026
dbf7060
fix(review): retain catalog concurrency capacity evidence
Trecek Oct 27, 2026
ef021c2
fix(review): require the resolved harness home
Trecek Oct 27, 2026
e641d09
fix(review): remove dead recipe local initializers
Trecek Oct 27, 2026
bef3b67
fix(review): match branch lookup HTTP 404 status
Trecek Oct 27, 2026
e6d8ec1
fix(review): wait for sandbox CI within a finite deadline
Trecek Oct 27, 2026
11846e1
fix(review): validate effective config and final issue payload
Trecek Oct 27, 2026
ed532f8
fix(review): validate session hook settings shapes
Trecek Oct 27, 2026
f30858c
fix(review): narrow pipeline pull request list projection
Trecek Oct 27, 2026
0b951ae
fix(review): assert exact listed-head mismatch
Trecek Oct 27, 2026
00c8ca6
fix(review): reuse recorded pull request close call
Trecek Oct 27, 2026
adaae55
fix(review): cover branch lookup cleanup failure
Trecek Oct 27, 2026
1e39dd5
fix(review): route seed fixture by its required URL field
Trecek Oct 27, 2026
ac088a8
fix(review): narrow redundant lifecycle hook exception tuple
Trecek Oct 27, 2026
4bb529b
refactor(review): trim unused discovery context fields
Trecek Oct 27, 2026
45592cd
fix(review): report missing native transcript once
Trecek Oct 27, 2026
fc773db
fix(review): report missing native path without mismatch noise
Trecek Oct 27, 2026
ba370e0
fix(review): contain native subagent transcript discovery
Trecek Oct 27, 2026
65569c5
refactor(review): validate and pair transcripts in one streaming pass
Trecek Oct 27, 2026
9f0c32b
fix(review): reject malformed UTF-8 in JSONL evidence
Trecek Oct 27, 2026
940689f
test(review): exercise symlink-loop missing check directly
Trecek Oct 27, 2026
f226444
test(review): select session index rows by identity
Trecek Oct 27, 2026
c16c3b3
refactor(review): remove unused envelope session parameter
Trecek Oct 27, 2026
0df758d
refactor(review): remove unused session fixture argument
Trecek Oct 27, 2026
e65ddc2
refactor(review): return fixture root path and centralize index reads
Trecek Oct 27, 2026
cd0a7e7
fix(review): share the exported E2E session log root
Trecek Oct 27, 2026
cdf54a0
refactor(review): share standalone script loading across test suites
Trecek Oct 27, 2026
683be9c
fix(review): seed log-root regression through the shared resolver
Trecek Oct 27, 2026
75eb1ec
fix(review): isolate MCP envelope contract under server fixtures
Trecek Oct 27, 2026
86ec033
fix(review): decode nested structured MCP result envelopes
Trecek Oct 27, 2026
90032a7
fix(review): name transcript context session_id
Trecek Oct 27, 2026
71528f8
fix(review): accept shorthand issue repository refs
Trecek Oct 27, 2026
0d4e5f7
fix(review): stream validation-only transcripts
Trecek Oct 27, 2026
7daf1dc
fix(review): contain transcript path resolution errors
Trecek Oct 27, 2026
ddf8f93
fix(review): type owned session records
Trecek Oct 27, 2026
26e8555
fix(review): retain summary parse failure reasons
Trecek Oct 27, 2026
29b6951
fix(review): fall back from null hook event names
Trecek Oct 27, 2026
5f316af
fix(review): serialize complete lifecycle trace appends
Trecek Oct 27, 2026
143b22c
fix(review): type transcript file-open wrapper
Trecek Oct 27, 2026
c95d4ee
test(review): pin compact ownership proof shape
Trecek Oct 27, 2026
d95b299
fix(review): remove remaining transcript aliases after stale-anchor r…
Trecek Oct 27, 2026
026c8c6
fix(review): classify CI validation failures
Trecek Oct 27, 2026
c0802a4
fix(review): remove unused pipeline home parameters
Trecek Oct 27, 2026
1e938b5
fix(review): copy recipe failures before matching
Trecek Oct 27, 2026
428c0ad
fix(review): recognize current gh 404 output
Trecek Oct 27, 2026
b44f9d7
fix(review): rename PR head validator
Trecek Oct 27, 2026
dcec8d7
fix(review): report PR precondition failures separately
Trecek Oct 27, 2026
e8d3406
fix(review): type ownership proof boundaries
Trecek Oct 27, 2026
7134e3c
fix(review): report JSON parse location
Trecek Oct 27, 2026
db6763c
fix(review): split pull request projections
Trecek Oct 27, 2026
70082c5
fix(review): annotate pipeline test helpers
Trecek Oct 27, 2026
46a1634
fix(review): parse fake gh api arguments
Trecek Oct 27, 2026
930beda
fix(review): scope branch delete faults
Trecek Oct 27, 2026
4fd55cc
fix(review): clarify issue repository constants
Trecek Oct 27, 2026
72c1902
fix(review): share failed pipeline test setup
Trecek Oct 27, 2026
4cd7b2a
fix(review): share retry exhausted envelope fixture
Trecek Oct 27, 2026
3a1c1e3
fix(review): assert exact invalid seed failure
Trecek Oct 27, 2026
4ca048e
fix(review): use the shared session log root in tests
Trecek Oct 27, 2026
d69ed4c
fix(review): use recorded branch delete call
Trecek Oct 27, 2026
029cf61
fix(review): unify expected failure parsing
Trecek Oct 27, 2026
6840e90
fix(review): identify invalid catalog list items
Trecek Oct 27, 2026
a1a6047
fix(review): share pull request test selection
Trecek Oct 27, 2026
c953eeb
fix(review): group execution fixture mappings
Trecek Oct 27, 2026
52d3368
fix(review): explain concurrency capacity
Trecek Oct 27, 2026
a2adeda
fix(review): identify recipe run outcome
Trecek Oct 27, 2026
15aa324
fix(review): name the commands covered by flag registry audit
Trecek Oct 27, 2026
1ba5d47
fix(review): assert builders honor requested output format
Trecek Oct 27, 2026
66fbec2
fix(review): identify command specs in flag failures
Trecek Oct 27, 2026
87014b1
fix(review): remove redundant workflow env assertion
Trecek Oct 27, 2026
caa3afa
fix(review): include plan parts and inventory in validation errors
Trecek Oct 27, 2026
40e1fc1
fix(review): align hook protocol comment with output fields
Trecek Oct 27, 2026
3c91bce
fix(review): cover completion receipt launch contract
Trecek Oct 27, 2026
7dc3a74
fix(review): move and share session log helpers
Trecek Oct 27, 2026
88ed822
fix(review): type the session locator stub
Trecek Oct 27, 2026
94dd972
fix(review): use the public execution identity export
Trecek Oct 27, 2026
821b616
fix(review): compare native agent records as lists
Trecek Oct 27, 2026
e718d0e
fix(review): share captured run materialization setup
Trecek Oct 27, 2026
1f884ba
fix(review): validate truncation markers in all transcripts
Trecek Oct 27, 2026
0ff8c4a
fix(review): explain missing captured tool replies
Trecek Oct 27, 2026
8b66909
fix(review): assert the reentry claim violation
Trecek Oct 27, 2026
f89b6bb
fix(review): spec the GitHub client mock
Trecek Oct 27, 2026
e0640bf
docs(review): preserve script loader isolation contract
Trecek Oct 27, 2026
e36a368
refactor(review): name the shared standalone script loader
Trecek Oct 27, 2026
64ab338
fix(review): retain one fleet output evidence format
Trecek Oct 27, 2026
f2b3d54
fix(review): assert the complete seed validation message
Trecek Oct 27, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .autoskillit/test-filter-manifest.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -246,6 +246,9 @@ tests/execution/fixtures/*.json:
tests/execution/fixtures/*.jsonl:
- execution/

tests/infra/fixtures/e2e_sessions/**:
- infra/

tests/config/golden_configs/*.yaml:
- config/

Expand Down
3 changes: 2 additions & 1 deletion .github/workflows/e2e.yml
Original file line number Diff line number Diff line change
Expand Up @@ -104,12 +104,13 @@ jobs:
env:
MINIMAX_API_KEY: ${{ matrix.kind != 'clean-install' && secrets.MINIMAX_API_KEY || '' }}
E2E_SANDBOX_TOKEN: ${{ matrix.kind != 'clean-install' && secrets.E2E_SANDBOX_TOKEN || '' }}
E2E_IMPLEMENTATION_ISSUE_URL: ${{ vars.E2E_IMPLEMENTATION_ISSUE_URL }}
TEST: ${{ matrix.test }}
run: |
set -euo pipefail
docker_args=(--shm-size=3g)
if [[ '${{ matrix.kind }}' != 'clean-install' ]]; then
docker_args+=(--env MINIMAX_API_KEY --env E2E_SANDBOX_TOKEN)
docker_args+=(--env MINIMAX_API_KEY --env E2E_SANDBOX_TOKEN --env E2E_IMPLEMENTATION_ISSUE_URL)
docker_args+=(--volume "$RUNNER_TEMP/e2e/data:/home/autoskillit/.local/share/autoskillit")
fi
if [[ "$TEST" == 'headless-smoke' ]]; then
Expand Down
157 changes: 146 additions & 11 deletions scripts/e2e/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ This directory holds everything the workflow runs:
| `e2e_catalog.py` | Stdlib loader and validator for the catalog |
| `e2e_select.py` | Picks the tests an event runs (`select`) and judges the required check (`gate`) |
| `e2e_harness.py` | Runs one test (`run`), performs exact sandbox teardown (`cleanup`) and redacts artifacts (`redact`) |
| `e2e_sessions.py` | Captures synchronous lifecycle events, measures concurrency and pairs ownership evidence |
| `autoskillit-config.yaml` | AutoSkillit user-layer config the harness installs |
| `post-e2e-failure.sh` | Appends a failure to the test's tracking issue |

Expand Down Expand Up @@ -40,6 +41,11 @@ model tests:
because MiniMax serves the Anthropic Messages API.
3. Create the `e2e` label in this repository.
4. In branch protection, mark only `e2e-gate` as required — never `select` or `run (…)`.
5. Before publishing a branch with the selectable `implementation` entry, obtain a passing
#5232 sandbox smoke CI run. A maintainer must seed a fixed OPEN sandbox issue describing
a small feature and its test, then set repository variable `E2E_IMPLEMENTATION_ISSUE_URL`
to its canonical `https://github.com/OWNER/REPO/issues/NUMBER` URL. Reuse that issue on
every run. Missing configuration fails preflight; it is not an expected product defect.

## Sandbox repository requirements

Expand Down Expand Up @@ -196,7 +202,11 @@ matrix is always parseable (`{"include":[]}` when nothing is selected).
bypasses sampling because a labelled pull request whose sample fell out would otherwise
never run for that commit.
- **Picking:** tests whose `trigger_paths` match a changed path, up to `MAX_TESTS` (2); when
none match, exactly one test. Ties are ordered by the SHA-256 hex digest of
none match, one eligible ordinary test. A pipeline requires actual size at or above a
threshold and a matching trigger path, even with the `e2e` label. This filter also applies
to fallback; a pipeline-only ineligible pool selects nothing. At most one pipeline is
selected within the total cap. Named workflow dispatch bypasses PR eligibility.
Ties are ordered by the SHA-256 hex digest of
`"{seed}:{name}"`, so the decision is a pure function of the payload and the changed paths.
Patterns use `fnmatch.fnmatchcase`, where `*` also matches `/` (`scripts/e2e/*` matches
every file below `scripts/e2e/`).
Expand All @@ -212,10 +222,53 @@ succeeded or nothing was selected and `run` was skipped. Every other combination
The `run` job uses one repository-wide concurrency group (`group: e2e`, `queue: max`,
`cancel-in-progress: false`) and `max-parallel: 1`, so every selected test of every run
waits its turn and never cancels another. `select` and `e2e-gate` are not queued. A
catalog entry's `peak_sessions` is capped at `MAX_PEAK_SESSIONS` (8): one step's fan-out of
up to six subagents plus the orchestrator and step session. With one test running at a
time, CI stays at or below eight of the roughly twelve concurrent MiniMax sessions.
`clean-install` requires exactly zero `peak_sessions`.
catalog entry's `peak_sessions` is capped at `MAX_PEAK_SESSIONS` (8). For an uncommissioned
pipeline this declares capacity, not a measured fan-out. A step can fan out to up to six
subagents alongside its orchestrator and step session. With runs serialized, CI stays at or
below eight of roughly twelve concurrent MiniMax sessions. `clean-install` requires exactly
zero `peak_sessions`.

The implementation pipeline installs synchronous command hooks for `SessionStart`,
`SessionEnd`, `SubagentStart` and `SubagentStop`, preserving existing hooks and MiniMax
settings. These hooks follow the [Claude lifecycle schema](https://code.claude.com/docs/en/hooks).
The agent hooks use `.+` to require a non-empty agent type: under the default main
agent, Claude Code also fires `SubagentStop` for internal agents with an empty type.
An omitted or empty matcher admits those callbacks; `.+` accepts every non-empty
built-in, custom or namespaced plugin agent name without an allowlist or product-registry
imports in these stdlib-only scripts. Session hooks have no matcher.
Sessions must run with the default main agent. The harness fails preflight if the
user-level settings file has an `agent` key; the fleet command builders pass neither
`--agent` nor `--settings`, and the bundled plugin sets no main agent. Project/local
settings in the sandbox, managed settings and other enabled plugins' main-agent defaults
are not checked. A main agent from those sources makes internal stops carry its name:
they fail coverage as a named stop without a start, without the blank-type `capture:` code.
Claude Code 2.1.275 fixed specific matchers admitting blank-type stops; the image's
2.1.280 pin includes that fix.
Each identity spans its first start through its final end, so compaction and resume yield
a conservative upper bound. Half-open intervals end before simultaneous starts. The
summary reconciles the dispatched root, managed lineage and enumerable native transcripts.
AutoSkillit writes `logs/sessions.jsonl` and each session's `summary.json`; their dispatch
and parent/caller fields are AutoSkillit provenance, separate from Claude Code transcript
and lifecycle hook fields. Empty links do not imply a Claude Code schema guarantee.
Managed sessions are adopted through declared parent fields or receipted `run_skill`,
`complete_run_skill_result` or `recover_run_skill_result` replies in an owned transcript.
Each must have exactly one index row with the run's `dispatch_id`. Parentless sessions
without receipts, missing or ambiguous index rows, and receipt-named sessions absent
from both trace and index fail coverage. Shared dispatch IDs or overlapping lifetimes
never adopt a session. Missing endpoints, identities or owned transcripts fail coverage;
transcript activity never infers a lifetime. The captured pre-fix trace contains a blank-type
stop immediately before a compact-source session start; its producing feature is unproven.
An owned agent record without a named type fails coverage with a `capture:` violation.
A named stop without its start or transcript still fails coverage, as do traces captured
before the matcher was installed. The collector retains delivered callbacks and adds
violations; it does not filter them out. Individual codes are in `out/session-concurrency.json`
under `violations`. A peak exceeding capacity fails validation.

Commission with a successful MiniMax CI run and repeat against the same seed. Inspect
planning, dry-walkthrough, implementation, test, integration, audit GO and open-PR evidence,
plus cleanup and the sandbox CI association. Replace provisional `peak_sessions` with
the observed conservative peak and set `peak_sessions_evidence` to the CI run URL, tested
commit and first-start/final-end method. Expected failures do not satisfy commissioning.

Every harness subprocess has a timeout that fires before the job's: `fleet run` gets the
test's `timeout_sec` plus `HARNESS_GRACE_SEC`, and the job gets that budget in minutes plus
Expand Down Expand Up @@ -259,6 +312,13 @@ digest, id, `sealed` state and coverage status remain in that session's
files survive `pre_review_cleanup`. A container killed before or during collection may
leave no evidence or only partial evidence.

Pipeline evidence includes `session-events.jsonl`, `session-concurrency.json`, compact
`ownership.json`, the seed, effective configuration, fleet stdout/stderr/envelope, captured
PR metadata/files, detached-head test commands, associated CI runs/jobs, and sandbox head-ref
listings before the run and after cleanup (`head-refs-before.*`, `head-refs-after.*`). Collection
occurs inside the container before transcript paths disappear. Incomplete evidence is
retained and fails the scenario; redaction and upload still run after failure.

`clean-install` mounts no home directories and retains evidence under `out/`. Its redactor
receives the same required `--secret-env` names with variables unset and skips the absent
home-log tree. Redaction and upload still run after test failures.
Expand Down Expand Up @@ -323,6 +383,57 @@ row and close the bug. Fleet translates the L3 result's
`reason` into the failed command envelope's `error` and `user_visible_message` fields;
the harness matches that envelope rather than the recipe payload's `failed_step`/`reason`.

For the implementation pipeline, the project-owned
[`DispatchCompleted.to_envelope`](../../src/autoskillit/fleet/campaign_state/state_outcomes.py)
must report success, `dispatch_status="success"` and root/dispatch identities. The nested
[`implementation` terminal](../../src/autoskillit/recipes/implementation.yaml) must have
`reason="implementation_complete"`. Successful no-change and already-done terminals remain
valid recipe behavior but fail this scenario. No PR URL is required in the envelope.

The harness resolves the configured seed once and dispatches its exact URL and title/body
task once. Effective project-aware config comes from `autoskillit config show`, whose
implementation calls `load_config(Path.cwd())`; both configured and sandbox default branches
must be `main`. No caller `base_branch` ingredient is sent. Sandbox setup and baseline tests
must succeed before launch.

Paired root transcript tool calls/results prove the created branch; paired root/managed
claim results prove any added claim label. The collector accepts the direct
`mcp__autoskillit__` and plugin `mcp__plugin_autoskillit_autoskillit__` namespaces;
a tracked tool under any other namespace fails the proof. Claude Code persisted
transcripts carry replies in row-level `mcpMeta.structuredContent` when `mcpMeta`
is present, otherwise in JSON block content. Malformed metadata or metadata on a
row with several results fails the paired resource call. Hook display text never
replaces unusable metadata. The pretty-output hook emits `updatedMCPToolOutput`;
`_result_object` remains the shared envelope decoder, and metadata presence alone
does not identify whether the hook ran. These shapes are pinned by
`tests/infra/fixtures/e2e_sessions/` from Claude Code 2.1.280; re-capture them when
an otherwise successful call reports a failed branch or claim result.

Discovery paginates through the PR baseline
and leaves unrelated PRs untouched. Require exactly one owned open, unmerged sandbox PR
targeting `main`, without auto-merge, with the canonical seed URL and non-removed changes
under both `sandbox/` and `tests/`. Fetch and detach at the captured head before setup and
tests. Require a successful associated sandbox `CI` run and `test` job whose nested PR
association has the exact branch revision; an outer merge SHA is not the branch SHA.
Finally re-read the head and state to reject movement during validation.

Teardown runs after validation, failure or exception. It independently collects metrics,
closes owned open PRs, deletes evidenced branches (including branches without a PR), and
removes only the effective claim label proven added by this run. The seed remains OPEN
and other labels are preserved. Missing ownership proof preserves uncertain resources and
fails cleanup. Cleanup errors remain fatal alongside the original failure.

Branch reconciliation lists the sandbox's branches before the run and after cleanup.
A branch created during the run that `ownership.json` does not explain is preserved
and fails cleanup; baseline branches are excluded from unexplained-branch reports.
Branches named by the ownership proof retain the existing cleanup rule. The original
API stdout saved as `head-refs-before.stdout.txt` and `head-refs-after.stdout.txt` supplies
the in-memory snapshots; accompanying stderr and command records diagnose failed requests.
This check only detects unexplained branches: the creating run records the failure once,
and the branch and any PR on it must be removed by hand. The next run includes it in
its baseline. A manual run overlapping CI can trigger the check because CI serialization
does not cover local runs.

## Catalog schema

```json
Expand Down Expand Up @@ -351,15 +462,39 @@ the harness matches that envelope rather than the recipe payload's `failed_step`
```

`recipe`, `ingredients` and `expected_pull_request_state` are required for `recipe` tests
and forbidden for `canary` and `clean-install` tests. `expected_failures` is optional for
`clean-install` and recipe tests. Clean-install rows use the exact doctor `severity`, `check`
and `message`; recipe rows require `severity: error`, `check` and `message` matching the failed
envelope. Every row includes an open bug issue URL
(`https://github.com/TalonT-Org/AutoSkillit/issues/<number>`). Duplicate diagnostic identities
and unknown keys are rejected. `peak_sessions` must be
and forbidden for `canary` and `clean-install` tests. Optional recipe-only fields are
`pipeline` (boolean, default false), `issue_url_env` (seed variable name),
`required_changed_paths` (nonempty relative prefixes), `test_command` (nonempty argv), and
`peak_sessions_evidence` (nonempty provenance string). Environment-owned seeds cannot also
specify literal `task` or `issue_url` ingredients. Path prefixes cannot be absolute or
traverse parents. For example, the implementation entry uses:

```json
{"pipeline": true, "issue_url_env": "E2E_IMPLEMENTATION_ISSUE_URL",
"required_changed_paths": ["sandbox/", "tests/"], "test_command": ["task", "test-check"]}
```

The recipe_fixture field names a staged YAML fixture for the headless smoke scenario.

The expected_failures field is optional for recipes and clean-install. Pipeline rows
contain exactly check, message, issue; ordinary recipe and clean-install rows also
contain severity. Ordinary recipes require severity: error and match the failed
envelope's error and user_visible_message; clean-install allows error or warning
and matches doctor diagnostics. Pipeline rows match structured implementation findings.
Every row links to an open AutoSkillit bug issue. Duplicate diagnostic identities and
unknown keys are rejected. The peak_sessions value must be
1–8 for model tests and exactly zero for clean-install. `timeout_sec` must keep the
derived job timeout within 360 minutes.

Recipe expectations match exact stable `(check, message)` product findings linked to real
bug issues. Setup, discovery, malformed evidence and cleanup failures cannot be admitted.
Unknown findings and absent expected findings fail. Each recipe run records `outcome` in
`result.json` as `passed`, `expected_failure` (with `passed=false`), or `failed`. A linked
known defect may
use the accepted CLI exit convention, but only `passed` with no expected findings counts
as successful commissioning. File or deduplicate observed product bugs separately; do not
predeclare a provider defect or replace the MiniMax settings-file route.

### Adding a test

1. Add the entry to `catalog.json` with a realistic `peak_sessions` and `timeout_sec`.
Expand Down
3 changes: 3 additions & 0 deletions scripts/e2e/autoskillit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,9 @@
quota_guard:
enabled: false

branching:
default_base_branch: main

features:
fleet: true
fleet_headless_run: true
Expand Down
29 changes: 29 additions & 0 deletions scripts/e2e/catalog.json
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,35 @@
"recipe_fixture": "sandbox-smoke.yaml",
"ingredients": {},
"expected_pull_request_state": "closed"
},
{
"name": "implementation",
"kind": "recipe",
"pipeline": true,
"peak_sessions": 8,
"timeout_sec": 7200,
"trigger_paths": [
"src/autoskillit/recipes/*",
"src/autoskillit/skills/*",
"src/autoskillit/skills_extended/*",
"src/autoskillit/server/*",
"src/autoskillit/execution/*"
],
"recipe": "implementation",
"ingredients": {
"source_dir": "/workspace/sandbox",
"open_pr": "true",
"auto_merge": "false",
"arch_lenses": "false",
"audit_impl": "true"
},
"expected_pull_request_state": "open",
"issue_url_env": "E2E_IMPLEMENTATION_ISSUE_URL",
"required_changed_paths": [
"sandbox/",
"tests/"
],
"test_command": ["task", "test-check"]
}
]
}
Loading
Loading