Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
58 changes: 41 additions & 17 deletions scripts/e2e/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -224,18 +224,36 @@ For model tests, the container's AutoSkillit data directory (`~/.local/share/aut
from the runner, so session logs survive a killed container; the harness also reads this
log tree after fleet runs. Those files belong to uid 1000
and are often `0600`, so a second container run copies `out/` and `data/logs/` into
`upload/` world-readable with both secrets replaced by `[REDACTED]`, skips symlinks, and
fails if any secret survives. Only `upload/` is uploaded, and only when redaction
succeeded.

When a test does not pass, the harness preserves plans from each pipeline clone under
`out/pipeline-artifacts/<clone>/<make-plan|rectify>/` and Claude Code JSONL under
`out/transcripts/`, retaining relative paths. The transcripts include the orchestrator,
skill sessions and their subagents; the orchestrator's transcript holds each raw
`bind_plan_set` response and its `plan_parts` argument. These files go through the same
redaction and upload steps as the other output artifacts. Collection also runs for an
expected failure or a harness exception. A container killed before or during collection
may leave no evidence or only partial evidence.
`upload/` world-readable, skipping symlinks. The workflow supplies `MINIMAX_API_KEY` and
`E2E_SANDBOX_TOKEN`; the redactor replaces their nonempty literal byte values in file
contents with `[REDACTED]`, re-reads all uploaded file contents, and fails closed if a
supplied literal survives. Encoded or transformed values, unknown credentials and file
names are outside this check. Only `upload/` is uploaded, and only when redaction succeeded.

Every test copies the stage directories selected by `PIPELINE_EVIDENCE` in
`e2e_harness.py`, on every outcome: pass, failure, expected failure or harness exception.
Sources are the `.autoskillit/temp` of the sandbox, each pipeline clone and each surviving
worktree. They land under `out/pipeline-artifacts/<path under /workspace>/<stage dir>/`.
Ledgers, kitchen and guard state, counters and the worktrees sidecar are not copied.
The `pipeline_files` evidence counter records copied stage files.

Claude Code `*.jsonl` and `*.txt` files under the projects tree land in
`out/transcripts/`, retaining relative paths for orchestrator, skill and subagent sessions.
The text-file selection includes `tool-results/*.txt`: a transcript's `bind_plan_set`
response may be a pointer to one of these retained payloads rather than inline bytes.
All collected files go through the same redaction and upload steps as other artifacts.

`evidence.unretained` lists existing files under any `.autoskillit/temp` mentioned by a
`sessions.jsonl` row that were not copied. An entry for a commissioning stage means that
stage's bytes are missing. No `evidence` key means nothing was found to collect.
Uploaded names escape `~` and upload-forbidden characters as `~XX`, so `sha256:` becomes
`sha256~3a`; paths quoted inside session logs and transcripts keep their original characters.

A merged run's worktree temp is deleted by `merge_worktree`. The implement step's binding
digest, id, `sealed` state and coverage status remain in that session's
`verified_plan_set_preflight` in `logs/sessions.jsonl`. Only the final review iteration's
files survive `pre_review_cleanup`. A container killed before or during collection may
leave no evidence or only partial evidence.

`clean-install` mounts no home directories and retains evidence under `out/`. Its redactor
receives the same required `--secret-env` names with variables unset and skips the absent
Expand Down Expand Up @@ -381,13 +399,19 @@ After the plan-set changes have merged, perform this check before CI commissioni
and the evidence collector. Build and run that revision with the Docker
invocation above, selecting `--test implementation`. Use the same revision for
commissioning and record the model.
2. If the run reaches implementation, the first bind succeeded. If the test does not
pass, inspect the orchestrator's `bind_plan_set` results in `out/transcripts/`.
2. Inspect the orchestrator's `bind_plan_set` results in `out/transcripts/` regardless of
outcome, including `tool-results/` when a transcript holds only a pointer. Inspect the
sealed records under
`out/pipeline-artifacts/autoskillit-runs/<clone>/plan-set-authority/`.
An observed `success: true` on the first or second bind is the pass condition.
A `coverage_failed` result explains the rejection in `coverage_gaps`, and
`out/pipeline-artifacts/` retains the rejected plan. Missing evidence is inconclusive
and does not satisfy this gate: require an observed successful bind. When a run
replans, also confirm the second bind's `plan_parts` names only the replacement plan.
`out/pipeline-artifacts/` retains the rejected plan. Treat an `evidence.unretained`
entry for a commissioning stage as missing evidence. An empty list covers existing
session-referenced temp files only; unmentioned producers depend on `PIPELINE_EVIDENCE`,
so inspect each required commissioning payload directly before signing off.
Missing evidence is inconclusive and does not satisfy this gate: require an observed
successful bind. When a run replans, also confirm the second bind's `plan_parts` names
only the replacement plan.
3. Record the model, plan path and bind result in issue #5259's body using
`gh issue edit --body-file`, never as a comment. Only an observed successful bind
unblocks `gh workflow run e2e.yml -f tests=implementation`.
Expand Down
203 changes: 157 additions & 46 deletions scripts/e2e/e2e_harness.py
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,32 @@
CONFIG_TEMPLATE = Path(__file__).with_name("autoskillit-config.yaml")
SANDBOX_CLONE = Path("/workspace/sandbox")
PIPELINE_RUNS_DIR = "autoskillit-runs"
PIPELINE_WORKTREES_DIR = "worktrees"
# Select stage artifacts by directory and file name; runtime state stays behind.
PIPELINE_EVIDENCE: tuple[tuple[str, str], ...] = (
("make-plan", "*"),
("rectify", "*"),
("plan-set-authority", "*"),
("audit-admission", "*.json"), # ledger.sqlite3 shares this directory
("audit-disposition", "*.json"),
("audit-impl", "*"),
("review-approach", "*"),
("arch-lens-*", "*"),
("prepare-pr", "*"),
("compose-pr", "*"),
("review-pr", "*"),
("resolve-review", "*"),
("diagnose-ci", "*"),
("resolve-failures", "*"),
("resolve-merge-conflicts", "*"),
("retry-worktree", "*"),
)
# Rejected in uploaded paths by actions/upload-artifact v4.6.2 (e2e.yml).
UPLOAD_FORBIDDEN_CHARACTERS = '":<>|*?\r\n'
_UPLOAD_ESCAPES = str.maketrans({c: f"~{ord(c):02x}" for c in "~" + UPLOAD_FORBIDDEN_CHARACTERS})
_TEMP_PATH = re.compile(
r"(?:/[^\s\"'`\\<>|*?,;()\[\]{}]*)?\.autoskillit/temp/[^\s\"'`\\<>|*?,;()\[\]{}]+"
)

_SCRUBBED_ENV = frozenset({"CLAUDE_CODE_OAUTH_TOKEN", *SECRET_ENV})
_STDERR_TAIL_CHARS = 2000
Expand Down Expand Up @@ -1105,70 +1131,148 @@ def _run_test(
token = env["E2E_SANDBOX_TOKEN"]
recipe_env = {
**scrubbed,
"AUTOSKILLIT_LOG_DIR": str(home / ".local" / "share" / "autoskillit" / "logs"),
"AUTOSKILLIT_LOG_DIR": str(autoskillit_log_dir(home)),
}
return run_recipe(test, catalog.sandbox_repository, token, out, recipe_env, runner)


def _copy_failure_files(
source: Path, dest: Path, boundary: Path, pattern: str, errors: list[str]
) -> int:
def autoskillit_log_dir(home: Path) -> Path:
return home / ".local" / "share" / "autoskillit" / "logs"


def _copy_evidence_files(
source: Path, dest: Path, boundary: Path, patterns: tuple[str, ...], errors: list[str]
) -> list[Path]:
relative = source.relative_to(boundary)
try:
if any(
boundary.joinpath(*relative.parts[:depth]).is_symlink()
for depth in range(len(relative.parts) + 1)
):
return 0
return []
if not source.exists():
return 0
return []
except OSError as exc:
errors.append(f"{source}: {exc}")
return 0
copied = 0
return []
copied = []
for root, dirs, files in os.walk(
source, followlinks=False, onerror=lambda exc: errors.append(f"{source}: {exc}")
):
dirs.sort()
for name in sorted(fnmatch.filter(files, pattern)):
for name in sorted(files):
if not any(fnmatch.fnmatch(name, pattern) for pattern in patterns):
continue
path = Path(root) / name
try:
if path.is_symlink() or not path.is_file():
continue
target = dest / path.relative_to(source)
target.parent.mkdir(parents=True, exist_ok=True)
shutil.copyfile(path, target)
copied += 1
copied.append(path)
except OSError as exc:
errors.append(f"{path}: {exc}")
return copied


def collect_failure_evidence(out: Path, home: Path) -> dict[str, Any]:
"""Keep pipeline plans and Claude transcripts for an unsuccessful E2E run."""
errors: list[str] = []
def _evidence_roots(errors: list[str]) -> list[Path]:
runs = SANDBOX_CLONE.parent / PIPELINE_RUNS_DIR
roots = [SANDBOX_CLONE]
for parent, skip in ((runs, PIPELINE_WORKTREES_DIR), (runs / PIPELINE_WORKTREES_DIR, None)):
try:
linked = runs.is_symlink() or parent.is_symlink()
entries = (
[] if linked else sorted(entry for entry in parent.iterdir() if entry.is_dir())
)
except FileNotFoundError:
entries = []
except OSError as exc:
errors.append(f"{parent}: {exc}")
entries = []
roots += [entry for entry in entries if entry.name != skip]
return roots


def _collect_stage_files(out: Path, roots: list[Path], errors: list[str]) -> list[Path]:
workspace = SANDBOX_CLONE.parent
copied = []
for root in roots:
temp = root / ".autoskillit" / "temp"
try:
# iterdir propagates scan errors instead of hiding unreadable directories.
entries = sorted(temp.iterdir())
except FileNotFoundError:
continue
except OSError as exc:
errors.append(f"{temp}: {exc}")
continue
for entry in entries:
for stage_pattern, file_pattern in PIPELINE_EVIDENCE:
if fnmatch.fnmatch(entry.name, stage_pattern):
copied.extend(
_copy_evidence_files(
entry,
out / "pipeline-artifacts" / root.relative_to(workspace) / entry.name,
workspace,
(file_pattern,),
errors,
)
)
return copied


def _recorded_temp_paths(line: str) -> list[Path]:
try:
row = json.loads(line)
except json.JSONDecodeError:
return []
cwd = row.get("cwd") if isinstance(row, dict) else None
paths = []
for match in _TEMP_PATH.finditer(line):
path = Path(match.group())
if path.is_absolute():
paths.append(path)
elif isinstance(cwd, str):
paths.append(Path(cwd) / path)
return paths


def _unretained_references(index: Path, retained: set[str], errors: list[str]) -> list[str]:
try:
clones = [] if runs.is_symlink() else sorted(runs.iterdir())
lines = index.read_text(encoding="utf-8").splitlines()
except FileNotFoundError:
clones = []
except OSError as exc:
errors.append(f"{runs}: {exc}")
clones = []
plan_files = 0
for clone in clones:
for kind in ("make-plan", "rectify"):
plan_files += _copy_failure_files(
clone / ".autoskillit" / "temp" / kind,
out / "pipeline-artifacts" / clone.name / kind,
clone,
"*",
errors,
)
transcripts = _copy_failure_files(
home / ".claude" / "projects", out / "transcripts", home, "*.jsonl", errors
return []
except (OSError, UnicodeDecodeError) as exc:
errors.append(f"{index}: {exc}")
return []
return sorted(
{
str(path)
for line in lines
for path in _recorded_temp_paths(line)
if path.is_file() and os.path.realpath(path) not in retained
}
)
return {"plan_files": plan_files, "transcripts": transcripts, "errors": errors}


def collect_run_evidence(out: Path, home: Path) -> dict[str, Any]:
"""Copy every run's stage artifacts and Claude transcripts into *out*, whatever the verdict."""
errors: list[str] = []
copied = _collect_stage_files(out, _evidence_roots(errors), errors)
projects = home / ".claude" / "projects"
transcripts = _copy_evidence_files(
projects, out / "transcripts", home, ("*.jsonl", "*.txt"), errors
)
unretained = _unretained_references(
autoskillit_log_dir(home) / "sessions.jsonl", {os.path.realpath(p) for p in copied}, errors
)
return {
"pipeline_files": len(copied),
"transcripts": len(transcripts),
"unretained": unretained,
"errors": errors,
}


def run_test(
Expand Down Expand Up @@ -1197,18 +1301,18 @@ def run_test(
result.update(
outcome=outcome, passed=outcome == "passed", expected_findings=matched_findings
)
if not result["passed"]:
try:
evidence = collect_failure_evidence(out, home)
except Exception as exc:
evidence = {
"plan_files": 0,
"transcripts": 0,
"errors": [f"{type(exc).__name__}: {exc}"],
"exception": traceback.format_exc(),
}
if any(evidence.values()):
result["evidence"] = evidence
try:
evidence = collect_run_evidence(out, home)
except Exception as exc:
evidence = {
"pipeline_files": 0,
"transcripts": 0,
"unretained": [],
"errors": [f"{type(exc).__name__}: {exc}"],
"exception": traceback.format_exc(),
}
if any(evidence.values()):
result["evidence"] = evidence
(out / "result.json").write_text(json.dumps(result, indent=2) + "\n", encoding="utf-8")
return failures

Expand All @@ -1224,16 +1328,23 @@ def _copy_redacted(source: Path, target: Path, needles: Sequence[bytes]) -> None
target.chmod(0o644)


def upload_name(name: str) -> str:
"""Escape ``~`` and upload-forbidden characters as ``~XX`` without name collisions."""
return name.translate(_UPLOAD_ESCAPES)


def _copy_tree_redacted(source: Path, target: Path, needles: Sequence[bytes]) -> None:
for root, _dirs, files in os.walk(source, followlinks=False):
root_path = Path(root)
target_dir = target / root_path.relative_to(source)
target_dir = target.joinpath(
*(upload_name(part) for part in root_path.relative_to(source).parts)
)
target_dir.mkdir(parents=True, exist_ok=True)
target_dir.chmod(0o755)
for name in files:
path = root_path / name
if not path.is_symlink() and path.is_file():
_copy_redacted(path, target_dir / name, needles)
_copy_redacted(path, target_dir / upload_name(name), needles)


def _find_leak(dest: Path, needles: Sequence[bytes]) -> Path | None:
Expand All @@ -1257,7 +1368,7 @@ def redact_tree(sources: Sequence[Path], dest: Path, secrets: Sequence[str]) ->
for source in sources:
if source.is_symlink() or not source.exists():
continue
target = dest / source.name
target = dest / upload_name(source.name)
if source.is_dir():
_copy_tree_redacted(source, target, needles)
else:
Expand Down
Loading
Loading