From 862916196c2d7fe702c3cc12adacd4fb2c6378ef Mon Sep 17 00:00:00 2001 From: tweinmann Date: Tue, 27 Oct 2026 22:08:24 +0000 Subject: [PATCH] Run level 2 when a pull request is ready, by path, and in full on main The first Phase 9 PR run took about 12 minutes on every push, most of it smoke-agent waiting for Flux's poll and for Helm to time out a broken upgrade. Now: - Pull requests run level 1 on every push, and level 2 when they are opened or reopened as ready, leave draft, or carry the label level-2; level-2-full runs everything in full. A new push cancels the run it replaces. - .github/scripts/select-smokes.sh picks the smoke tests the changed paths need; docs and planning need none, unlisted paths need all. - smoke-agent asks Flux to fetch each new artifact at once, and skips the fix-after-failure step with SMOKE_FULL=0 (pull requests). - main runs everything in full after each merge and opens or comments on a ci-main-red issue when it fails; release.yml publishes only a commit whose CI run on main succeeded. - just test keeps Go's caches in .cache/go via actions/cache; gofmt checks tracked files only. Spec delta for dev-environment, design decision 8 with the measured job times. Locally smoke-agent takes 48 s with SMOKE_FULL=0 and 380 s in full (was about 10 minutes). Co-Authored-By: Claude Opus 5.5 --- .github/scripts/select-smokes.sh | 45 ++++++++ .github/workflows/ci.yml | 102 ++++++++++++++++-- .github/workflows/release.yml | 22 ++++ .gitignore | 3 + CLAUDE.md | 10 +- hack/smoke/agent.sh | 23 +++- justfile | 3 +- .../changes/phase-9-agent-interface/design.md | 40 +++++++ .../phase-9-agent-interface/proposal.md | 6 ++ .../specs/dev-environment/spec.md | 33 ++++++ .../changes/phase-9-agent-interface/tasks.md | 12 ++- openspec/config.yaml | 3 +- 12 files changed, 287 insertions(+), 15 deletions(-) create mode 100755 .github/scripts/select-smokes.sh create mode 100644 openspec/changes/phase-9-agent-interface/specs/dev-environment/spec.md diff --git a/.github/scripts/select-smokes.sh b/.github/scripts/select-smokes.sh new file mode 100755 index 0000000..17c601c --- /dev/null +++ b/.github/scripts/select-smokes.sh @@ -0,0 +1,45 @@ +#!/usr/bin/env bash +# Prints the level-2 smoke tests a change needs, as a JSON array for the CI matrix. Reads the +# changed paths from stdin, one per line. Runs on the CI runner, not in the devcontainer. +# +# A path that is documentation or planning needs none. A path listed below needs the tests +# named for it. Anything else needs all of them: an unlisted path is a reason to test, never a +# reason to skip. +set -euo pipefail + +all=(smoke-secrets smoke-chart smoke-init smoke-apps smoke-agent) +declare -A need=() + +for_path() { + case "$1" in + # Planning, documentation and examples (the examples are validated by `just test`). + openspec/* | *.md | LICENSE | NOTICE | examples/* | .claude/* | schema/*) ;; + # Workflows that are checked by running them, not by the smoke tests. + .github/workflows/build.yml | .github/workflows/release.yml) ;; + # Code no smoke test reaches: the admin UI and the APIs of Cloudflare and GitHub. + internal/server/* | internal/cloudflare/* | internal/github/*) ;; + hack/smoke/secret-expansion*) echo smoke-secrets ;; + hack/smoke/chart.sh) echo smoke-chart ;; + charts/* | internal/chart/*) echo smoke-chart smoke-apps smoke-agent ;; + hack/smoke/init.sh | internal/cli/init.go | internal/cli/init_test.go) echo smoke-init ;; + hack/smoke/apps.sh) echo smoke-apps ;; + hack/smoke/agent.* | internal/cli/json* | internal/cli/open* | internal/cli/wait* | internal/cli/tenant_test.go) + echo smoke-agent ;; + internal/cli/* | internal/ops/* | internal/progress/* | internal/secrets/* | internal/hostcfg/*) + echo smoke-apps smoke-agent ;; + *) echo "${all[@]}" ;; + esac +} + +while IFS= read -r path; do + [[ -n "$path" ]] || continue + for smoke in $(for_path "$path"); do + need[$smoke]=1 + done +done + +out=() +for smoke in "${all[@]}"; do + [[ -n "${need[$smoke]:-}" ]] && out+=("\"$smoke\"") +done +(IFS=,; echo "[${out[*]}]") diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index f574a4f..6771203 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -1,40 +1,105 @@ # Level-1 checks, and level-2 smoke tests against k3d. Both run inside the devcontainer image, so # CI uses exactly the tool versions pinned in .devcontainer/Dockerfile. +# +# Level 1 runs on every push. Level 2 is slow, so a pull request runs it when it is ready, not on +# every push while it is being worked on: +# - when a pull request is opened or reopened as ready, or leaves draft; +# - on every push while it carries the label `level-2`, and when the label is added, also as a +# draft; +# - with the label `level-2-full`, the full set, as on main. +# A pull request runs only the smoke tests its changed paths need (.github/scripts/ +# select-smokes.sh), and skips the slowest step of smoke-agent (SMOKE_FULL=0). main runs all of +# them in full after every merge, and opens an issue when that fails; release.yml publishes only a +# commit whose run on main succeeded. name: ci on: push: branches: [main] pull_request: + types: [opened, synchronize, reopened, ready_for_review, labeled] permissions: contents: read +# A new push to a pull request cancels the run it replaces. A label that has nothing to do with CI +# gets a group of its own, so that adding it cancels nothing; runs on main are never cancelled. +concurrency: + group: >- + ci-${{ github.event.pull_request.number || github.sha }}-${{ + (github.event.action == 'labeled' && !startsWith(github.event.label.name, 'level-2')) && github.run_id || 'run' }} + cancel-in-progress: ${{ github.event_name == 'pull_request' }} + jobs: test: + # A label that has nothing to do with CI changes no code. A level-2 label runs it again: its run + # may have cancelled the one of the push before it. + if: github.event.action != 'labeled' || startsWith(github.event.label.name, 'level-2') # The same architecture as the Mac and the mini. runs-on: ubuntu-24.04-arm timeout-minutes: 20 steps: - uses: actions/checkout@v7 + # Go's build and module cache, kept in the workspace so the devcontainer sees it. + - uses: actions/cache@v4 + with: + path: .cache/go + key: go-${{ runner.arch }}-${{ hashFiles('go.sum', '.devcontainer/Dockerfile') }}-${{ github.sha }} + restore-keys: go-${{ runner.arch }}-${{ hashFiles('go.sum', '.devcontainer/Dockerfile') }}- - name: just test (in the devcontainer) uses: devcontainers/ci@v0.3 with: push: never - runCmd: just test + runCmd: | + export GOCACHE="$PWD/.cache/go/build" GOMODCACHE="$PWD/.cache/go/mod" + just test + + # Which smoke tests to run, and how: none, those the changed paths need, or all in full. + plan: + runs-on: ubuntu-24.04-arm + timeout-minutes: 5 + outputs: + smokes: ${{ steps.plan.outputs.smokes }} + full: ${{ steps.plan.outputs.full }} + steps: + - uses: actions/checkout@v7 + with: + fetch-depth: 0 + - id: plan + env: + EVENT: ${{ github.event_name }} + ACTION: ${{ github.event.action }} + DRAFT: ${{ github.event.pull_request.draft }} + LABEL: ${{ github.event.label.name }} + LABELS: ${{ toJSON(github.event.pull_request.labels.*.name) }} + BASE: ${{ github.event.pull_request.base.sha }} + run: | + all='["smoke-secrets","smoke-chart","smoke-init","smoke-apps","smoke-agent"]' + has() { jq -e --arg l "$1" 'index($l) != null' <<<"$LABELS" >/dev/null; } + smokes='[]' full=0 + if [[ "$EVENT" == push ]]; then + smokes="$all" full=1 + elif has level-2-full && [[ "$ACTION" != labeled || "$LABEL" == level-2-full ]]; then + smokes="$all" full=1 + elif { has level-2 && [[ "$ACTION" == synchronize || "$LABEL" == level-2 ]]; } || + { [[ "$DRAFT" != true && "$ACTION" =~ ^(opened|reopened|ready_for_review)$ ]]; }; then + smokes="$(git diff --name-only "$BASE" HEAD | .github/scripts/select-smokes.sh)" + fi + echo "smokes=$smokes" >>"$GITHUB_OUTPUT" + echo "full=$full" >>"$GITHUB_OUTPUT" + echo "level 2: $smokes (full=$full)" | tee -a "$GITHUB_STEP_SUMMARY" # Level-2 checks against a k3d cluster inside the devcontainer, as locally. Each smoke test # gets a fresh cluster of its own, so they run side by side; none depends on another's state. - # Pull requests only: main receives what a pull request already checked, so after a merge - # level 1 is enough. cluster: - if: github.event_name == 'pull_request' + needs: plan + if: needs.plan.outputs.smokes != '[]' runs-on: ubuntu-24.04-arm timeout-minutes: 30 strategy: fail-fast: false matrix: - smoke: [smoke-secrets, smoke-chart, smoke-init, smoke-apps, smoke-agent] + smoke: ${{ fromJSON(needs.plan.outputs.smokes) }} name: cluster (${{ matrix.smoke }}) steps: - uses: actions/checkout@v7 @@ -45,4 +110,29 @@ jobs: runCmd: | set -e just cluster-up - just ${{ matrix.smoke }} + SMOKE_FULL=${{ needs.plan.outputs.full }} just ${{ matrix.smoke }} + + # A failure on main is easy to miss in a mailbox: keep it as an issue until someone closes it. + # One open issue collects every failure; each one adds a comment. + report: + needs: [test, plan, cluster] + if: failure() && github.event_name == 'push' + runs-on: ubuntu-24.04-arm + timeout-minutes: 5 + permissions: + issues: write + steps: + - env: + GH_TOKEN: ${{ github.token }} + GH_REPO: ${{ github.repository }} + RUN: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} + run: | + body="CI failed on main at ${GITHUB_SHA::12}: $RUN" + issue="$(gh issue list --label ci-main-red --state open --json number --jq '.[0].number // empty')" + if [[ -n "$issue" ]]; then + gh issue comment "$issue" --body "$body" + else + gh label create ci-main-red --color B60205 --description "CI on main failed" 2>/dev/null || true + gh issue create --title "CI on main failed" --label ci-main-red \ + --body "$body"$'\n\n'"Close this issue once main is green again." + fi diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 3b61404..1e64ea3 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -13,7 +13,29 @@ permissions: packages: write jobs: + # A release ships what main tested in full: the CI run of the tagged commit on main, with every + # smoke test, must have succeeded. Pull requests skip the slowest checks; this is where they + # count. + gate: + runs-on: ubuntu-24.04-arm + timeout-minutes: 5 + permissions: + actions: read + steps: + - env: + GH_TOKEN: ${{ github.token }} + GH_REPO: ${{ github.repository }} + run: | + conclusion="$(gh api "repos/$GH_REPO/actions/workflows/ci.yml/runs?head_sha=$GITHUB_SHA&event=push&branch=main" \ + --jq '.workflow_runs | sort_by(.run_started_at) | last | .conclusion // .status // "none"')" + if [[ "$conclusion" != success ]]; then + echo "::error::CI on main for ${GITHUB_SHA::12} is '$conclusion', not success. Tag a commit of main whose CI run succeeded, or wait for it to finish." + exit 1 + fi + echo "CI on main for ${GITHUB_SHA::12} succeeded" + release: + needs: gate runs-on: ubuntu-24.04-arm timeout-minutes: 30 env: diff --git a/.gitignore b/.gitignore index cca9735..7942c45 100644 --- a/.gitignore +++ b/.gitignore @@ -4,3 +4,6 @@ # Build output /bin/ /dist/ + +# Go build and module cache of CI (the test job keeps it in the workspace) +/.cache/ diff --git a/CLAUDE.md b/CLAUDE.md index 5a6f84a..74646e0 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -135,6 +135,14 @@ All development happens inside the devcontainer on Docker Desktop. It runs its o daemon (Docker-in-Docker), which also hosts the k3d cluster. Nothing is installed on the host Mac. Tool versions are pinned in `.devcontainer/Dockerfile`. +CI runs level 1 on every push. Level 2 runs when a pull request is opened or reopened as ready +or leaves draft, and on every push while it carries the label `level-2`; only the smoke tests its +changed paths need (`.github/scripts/select-smokes.sh`), with `SMOKE_FULL=0`. The label +`level-2-full` and every merge to main run all of them in full. A red main opens an issue +labelled `ci-main-red`, and `release.yml` publishes only a commit whose CI run on main succeeded. +A new path that is neither documentation nor listed in `select-smokes.sh` runs everything; list +it there when it belongs to one smoke test. + | Command | Run from | Purpose | |---|---|---| | `just test` | devcontainer | level-1 checks, same as CI (gofmt, vet, `go test -race`, helm lint, kubeconform, examples) | @@ -154,7 +162,7 @@ Mac. Tool versions are pinned in `.devcontainer/Dockerfile`. | `just smoke-chart` | devcontainer | install `examples/hello` with the chart, check the Phase 2 acceptance criteria (needs Docker Hub) | | `just smoke-init` | devcontainer | Phase 3 acceptance: init twice, re-push, routing and `stripPrefix` through Traefik (run `just cluster-reset` first; needs network) | | `just smoke-apps` | devcontainer | Phase 4: `shelf app add`/`rm`, rollout by polling, tampered artifact refused, restore from backup (needs network) | -| `just smoke-agent` | devcontainer | Phase 9: `--json` documents, `--wait --commit` (Ready only after the commit rolled out, Failed for an image that cannot be pulled, TimedOut for a commit that never arrives, Ready again for a fix after a failure; takes up to 10 minutes because Helm times out the broken upgrade first) and `shelf app open` with the app's Host and `stripPrefix` (needs Docker Hub) | +| `just smoke-agent` | devcontainer | Phase 9: `--json` documents, `--wait --commit` (Ready only after the commit rolled out, Failed for an image that cannot be pulled, TimedOut for a commit that never arrives; with `SMOKE_FULL=1`, the default, also Ready again for a fix after a failure, which takes about 5 more minutes because Helm times out the broken upgrade first) and `shelf app open` with the app's Host and `stripPrefix` (needs Docker Hub) | | `just smoke-tenant ` | devcontainer | Phase 4 acceptance with a real tenant repo and GHCR; asks for the GHCR login, waits for a push | | `just smoke-expose ` | devcontainer | Phase 8b acceptance: the app gets the Cloudflare connection `smoke`, its own tunnel, DNS record and HTTPS, then is taken off again; asks for the Cloudflare API token (set `CF_ACCOUNT_ID` if it sees several accounts); needs a cluster with a host suffix | | `just smoke-quick [app]` | devcontainer | Phase 8d acceptance: `examples/hello` as `hello-quick` with `--quick` gets a quick tunnel, answers with its own Host, gets a new name after a restart, `--private`/`--quick`, rm; keeps the cluster's host suffix; opens a public trycloudflare.com tunnel for a few minutes | diff --git a/hack/smoke/agent.sh b/hack/smoke/agent.sh index 981aded..0d5d94a 100755 --- a/hack/smoke/agent.sh +++ b/hack/smoke/agent.sh @@ -3,7 +3,13 @@ # for GHCR: `shelf app status --json`, `--wait --commit` that returns Ready only after the # awaited commit rolled out, Failed (exit 1) for an image that cannot be pulled, TimedOut # (exit 2) for a commit that never arrives, and `shelf app open` with the app's Host and -# stripPrefix through Traefik. +# stripPrefix through Traefik. With SMOKE_FULL=1 (the default) it also checks that a fixed commit +# after the failed one is Ready again, which waits for Helm to time out the broken upgrade +# (about 5 minutes). CI runs that part on main only; pull requests set SMOKE_FULL=0, leave the +# broken app in place and discard the cluster. +# +# The script asks Flux to fetch each new artifact at once instead of waiting for its poll: what +# it checks is how `--wait` follows a rollout, not Flux's interval. `--wait` itself only reads. # # Installs or updates the platform first (idempotent). Needs network access for images. set -euo pipefail @@ -15,14 +21,15 @@ work="$(mktemp -d)" app=agent-smoke artifact="oci://$SHELF_REGISTRY_HOST/smoke/$app-deploy" open_pid="" +full="${SMOKE_FULL:-1}" export SHELF_HOME="$work/home" shelf() { "$work/shelf" "$@"; } cleanup() { [[ -n "$open_pid" ]] && kill "$open_pid" 2>/dev/null || true - if [[ "${KEEP:-}" == 1 ]]; then - echo "KEEP=1: leaving app $app in place; its secret backup is in $SHELF_HOME" + if [[ "${KEEP:-}" == 1 || "$full" != 1 ]]; then + echo "leaving app $app in place (KEEP=1 or SMOKE_FULL=0); its secret backup is in $SHELF_HOME" return fi shelf app rm "$app" --yes >/dev/null 2>&1 || true @@ -35,10 +42,12 @@ step() { echo "--- $*"; } commit() { head -c 20 /dev/urandom | od -An -tx1 | tr -d ' \n'; } # publish : pushes the files in dir as the deploy artifact, tag main, built from -# commit. +# commit, and asks Flux to fetch it once the app exists. publish() { flux push artifact "$artifact:main" --path "$1" --source local --revision "main@sha1:$2" \ --insecure-registry >/dev/null 2>&1 || die "push failed" + kubectl -n "$app" annotate ocirepository deploy --overwrite \ + "reconcile.fluxcd.io/requestedAt=$(date +%s%N)" >/dev/null 2>&1 || true } # render : writes the deploy manifest of an app.yaml into dir. @@ -141,6 +150,12 @@ shelf app status "$app" --wait --commit deadbeef --timeout 5s --json >"$work/wai verdict "$work/wait4.json" .stage.message | grep -q "waiting for commit deadbee" \ || die "the timeout does not say what it waited for: $(cat "$work/wait4.json")" +if [[ "$full" != 1 ]]; then + echo "SMOKE_FULL=0: skipping the fix after the failure; the broken app stays (its upgrade holds up removal)" + echo "smoke-agent passed" + exit 0 +fi + # The agent's loop ends with a fix: a working commit after the broken one is Ready again. It # also leaves no release stuck in an upgrade, which would hold up the removal. step "a fixed commit after the failure is Ready again" diff --git a/justfile b/justfile index eed4c08..7f70bbc 100644 --- a/justfile +++ b/justfile @@ -13,7 +13,8 @@ default: test: #!/usr/bin/env bash set -euo pipefail - unformatted="$(gofmt -l .)" + # Only the repository's own files: CI keeps a Go cache under .cache in the workspace. + unformatted="$(git ls-files -z '*.go' | xargs -0 gofmt -l)" if [[ -n "$unformatted" ]]; then echo "gofmt needed:" >&2 echo "$unformatted" >&2 diff --git a/openspec/changes/phase-9-agent-interface/design.md b/openspec/changes/phase-9-agent-interface/design.md index 0cc28ff..8518cbf 100644 --- a/openspec/changes/phase-9-agent-interface/design.md +++ b/openspec/changes/phase-9-agent-interface/design.md @@ -207,6 +207,42 @@ The new smoke test does the following: It runs in CI with the other level-2 smoke tests. +### 8. CI runs level 2 when a pull request is ready, chosen by paths, and in full on main + +Added after the first PR run, when `smoke-agent` made each pull-request run about 12 minutes +(33 runner minutes), on every push. Measured on the PR runs of 2026-10-19 and 2026-10-27: test +2.7 min, smoke-secrets 2, smoke-chart 4.5, smoke-init 4.7, smoke-apps 6.8, smoke-agent about +12, of which about 5 are Helm's upgrade timeout in its last step. + +- **When.** Level 2 runs when a pull request is opened or reopened as ready, or leaves draft; + on every push and when the label is added while it carries the label `level-2` (also as a + draft); and in full with the label `level-2-full`. Other pushes run level 1 only. A new push + cancels the run it replaces, while a label unrelated to CI cancels nothing. +- **What.** `.github/scripts/select-smokes.sh` maps changed paths to smoke tests. Planning, + docs, examples, the admin UI and the Cloudflare and GitHub clients need none. A path it does + not list needs all of them, so a forgotten entry costs time, never coverage. +- **How much.** Pull requests run `smoke-agent` with `SMOKE_FULL=0`, which skips the fixed + commit after the broken one (Helm's 5-minute timeout). `smoke-agent` asks Flux to fetch each + new artifact at once; `--wait` stays read-only, and what is checked is how it follows a + rollout, not Flux's interval. `smoke-apps` keeps its one step that waits for the poll on + purpose. +- **main.** Every merge runs everything with `SMOKE_FULL=1`, so a failure belongs to one commit. + A failure opens an issue labelled `ci-main-red`, or comments on the open one, because the + e-mail GitHub sends goes to one person and is easy to miss (the maintainer chose the issue). + `release.yml` publishes only a commit whose CI run on main succeeded. +- **Cache.** `just test` keeps Go's build and module cache in `.cache/go` through + `actions/cache`. `gofmt` therefore checks the tracked files only. + +Expected: about 2 minutes for a push to a pull request, about 4–5 when it is ready (smoke-apps +and smoke-agent decide), about 7 in full on main. + +*Alternatives:* a nightly full run instead of one per merge (it was rejected: the result belongs +to no commit, GitHub mails only the last editor of the schedule, and it disables schedules in +public repositories after 60 days without activity). A cached devcontainer image in GHCR (about +a minute per job, but it publishes a new package; deferred to the maintainer). A shorter Helm +timeout or Flux interval for every app (product changes, not CI ones). One job for all smoke +tests (fewer runner minutes, but no shorter wait, and the tests would share a cluster). + ## Risks / Trade-offs - **The values digest reproduces helm-controller's.** If a Flux release changes how @@ -229,6 +265,10 @@ It runs in CI with the other level-2 smoke tests. - **Absolute URLs an app builds from the Host header point at `.shelf.internal`.** Quick tunnels have the same issue. → Rewrite `Location` headers (decision 5). Links inside page bodies stay the app's responsibility, as they do behind any proxy. +- **A regression in the fix-after-failure path shows on main, not in the pull request.** → The + `ci-main-red` issue and the release gate; `level-2-full` runs it on a pull request on demand. +- **The path map can miss a dependency.** → Unlisted paths run everything, and main runs + everything after each merge. - **The JSON contract can't be made stable after the fact.** → Golden files per kind from the first release, and the rule in the spec "Stable fields within a version". diff --git a/openspec/changes/phase-9-agent-interface/proposal.md b/openspec/changes/phase-9-agent-interface/proposal.md index 6dbce4a..8d0392a 100644 --- a/openspec/changes/phase-9-agent-interface/proposal.md +++ b/openspec/changes/phase-9-agent-interface/proposal.md @@ -30,6 +30,10 @@ verdict, and if a person can look at it without exposing it. section that explain the loop: push, wait for the commit, read the verdict, fix. There is no hook. +- CI runs level 2 when a pull request is ready rather than on every push, only the smoke tests its + paths need, and everything in full after each merge to main, which opens an issue when it fails; + a release needs a green main. + ## Capabilities ### New Capabilities @@ -42,6 +46,8 @@ verdict, and if a person can look at it without exposing it. - `app-delivery`: the app status command also shows the commit the running artifact was built from. +- `dev-environment`: when CI runs which level-2 checks, the issue for a red main, the release + gate. - `app-exposure`: a new way to reach an app: a local port-forward to Traefik with the app's Host, for private and exposed apps alike, on the loopback interface only. diff --git a/openspec/changes/phase-9-agent-interface/specs/dev-environment/spec.md b/openspec/changes/phase-9-agent-interface/specs/dev-environment/spec.md new file mode 100644 index 0000000..b59e877 --- /dev/null +++ b/openspec/changes/phase-9-agent-interface/specs/dev-environment/spec.md @@ -0,0 +1,33 @@ +# Spec Delta + +## MODIFIED Requirements + +### Requirement: Three test levels +Checks SHALL be split into level 1 (no cluster: `just test`, which runs gofmt, `go vet`, +`go test -race`, Helm lint, kubeconform, example validation and `openspec validate --all --strict`), +level 2 (the k3d cluster: the `just smoke-*` recipes), and level 3 (the Mac mini and real +Cloudflare), and a task that needs level 2 or 3 MUST say so. + +#### Scenario: Level 1 in CI +- **WHEN** a commit is pushed to `main` or a pull request is opened or pushed to +- **THEN** CI runs `just test` inside the devcontainer image on an arm64 runner + +#### Scenario: Level 2 in CI +- **WHEN** a pull request is opened or reopened as ready, leaves draft, or is pushed to or labelled while it carries the label `level-2` +- **THEN** CI also runs the smoke tests its changed paths need, chosen by `.github/scripts/select-smokes.sh`, each against a fresh k3d cluster of its own, side by side, with `SMOKE_FULL=0` + +#### Scenario: Documentation only +- **WHEN** a pull request changes only planning, documentation or examples +- **THEN** CI runs no smoke test, and a path the selection does not know runs all of them + +#### Scenario: Level 2 in full +- **WHEN** a commit is pushed to `main`, or a pull request carries the label `level-2-full` +- **THEN** CI runs `smoke-secrets`, `smoke-chart`, `smoke-init`, `smoke-apps` and `smoke-agent` with `SMOKE_FULL=1` + +#### Scenario: main fails +- **WHEN** CI fails on `main` +- **THEN** it opens an issue labelled `ci-main-red`, or comments on the one that is open, with the commit and the run + +#### Scenario: Release gate +- **WHEN** a release tag is pushed for a commit whose CI run on `main` did not succeed +- **THEN** `release.yml` stops before it publishes anything diff --git a/openspec/changes/phase-9-agent-interface/tasks.md b/openspec/changes/phase-9-agent-interface/tasks.md index e08bfde..baca79e 100644 --- a/openspec/changes/phase-9-agent-interface/tasks.md +++ b/openspec/changes/phase-9-agent-interface/tasks.md @@ -35,10 +35,18 @@ ## 6. Level-2 acceptance -- [ ] 6.1 Add `hack/smoke/agent.sh` and `just smoke-agent` as in design decision 7 (status JSON, wait for a new commit with exit 0 only after it rolled out, failing image with exit 1, unknown commit with exit 2, `open` against a whoami fixture with `/api` + `stripPrefix`, cleanup); verify that `just smoke-agent` passes on a fresh `just cluster-reset` -- [ ] 6.2 Add `smoke-agent` to the level-2 matrix in `.github/workflows/ci.yml` and to the CLAUDE.md dev-environment table; verify that the CI job runs it on a pull request +- [x] 6.1 Add `hack/smoke/agent.sh` and `just smoke-agent` as in design decision 7 (status JSON, wait for a new commit with exit 0 only after it rolled out, failing image with exit 1, unknown commit with exit 2, `open` against a whoami fixture with `/api` + `stripPrefix`, cleanup); verify that `just smoke-agent` passes on a fresh `just cluster-reset` +- [x] 6.2 Add `smoke-agent` to the level-2 matrix in `.github/workflows/ci.yml` and to the CLAUDE.md dev-environment table; verify that the CI job runs it on a pull request - [x] 6.3 Run `just test` and `openspec validate --all --strict`; both pass +## 7. Faster CI + +- [x] 7.1 Add `.github/scripts/select-smokes.sh` (changed paths to smoke tests, unlisted paths to all of them); verify on the paths of both commits of this branch (the roadmap commit selects none) +- [x] 7.2 Rework `.github/workflows/ci.yml` as in design decision 8 (plan job, label and draft rules, concurrency, `SMOKE_FULL`, Go cache, `ci-main-red` issue on main); verify the plan logic by running its script for every event kind +- [x] 7.3 Add `SMOKE_FULL` and the Flux nudge to `hack/smoke/agent.sh`, and limit `gofmt` in `just test` to tracked files; verify with `just test` and `just smoke-agent` with `SMOKE_FULL=0` and `1` +- [x] 7.4 Gate `release.yml` on a successful CI run on main; verify the query against the GitHub API for a green commit and an unknown one +- [ ] 7.5 Update the spec `dev-environment` (delta), CLAUDE.md and the context; verify `openspec validate --all --strict` and that a pull request run shows the selected matrix in its summary + ## Workflow follow-up - Acceptance by the maintainer: an agent in a copy of `examples/tenant` pushes, waits and reads the verdict without parsing text; `shelf app open` serves a private app in the Mac's browser through the devcontainer's forwarded port. diff --git a/openspec/config.yaml b/openspec/config.yaml index fbe2dc0..c372ba6 100644 --- a/openspec/config.yaml +++ b/openspec/config.yaml @@ -48,7 +48,8 @@ context: | operator's decision, never part of app.yaml. Previews are reached by port-forward through the API server to Traefik (`shelf app open`, Phase 9); LAN access stays rejected. - Test levels: 1 is `just test` without a cluster (also CI), 2 the k3d cluster `shelf-dev` in - the devcontainer (`just smoke-*`, in CI on pull requests), 3 the mini and real Cloudflare. + the devcontainer (`just smoke-*`; in CI when a pull request is ready or labelled `level-2`, + chosen by changed paths, and in full on main), 3 the mini and real Cloudflare. The rules are in the spec `dev-environment`. ## Rules for proposals