Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 45 additions & 0 deletions .github/scripts/select-smokes.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
#!/usr/bin/env bash
# Prints the level-2 smoke tests a change needs, as a JSON array for the CI matrix. Reads the
# changed paths from stdin, one per line. Runs on the CI runner, not in the devcontainer.
#
# A path that is documentation or planning needs none. A path listed below needs the tests
# named for it. Anything else needs all of them: an unlisted path is a reason to test, never a
# reason to skip.
set -euo pipefail

all=(smoke-secrets smoke-chart smoke-init smoke-apps smoke-agent)
declare -A need=()

for_path() {
case "$1" in
# Planning, documentation and examples (the examples are validated by `just test`).
openspec/* | *.md | LICENSE | NOTICE | examples/* | .claude/* | schema/*) ;;
# Workflows that are checked by running them, not by the smoke tests.
.github/workflows/build.yml | .github/workflows/release.yml) ;;
# Code no smoke test reaches: the admin UI and the APIs of Cloudflare and GitHub.
internal/server/* | internal/cloudflare/* | internal/github/*) ;;
hack/smoke/secret-expansion*) echo smoke-secrets ;;
hack/smoke/chart.sh) echo smoke-chart ;;
charts/* | internal/chart/*) echo smoke-chart smoke-apps smoke-agent ;;
hack/smoke/init.sh | internal/cli/init.go | internal/cli/init_test.go) echo smoke-init ;;
hack/smoke/apps.sh) echo smoke-apps ;;
hack/smoke/agent.* | internal/cli/json* | internal/cli/open* | internal/cli/wait* | internal/cli/tenant_test.go)
echo smoke-agent ;;
internal/cli/* | internal/ops/* | internal/progress/* | internal/secrets/* | internal/hostcfg/*)
echo smoke-apps smoke-agent ;;
*) echo "${all[@]}" ;;
esac
}

while IFS= read -r path; do
[[ -n "$path" ]] || continue
for smoke in $(for_path "$path"); do
need[$smoke]=1
done
done

out=()
for smoke in "${all[@]}"; do
[[ -n "${need[$smoke]:-}" ]] && out+=("\"$smoke\"")
done
(IFS=,; echo "[${out[*]}]")
102 changes: 96 additions & 6 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -1,40 +1,105 @@
# Level-1 checks, and level-2 smoke tests against k3d. Both run inside the devcontainer image, so
# CI uses exactly the tool versions pinned in .devcontainer/Dockerfile.
#
# Level 1 runs on every push. Level 2 is slow, so a pull request runs it when it is ready, not on
# every push while it is being worked on:
# - when a pull request is opened or reopened as ready, or leaves draft;
# - on every push while it carries the label `level-2`, and when the label is added, also as a
# draft;
# - with the label `level-2-full`, the full set, as on main.
# A pull request runs only the smoke tests its changed paths need (.github/scripts/
# select-smokes.sh), and skips the slowest step of smoke-agent (SMOKE_FULL=0). main runs all of
# them in full after every merge, and opens an issue when that fails; release.yml publishes only a
# commit whose run on main succeeded.
name: ci

on:
push:
branches: [main]
pull_request:
types: [opened, synchronize, reopened, ready_for_review, labeled]

permissions:
contents: read

# A new push to a pull request cancels the run it replaces. A label that has nothing to do with CI
# gets a group of its own, so that adding it cancels nothing; runs on main are never cancelled.
concurrency:
group: >-
ci-${{ github.event.pull_request.number || github.sha }}-${{
(github.event.action == 'labeled' && !startsWith(github.event.label.name, 'level-2')) && github.run_id || 'run' }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}

jobs:
test:
# A label that has nothing to do with CI changes no code. A level-2 label runs it again: its run
# may have cancelled the one of the push before it.
if: github.event.action != 'labeled' || startsWith(github.event.label.name, 'level-2')
# The same architecture as the Mac and the mini.
runs-on: ubuntu-24.04-arm
timeout-minutes: 20
steps:
- uses: actions/checkout@v7
# Go's build and module cache, kept in the workspace so the devcontainer sees it.
- uses: actions/cache@v4
with:
path: .cache/go
key: go-${{ runner.arch }}-${{ hashFiles('go.sum', '.devcontainer/Dockerfile') }}-${{ github.sha }}
restore-keys: go-${{ runner.arch }}-${{ hashFiles('go.sum', '.devcontainer/Dockerfile') }}-
- name: just test (in the devcontainer)
uses: devcontainers/ci@v0.3
with:
push: never
runCmd: just test
runCmd: |
export GOCACHE="$PWD/.cache/go/build" GOMODCACHE="$PWD/.cache/go/mod"
just test

# Which smoke tests to run, and how: none, those the changed paths need, or all in full.
plan:
runs-on: ubuntu-24.04-arm
timeout-minutes: 5
outputs:
smokes: ${{ steps.plan.outputs.smokes }}
full: ${{ steps.plan.outputs.full }}
steps:
- uses: actions/checkout@v7
with:
fetch-depth: 0
- id: plan
env:
EVENT: ${{ github.event_name }}
ACTION: ${{ github.event.action }}
DRAFT: ${{ github.event.pull_request.draft }}
LABEL: ${{ github.event.label.name }}
LABELS: ${{ toJSON(github.event.pull_request.labels.*.name) }}
BASE: ${{ github.event.pull_request.base.sha }}
run: |
all='["smoke-secrets","smoke-chart","smoke-init","smoke-apps","smoke-agent"]'
has() { jq -e --arg l "$1" 'index($l) != null' <<<"$LABELS" >/dev/null; }
smokes='[]' full=0
if [[ "$EVENT" == push ]]; then
smokes="$all" full=1
elif has level-2-full && [[ "$ACTION" != labeled || "$LABEL" == level-2-full ]]; then
smokes="$all" full=1
elif { has level-2 && [[ "$ACTION" == synchronize || "$LABEL" == level-2 ]]; } ||
{ [[ "$DRAFT" != true && "$ACTION" =~ ^(opened|reopened|ready_for_review)$ ]]; }; then
smokes="$(git diff --name-only "$BASE" HEAD | .github/scripts/select-smokes.sh)"
fi
echo "smokes=$smokes" >>"$GITHUB_OUTPUT"
echo "full=$full" >>"$GITHUB_OUTPUT"
echo "level 2: $smokes (full=$full)" | tee -a "$GITHUB_STEP_SUMMARY"

# Level-2 checks against a k3d cluster inside the devcontainer, as locally. Each smoke test
# gets a fresh cluster of its own, so they run side by side; none depends on another's state.
# Pull requests only: main receives what a pull request already checked, so after a merge
# level 1 is enough.
cluster:
if: github.event_name == 'pull_request'
needs: plan
if: needs.plan.outputs.smokes != '[]'
runs-on: ubuntu-24.04-arm
timeout-minutes: 30
strategy:
fail-fast: false
matrix:
smoke: [smoke-secrets, smoke-chart, smoke-init, smoke-apps, smoke-agent]
smoke: ${{ fromJSON(needs.plan.outputs.smokes) }}
name: cluster (${{ matrix.smoke }})
steps:
- uses: actions/checkout@v7
Expand All @@ -45,4 +110,29 @@ jobs:
runCmd: |
set -e
just cluster-up
just ${{ matrix.smoke }}
SMOKE_FULL=${{ needs.plan.outputs.full }} just ${{ matrix.smoke }}

# A failure on main is easy to miss in a mailbox: keep it as an issue until someone closes it.
# One open issue collects every failure; each one adds a comment.
report:
needs: [test, plan, cluster]
if: failure() && github.event_name == 'push'
runs-on: ubuntu-24.04-arm
timeout-minutes: 5
permissions:
issues: write
steps:
- env:
GH_TOKEN: ${{ github.token }}
GH_REPO: ${{ github.repository }}
RUN: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
run: |
body="CI failed on main at ${GITHUB_SHA::12}: $RUN"
issue="$(gh issue list --label ci-main-red --state open --json number --jq '.[0].number // empty')"
if [[ -n "$issue" ]]; then
gh issue comment "$issue" --body "$body"
else
gh label create ci-main-red --color B60205 --description "CI on main failed" 2>/dev/null || true
gh issue create --title "CI on main failed" --label ci-main-red \
--body "$body"$'\n\n'"Close this issue once main is green again."
fi
22 changes: 22 additions & 0 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,29 @@ permissions:
packages: write

jobs:
# A release ships what main tested in full: the CI run of the tagged commit on main, with every
# smoke test, must have succeeded. Pull requests skip the slowest checks; this is where they
# count.
gate:
runs-on: ubuntu-24.04-arm
timeout-minutes: 5
permissions:
actions: read
steps:
- env:
GH_TOKEN: ${{ github.token }}
GH_REPO: ${{ github.repository }}
run: |
conclusion="$(gh api "repos/$GH_REPO/actions/workflows/ci.yml/runs?head_sha=$GITHUB_SHA&event=push&branch=main" \
--jq '.workflow_runs | sort_by(.run_started_at) | last | .conclusion // .status // "none"')"
if [[ "$conclusion" != success ]]; then
echo "::error::CI on main for ${GITHUB_SHA::12} is '$conclusion', not success. Tag a commit of main whose CI run succeeded, or wait for it to finish."
exit 1
fi
echo "CI on main for ${GITHUB_SHA::12} succeeded"

release:
needs: gate
runs-on: ubuntu-24.04-arm
timeout-minutes: 30
env:
Expand Down
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -4,3 +4,6 @@
# Build output
/bin/
/dist/

# Go build and module cache of CI (the test job keeps it in the workspace)
/.cache/
10 changes: 9 additions & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -135,6 +135,14 @@ All development happens inside the devcontainer on Docker Desktop. It runs its o
daemon (Docker-in-Docker), which also hosts the k3d cluster. Nothing is installed on the host
Mac. Tool versions are pinned in `.devcontainer/Dockerfile`.

CI runs level 1 on every push. Level 2 runs when a pull request is opened or reopened as ready
or leaves draft, and on every push while it carries the label `level-2`; only the smoke tests its
changed paths need (`.github/scripts/select-smokes.sh`), with `SMOKE_FULL=0`. The label
`level-2-full` and every merge to main run all of them in full. A red main opens an issue
labelled `ci-main-red`, and `release.yml` publishes only a commit whose CI run on main succeeded.
A new path that is neither documentation nor listed in `select-smokes.sh` runs everything; list
it there when it belongs to one smoke test.

| Command | Run from | Purpose |
|---|---|---|
| `just test` | devcontainer | level-1 checks, same as CI (gofmt, vet, `go test -race`, helm lint, kubeconform, examples) |
Expand All @@ -154,7 +162,7 @@ Mac. Tool versions are pinned in `.devcontainer/Dockerfile`.
| `just smoke-chart` | devcontainer | install `examples/hello` with the chart, check the Phase 2 acceptance criteria (needs Docker Hub) |
| `just smoke-init` | devcontainer | Phase 3 acceptance: init twice, re-push, routing and `stripPrefix` through Traefik (run `just cluster-reset` first; needs network) |
| `just smoke-apps` | devcontainer | Phase 4: `shelf app add`/`rm`, rollout by polling, tampered artifact refused, restore from backup (needs network) |
| `just smoke-agent` | devcontainer | Phase 9: `--json` documents, `--wait --commit` (Ready only after the commit rolled out, Failed for an image that cannot be pulled, TimedOut for a commit that never arrives, Ready again for a fix after a failure; takes up to 10 minutes because Helm times out the broken upgrade first) and `shelf app open` with the app's Host and `stripPrefix` (needs Docker Hub) |
| `just smoke-agent` | devcontainer | Phase 9: `--json` documents, `--wait --commit` (Ready only after the commit rolled out, Failed for an image that cannot be pulled, TimedOut for a commit that never arrives; with `SMOKE_FULL=1`, the default, also Ready again for a fix after a failure, which takes about 5 more minutes because Helm times out the broken upgrade first) and `shelf app open` with the app's Host and `stripPrefix` (needs Docker Hub) |
| `just smoke-tenant <app> <artifact>` | devcontainer | Phase 4 acceptance with a real tenant repo and GHCR; asks for the GHCR login, waits for a push |
| `just smoke-expose <app> <domain>` | devcontainer | Phase 8b acceptance: the app gets the Cloudflare connection `smoke`, its own tunnel, DNS record and HTTPS, then is taken off again; asks for the Cloudflare API token (set `CF_ACCOUNT_ID` if it sees several accounts); needs a cluster with a host suffix |
| `just smoke-quick [app]` | devcontainer | Phase 8d acceptance: `examples/hello` as `hello-quick` with `--quick` gets a quick tunnel, answers with its own Host, gets a new name after a restart, `--private`/`--quick`, rm; keeps the cluster's host suffix; opens a public trycloudflare.com tunnel for a few minutes |
Expand Down
23 changes: 19 additions & 4 deletions hack/smoke/agent.sh
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,13 @@
# for GHCR: `shelf app status --json`, `--wait --commit` that returns Ready only after the
# awaited commit rolled out, Failed (exit 1) for an image that cannot be pulled, TimedOut
# (exit 2) for a commit that never arrives, and `shelf app open` with the app's Host and
# stripPrefix through Traefik.
# stripPrefix through Traefik. With SMOKE_FULL=1 (the default) it also checks that a fixed commit
# after the failed one is Ready again, which waits for Helm to time out the broken upgrade
# (about 5 minutes). CI runs that part on main only; pull requests set SMOKE_FULL=0, leave the
# broken app in place and discard the cluster.
#
# The script asks Flux to fetch each new artifact at once instead of waiting for its poll: what
# it checks is how `--wait` follows a rollout, not Flux's interval. `--wait` itself only reads.
#
# Installs or updates the platform first (idempotent). Needs network access for images.
set -euo pipefail
Expand All @@ -15,14 +21,15 @@ work="$(mktemp -d)"
app=agent-smoke
artifact="oci://$SHELF_REGISTRY_HOST/smoke/$app-deploy"
open_pid=""
full="${SMOKE_FULL:-1}"
export SHELF_HOME="$work/home"

shelf() { "$work/shelf" "$@"; }

cleanup() {
[[ -n "$open_pid" ]] && kill "$open_pid" 2>/dev/null || true
if [[ "${KEEP:-}" == 1 ]]; then
echo "KEEP=1: leaving app $app in place; its secret backup is in $SHELF_HOME"
if [[ "${KEEP:-}" == 1 || "$full" != 1 ]]; then
echo "leaving app $app in place (KEEP=1 or SMOKE_FULL=0); its secret backup is in $SHELF_HOME"
return
fi
shelf app rm "$app" --yes >/dev/null 2>&1 || true
Expand All @@ -35,10 +42,12 @@ step() { echo "--- $*"; }
commit() { head -c 20 /dev/urandom | od -An -tx1 | tr -d ' \n'; }

# publish <dir> <commit>: pushes the files in dir as the deploy artifact, tag main, built from
# commit.
# commit, and asks Flux to fetch it once the app exists.
publish() {
flux push artifact "$artifact:main" --path "$1" --source local --revision "main@sha1:$2" \
--insecure-registry >/dev/null 2>&1 || die "push failed"
kubectl -n "$app" annotate ocirepository deploy --overwrite \
"reconcile.fluxcd.io/requestedAt=$(date +%s%N)" >/dev/null 2>&1 || true
}

# render <app.yaml> <dir>: writes the deploy manifest of an app.yaml into dir.
Expand Down Expand Up @@ -141,6 +150,12 @@ shelf app status "$app" --wait --commit deadbeef --timeout 5s --json >"$work/wai
verdict "$work/wait4.json" .stage.message | grep -q "waiting for commit deadbee" \
|| die "the timeout does not say what it waited for: $(cat "$work/wait4.json")"

if [[ "$full" != 1 ]]; then
echo "SMOKE_FULL=0: skipping the fix after the failure; the broken app stays (its upgrade holds up removal)"
echo "smoke-agent passed"
exit 0
fi

# The agent's loop ends with a fix: a working commit after the broken one is Ready again. It
# also leaves no release stuck in an upgrade, which would hold up the removal.
step "a fixed commit after the failure is Ready again"
Expand Down
3 changes: 2 additions & 1 deletion justfile
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,8 @@ default:
test:
#!/usr/bin/env bash
set -euo pipefail
unformatted="$(gofmt -l .)"
# Only the repository's own files: CI keeps a Go cache under .cache in the workspace.
unformatted="$(git ls-files -z '*.go' | xargs -0 gofmt -l)"
if [[ -n "$unformatted" ]]; then
echo "gofmt needed:" >&2
echo "$unformatted" >&2
Expand Down
40 changes: 40 additions & 0 deletions openspec/changes/phase-9-agent-interface/design.md
Original file line number Diff line number Diff line change
Expand Up @@ -207,6 +207,42 @@ The new smoke test does the following:

It runs in CI with the other level-2 smoke tests.

### 8. CI runs level 2 when a pull request is ready, chosen by paths, and in full on main

Added after the first PR run, when `smoke-agent` made each pull-request run about 12 minutes
(33 runner minutes), on every push. Measured on the PR runs of 2026-10-19 and 2026-10-27: test
2.7 min, smoke-secrets 2, smoke-chart 4.5, smoke-init 4.7, smoke-apps 6.8, smoke-agent about
12, of which about 5 are Helm's upgrade timeout in its last step.

- **When.** Level 2 runs when a pull request is opened or reopened as ready, or leaves draft;
on every push and when the label is added while it carries the label `level-2` (also as a
draft); and in full with the label `level-2-full`. Other pushes run level 1 only. A new push
cancels the run it replaces, while a label unrelated to CI cancels nothing.
- **What.** `.github/scripts/select-smokes.sh` maps changed paths to smoke tests. Planning,
docs, examples, the admin UI and the Cloudflare and GitHub clients need none. A path it does
not list needs all of them, so a forgotten entry costs time, never coverage.
- **How much.** Pull requests run `smoke-agent` with `SMOKE_FULL=0`, which skips the fixed
commit after the broken one (Helm's 5-minute timeout). `smoke-agent` asks Flux to fetch each
new artifact at once; `--wait` stays read-only, and what is checked is how it follows a
rollout, not Flux's interval. `smoke-apps` keeps its one step that waits for the poll on
purpose.
- **main.** Every merge runs everything with `SMOKE_FULL=1`, so a failure belongs to one commit.
A failure opens an issue labelled `ci-main-red`, or comments on the open one, because the
e-mail GitHub sends goes to one person and is easy to miss (the maintainer chose the issue).
`release.yml` publishes only a commit whose CI run on main succeeded.
- **Cache.** `just test` keeps Go's build and module cache in `.cache/go` through
`actions/cache`. `gofmt` therefore checks the tracked files only.

Expected: about 2 minutes for a push to a pull request, about 4–5 when it is ready (smoke-apps
and smoke-agent decide), about 7 in full on main.

*Alternatives:* a nightly full run instead of one per merge (it was rejected: the result belongs
to no commit, GitHub mails only the last editor of the schedule, and it disables schedules in
public repositories after 60 days without activity). A cached devcontainer image in GHCR (about
a minute per job, but it publishes a new package; deferred to the maintainer). A shorter Helm
timeout or Flux interval for every app (product changes, not CI ones). One job for all smoke
tests (fewer runner minutes, but no shorter wait, and the tests would share a cluster).

## Risks / Trade-offs

- **The values digest reproduces helm-controller's.** If a Flux release changes how
Expand All @@ -229,6 +265,10 @@ It runs in CI with the other level-2 smoke tests.
- **Absolute URLs an app builds from the Host header point at `<app>.shelf.internal`.** Quick
tunnels have the same issue. → Rewrite `Location` headers (decision 5). Links inside page
bodies stay the app's responsibility, as they do behind any proxy.
- **A regression in the fix-after-failure path shows on main, not in the pull request.** → The
`ci-main-red` issue and the release gate; `level-2-full` runs it on a pull request on demand.
- **The path map can miss a dependency.** → Unlisted paths run everything, and main runs
everything after each merge.
- **The JSON contract can't be made stable after the fact.** → Golden files per kind from the
first release, and the rule in the spec "Stable fields within a version".

Expand Down
Loading
Loading