From fdd1c9af4e6a06dc7b1a08761cf994ba16720fb2 Mon Sep 17 00:00:00 2001 From: tweinmann Date: Wed, 28 Oct 2026 20:11:57 +0000 Subject: [PATCH] Archive Phase 12: hardening The level-3 acceptance passed with tweinmann/shelf-hello on release v0.9.0: a curl test on main was Ready, a broken test on a branch kept a red preview running and showed in shelf app report, and on main it rolled gretter back while gretter kept answering. gretter is exposed again. What the acceptance taught, now in the README and the tenant skill: a test has to be able to fail (an app that answers every path with 200 makes curl -f on a wrong path pass), and curl with several URLs needs --fail-early. shelf reads the resolved app.yaml in the cluster leniently (schema.ParseLenient), so an older shelf still shows an app built by a newer workflow; an admin UI from before Phase 12 failed on the app page of gretter once its values held tests. validate and render stay strict. The deltas are merged into openspec/specs, with the new specs isolation and app-tests. Co-Authored-By: Claude Opus 5.5 --- CLAUDE.md | 12 ++- README.md | 13 ++-- examples/tenant/.claude/skills/shelf/SKILL.md | 3 + internal/render/render.go | 9 +-- internal/schema/parse.go | 34 ++++++++ internal/schema/parse_test.go | 39 ++++++++++ .../.openspec.yaml | 0 .../2026-10-28-phase-12-hardening}/design.md | 34 ++++++++ .../proposal.md | 0 .../specs/admin-ui/spec.md | 0 .../specs/agent-interface/spec.md | 0 .../specs/app-chart/spec.md | 0 .../specs/app-delivery/spec.md | 0 .../specs/app-manifest/spec.md | 0 .../specs/app-tests/spec.md | 0 .../specs/branch-previews/spec.md | 0 .../specs/dev-environment/spec.md | 0 .../specs/isolation/spec.md | 0 .../2026-10-28-phase-12-hardening}/tasks.md | 2 +- openspec/config.yaml | 4 +- openspec/specs/admin-ui/spec.md | 16 +++- openspec/specs/agent-interface/spec.md | 36 ++++++++- openspec/specs/app-chart/spec.md | 28 +++++++ openspec/specs/app-delivery/spec.md | 22 +++--- openspec/specs/app-manifest/spec.md | 59 +++++++++++--- openspec/specs/app-tests/spec.md | 51 ++++++++++++ openspec/specs/branch-previews/spec.md | 13 +++- openspec/specs/dev-environment/spec.md | 2 +- openspec/specs/isolation/spec.md | 78 +++++++++++++++++++ 29 files changed, 411 insertions(+), 44 deletions(-) rename openspec/changes/{phase-12-hardening => archive/2026-10-28-phase-12-hardening}/.openspec.yaml (100%) rename openspec/changes/{phase-12-hardening => archive/2026-10-28-phase-12-hardening}/design.md (87%) rename openspec/changes/{phase-12-hardening => archive/2026-10-28-phase-12-hardening}/proposal.md (100%) rename openspec/changes/{phase-12-hardening => archive/2026-10-28-phase-12-hardening}/specs/admin-ui/spec.md (100%) rename openspec/changes/{phase-12-hardening => archive/2026-10-28-phase-12-hardening}/specs/agent-interface/spec.md (100%) rename openspec/changes/{phase-12-hardening => archive/2026-10-28-phase-12-hardening}/specs/app-chart/spec.md (100%) rename openspec/changes/{phase-12-hardening => archive/2026-10-28-phase-12-hardening}/specs/app-delivery/spec.md (100%) rename openspec/changes/{phase-12-hardening => archive/2026-10-28-phase-12-hardening}/specs/app-manifest/spec.md (100%) rename openspec/changes/{phase-12-hardening => archive/2026-10-28-phase-12-hardening}/specs/app-tests/spec.md (100%) rename openspec/changes/{phase-12-hardening => archive/2026-10-28-phase-12-hardening}/specs/branch-previews/spec.md (100%) rename openspec/changes/{phase-12-hardening => archive/2026-10-28-phase-12-hardening}/specs/dev-environment/spec.md (100%) rename openspec/changes/{phase-12-hardening => archive/2026-10-28-phase-12-hardening}/specs/isolation/spec.md (100%) rename openspec/changes/{phase-12-hardening => archive/2026-10-28-phase-12-hardening}/tasks.md (99%) create mode 100644 openspec/specs/app-tests/spec.md create mode 100644 openspec/specs/isolation/spec.md diff --git a/CLAUDE.md b/CLAUDE.md index afe483f..66e4b84 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -114,7 +114,17 @@ the app page show the chain. Release v0.8.0. Accepted with `tweinmann/shelf-hell 2 min 36 s (Phase 10: 1 min 33 s), an unsigned artifact and an unsigned image were refused, and `gretter` was exposed only once verified. `shelf serve` now follows a rewritten kubeconfig. Results in `openspec/changes/archive/2026-10-28-phase-11-verification-chain`. -Next: Phase 12 (hardening). +Phase 12 complete (2026-10-28): hardening. App and preview namespaces enforce Pod Security +`baseline` (and list what `restricted` would refuse), and the NetworkPolicy `shelf-isolation` +lets pods reach their own namespace, DNS and the internet only; only Traefik reaches in, only +cloudflared reaches Traefik. LAN ranges are per app and the operator's (`--allow-lan`), never a +preview's. `tests:` in app.yaml run as Helm test hooks after every rollout: a failing test rolls +an app back and leaves a preview red, and `--wait` is Ready only once they pass; `shelf app +report` writes the per-branch report for a pull request. Release v0.9.0. Accepted with +`tweinmann/shelf-hello`: a test on main was Ready, a broken test on a branch kept a red preview +running, and on main it rolled `gretter` back within 32 s while it kept answering. Results in +`openspec/changes/archive/2026-10-28-phase-12-hardening`. +Next: Phase 13 (host). ## Working agreements diff --git a/README.md b/README.md index e7e3c34..3f5dc4e 100644 --- a/README.md +++ b/README.md @@ -17,9 +17,10 @@ your own Cloudflare account, or a temporary `*.trycloudflare.com` name that need > shelf is a learning and reference project for platform engineering, and it is > **pre-production**: production is a separate cluster reached through eject (planned). It is > **still under construction**: today a push to your repository deploys your app, and the CLI or -> an admin UI in the browser adds, changes and removes apps, every branch runs as a preview, and -> only a signed and scanned app can be exposed. The host setup, the installer and eject are -> planned, not built. See [Status](#status). +> an admin UI in the browser adds, changes and removes apps, every branch runs as an isolated +> preview that counts as green once its tests pass, and only a signed and scanned app can be +> exposed. The host setup, the installer and eject are planned, not built. See +> [Status](#status). Where this is going: you run Claude Code (other agents later) in several worktrees at once. Every branch your agents push gets its own preview of the complete stack, database included, in a @@ -113,7 +114,7 @@ one starts. | 9 | Agent interface: `--json`, `shelf app status --wait`, `shelf app open` | ✅ done | | 10 | Branch previews: one isolated preview per branch, discovered from the registry | ✅ done | | 11 | Verification chain: signatures, scans, provenance; exposure only for verified digests | ✅ done | -| 12 | Hardening: Pod Security, NetworkPolicies, smoke tests per preview | planned | +| 12 | Hardening: Pod Security, NetworkPolicies, smoke tests per preview | ✅ done | | 13 | Host: macOS with Colima and Linux with k3s, `shelf doctor`, `shelf destroy` | planned | | 14 | Service and one-command installer | planned | | 15 | Logs, resource usage, self-update | planned | @@ -465,7 +466,9 @@ A test passes when its container exits with 0. It runs inside the app's namespac the components by their references (`http://${web.host}:${web.port}/`), not through a route, and it can read the app's secrets. Let it retry its first connection (`curl --retry 5 --retry-connrefused`): for a moment after a pod starts, the network policy may not know it yet, -and refuses it. Until every test passed, the app is not Ready: +and refuses it. A curl test that checks several URLs needs `--fail-early`: without it, curl +exits with the status of the last URL only, so a failure before it passes unnoticed. Until every +test passed, the app is not Ready: - on the app, a failing test fails the upgrade, so the previous version comes back; - on a preview, the failing version keeps running, so you can look at it with diff --git a/examples/tenant/.claude/skills/shelf/SKILL.md b/examples/tenant/.claude/skills/shelf/SKILL.md index 88ac0e5..c51427d 100644 --- a/examples/tenant/.claude/skills/shelf/SKILL.md +++ b/examples/tenant/.claude/skills/shelf/SKILL.md @@ -97,6 +97,9 @@ The first stage that is not Ready is the cause; everything after it is a consequ its namespace. A test reaches the components by their references, such as `http://${web.host}:${web.port}/`, never through a route, and passes when it exits with 0. Add one for what a change must not break; the wait counts the commit as Ready only once they pass. +With curl, pass `--retry 5 --retry-connrefused` (a new pod's first connection can be refused for +a moment), and `--fail-early` when it checks several URLs, or only the last one counts. Check +that a test can fail: an app that answers every path with 200 makes `-f` on a wrong path pass. ## What you cannot do diff --git a/internal/render/render.go b/internal/render/render.go index 86af003..1c19893 100644 --- a/internal/render/render.go +++ b/internal/render/render.go @@ -315,13 +315,10 @@ func encodeYAML(v any) ([]byte, error) { const ConfigMapName = "shelf-values" // ParseApp reads back what MarshalApp wrote, so that anyone holding the ConfigMap of an app can -// see what it is made of without going to the registry for the artifact. +// see what it is made of without going to the registry for the artifact. The artifact may come +// from a newer workflow than this shelf, so fields it does not know are left out. func ParseApp(data []byte) (*schema.App, error) { - doc, err := schema.Parse(ConfigMapKey, data) - if err != nil { - return nil, err - } - return doc.App, nil + return schema.ParseLenient(ConfigMapKey, data) } // ConfigMapKey is the data key holding the resolved app.yaml; the HelmRelease reads it through diff --git a/internal/schema/parse.go b/internal/schema/parse.go index 0d94aa1..ea57959 100644 --- a/internal/schema/parse.go +++ b/internal/schema/parse.go @@ -43,6 +43,40 @@ func Parse(file string, data []byte) (*Document, error) { return &Document{File: file, App: &app, root: &root}, nil } +// ParseLenient decodes what another version of shelf may have written, such as the resolved +// app.yaml in an app's ConfigMap: fields this version does not know are left out instead of +// being errors, so an older shelf can still show an app whose workflow is newer. Everything a +// person writes goes through Parse. +func ParseLenient(file string, data []byte) (*App, error) { + var root yaml.Node + if err := yaml.Unmarshal(data, &root); err != nil { + return nil, fmt.Errorf("%s: %w", file, err) + } + if root.Kind == 0 { + return nil, fmt.Errorf("%s: file is empty", file) + } + // Route checks its own keys, since node.Decode knows no KnownFields; drop the ones it + // would refuse. + if components := mappingValue(root.Content[0], "components"); components != nil { + for i := 1; i < len(components.value.Content); i += 2 { + if route := mappingValue(components.value.Content[i], "route"); route != nil && route.value.Kind == yaml.MappingNode { + kept := route.value.Content[:0] + for j := 0; j+1 < len(route.value.Content); j += 2 { + if slices.Contains(routeKeys, route.value.Content[j].Value) { + kept = append(kept, route.value.Content[j], route.value.Content[j+1]) + } + } + route.value.Content = kept + } + } + } + var app App + if err := root.Decode(&app); err != nil { + return nil, fmt.Errorf("%s: %w", file, err) + } + return &app, nil +} + // Line returns the line of the node at path, e.g. ("components", "web", "port"). If the path // does not exist, it returns the line of the deepest ancestor that does, and 0 without a tree. func (d *Document) Line(path ...string) int { diff --git a/internal/schema/parse_test.go b/internal/schema/parse_test.go index c73b1bd..f50de3e 100644 --- a/internal/schema/parse_test.go +++ b/internal/schema/parse_test.go @@ -136,3 +136,42 @@ func TestSecretEnvName(t *testing.T) { t.Errorf("got %q", got) } } + +// TestParseLenient covers what an older shelf reads from the cluster: the resolved app.yaml of a +// newer workflow, with fields it does not know, at the top, in a component and in a route. +func TestParseLenient(t *testing.T) { + t.Parallel() + app, err := ParseLenient("app.yaml", []byte(`apiVersion: shelf.dev/v1alpha1 +name: greeter +components: + web: + image: nginx@sha256:0000000000000000000000000000000000000000000000000000000000000000 + ports: { main: 8080 } + route: { path: /, port: main, rewrite: true } + sidecar: { image: envoy } + db: + image: postgres:16 + route: /db +futureSection: + anything: [1, 2] +`)) + if err != nil { + t.Fatal(err) + } + web := app.Components["web"] + if web == nil || web.Ports["main"] != 8080 || web.Route == nil || web.Route.Path != "/" || web.Route.Port != "main" { + t.Errorf("web %+v", web) + } + if db := app.Components["db"]; db == nil || db.Route == nil || db.Route.Path != "/db" { + t.Errorf("db %+v", db) + } + // What a person writes stays strict. + if _, err := Parse("app.yaml", []byte("apiVersion: shelf.dev/v1alpha1\nname: a\nfutureSection: {}\n")); err == nil { + t.Error("Parse accepts an unknown field") + } + for _, bad := range []string{"", "components: [\n", "components: 5\n"} { + if _, err := ParseLenient("app.yaml", []byte(bad)); err == nil { + t.Errorf("ParseLenient accepts %q", bad) + } + } +} diff --git a/openspec/changes/phase-12-hardening/.openspec.yaml b/openspec/changes/archive/2026-10-28-phase-12-hardening/.openspec.yaml similarity index 100% rename from openspec/changes/phase-12-hardening/.openspec.yaml rename to openspec/changes/archive/2026-10-28-phase-12-hardening/.openspec.yaml diff --git a/openspec/changes/phase-12-hardening/design.md b/openspec/changes/archive/2026-10-28-phase-12-hardening/design.md similarity index 87% rename from openspec/changes/phase-12-hardening/design.md rename to openspec/changes/archive/2026-10-28-phase-12-hardening/design.md index f8fc7b1..127a49d 100644 --- a/openspec/changes/phase-12-hardening/design.md +++ b/openspec/changes/archive/2026-10-28-phase-12-hardening/design.md @@ -378,3 +378,37 @@ What apply found: Also confirmed: `shelf serve` kept working through two `cluster-reset`s without a restart, the follow-up of Phase 11. + +### Level 3 (`tweinmann/shelf-hello` as `gretter`, release v0.9.0, 2026-10-28) + +The dev cluster was installed again from the release, keyless, and `gretter` added privately with +previews. Both release artifacts carried keyless signatures of `release.yml`. + +| Check | Result | +|---|---| +| `gretter` on the release platform | verified; Pod Security baseline; for `restricted`, only "unrestricted capabilities, runAsNonRoot" are left per pod (decision 1b), Postgres and Mongo run | +| Push of a curl test to `main` (`33f8756`) | CI 1 min 58 s (Phase 11: 2 min 36 s, cold cache); Ready, "1 passed: smoke"; the scan covers the test's image | +| A branch with a test that fails (`3a40f54`) | CI 1 min 48 s; `--wait --commit` ends `Failed` at "the tests" with exit code 7 and the log; the preview keeps running that commit; `shelf app report --preview` shows it red, with the log, the scan and the comparison with `main` | +| The same commit merged to `main` | CI 1 min 35 s; `--wait` ends `Failed` after 22 s; Helm rolled back 10 s later; after four attempts (about 2 min) `gretter` runs the values of `33f8756` again, `/` answered 200 throughout; "the app" Ready, "the tests" Failed | + +Two things the acceptance taught, both now in the README and the tenant skill: +- **A test has to be able to fail.** The web component of `shelf-hello` answers every path with + 200, so `curl -f …/nope` passed. Breaking the port did fail. +- **curl with several URLs** exits with the status of the last one; a failure before it passes + unnoticed without `--fail-early`. + +As foreseen in the risks, a failing test on `main` makes helm-controller retry the upgrade three +times before it stalls, replacing the pods each time; the verdict does not wait for that. + +After the acceptance, the test was fixed on `main` (`801843a`): Ready, verified, and the release +left its stalled state with the new values. `gretter` was then exposed again with +`--cloudflare tobile --domain tobile.ch`: deployed privately, verified, then tunnel and record +created; `https://gretter.tobile.ch/` answered 200, so cloudflared reaches Traefik under +`shelf-cloudflared` on the released platform. + +**Found after the archive, and fixed with it:** an admin UI started before Phase 12 failed on the +app page of `gretter` once its values held `tests`: shelf read the resolved app.yaml from the +cluster as strictly as one a person writes. `render.ParseApp` now uses `schema.ParseLenient`, +which leaves out fields it does not know (also in a long-form `route`), so an older shelf can +still show an app built by a newer workflow; `shelf validate` and `render` stay strict. The main +spec `app-manifest` has the requirement "Reading a resolved app.yaml from the cluster". diff --git a/openspec/changes/phase-12-hardening/proposal.md b/openspec/changes/archive/2026-10-28-phase-12-hardening/proposal.md similarity index 100% rename from openspec/changes/phase-12-hardening/proposal.md rename to openspec/changes/archive/2026-10-28-phase-12-hardening/proposal.md diff --git a/openspec/changes/phase-12-hardening/specs/admin-ui/spec.md b/openspec/changes/archive/2026-10-28-phase-12-hardening/specs/admin-ui/spec.md similarity index 100% rename from openspec/changes/phase-12-hardening/specs/admin-ui/spec.md rename to openspec/changes/archive/2026-10-28-phase-12-hardening/specs/admin-ui/spec.md diff --git a/openspec/changes/phase-12-hardening/specs/agent-interface/spec.md b/openspec/changes/archive/2026-10-28-phase-12-hardening/specs/agent-interface/spec.md similarity index 100% rename from openspec/changes/phase-12-hardening/specs/agent-interface/spec.md rename to openspec/changes/archive/2026-10-28-phase-12-hardening/specs/agent-interface/spec.md diff --git a/openspec/changes/phase-12-hardening/specs/app-chart/spec.md b/openspec/changes/archive/2026-10-28-phase-12-hardening/specs/app-chart/spec.md similarity index 100% rename from openspec/changes/phase-12-hardening/specs/app-chart/spec.md rename to openspec/changes/archive/2026-10-28-phase-12-hardening/specs/app-chart/spec.md diff --git a/openspec/changes/phase-12-hardening/specs/app-delivery/spec.md b/openspec/changes/archive/2026-10-28-phase-12-hardening/specs/app-delivery/spec.md similarity index 100% rename from openspec/changes/phase-12-hardening/specs/app-delivery/spec.md rename to openspec/changes/archive/2026-10-28-phase-12-hardening/specs/app-delivery/spec.md diff --git a/openspec/changes/phase-12-hardening/specs/app-manifest/spec.md b/openspec/changes/archive/2026-10-28-phase-12-hardening/specs/app-manifest/spec.md similarity index 100% rename from openspec/changes/phase-12-hardening/specs/app-manifest/spec.md rename to openspec/changes/archive/2026-10-28-phase-12-hardening/specs/app-manifest/spec.md diff --git a/openspec/changes/phase-12-hardening/specs/app-tests/spec.md b/openspec/changes/archive/2026-10-28-phase-12-hardening/specs/app-tests/spec.md similarity index 100% rename from openspec/changes/phase-12-hardening/specs/app-tests/spec.md rename to openspec/changes/archive/2026-10-28-phase-12-hardening/specs/app-tests/spec.md diff --git a/openspec/changes/phase-12-hardening/specs/branch-previews/spec.md b/openspec/changes/archive/2026-10-28-phase-12-hardening/specs/branch-previews/spec.md similarity index 100% rename from openspec/changes/phase-12-hardening/specs/branch-previews/spec.md rename to openspec/changes/archive/2026-10-28-phase-12-hardening/specs/branch-previews/spec.md diff --git a/openspec/changes/phase-12-hardening/specs/dev-environment/spec.md b/openspec/changes/archive/2026-10-28-phase-12-hardening/specs/dev-environment/spec.md similarity index 100% rename from openspec/changes/phase-12-hardening/specs/dev-environment/spec.md rename to openspec/changes/archive/2026-10-28-phase-12-hardening/specs/dev-environment/spec.md diff --git a/openspec/changes/phase-12-hardening/specs/isolation/spec.md b/openspec/changes/archive/2026-10-28-phase-12-hardening/specs/isolation/spec.md similarity index 100% rename from openspec/changes/phase-12-hardening/specs/isolation/spec.md rename to openspec/changes/archive/2026-10-28-phase-12-hardening/specs/isolation/spec.md diff --git a/openspec/changes/phase-12-hardening/tasks.md b/openspec/changes/archive/2026-10-28-phase-12-hardening/tasks.md similarity index 99% rename from openspec/changes/phase-12-hardening/tasks.md rename to openspec/changes/archive/2026-10-28-phase-12-hardening/tasks.md index d5f0ad6..40bbc4d 100644 --- a/openspec/changes/phase-12-hardening/tasks.md +++ b/openspec/changes/archive/2026-10-28-phase-12-hardening/tasks.md @@ -117,7 +117,7 @@ - `smoke-quick` checks that the quick tunnel's name is still read through the pod proxy. Record the turnaround with and without a test in design.md under "Results". -- [ ] 9.2 Level 3 with `tweinmann/shelf-hello`: +- [x] 9.2 Level 3 with `tweinmann/shelf-hello`: - add a curl test and push to main: Ready; - a branch whose test fails: a red preview that keeps running, then `shelf app report` for it; - merge a failing test: `gretter` keeps the previous version. diff --git a/openspec/config.yaml b/openspec/config.yaml index ba40231..1a4b16e 100644 --- a/openspec/config.yaml +++ b/openspec/config.yaml @@ -17,8 +17,8 @@ context: | artefact becomes reachable beyond a preview, and the same verified state goes to any Kubernetes cloud as a pull request (eject), without depending on shelf. shelf runs on any Linux host with k3s, with a one-step install on a Mac (Colima). Today it runs against any kubecontext, developed - on the dev cluster, with branch previews and the verification chain; the host setup and eject - are planned. shelf is pre-production: an app's "main" is its stable, verified state, not + on the dev cluster, with isolated, tested branch previews and the verification chain; the host + setup and eject are planned. shelf is pre-production: an app's "main" is its stable, verified state, not production. Learning and reference project for platform engineering. OpenSpec is the only planning source: `openspec/specs/` is what shelf does today, diff --git a/openspec/specs/admin-ui/spec.md b/openspec/specs/admin-ui/spec.md index f6ef588..8a8ff43 100644 --- a/openspec/specs/admin-ui/spec.md +++ b/openspec/specs/admin-ui/spec.md @@ -163,8 +163,9 @@ reason of the first broken step of its diagnosis. The system SHALL show at `/apps/` the app's phase, the first stage that is not ready with its message and hint, its address, artifact, revision and deploy time, its components each with its phase and address (routed ones as links, the others marked `inside the cluster only` or `not -exposed`), its access settings, the names of its generated secrets, its verification as `shelf -app status` prints it, and the diagnosis chain under `How it gets here`. +exposed`), its access settings, the names of its generated secrets, its verification and its +isolation as `shelf app status` prints them, and the diagnosis chain with the tests under `How +it gets here`. #### Scenario: Routed and internal components - **WHEN** an app has a routed `web` component and an unrouted `db` component @@ -180,6 +181,10 @@ app status` prints it, and the diagnosis chain under `How it gets here`. - **WHEN** the running artifact of an app is unsigned - **THEN** the page says it is not verified and why, and that it cannot be exposed +#### Scenario: Failing test +- **WHEN** the test `smoke` of an app failed +- **THEN** the page shows the tests stage as Failed with the test's name and the end of its log + ### Requirement: Changes run as jobs The system SHALL run every change made from the UI (add an app, deploy another artifact or tag, deploy again, change an app's connections and domain, remove an app, save or remove a connection) @@ -242,7 +247,8 @@ report a failed audit write instead of hiding it. The system SHALL offer on the app page a form that deploys the app from another artifact or tag (`POST /apps//deploy`), a button that deploys the same tag again (`POST /apps//redeploy`), and a form that changes the registry connection, the Cloudflare -connection or quick tunnel, and the domain (`POST /apps//access`), each running as a job. +connection or quick tunnel, the domain and the LAN exceptions (`POST /apps//access`), each +running as a job. #### Scenario: Roll back to a commit tag - **WHEN** the deploy form is submitted with `oci://ghcr.io/owner/greeter:sha-905acce` @@ -254,6 +260,10 @@ connection or quick tunnel, and the domain (`POST /apps//access`), each ru - **THEN** a job `Change the connections of ` makes the app private; the choice `@quick` asks for a quick tunnel instead +#### Scenario: LAN exception from the browser +- **WHEN** the access form is submitted with the LAN exceptions `192.168.1.10/32` +- **THEN** the job gives the app that exception, as `shelf app credentials --allow-lan 192.168.1.10/32` does; an empty field takes the exceptions away, and an invalid range is answered with status 400 and the reason, without a job + ### Requirement: Removing an app needs its name typed The system SHALL remove an app with its namespace and volumes at `POST /apps//delete` only when the field `confirm` equals the app's name. diff --git a/openspec/specs/agent-interface/spec.md b/openspec/specs/agent-interface/spec.md index 0151b6a..358a8e5 100644 --- a/openspec/specs/agent-interface/spec.md +++ b/openspec/specs/agent-interface/spec.md @@ -48,12 +48,12 @@ know. A field without a value SHALL be omitted, never set to `null`. ### Requirement: Commands with JSON output `--json` SHALL be offered by `shelf app status` (kind `AppList` without a name, `AppStatus` with -one, `WaitResult` with `--wait`), `shelf app tags` (`TagList`), `shelf connection list` -(`ConnectionList`) and `shelf version` (`Version`). +one, `WaitResult` with `--wait`), `shelf app report` (`Report`), `shelf app tags` (`TagList`), +`shelf connection list` (`ConnectionList`) and `shelf version` (`Version`). #### Scenario: One app in detail - **WHEN** a user runs `shelf app status hello --json` for a healthy app -- **THEN** the `AppStatus` holds the app's name, phase, address, artifact, registry connection, exposure, revision, commit, deploy time, components with their addresses, and the five stages each with name, object, phase and, where present, reason, message and hint +- **THEN** the `AppStatus` holds the app's name, phase, address, artifact, registry connection, exposure, revision, commit, deploy time, components with their addresses, and the six stages each with name, object, phase and, where present, reason, message and hint #### Scenario: Tags - **WHEN** a user runs `shelf app tags hello --json` @@ -193,3 +193,33 @@ with `component`, `digest`, `scanned`, `critical`, `high`, `scanner` and `scanne #### Scenario: Agent reads why it cannot be exposed - **WHEN** the running artifact has no scan summary - **THEN** `verification.verified` is false and `reason` names the missing scan + +### Requirement: Tests and isolation in the agent interface +The `AppStatus` of an app or preview SHALL hold `tests`, one entry per test of the running +revision with `name`, `phase` and, when it failed, `exitCode` and `log` (its last lines), and +`isolation` with `podSecurity`, `restrictedViolations` (each with `pod` and `reason`) and +`lan` (the exceptions). + +#### Scenario: Failed test as JSON +- **WHEN** an agent runs `shelf app status hello --preview feature-login --json` and the test `smoke` failed +- **THEN** `tests` holds `smoke` with phase `Failed`, its exit code and its log, and the stage "the tests" is Failed + +### Requirement: Verification report +The system SHALL provide `shelf app report [--preview ]`, read-only, which prints a +Markdown report and with `--json` a `Report` document: the app or preview, its address and how to +open it, the commit, the state with the first stage that is not Ready, the tests, the +verification with the scan per component, and, for a preview, the comparison with the app: each +component whose image digest differs, and the CRITICAL and HIGH counts of the preview against +the app's. + +#### Scenario: Report for a pull request +- **WHEN** an agent runs `shelf app report hello --preview feature-login` for a Ready, verified preview whose `web` image differs from the app's +- **THEN** the Markdown names the commit, says Ready, lists the passed tests, the signature and scan, and lists `web` as changed with the findings of both + +#### Scenario: Report of an app +- **WHEN** the report is printed without `--preview` +- **THEN** it has no comparison section + +#### Scenario: Report as JSON +- **WHEN** `--json` is given +- **THEN** a `Report` document is printed with the same content, and no Markdown diff --git a/openspec/specs/app-chart/spec.md b/openspec/specs/app-chart/spec.md index 585537c..8e1dc9f 100644 --- a/openspec/specs/app-chart/spec.md +++ b/openspec/specs/app-chart/spec.md @@ -211,3 +211,31 @@ Without `platform.preview`, the chart MUST NOT render `shelf-secrets`. #### Scenario: Golden files - **WHEN** the chart tests render preview mode - **THEN** the golden files hold a placeholder instead of the random value, and no generated value is written to any file in the repository + +### Requirement: Tests as Helm test hooks +The chart SHALL render one Pod per entry in `tests`, named `-test-` and labelled +`shelf.dev/app` and `shelf.dev/test` but not `shelf.dev/component`, the annotations `helm.sh/hook: test` and +`helm.sh/hook-delete-policy: before-hook-creation`, `restartPolicy: Never`, the test's image, +command, args, env with references and secrets expanded as for a component, its resources, and +the image pull secret; it MUST NOT render a Service, Ingress or volume for a test. + +#### Scenario: Test pod +- **WHEN** the values of `hello` declare a test `smoke` with `image` and `args` +- **THEN** the chart renders the Pod `hello-test-smoke` with the hook annotations and `restartPolicy: Never`, and no other object for `smoke` + +#### Scenario: Test reads a secret +- **WHEN** the test `smoke` sets `env: { TOKEN: ${secrets.token} }` +- **THEN** the Pod reads `TOKEN` from the Secret `shelf-secrets`, as a component does + +### Requirement: Security defaults of every pod +The chart SHALL give every pod it renders, components and tests, the seccomp profile +`RuntimeDefault` and every container `allowPrivilegeEscalation: false`, and SHALL NOT set a user, +group or capabilities, which depend on the image. + +#### Scenario: Component +- **WHEN** the chart renders the component `web` +- **THEN** its pod template has `securityContext.seccompProfile.type: RuntimeDefault` and its container `securityContext.allowPrivilegeEscalation: false` + +#### Scenario: Image that starts as root +- **WHEN** an app runs `postgres:16`, whose entrypoint switches from root to the user postgres +- **THEN** it runs with these defaults, and Pod Security still lists it for `restricted`, because it runs as root diff --git a/openspec/specs/app-delivery/spec.md b/openspec/specs/app-delivery/spec.md index 6ee4b09..3a30cf0 100644 --- a/openspec/specs/app-delivery/spec.md +++ b/openspec/specs/app-delivery/spec.md @@ -309,11 +309,11 @@ omitting `sha256-` signature and attestation tags; no deploy path depends on thi - **WHEN** the app runs tag `main` - **THEN** the `main` line is marked with `*` -### Requirement: Five-stage diagnosis -The system SHALL diagnose an app as five ordered stages: the deploy artifact (OCIRepository +### Requirement: Stage diagnosis +The system SHALL diagnose an app as six ordered stages: the deploy artifact (OCIRepository `deploy`), the app values (Kustomization `deploy`), the chart (OCIRepository `shelf-app`), the -app (HelmRelease), and the pods; each stage is Ready, Working, Failed or Missing, and the app's -state is that of the first stage that is not Ready. +app (HelmRelease), the pods, and the tests; each stage is Ready, Working, Failed or Missing, and +the app's state is that of the first stage that is not Ready. #### Scenario: Image cannot be pulled - **WHEN** all Flux stages are Ready and a pod waits in `ImagePullBackOff` @@ -327,12 +327,16 @@ state is that of the first stage that is not Ready. - **WHEN** the platform has not created the app's objects yet - **THEN** the first stage is Missing with the message that the platform has not created the app yet +#### Scenario: Failing test +- **WHEN** every pod is ready and the test `smoke` exited with 1 +- **THEN** the tests stage is Failed naming `smoke`, and the app is Failed + ### Requirement: App status command `shelf app status` SHALL list every app with its state and, when not Ready, the failing stage and reason in one line; `shelf app status ` SHALL print the app's address, artifact, registry -connection, running revision and commit, deploy time, components, its verification, every stage -with its message and, if not Ready, its hint, and its previews with tag, state and commit; -`--preview ` SHALL print the same for one preview. +connection, running revision and commit, deploy time, components, its verification, its +isolation, every stage with its message and, if not Ready, its hint, and its previews with tag, +state and commit; `--preview ` SHALL print the same for one preview. #### Scenario: No apps - **WHEN** the cluster has no registered apps @@ -340,7 +344,7 @@ with its message and, if not Ready, its hint, and its previews with tag, state a #### Scenario: One app in detail - **WHEN** a user runs `shelf app status hello` for a healthy app -- **THEN** the output shows `hello: Ready`, the artifact reference, the revision, and all five stages as Ready +- **THEN** the output shows `hello: Ready`, the artifact reference, the revision, and all six stages as Ready #### Scenario: Commit of the running artifact - **WHEN** the app's deploy artifact was pushed by the reusable workflow for commit `42a964eb0b4b…` @@ -356,7 +360,7 @@ with its message and, if not Ready, its hint, and its previews with tag, state a #### Scenario: One preview - **WHEN** a user runs `shelf app status hello --preview feature/login` -- **THEN** the output shows the preview's namespace `hello--feature-login`, its host, artifact tag, commit, components and five stages +- **THEN** the output shows the preview's namespace `hello--feature-login`, its host, artifact tag, commit, components and six stages ### Requirement: App operations require the platform `shelf app add`, `shelf app credentials` and `shelf app redeploy`, and their admin UI jobs, MUST diff --git a/openspec/specs/app-manifest/spec.md b/openspec/specs/app-manifest/spec.md index 3df983e..e573b3f 100644 --- a/openspec/specs/app-manifest/spec.md +++ b/openspec/specs/app-manifest/spec.md @@ -11,8 +11,8 @@ and the platform, carried in the deploy artifact as the ConfigMap `shelf-values` ### Requirement: Document structure and strict parsing The system SHALL accept an app.yaml only as a single YAML document with the top-level fields -`apiVersion`, `name`, `components` and optional `secrets`, and SHALL reject unknown fields, -duplicate keys, wrong types, empty files and multiple documents as parse errors. +`apiVersion`, `name`, `components` and optional `secrets` and `tests`, and SHALL reject unknown +fields, duplicate keys, wrong types, empty files and multiple documents as parse errors. #### Scenario: Unknown field - **WHEN** an app.yaml contains a field that is not part of the schema, at any level including inside a long-form `route` @@ -209,11 +209,25 @@ image's TCP `EXPOSE` list (images without any `EXPOSE` are not checked). - **WHEN** a component declares port 8080 and its image exposes only 80 - **THEN** a warning `port 8080 is not exposed by image ... (EXPOSE 80)` is printed and rendering continues +### Requirement: Reading a resolved app.yaml from the cluster +The system SHALL read the resolved app.yaml in an app's ConfigMap `shelf-values` leniently, +leaving out fields it does not know, at any level including a long-form `route`, so that a shelf +older than the workflow that built the artifact can still show the app; `shelf validate` and +`shelf render` stay strict. + +#### Scenario: Artifact from a newer workflow +- **WHEN** the ConfigMap of an app holds a section this shelf does not know, such as `tests` for a shelf from before Phase 12 +- **THEN** `shelf app status` and the app page show the app and its components as before + +#### Scenario: What a person writes +- **WHEN** `shelf validate` reads an app.yaml with an unknown field +- **THEN** it still reports a parse error + ### Requirement: Built components and --image -The system SHALL take the image of every component with `build` from a repeatable flag -`--image =` and SHALL fail when a built component has no `--image`, when -`--image` names an unknown component or a component with `image`, when a component is given -twice, or when the flag is not of the form `=`. +The system SHALL take the image of every component and test with `build` from a repeatable flag +`--image =` and SHALL fail when a built component or test has no `--image`, when +`--image` names an unknown component or test or one with `image`, when a name is given twice, or +when the flag is not of the form `=`. #### Scenario: Built image passed back - **WHEN** `shelf render --image web=ghcr.io/owner/greeter/web:sha-abc1234` is run for `examples/tenant/app.yaml` @@ -223,6 +237,10 @@ twice, or when the flag is not of the form `=`. - **WHEN** a component has `build: ./web` and no `--image web=...` is given - **THEN** `shelf render` fails with an error telling the user to pass `--image web=` +#### Scenario: Built test +- **WHEN** a test `e2e` has `build: ./e2e` and `--image e2e=ghcr.io/owner/greeter/e2e:sha-abc1234` is given +- **THEN** the resolved test `e2e` has that image pinned to its digest + ### Requirement: Render output The system SHALL write the render result to stdout and everything else to stderr: by default (`--output configmap`) a ConfigMap `v1` named `shelf-values` without namespace or labels whose @@ -253,9 +271,10 @@ The system SHALL deliver each secret referenced by a component as the environmen ### Requirement: shelf build-plan The system SHALL provide `shelf build-plan ` that prints JSON -`{"package": , "builds": [{"component", "context"}]}` listing every component with `build` -in component name order, with each context relative to the working directory, an empty list when -nothing is built, and an error when a build directory does not exist. +`{"package": , "builds": [{"component", "context"}]}` listing every component and test with +`build`, components first, each in name order, with each context relative to the working +directory, an empty list when nothing is built, and an error when a build directory does not +exist. #### Scenario: Tenant example - **WHEN** `shelf build-plan examples/tenant/app.yaml` is run @@ -265,6 +284,10 @@ nothing is built, and an error when a build directory does not exist. - **WHEN** a component's `build` directory does not exist - **THEN** the command fails with an error naming the component and the directory +#### Scenario: Built test +- **WHEN** an app.yaml has a built component `web` and a built test `e2e` +- **THEN** the plan lists `web`, then `e2e`, so the workflow builds and pushes `ghcr.io///e2e` + ### Requirement: shelf schema The system SHALL provide `shelf schema` that prints the JSON Schema (draft 2020-12) for app.yaml, generated from the same definitions as the parser and committed as `schema/app.schema.json`, @@ -275,3 +298,21 @@ including the name patterns, the exclusive choice of `image` or `build`, and the - **WHEN** `shelf schema` is run - **THEN** it prints a JSON Schema with `$id` `https://raw.githubusercontent.com/tweinmann/shelf/main/schema/app.schema.json` - **AND** the schema accepts every app.yaml under `examples/` + +### Requirement: Tests section +The system SHALL accept an optional top-level `tests` map; each test name MUST be a DNS-1035 label +of at most 40 characters that is not also a component name, and each test MUST have exactly one of +`image` or `build` and MAY have `command`, `args`, `env` and `resources`, with the same rules and +references as a component; a test MUST NOT have ports, a route, volumes, health or instances. + +#### Scenario: Test against the web component +- **WHEN** an app.yaml has `tests: { smoke: { image: curlimages/curl:8.10.1, args: [-fsS, "http://${web.host}:${web.port}/healthz"] } }` +- **THEN** validation passes, and `shelf render` pins the test image to its digest like a component image + +#### Scenario: Name taken by a component +- **WHEN** a test and a component are both named `web` +- **THEN** validation reports an error at `tests.web` + +#### Scenario: Test with a route +- **WHEN** a test declares `route: /` +- **THEN** parsing fails, as `route` is not a field of a test diff --git a/openspec/specs/app-tests/spec.md b/openspec/specs/app-tests/spec.md new file mode 100644 index 0000000..4dc7af3 --- /dev/null +++ b/openspec/specs/app-tests/spec.md @@ -0,0 +1,51 @@ +# app-tests Specification + +## Purpose +Lets an app check itself after every rollout: the tests its `app.yaml` declares run against the +app or preview in its own namespace, and an app counts as green only when they pass. + +## Requirements + +### Requirement: Tests run after every rollout +The platform SHALL run every test an app declares after each install and upgrade of the app's or +preview's release, in its namespace, one pod per test with the app's resources defaults and a +timeout of 5 minutes for all of them together; a test passes when its container exits with 0. + +#### Scenario: New commit with a test +- **WHEN** a new artifact of an app with a test `smoke` rolls out +- **THEN** the pod for `smoke` runs after the components are ready, and its result is recorded for that release revision + +#### Scenario: App without tests +- **WHEN** an app declares no tests +- **THEN** its rollout runs no test, and the tests stage is Ready + +### Requirement: A failing test on an app keeps the previous version +When a test fails after an upgrade of an app, the platform SHALL count the upgrade as failed, so +the release returns to the last revision whose tests passed; after a failed first install the +failing version keeps running. + +#### Scenario: Broken commit on main +- **WHEN** a new artifact of `hello` makes its test `smoke` exit with 1 +- **THEN** `hello` runs its previous version again, and the tests stage is Failed naming `smoke` + +### Requirement: A failing test on a preview keeps the preview +When a test fails on a preview, the platform SHALL keep the version that failed running, so the +branch can be looked at with `shelf app open --preview`. + +#### Scenario: Broken branch +- **WHEN** the artifact of `branch-feature-login` makes the test `smoke` fail +- **THEN** the preview runs that artifact, and its tests stage is Failed naming `smoke` + +### Requirement: Tests in the status and the wait +The tests stage SHALL follow the pods stage: Ready when every test of the running revision passed +or there are none, Working while they run, Failed when one failed, with the failing test's name, +its exit code and the last lines of its log as the message; `shelf app status --wait --commit` +SHALL end with `Ready` only after the tests of that commit passed. + +#### Scenario: Waiting for a commit whose test fails +- **WHEN** an agent runs `shelf app status hello --preview feature-login --wait --commit ` and the test `smoke` fails for that commit +- **THEN** the verdict is `Failed` with the stage "the tests", the test name and the end of its log, and the exit status is 1 + +#### Scenario: Tests still running +- **WHEN** the pods are ready and the test `smoke` still runs +- **THEN** the app is Working at the stage "the tests" diff --git a/openspec/specs/branch-previews/spec.md b/openspec/specs/branch-previews/spec.md index 6a87a24..2b7dfc1 100644 --- a/openspec/specs/branch-previews/spec.md +++ b/openspec/specs/branch-previews/spec.md @@ -76,10 +76,11 @@ answer at `--.shelf.internal` inside the cluster, and carry the label ### Requirement: Preview contents The platform SHALL give each preview the same deploy chain as an app (its deploy artifact at the -branch tag, checked every 30 seconds, the app values, the chart, the release), verified against -the cluster's signer, the app's registry login, a ResourceQuota (requests of 1 CPU, 2Gi memory, -5Gi storage) and a LimitRange that gives containers without requests a default; it MUST NOT give -a preview a tunnel, cloudflared or a DNS record. +branch tag, checked every 30 seconds, the app values, the chart, the release, the tests), +verified against the cluster's signer, the app's registry login, the isolation of an app without +LAN exceptions, a ResourceQuota (requests of 1 CPU, 2Gi memory, 5Gi storage) and a LimitRange that +gives containers without requests a default; it MUST NOT give a preview a tunnel, cloudflared or a +DNS record. #### Scenario: Preview of an exposed app - **WHEN** an app exposed through a Cloudflare connection has a preview @@ -97,6 +98,10 @@ a preview a tunnel, cloudflared or a DNS record. - **WHEN** a preview's values name an unsigned built image of the app's package - **THEN** its pod is denied by the app's admission policy, as in the app's namespace +#### Scenario: Preview of an app with a LAN exception +- **WHEN** `hello` may reach `192.168.1.10/32` and has the preview `hello--feature-login` +- **THEN** the preview's pods cannot reach `192.168.1.10`, nor the pods of `hello` + ### Requirement: A preview's own state A preview SHALL have its own volumes and its own secret values, generated in the cluster when the preview is first created and kept for its lifetime; it MUST NOT share a volume, a secret value or a diff --git a/openspec/specs/dev-environment/spec.md b/openspec/specs/dev-environment/spec.md index 09ea6dd..1aca12d 100644 --- a/openspec/specs/dev-environment/spec.md +++ b/openspec/specs/dev-environment/spec.md @@ -80,7 +80,7 @@ Cloudflare), and a task that needs level 2 or 3 MUST say so. #### Scenario: Level 2 in full - **WHEN** a commit is pushed to `main`, or a pull request carries the label `level-2-full` -- **THEN** CI runs `smoke-secrets`, `smoke-chart`, `smoke-init`, `smoke-apps`, `smoke-agent`, `smoke-previews` and `smoke-verify` with `SMOKE_FULL=1` +- **THEN** CI runs `smoke-secrets`, `smoke-chart`, `smoke-init`, `smoke-apps`, `smoke-agent`, `smoke-previews`, `smoke-verify` and `smoke-hardening` with `SMOKE_FULL=1` #### Scenario: main fails - **WHEN** CI fails on `main` diff --git a/openspec/specs/isolation/spec.md b/openspec/specs/isolation/spec.md new file mode 100644 index 0000000..2bdc8b5 --- /dev/null +++ b/openspec/specs/isolation/spec.md @@ -0,0 +1,78 @@ +# isolation Specification + +## Purpose +Fences in what an app or preview runs: which pods its namespace admits, and which addresses its +pods can reach, so that code an agent wrote reaches neither other apps nor the operator's LAN +unless the operator allowed it for one app. + +## Requirements + +### Requirement: Pod Security of app and preview namespaces +The platform SHALL label every app and preview namespace to enforce the Pod Security level +`baseline` and to warn and audit `restricted`, both at the version `latest`. + +#### Scenario: Privileged pod +- **WHEN** an app's values would start a privileged container or one with a `hostPath` volume +- **THEN** the pod is refused, and the pods stage shows the refusal with a hint about Pod Security + +#### Scenario: Image that starts as root +- **WHEN** an app runs `postgres:16`, which starts as root +- **THEN** the pod runs + +### Requirement: Network isolation of app and preview namespaces +The platform SHALL give every app and preview namespace a NetworkPolicy that selects all its pods +and allows ingress only from pods in the same namespace and from the Traefik namespace, and +egress only to pods in the same namespace, to the cluster DNS on port 53, and to any address +outside `10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16`, `100.64.0.0/10`, `169.254.0.0/16`, +`fc00::/7` and `fe80::/10`. + +#### Scenario: Another app +- **WHEN** a pod of the preview `hello--feature-login` connects to the service of the app `greeter` +- **THEN** the connection fails + +#### Scenario: Traffic within the app +- **WHEN** the `web` pod of `hello` connects to `db:5432` in its own namespace +- **THEN** the connection succeeds + +#### Scenario: LAN and metadata +- **WHEN** a pod connects to `192.168.1.10` or `169.254.169.254` +- **THEN** the connection fails, and a connection to a public address succeeds + +#### Scenario: Routed and exposed +- **WHEN** a request reaches the app through Traefik, or through cloudflared in the app namespace +- **THEN** it is answered as before + +### Requirement: LAN exceptions are the operator's decision +`shelf app add` and `shelf app credentials` SHALL take `--allow-lan `, repeatable, which +replaces the app's list of private address ranges it may reach, and `--deny-lan`, which clears +the list; each range MUST lie inside one of the private ranges of the isolation, and the +exceptions MUST NOT be part of `app.yaml` and MUST NOT apply to the app's previews. + +#### Scenario: NAS +- **WHEN** an operator runs `shelf app credentials hello --allow-lan 192.168.1.10/32` +- **THEN** pods of `hello` reach `192.168.1.10` and no other private address, and pods of its previews reach none + +#### Scenario: Not a private range +- **WHEN** `--allow-lan 8.8.8.0/24` or `--allow-lan nas.local` is given +- **THEN** the command fails before it changes anything, saying the value must be a private CIDR + +#### Scenario: Range of the cluster's pods +- **WHEN** `--allow-lan 10.42.0.0/16` is given on a cluster whose nodes have pod ranges inside it +- **THEN** the command fails before it changes anything, saying the range would reach other apps + +#### Scenario: Unchanged by redeploy +- **WHEN** an app with an exception is deployed again or pointed at another tag without `--allow-lan` +- **THEN** it keeps its exception + +### Requirement: Isolation in the status +`shelf app status ` and `--preview` SHALL show the Pod Security level, the pods that would +violate `restricted` with the first reason for each, and the LAN exceptions, or `none`; reading +them MUST NOT change the cluster. + +#### Scenario: Image that starts as root +- **WHEN** `hello` runs `postgres:16` in its `db` component +- **THEN** the status shows `pod security baseline`, and lists the `db` pod as violating `restricted` with its reason + +#### Scenario: No exceptions +- **WHEN** an app has no LAN exceptions +- **THEN** the status shows `lan none`