diff --git a/AGENTS.md b/AGENTS.md index 9835452..15dae22 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -80,12 +80,18 @@ docker compose up -d --build --no-deps control ``` Then confirm the server is actually serving your change — `curl` the asset and -grep it — before concluding anything about whether the change worked. The -browser caches these aggressively too, so a hard reload is part of the loop. +grep it — before concluding anything about whether the change worked. This has already sent one session off to debug a fix that was correct the whole time. +A hard reload used to be part of that loop as well, because the assets went out +with no `ETag` and no `Cache-Control` and the browser was free to keep whatever +it had. It no longer is: every asset is tagged with a hash of its own bytes and +marked `no-cache`, so an ordinary reload revalidates and picks up the rebuild. A +304 keeps that cheap. If you find yourself reaching for Cmd+Shift+R to see a +change, that is a bug worth reporting rather than a step worth remembering. + ## Verify in the lab, not only in the tests `make check` runs without Docker and finishes in about a second. It is necessary diff --git a/CLAUDE.md b/CLAUDE.md index 387504c..f31902f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -31,9 +31,10 @@ You can run the lab, drive a browser and call the API, which means "I believe this works" is almost never the best sentence available to you. Run it, and report what came back. -Two traps that make a correct change look broken, both covered in AGENTS.md: -the UI is `//go:embed`-ed and needs the control container rebuilt, and the -browser caches the assets hard. +One trap that makes a correct change look broken, covered in AGENTS.md: the UI +is `//go:embed`-ed and needs the control container rebuilt. The browser no +longer needs a hard reload on top of that — the assets carry an `ETag` and +revalidate — but the rebuild is still yours to run. ## The lab on this machine is probably in use diff --git a/lab/README.md b/lab/README.md index 28654a7..00a2c2b 100644 --- a/lab/README.md +++ b/lab/README.md @@ -347,6 +347,48 @@ key the coordination server holds before and after, so "it re-keyed" is something the run found rather than something the button claims. It takes about a minute. +#### Every attack that crosses the public segment measures the route first + +`make attack` creates two containers, not one: `evil-box` and `nat-evil`, the +router that carries its packets off its own segment. Stop the router and the +attacker keeps running, keeps its default route, and reaches nothing — +`ping 203.0.113.3` to the lab's own relay is 100% loss while +`docker ps` and `/api/state` both still say `online`. + +That state used to score. `nmap` prints `filtered` both for "a firewall dropped +this" and for "nothing carried this", and at rung 4 there is nothing else to +tell them apart, so **Scan from the open internet** answered a stopped router +with `ok`, rung 4 and `ufw default incoming policy: deny` — byte for byte the +verdict a run where the firewall really did the work gets. Measured on Docker +29.7.2 on 2026-09-12, and it was the same for the other six attacks that have +to leave that segment, and for five of the eleven audit checks, where five +failures to reach read as five passes. + +So the precondition is no longer "the attacker's container is running". Before +any of those seven scores, one packet goes from `evil-box` to the relay at +`203.0.113.3`, and a run that gets no answer refuses at rung 1, naming the +router rather than a defence: + +``` +{"ok": false, "rung": 1, "rule": "nat-evil is stopped", + "why": "The attacker's own router is down, so it reaches nothing at all — not + lab-vps, not the coordination server, not the relay. Its route survived + the outage; the gateway at 10.0.66.254 did not. …"} +``` + +The relay is the target on purpose: it sits on the public segment, holds none of +your firewall rules, and no configuration in this lab changes it. Aiming at +`lab-vps` would ask two questions in one packet. The router is not restarted for +you either — a stopped router is a network condition, like a downed link, and +this lab's job with those is to measure them. The one control that does put it +back is `evil-box`'s own switch in the **Machines** panel, which brings up the +pair and says so, and `/api/state` now carries `routerDown` for any machine +whose NAT box is stopped, which the drawing dims and labels. + +**Sit on the wire and capture** and **Replay a captured frame** are the two that +still ask only for `evil-box`, because `nat-evil` carries none of their packets: +both work on the segment `evil-box` already shares with `lab-ubuntu`. + #### The capture has a control experiment in it "We captured the traffic and could not read it" proves very little on its own: @@ -1274,6 +1316,11 @@ that carries everything. its log, and `make reset && make up` is the clean way through. - A machine never joins. Try `docker compose logs lab-ubuntu`. The usual cause is that the lab CA was not there when it booted; `make reset && make up`. +- Every attack refuses at rung 1 with `nat-evil is stopped` or + `nat-evil is not in this stack`. That is the refusal working: the attacker + cannot reach anything, so nothing it does can be scored. `docker start + nat-evil`, or `make attack` if it was never created — starting `evil-box` + on its own with `--no-deps` leaves exactly this. - Every probe dies at rung 3. Check `docker exec headscale headscale policy get`, then `docker exec headscale headscale nodes list` and confirm each machine carries the tag the policy names. See the Headscale wrinkle above. diff --git a/lab/checks/reading.test.mjs b/lab/checks/reading.test.mjs index 9f97b9e..92c7c1d 100644 --- a/lab/checks/reading.test.mjs +++ b/lab/checks/reading.test.mjs @@ -27,8 +27,8 @@ import test from "node:test"; import assert from "node:assert/strict"; import { - ev, rung, tone, ladderMark, neverRan, head, scanReading, sniffReading, - tailcatReading + ev, rung, tone, ladderMark, neverRan, head, whyNothingRan, scanReading, + sniffReading, tailcatReading } from "../control/ui/hud/reading.js"; /* ---------- tone, and the rule that decides it ------------------- */ @@ -79,6 +79,20 @@ test("neverRan separates the attacker not starting from the defence holding", () assert.equal(neverRan({ ok: false, danger: true, rung: 1 }), false); }); +test("a run that never ran says which of the three reasons it was", () => { + /* The attacker not being there was once the only way an attack could + refuse, so the heads asked "is evil-box running?" and were right. A + stopped nat-evil refuses with evil-box running, and a head that still + asked that question would be pointing at the wrong container. */ + assert.equal(whyNothingRan({ rung: 1, rule: "evil-box is not in this stack" }), + "evil-box is not in this stack"); + assert.equal(whyNothingRan({ rung: 1, rule: "nat-evil is stopped" }), + "nat-evil is stopped"); + /* And a refusal that named no rule at all still asks something useful. */ + assert.equal(whyNothingRan({ rung: 1 }), "is evil-box running? `make attack`"); + assert.equal(whyNothingRan(null), "is evil-box running? `make attack`"); +}); + test("absent evidence is zero, not missing and not some", () => { assert.equal(ev({ evidence: { frames: 12 } }, "frames"), 12); assert.equal(ev({ evidence: { frames: 12 } }, "cleartext"), 0); diff --git a/lab/control/actions.go b/lab/control/actions.go index 489aadd..788c7a2 100644 --- a/lab/control/actions.go +++ b/lab/control/actions.go @@ -345,6 +345,29 @@ func (c *Controller) setOnline(ctx context.Context, id string, up bool) Result { res.Cmds = []string{"docker compose start " + id} err = c.lab.Start(ctx, id) res.Why = id + " is running" + // A machine and the router in front of it come up together in compose, + // and this switch used to know only about the machine. Starting + // evil-box while nat-evil is stopped put an attacker on the board that + // could reach nothing — which every attack then scored as a firewall + // holding. So the pair goes back up together, and when the router is + // not there to start, the switch says that instead of leaving the + // reader to find the other half. + m, known := machineByID(id) + switch { + case err != nil || !known || m.Router == "" || c.lab.Running(ctx, m.Router): + case !c.lab.Exists(ctx, m.Router): + res.Why = id + " is running, and " + m.Router + " — the router in front of it —" + + " does not exist, so it reaches nothing off its own segment. That is what " + + "`--no-deps` leaves behind; `make attack` brings up the pair." + default: + if rerr := c.lab.Start(ctx, m.Router); rerr == nil { + res.Cmds = append(res.Cmds, "docker compose start "+m.Router+ + " # its router, or it reaches nothing") + res.Why = id + " is running, and so is " + m.Router + " — the router in " + + "front of it, which was stopped. Without that one the machine is up " + + "and reaching nothing, and nothing measured from there is worth a score." + } + } } else { res.Cmds = []string{"docker compose stop " + id} err = c.lab.Stop(ctx, id) @@ -387,6 +410,15 @@ func (c *Controller) evilJoin(ctx context.Context, join bool) Result { "a handshake with it. Turn lock off and try the same thing again." return res } + // Lock first, because that refusal is a fact about the lab whether or not + // anything could have been reached. After it, a join is a packet to the + // coordination server on the public segment — and without a route there + // `tailscale up` spends its thirty seconds and reports a timeout, which is + // true and no help at all beside the one fact that explains it. + if r := c.needRouteOut(ctx); r != nil { + r.Rule = "it cannot reach the coordination server: " + r.Rule + return *r + } if c.untrustedKey == "" { res.Why = "no spare key was created at boot — try `make reset`" return res @@ -458,16 +490,12 @@ func (c *Controller) setOnPath(ctx context.Context, on bool) Result { return res } +// gatewayFor is the address the machine's own router answers on. It reads the +// catalog rather than repeating it, because the same three addresses are now +// also what names the fault when one of those routers is down. func (c *Controller) gatewayFor(id string) string { - switch id { - case "lab-ubuntu": - return "10.0.13.254" - case "lab-roam": - return "10.0.27.254" - case "evil-box": - return "10.0.66.254" - } - return "" + m, _ := machineByID(id) + return m.Gateway } // --------------------------------------------------------------------------- diff --git a/lab/control/assets.go b/lab/control/assets.go new file mode 100644 index 0000000..c99101c --- /dev/null +++ b/lab/control/assets.go @@ -0,0 +1,89 @@ +// SPDX-License-Identifier: GPL-3.0-or-later +// Copyright (C) 2026 Hashem Aldhaheri + +package main + +import ( + "crypto/sha256" + "encoding/hex" + "io/fs" + "net/http" + "path" + "strings" +) + +// The UI is //go:embed-ed, so the server knows the exact bytes of every asset +// at build time and can say so. Without that, style.css and app.js went out +// with no Cache-Control, no ETag and no Last-Modified — embed.FS reports a zero +// modtime, so net/http had nothing to put there — and a browser handed a +// response with no freshness information at all is free to invent one. The loop +// this lab is built around is edit, `docker compose up -d --build --no-deps +// control`, look at the page; a rebuilt server that does not change the page +// reads as a broken fix, and that cost a session on #68. +// +// So every asset carries an ETag over its own bytes and Cache-Control: +// no-cache, which does not mean "do not store" — it means store it and ask +// before using it. The browser revalidates on every load and gets a 304 with no +// body back when nothing moved. One round trip against a server on loopback, +// and the page can never be a new app.js against yesterday's style.css. +// +// The hash is computed once at startup rather than per request: the bytes are +// in the binary and cannot change while it runs. + +// staticUI serves fsys with a per-file ETag and Cache-Control: no-cache. +// +// It sets the header and then hands off to http.FileServer, which is not a +// trick: http.ServeContent reads the ETag already on the ResponseWriter when it +// evaluates If-None-Match, so the 304 handling, range requests and content +// sniffing all stay where they were. +func staticUI(fsys fs.FS) (http.Handler, error) { + tags, err := assetETags(fsys) + if err != nil { + return nil, err + } + files := http.FileServer(http.FS(fsys)) + return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if tag, ok := tags[assetName(r.URL.Path)]; ok { + w.Header().Set("ETag", tag) + w.Header().Set("Cache-Control", "no-cache") + } + files.ServeHTTP(w, r) + }), nil +} + +// assetETags hashes every file in fsys, keyed by its path within fsys. +func assetETags(fsys fs.FS) (map[string]string, error) { + tags := map[string]string{} + err := fs.WalkDir(fsys, ".", func(p string, d fs.DirEntry, err error) error { + if err != nil || d.IsDir() { + return err + } + b, err := fs.ReadFile(fsys, p) + if err != nil { + return err + } + sum := sha256.Sum256(b) + // Half of SHA-256 is far more than enough to tell two versions of one + // stylesheet apart, and a shorter header is easier to read in curl -I. + tags[p] = `"` + hex.EncodeToString(sum[:16]) + `"` + return nil + }) + if err != nil { + return nil, err + } + return tags, nil +} + +// assetName maps a request path to the name of the embedded file +// http.FileServer will serve for it, so the two agree on what is being tagged. +// A directory gets its index.html, which is how / becomes index.html. +func assetName(p string) string { + if !strings.HasPrefix(p, "/") { + p = "/" + p + } + // Before Clean, which strips the trailing slash that marks a directory. + if strings.HasSuffix(p, "/") { + p += "index.html" + } + return strings.TrimPrefix(path.Clean(p), "/") +} diff --git a/lab/control/assets_test.go b/lab/control/assets_test.go new file mode 100644 index 0000000..8902515 --- /dev/null +++ b/lab/control/assets_test.go @@ -0,0 +1,192 @@ +// SPDX-License-Identifier: GPL-3.0-or-later +// Copyright (C) 2026 Hashem Aldhaheri + +package main + +import ( + "io/fs" + "net/http" + "net/http/httptest" + "testing" + "testing/fstest" +) + +func testUI() fs.FS { + return fstest.MapFS{ + "index.html": {Data: []byte("")}, + "style.css": {Data: []byte(".panel { color: red }")}, + "app.js": {Data: []byte("export const x = 1\n")}, + "hud/scene.js": {Data: []byte("export function scene() {}\n")}, + } +} + +func serveUI(t *testing.T, fsys fs.FS) http.Handler { + t.Helper() + h, err := staticUI(fsys) + if err != nil { + t.Fatalf("staticUI: %v", err) + } + return h +} + +func get(t *testing.T, h http.Handler, path string, headers map[string]string) *http.Response { + t.Helper() + r := httptest.NewRequest(http.MethodGet, path, nil) + for k, v := range headers { + r.Header.Set(k, v) + } + w := httptest.NewRecorder() + h.ServeHTTP(w, r) + return w.Result() +} + +// Every asset answers with something the browser can revalidate against. This +// is the whole bug in #70: with no ETag and no Cache-Control, a rebuilt server +// does not change what the reader sees. +func TestAssetsCarryAnETagAndRevalidate(t *testing.T) { + h := serveUI(t, testUI()) + + // Not /index.html: http.FileServer redirects that to / on its own, and + // that predates this change. + for _, path := range []string{"/", "/style.css", "/app.js", "/hud/scene.js"} { + res := get(t, h, path, nil) + if res.StatusCode != http.StatusOK { + t.Fatalf("%s: status %d, want 200", path, res.StatusCode) + } + if got := res.Header.Get("ETag"); got == "" { + t.Errorf("%s: no ETag", path) + } + if got := res.Header.Get("Cache-Control"); got != "no-cache" { + t.Errorf("%s: Cache-Control %q, want %q", path, got, "no-cache") + } + } +} + +// The cheap half: an unchanged asset comes back as a bodiless 304, so +// revalidating on every load costs a round trip and not a stylesheet. +func TestUnchangedAssetIsNotModified(t *testing.T) { + h := serveUI(t, testUI()) + + first := get(t, h, "/style.css", nil) + tag := first.Header.Get("ETag") + if tag == "" { + t.Fatal("no ETag on the first response") + } + + second := get(t, h, "/style.css", map[string]string{"If-None-Match": tag}) + if second.StatusCode != http.StatusNotModified { + t.Fatalf("status %d, want 304", second.StatusCode) + } + if n := second.ContentLength; n > 0 { + t.Errorf("304 carried %d bytes of body", n) + } + if got := second.Header.Get("ETag"); got != tag { + t.Errorf("304 ETag %q, want %q", got, tag) + } +} + +// The point of hashing the bytes rather than stamping a build: edit the file, +// and the tag the browser is holding stops matching. +func TestEditedAssetGetsANewETag(t *testing.T) { + before := get(t, serveUI(t, testUI()), "/style.css", nil).Header.Get("ETag") + + edited := testUI().(fstest.MapFS) + edited["style.css"] = &fstest.MapFile{Data: []byte(".panel { color: blue }")} + after := get(t, serveUI(t, edited), "/style.css", nil).Header.Get("ETag") + + if before == "" || after == "" { + t.Fatalf("missing ETag: before %q, after %q", before, after) + } + if before == after { + t.Errorf("ETag %q survived an edit to the file", before) + } + + // And the stale tag no longer satisfies the request, so the browser is + // handed the new bytes rather than told to keep the old ones. + res := get(t, serveUI(t, edited), "/style.css", map[string]string{"If-None-Match": before}) + if res.StatusCode != http.StatusOK { + t.Errorf("status %d for a stale ETag, want 200", res.StatusCode) + } +} + +// Two assets that differ must not share a tag, or a page can be a new app.js +// against an old style.css with nothing indicating it. +func TestETagsAreNotSharedBetweenAssets(t *testing.T) { + h := serveUI(t, testUI()) + + seen := map[string]string{} + for _, path := range []string{"/", "/style.css", "/app.js", "/hud/scene.js"} { + tag := get(t, h, path, nil).Header.Get("ETag") + if other, clash := seen[tag]; clash { + t.Errorf("%s and %s share the ETag %s", other, path, tag) + } + seen[tag] = path + } +} + +// The tag on / has to be the tag of index.html, the file / actually serves, or +// the root document is the one page that never revalidates correctly. +func TestRootIsTaggedWithIndexHTML(t *testing.T) { + fsys := testUI() + tags, err := assetETags(fsys) + if err != nil { + t.Fatalf("assetETags: %v", err) + } + + root := get(t, serveUI(t, fsys), "/", nil).Header.Get("ETag") + if root != tags["index.html"] { + t.Errorf("/ tagged %q, index.html hashes to %q", root, tags["index.html"]) + } +} + +func TestAssetName(t *testing.T) { + cases := map[string]string{ + "/": "index.html", + "/index.html": "index.html", + "/style.css": "style.css", + "/hud/": "hud/index.html", + "/hud/scene.js": "hud/scene.js", + "/./style.css": "style.css", + "/hud/../app.js": "app.js", + "/../../etc/pass": "etc/pass", + "style.css": "style.css", + } + for in, want := range cases { + if got := assetName(in); got != want { + t.Errorf("assetName(%q) = %q, want %q", in, got, want) + } + } +} + +// A path with no file behind it is served exactly as it was before: no tag +// invented for something that does not exist, and net/http still 404s. +func TestMissingAssetIsUntagged(t *testing.T) { + res := get(t, serveUI(t, testUI()), "/nope.css", nil) + if res.StatusCode != http.StatusNotFound { + t.Fatalf("status %d, want 404", res.StatusCode) + } + if got := res.Header.Get("ETag"); got != "" { + t.Errorf("404 carried ETag %q", got) + } +} + +// The real embedded UI, not a fixture: the files this lab actually ships are +// the ones that were going out unrevalidatable. +func TestEmbeddedUIIsTagged(t *testing.T) { + ui, err := fs.Sub(uiFS, "ui") + if err != nil { + t.Fatalf("embedded UI: %v", err) + } + h := serveUI(t, ui) + + for _, path := range []string{"/", "/style.css", "/app.js"} { + res := get(t, h, path, nil) + if res.StatusCode != http.StatusOK { + t.Errorf("%s: status %d, want 200", path, res.StatusCode) + } + if res.Header.Get("ETag") == "" || res.Header.Get("Cache-Control") != "no-cache" { + t.Errorf("%s: ETag %q, Cache-Control %q", + path, res.Header.Get("ETag"), res.Header.Get("Cache-Control")) + } + } +} diff --git a/lab/control/attacks.go b/lab/control/attacks.go index 2f6b5c2..963b0d0 100644 --- a/lab/control/attacks.go +++ b/lab/control/attacks.go @@ -104,7 +104,8 @@ func (c *Controller) RunAttack(ctx context.Context, id string) Result { return Result{Why: "no attack called " + id} } -// needEvil is the one precondition every attack shares. +// needEvil is the one precondition every attack shares: the attacker has to +// exist. It is not the whole precondition for most of them — see needEvilOut. func (c *Controller) needEvil(ctx context.Context) *Result { if c.lab.Running(ctx, "evil-box") { return nil @@ -119,6 +120,113 @@ func (c *Controller) needEvil(ctx context.Context) *Result { return &r } +// needEvilOut is that precondition plus the half a stopped router walked +// straight through: the attacker also has to be able to reach this stack. +// +// `nmap` prints `filtered` both for "a firewall dropped this" and for "nothing +// carried this", and at rung 4 there is nothing else to separate them. So an +// attack run with nat-evil stopped measured no packet at all and credited +// `ufw default incoming policy: deny` by name for the silence — the same +// verdict, byte for byte, as a run where the firewall really did the work. +// That is a block nobody measured, and refusing to score is the right failure: +// "I could not reach anything" teaches more than "the firewall held". +// +// The two attacks that work on the wire evil-box shares with lab-ubuntu keep +// asking for needEvil instead, because no router carries those packets. +func (c *Controller) needEvilOut(ctx context.Context) *Result { + if r := c.needEvil(ctx); r != nil { + return r + } + return c.needRouteOut(ctx) +} + +// needRouteOut measures the route rather than inferring it from the router's +// state, because a running router is not the same claim as a working one — a +// link this lab can take down with a switch sits between them. The router is +// still asked about afterwards, but only to name the fault. +// +// It deliberately does not start nat-evil the way needEvil starts evil-box. A +// stopped router is a network condition, like a downed link, and the lab's job +// with those is to measure and report them rather than to tidy them away +// behind the reader's back. +func (c *Controller) needRouteOut(ctx context.Context) *Result { + evil, _ := machineByID("evil-box") + ping, _ := c.lab.Sh(ctx, "evil-box", + fmt.Sprintf("ping -c 1 -W 2 %s 2>&1", labRelayAddr)) + if pingGotThrough(ping.Out()) { + return nil + } + r := evilStranded(evil.Router, evil.Gateway, + c.lab.Exists(ctx, evil.Router), c.lab.Running(ctx, evil.Router), + strings.TrimSpace(ping.Out())) + return &r +} + +// labRelayAddr is the lab's own DERP relay, and the reachability probe aims at +// it on purpose. It sits on the segment standing in for the public internet, it +// holds no policy and none of the reader's firewall rules, and no configuration +// in this lab changes it. Aiming at lab-vps instead would ask two questions in +// one packet — "is there a route" and "did the host firewall allow it" — and +// then hand the answer to whichever one you already believed. +const labRelayAddr = "203.0.113.3" + +// pingGotThrough reads ping's own summary line rather than an exit code. The +// code says 1 for a fully lost run and for a name that would not resolve, and +// what matters here is a measurement, not an error. +func pingGotThrough(out string) bool { + m := lossRe.FindStringSubmatch(out) + if len(m) != 2 { + return false + } + loss, err := strconv.ParseFloat(m[1], 64) + return err == nil && loss < 100 +} + +// evilStranded is the sentence for "the attacker reached nothing", which is the +// answer a firewall's name used to be given for. Rung 1: the ladder's first +// question is whether anything on the path is alive, and on this path nothing +// is — so no higher rung was consulted and none may be reported. +// +// Three faults land here and they are three different problems, so they get +// three different sentences: the router was never created, the router is +// stopped, or the router is up and the wire still carried nothing. +func evilStranded(router, gw string, exists, running bool, raw string) Result { + res := Result{ + Rung: 1, From: "evil-box", Path: "public", Raw: raw, + Cmds: []string{"ping -c 1 -W 2 " + labRelayAddr + " # on evil-box, the lab's relay"}, + } + // The same finding as Why, in numbers, so a drawing never has to read the + // prose to find out that nothing was measured. + res.Evidence = map[string]int{"reached": 0} + + unscored := " Nothing is scored from a run like this, on purpose. A packet nobody " + + "carried draws the same silence a dropped one does — `nmap` prints `filtered` " + + "for both — so a defence credited here would be a block nobody measured." + + switch { + case !exists: + res.Rule = router + " is not in this stack" + res.Why = "evil-box is running, its default route still points at " + gw + ", and " + + "nothing is on the other end: " + router + " was never created. That is what " + + "starting the attacker on its own leaves behind — `--no-deps`. Bring the pair " + + "up together with `docker compose --profile attack up -d`, or `make attack`." + + unscored + case !running: + res.Rule = router + " is stopped" + res.Why = "The attacker's own router is down, so it reaches nothing at all — not " + + "lab-vps, not the coordination server, not the relay. Its route survived the " + + "outage; the gateway at " + gw + " did not. Start it with `docker start " + + router + "` and run this again." + unscored + res.Cmds = append([]string{"docker start " + router}, res.Cmds...) + default: + res.Rule = "no route out of evil-box's segment" + res.Why = router + " is running and the relay at " + labRelayAddr + " still " + + "answered nothing. Check evil-box's own link in the network panel: an " + + "interface that is down takes every route through it with it." + unscored + } + return res +} + // evilNotHere is the one sentence for "the attacker was never created". Two // different controls land on it — every attack button, and the machines switch // — and a reader who pressed the switch first should not get a rawer answer @@ -135,7 +243,7 @@ func evilNotHere() Result { // --------------------------------------------------------------------------- func (c *Controller) atkScanPublic(ctx context.Context) Result { - if r := c.needEvil(ctx); r != nil { + if r := c.needEvilOut(ctx); r != nil { return *r } target := c.lab.IPOn(ctx, "lab-vps", "wan") @@ -166,7 +274,7 @@ func (c *Controller) atkScanPublic(ctx context.Context) Result { } func (c *Controller) atkScanTailnet(ctx context.Context) Result { - if r := c.needEvil(ctx); r != nil { + if r := c.needEvilOut(ctx); r != nil { return *r } // The defence under test is the policy, so make sure the machine is on the @@ -220,6 +328,10 @@ func (c *Controller) atkScanTailnet(ctx context.Context) Result { // broken. So the lab sends the same marker twice — once in the clear, once // through the tunnel — and reports both counts. One appears, one does not. func (c *Controller) atkSniff(ctx context.Context) Result { + // needEvil rather than needEvilOut: nothing here leaves the segment + // evil-box shares with lab-ubuntu, so nat-evil carries none of it and a + // stopped router is not this attack's problem. What would be — a capture + // that saw nothing — this one already measures with its control marker. if r := c.needEvil(ctx); r != nil { return *r } @@ -298,6 +410,9 @@ func (c *Controller) atkSniff(ctx context.Context) Result { } func (c *Controller) atkReplay(ctx context.Context) Result { + // Same segment, same reason as atkSniff: the frames go back onto the wire + // evil-box is already attached to, and the count lab-vps's own interface + // moved by is the measurement. if r := c.needEvil(ctx); r != nil { return *r } @@ -348,7 +463,12 @@ func (c *Controller) atkReplay(ctx context.Context) Result { } func (c *Controller) atkStolenKey(ctx context.Context) Result { - if r := c.needEvil(ctx); r != nil { + // Both halves of this one cross the public segment — the join, and the + // ordinary-network probe that runs whether or not the key was accepted. A + // stranded attacker fails both and the second failure used to read as "the + // ordinary network gave its holder nothing either", which is the same + // unmeasured credit in the shape of a compliment. + if r := c.needEvilOut(ctx); r != nil { return *r } join := c.evilJoin(ctx, true) @@ -771,7 +891,7 @@ func shortKey(k string) string { } func (c *Controller) atkRogueExit(ctx context.Context) Result { - if r := c.needEvil(ctx); r != nil { + if r := c.needEvilOut(ctx); r != nil { return *r } if join := c.evilJoin(ctx, true); !join.OK { @@ -805,7 +925,7 @@ func (c *Controller) atkRogueExit(ctx context.Context) Result { } func (c *Controller) atkDockerBypass(ctx context.Context) Result { - if r := c.needEvil(ctx); r != nil { + if r := c.needEvilOut(ctx); r != nil { return *r } _ = c.Set(ctx, "vps.ufwDefaultDeny", true) @@ -848,9 +968,18 @@ func (c *Controller) atkLockOut(ctx context.Context) Result { res.Cmds = append(r1.Cmds, r2.Cmds...) overTailnet := c.Probe(ctx, "lab-ubuntu", "lab-vps", "22") - overPublic := c.Probe(ctx, "evil-box", "lab-vps", "22") - if !c.lab.Running(ctx, "evil-box") { - overPublic = Result{Rung: 1, Why: "no machine outside to try from — evil-box is not running"} + + // The outside half is a courtesy line rather than the verdict — the verdict + // is whether *you* are still in — but it is printed as a rung, and a rung + // printed from a machine that could not have reached anything is a reading + // of nothing. So establish the route first and print the reason instead. + overPublic := Result{Rung: 1, Why: "no machine outside to try from — evil-box is not running"} + if c.lab.Running(ctx, "evil-box") { + if r := c.needRouteOut(ctx); r != nil { + overPublic = *r + } else { + overPublic = c.Probe(ctx, "evil-box", "lab-vps", "22") + } } res.Raw = fmt.Sprintf("over the tailnet: rung %d — %s\nfrom outside: rung %d — %s", overTailnet.Rung, overTailnet.Why, overPublic.Rung, overPublic.Why) @@ -1053,7 +1182,11 @@ echo "--- mosh, last lines ---"; tr -d '\r' < /tmp/mosh.out 2>/dev/null | grep - // would be wrong, and this lab can show it is wrong: the firewall is still // there, still default-deny, still right, and was never in the path. func (c *Controller) atkTailcatTunnel(ctx context.Context) Result { - if r := c.needEvil(ctx); r != nil { + // The tunnel meets in the middle, at the relay on the public segment, so + // evil-box has to be able to get there. A tunnel that would not dial is + // already a rung-1 answer here; a tunnel that could not have dialled is + // the same answer earlier, with the reason attached. + if r := c.needEvilOut(ctx); r != nil { return *r } st := c.Snapshot().Tailcat diff --git a/lab/control/attacks_test.go b/lab/control/attacks_test.go index ae5fbaa..12a02a4 100644 --- a/lab/control/attacks_test.go +++ b/lab/control/attacks_test.go @@ -752,3 +752,117 @@ func TestEvilNotHereNamesTheCommand(t *testing.T) { } } } + +// ping's own summary, from evil-box with nat-evil down and then up. The exit +// code is not the measurement — busybox exits 1 for a lost packet and for an +// address it could not parse — so the summary line is, and it is the reason +// this parser exists rather than a `.Code != 0`. +const pingLost = `PING 203.0.113.3 (203.0.113.3): 56 data bytes + +--- 203.0.113.3 ping statistics --- +1 packets transmitted, 0 received, +1 errors, 100% packet loss +` + +const pingFine = `PING 203.0.113.3 (203.0.113.3): 56 data bytes +64 bytes from 203.0.113.3: seq=0 ttl=63 time=0.151 ms + +--- 203.0.113.3 ping statistics --- +1 packets transmitted, 1 received, 0% packet loss, round-trip min/avg/max = 0.151/0.151/0.151 ms +` + +func TestPingGotThrough(t *testing.T) { + for _, tc := range []struct { + name string + in string + want bool + }{ + {"a reply", pingFine, true}, + {"every packet lost", pingLost, false}, + // Partial loss is still a route. A lossy wire is a condition this lab + // sets on purpose, and refusing to measure on one would be refusing to + // measure most of chapter 03. + {"lossy but carrying", "5 packets transmitted, 2 received, 60% packet loss", true}, + {"no summary at all", "ping: bad address '203.0.113.3'", false}, + {"nothing", "", false}, + } { + t.Run(tc.name, func(t *testing.T) { + if got := pingGotThrough(tc.in); got != tc.want { + t.Fatalf("pingGotThrough() = %v, want %v", got, tc.want) + } + }) + } +} + +// The bug this guards is the one the lab is least able to notice about itself: +// a stopped nat-evil reading as a firewall that held. Every attack that crosses +// the public segment now refuses instead, and the refusal has three jobs — stop +// at rung 1, claim nothing, and name the router — so all three are pinned here. +func TestEvilStranded(t *testing.T) { + for _, tc := range []struct { + name string + exists, running bool + wantRule, wantInWhy string + }{ + {"the router was never created", false, false, + "nat-evil is not in this stack", "--no-deps"}, + {"the router is stopped", true, false, + "nat-evil is stopped", "docker start nat-evil"}, + // Running, and still nothing came back: the lab's own link switches can + // do that, and pointing at the router would be pointing at the wrong + // thing. + {"the router is up and the wire is not", true, true, + "no route out of evil-box's segment", "network panel"}, + } { + t.Run(tc.name, func(t *testing.T) { + r := evilStranded("nat-evil", "10.0.66.254", tc.exists, tc.running, pingLost) + + if r.OK || r.Danger { + t.Error("a run that reached nothing is neither a pass nor a breach") + } + if r.Rung != 1 { + t.Errorf("rung = %d, want 1 — nothing on this path was alive, so no higher "+ + "rung was consulted", r.Rung) + } + if r.Rule != tc.wantRule { + t.Errorf("rule = %q, want %q", r.Rule, tc.wantRule) + } + if !strings.Contains(r.Why, tc.wantInWhy) { + t.Errorf("the message does not name %q:\n%s", tc.wantInWhy, r.Why) + } + // The whole point: no defence is credited by name, and the numbers + // say nothing was reached rather than leaving a drawing to guess. + if strings.Contains(r.Rule, "ufw") || strings.Contains(r.Why, "firewall held") { + t.Errorf("a defence took credit for a packet nobody carried: %q / %s", r.Rule, r.Why) + } + if r.Evidence["reached"] != 0 { + t.Errorf("evidence claims something was reached: %v", r.Evidence) + } + if !strings.Contains(r.Why, "nobody measured") { + t.Errorf("the message does not say why nothing is scored:\n%s", r.Why) + } + if r.Raw == "" { + t.Error("the refusal carries no measurement to show for itself") + } + }) + } +} + +// nat-evil is the only one of the three routers a reader is ever told to stop, +// but all three strand the machine behind them the same way, and the state has +// to be able to say so. lab-vps is the machine with no router at all: a field +// that went true for it would be a lie in the other direction. +func TestEveryMachineBehindNATHasItsRouterNamed(t *testing.T) { + for _, m := range Catalog { + if m.ID == "lab-vps" { + if m.Router != "" || m.Gateway != "" { + t.Errorf("lab-vps has an address on the public segment itself and needs no "+ + "router, but the catalog gives it %q / %q", m.Router, m.Gateway) + } + continue + } + if m.Router == "" || m.Gateway == "" { + t.Errorf("%s is behind a NAT with no router or gateway named: %q / %q", + m.ID, m.Router, m.Gateway) + } + } +} diff --git a/lab/control/audit.go b/lab/control/audit.go index af71a86..15ca619 100644 --- a/lab/control/audit.go +++ b/lab/control/audit.go @@ -174,18 +174,20 @@ func (c *Controller) Audit(ctx context.Context) AuditReport { c.loadDesired() rep := AuditReport{} - // Checks 3 to 7 need somewhere hostile to fire from. Without it the score - // is not comparable to anything, so say so rather than scoring five checks - // out of thin air. - if !c.lab.Running(ctx, "evil-box") { - if r := c.needEvil(ctx); r != nil { - rep.Tone = "warn" - rep.Verdict = "Five of the eleven checks fire an attacker at this stack, and there " + - "is no attacker in it. Start one with `make attack` (or " + - "`docker compose --profile attack up -d`) and run the audit again — a " + - "score with five holes in it cannot be compared with the sandbox's." - return rep - } + // Checks 3 to 7 need somewhere hostile to fire from, and need it to be able + // to reach this stack. Missing either one, the score is not comparable to + // anything, so say so rather than scoring five checks out of thin air. + // + // The second half is the worse of the two and was the one nothing checked. + // An attacker with no route out fails all five, five failures to reach are + // five passes, and the report would have said so in the same words it uses + // for a stack that actually held. + if r := c.needEvilOut(ctx); r != nil { + rep.Tone = "warn" + rep.Verdict = "Five of the eleven checks fire an attacker at this stack, and this " + + "run could not. " + r.Why + "\n\nA score with five holes in it cannot be " + + "compared with the sandbox's, so it is withheld rather than filled in." + return rep } // Remember what the attacker was doing, and put it back afterwards. diff --git a/lab/control/main.go b/lab/control/main.go index 9915b21..d93fc39 100644 --- a/lab/control/main.go +++ b/lab/control/main.go @@ -500,7 +500,11 @@ func routes(c *Controller) http.Handler { if err != nil { log.Fatalf("embedded UI: %v", err) } - mux.Handle("/", http.FileServer(http.FS(ui))) + assets, err := staticUI(ui) + if err != nil { + log.Fatalf("embedded UI: %v", err) + } + mux.Handle("/", assets) writeJSON := func(w http.ResponseWriter, v any) { w.Header().Set("Content-Type", "application/json") diff --git a/lab/control/state.go b/lab/control/state.go index 833f8a1..2db0d01 100644 --- a/lab/control/state.go +++ b/lab/control/state.go @@ -33,6 +33,16 @@ type Machine struct { NAT string `json:"nat"` // none | easy | hard Hostile bool `json:"hostile,omitempty"` + // Router is the container that carries this machine's packets off its own + // segment, and Gateway is the address it answers on. lab-vps has neither: + // it is the one machine with an address on the public segment itself. + // + // Worth naming rather than leaving implicit, because a stopped router does + // not stop the machine behind it — it strands it, which for a long time + // this lab had no way to say and scored straight through. + Router string `json:"router,omitempty"` + Gateway string `json:"gateway,omitempty"` + // Which compose network carries this machine's own segment, and which // carries the one standing in for the public internet. LANNet string `json:"-"` @@ -43,11 +53,14 @@ var Catalog = []Machine{ {ID: "lab-vps", Label: "lab-vps", Role: "the public box", Chapter: "08", Tag: "tag:server", NAT: "none", WANNet: "wan", LANNet: "vpslan"}, {ID: "lab-ubuntu", Label: "lab-ubuntu", Role: "the laptop", Chapter: "09", - Tag: "tag:laptop", NAT: "easy", LANNet: "lan-ubuntu"}, + Tag: "tag:laptop", NAT: "easy", LANNet: "lan-ubuntu", + Router: "nat-ubuntu", Gateway: "10.0.13.254"}, {ID: "lab-roam", Label: "lab-roam", Role: "the phone's stand-in", Chapter: "03", - Tag: "tag:roam", NAT: "hard", LANNet: "lan-roam"}, + Tag: "tag:roam", NAT: "hard", LANNet: "lan-roam", + Router: "nat-roam", Gateway: "10.0.27.254"}, {ID: "evil-box", Label: "evil-box", Role: "a hostile machine", Chapter: "10", - Tag: "tag:untrusted", NAT: "easy", LANNet: "lan-evil", Hostile: true}, + Tag: "tag:untrusted", NAT: "easy", LANNet: "lan-evil", Hostile: true, + Router: "nat-evil", Gateway: "10.0.66.254"}, } func machineByID(id string) (Machine, bool) { @@ -89,9 +102,16 @@ type LinkState struct { } type MachineState struct { - Online bool `json:"online"` - OnTailnet bool `json:"onTailnet"` - Signed bool `json:"signed"` + Online bool `json:"online"` + OnTailnet bool `json:"onTailnet"` + Signed bool `json:"signed"` + // RouterDown is the NAT box in front of this machine not running, while the + // machine itself is. It is named after what was read from Docker rather + // than after the conclusion — "stranded" would overstate it for evil-box, + // which keeps a second interface on lab-ubuntu's segment — but it is the + // difference between a machine that is online and a machine that can + // reach anything, and this state used to report only the first. + RouterDown bool `json:"routerDown,omitempty"` KeyExpired bool `json:"keyExpired"` TSAddr string `json:"tsAddr,omitempty"` LANAddr string `json:"lanAddr,omitempty"` @@ -367,6 +387,11 @@ func (c *Controller) Observe(ctx context.Context) { ms := &MachineState{Online: running} if running { + // Asked about every machine, not only the attacker. Each of the + // three behind a NAT can be left stranded the same way, and a + // machine whose router is down looks identical to one that is + // merely idle unless the page is told to look. + ms.RouterDown = m.Router != "" && !c.lab.Running(ctx, m.Router) ms.LANAddr = c.lab.IPOn(ctx, m.ID, m.LANNet) if m.WANNet != "" { ms.WANAddr = c.lab.IPOn(ctx, m.ID, m.WANNet) diff --git a/lab/control/ui/app.js b/lab/control/ui/app.js index e3099d8..2afd1f5 100644 --- a/lab/control/ui/app.js +++ b/lab/control/ui/app.js @@ -503,7 +503,10 @@ var bits = meta.catalog.map(function (m) { var ms = state.machines[m.id] || {}; if (!ms.online) return m.label + " off"; + /* A machine whose router is stopped is online and reaches nothing, and + this line used to read identically to one that was simply idle. */ return m.label + " " + (ms.tsAddr || ms.wanAddr || ms.lanAddr || "?") + + (ms.routerDown ? " (" + m.router + " down)" : "") + (ms.pathTo ? " (" + ms.pathTo + ")" : ""); }); $("#status-out").setAttribute("data-machines", bits.join(" · ")); @@ -575,7 +578,13 @@ g.classList.toggle("is-hidden", gone); } var nat = el("nat-" + m.id); - if (nat) nat.classList.toggle("is-hidden", gone); + if (nat) { + nat.classList.toggle("is-hidden", gone); + /* Dimmed for the same reason a stopped machine is: the box is there and + nothing is going through it. Everything behind it is stranded, and a + NAT drawn at full strength says the opposite. */ + nat.classList.toggle("is-off", !!ms.routerDown); + } /* The address a machine answers on is not a fact about the machine, it is a fact about the path you are taking to it. Show the tailnet address @@ -592,6 +601,9 @@ if (cond) { var bits = []; if (!link.up) bits.push("eth0 DOWN"); + /* The condition the state had no word for: the machine is up, its + router is not, and it reaches nothing off its own segment. */ + if (ms.routerDown) bits.push(m.router + " DOWN"); if (link.loss) bits.push(link.loss + "% loss"); if (link.delay) bits.push(link.delay + "ms"); if (link.udpBlocked) bits.push("udp dropped"); diff --git a/lab/control/ui/hud/reading.js b/lab/control/ui/hud/reading.js index 66f765e..a1853ce 100644 --- a/lab/control/ui/hud/reading.js +++ b/lab/control/ui/hud/reading.js @@ -53,6 +53,15 @@ export const ladderMark = (res) => ((res && res.danger) ? "breached" : "stopped" eleven at night. */ export const neverRan = (res) => !!res && !res.ok && !res.danger && rung(res) <= 1; +/* Why a run never ran, in the refusal's own words rather than a guess. + Three faults land on neverRan now — the attacker was never created, the + router in front of it is stopped, or it is up and the wire carried nothing + — and a head that names one of them by hand names the wrong one two times + in three. Every refusal carries the rule it refused with; read that, and + keep the old question as the fallback for a Result that carries none. */ +export const whyNothingRan = (res) => + (res && res.rule) || "is evil-box running? `make attack`"; + /* The three outcomes an attack can have, phrased so the head always names something rather than reporting a boolean. */ export function head(res, defence, through, inconclusive) { @@ -201,7 +210,7 @@ export function tailcatReading(res) { (ev(res, "ordinaryOK") > 0 ? " · in" : " · stopped") ], head: - neverRan(res) ? "INCONCLUSIVE · no tunnel ran — is evil-box running? `make attack`" : + neverRan(res) ? "INCONCLUSIVE · no tunnel ran — " + whyNothingRan(res) : refused ? "HELD · --allow pinned the server to one client key" : shell ? "THROUGH · a shell, and nothing you configured was asked" : "HELD · an SSH key, at rung 5, with rungs 3 and 4 never asked" diff --git a/lab/control/ui/hud/setpieces.js b/lab/control/ui/hud/setpieces.js index 0211bfb..93b104c 100644 --- a/lab/control/ui/hud/setpieces.js +++ b/lab/control/ui/hud/setpieces.js @@ -54,8 +54,8 @@ ============================================================ */ import { GEOM } from "./scene.js"; -import { ev, tone, ladderMark, neverRan, head, scanReading, sniffReading, - tailcatReading } from "./reading.js"; +import { ev, tone, ladderMark, neverRan, head, whyNothingRan, scanReading, + sniffReading, tailcatReading } from "./reading.js"; const { POS, GX, ANY, Y_PUB, Y_MACH, Y_NET, Y_DERP, TN_Y } = GEOM; @@ -201,7 +201,7 @@ function scanPublic(board, res) { head: head(res, "ufw default incoming policy: deny", "the host firewall let them through", - "nothing was scanned — is evil-box running?"), + "nothing was scanned — " + whyNothingRan(res)), nums: scan.nums, chip: "public segment · in the clear", chipKind: "pub" }); @@ -826,7 +826,7 @@ function dockerBypass(board, res) { held(board, res, { head: head(res, "nothing was listening on :8080, so there was nothing to reach", "DOCKER-USER accepted it before ufw's chain ever ran", - "the trap was never set up — is evil-box running? `make attack`"), + "the trap was never set up — " + whyNothingRan(res)), nums: ["port 8080", through ? "answered" : "no answer"], chip: "public segment · in the clear", chipKind: "pub" }); diff --git a/lab/control/ui/index.html b/lab/control/ui/index.html index bdcf682..93b84a4 100644 --- a/lab/control/ui/index.html +++ b/lab/control/ui/index.html @@ -135,15 +135,21 @@ rung and a round-trip time and grows into the middle of the row. --> THE PUBLIC SEGMENT — anyone on it, to lab-vps - - - NAT + + + + NAT + - - NAT + + + NAT +