From a223d725e1ddfd711f3c10276c939fcc2b3dddb9 Mon Sep 17 00:00:00 2001 From: madeofpendletonwool Date: Wed, 14 Oct 2026 16:30:54 -0500 Subject: [PATCH] feat: box watchdogs and self-healing --- README.md | 55 ++++ boxd/cmd/boxd/main.go | 3 +- boxd/internal/daemon/bridge.go | 24 +- boxd/internal/daemon/daemon.go | 89 +++---- boxd/internal/daemon/screens.go | 15 +- boxd/internal/daemon/setup.go | 4 +- boxd/internal/daemon/thermals.go | 133 ++++++++++ boxd/internal/daemon/thermals_test.go | 145 +++++++++++ boxd/internal/daemon/watchdog.go | 234 ++++++++++++++++++ boxd/internal/daemon/watchdog_test.go | 220 ++++++++++++++++ docs/adr/0008-watchdogs-and-self-healing.md | 108 ++++++++ docs/testing.md | 84 +++++++ scripts/smoke-test.sh | 41 ++- .../01-kiosk/files/syncphony-kiosk.service | 7 +- stage-syncphony/07-watchdog/00-packages | 6 + stage-syncphony/07-watchdog/00-run.sh | 25 ++ .../files/50-syncphony-watchdog.conf | 8 + 17 files changed, 1120 insertions(+), 81 deletions(-) create mode 100644 boxd/internal/daemon/thermals.go create mode 100644 boxd/internal/daemon/thermals_test.go create mode 100644 boxd/internal/daemon/watchdog.go create mode 100644 boxd/internal/daemon/watchdog_test.go create mode 100644 docs/adr/0008-watchdogs-and-self-healing.md create mode 100644 stage-syncphony/07-watchdog/00-packages create mode 100644 stage-syncphony/07-watchdog/00-run.sh create mode 100644 stage-syncphony/07-watchdog/files/50-syncphony-watchdog.conf diff --git a/README.md b/README.md index 1a9a6ae..bd96c17 100644 --- a/README.md +++ b/README.md @@ -13,6 +13,10 @@ The box is **client only**: it runs no Syncphony server. Point it at yours. - Raspberry Pi 4 or Pi 5, 2 GB RAM or more - 16 GB+ SD card - arm64 only. Pi 3, Pi Zero 2 W and armhf are not supported. +- A case with a heatsink or a fan. A long party with the visualizer on + warms a Pi in a closed case past its soft temperature limit, and the + firmware silently halves the CPU first (the box logs temperature and + throttling — see [Self-healing](#self-healing-watchdogs) below). ## Building @@ -164,6 +168,42 @@ sudo syncphony-box chroot # shell on the real root, read-write Reboot afterwards. Changes to the pairing, config and anything else on `/data` need no chroot — that partition is writable at runtime. +## Self-healing: watchdogs + +The box notices when it's stuck and fixes itself — nobody walks over to +it. Three watchdogs, each owning one failure class +([ADR 0008](docs/adr/0008-watchdogs-and-self-healing.md)): + +- **Hardware watchdog (kernel/systemd hangs).** systemd arms the Pi's + hardware watchdog at 15 s. A kernel or systemd hang stops the petting + and the hardware hard-resets the box; with the read-only root it comes + back clean, paired, playing. +- **Kiosk watchdog in boxd (a wedged page).** The `/tv` page heartbeats + to boxd every 30 s. If the heartbeat stops for 90 s while the server + still answers `/healthz`, boxd restarts the kiosk — Chromium comes + back and the speaker resumes by itself. If that happens 3 times in 15 + minutes, the box reboots instead. When the server is *unreachable*, + the watchdog stays quiet: that's the offline screen's job, not a + stuck page. +- **Time.** `systemd-timesyncd` runs and the kiosk waits for + `time-sync.target`: the server's playback clock sync trusts the + browser's clock, and a Pi has no RTC battery by default. + +Two hygiene loops ride along: + +- **Thermals.** boxd logs `vcgencmd measure_temp` and the throttling + flags to the journal every minute (`journalctl -u boxd | grep + thermals`) and reports them in `/v1/info` for the room's Screens list + (stage 4). A persistent `throttled=0x50000`-style value means the + power supply is weak; a hot `temp=` means the case needs airflow. +- **Nightly reload.** Once a day around 05:00 local, while the page is + up and nothing is playing, boxd restarts the kiosk so a long-lived + Chromium gives its memory back. A party that runs through the night + is never interrupted. + +The freeze and kernel-hang procedures that verify all this are in +[docs/testing.md](docs/testing.md). + ## Audio The box is the room's speaker: Chromium plays everything through the @@ -273,6 +313,21 @@ has its own page: [docs/testing.md](docs/testing.md). - Pull the server's network cable: the offline screen appears within a couple of minutes (or at the next kiosk start) with the box's IP and a countdown; plugging it back returns to the room with no touch. +- Freeze Chromium's renderer (`pkill -STOP -f 'type=renderer'`): the + room comes back by itself within about 2 minutes (the boxd kiosk + watchdog), and music resumes. +- Kernel-hang reboot (destructive; see docs/testing.md): + `echo c > /proc/sysrq-trigger` — the hardware watchdog resets the box + and it comes back paired and playing. +- `journalctl -u boxd | grep thermals` shows a temperature line every + minute; `curl 127.0.0.1:8099/v1/info` includes `temperatureC`. Check + on a Pi 4 and a Pi 5. +- `timedatectl` shows the clock synchronized and + `systemctl status systemd-timesyncd` is active, on both a Pi 4 and a + Pi 5 (which has an RTC header but no battery by default). +- A 12-hour playback soak with the visualizer on, on a 2 GB Pi 4 (the + procedure is in docs/testing.md): memory plateaus, and the 05:00 + nightly reload happens only while idle. - An hour of real playback as the room's speaker — Navidrome (MP3 and FLAC) and Spotify (Ogg): no gaps between songs, no drift, no clipped first notes. diff --git a/boxd/cmd/boxd/main.go b/boxd/cmd/boxd/main.go index 976426a..61acbde 100644 --- a/boxd/cmd/boxd/main.go +++ b/boxd/cmd/boxd/main.go @@ -1,7 +1,8 @@ // boxd is the Syncphony box's helper daemon (MAD-803, ADR 0005 in the box // repo; the browser-facing bridge contract is Syncphony ADR 0016). It owns // everything on the box that isn't Chromium: the syncphony.txt config, the -// box bridge on loopback, and the local setup and offline screens. +// box bridge on loopback, the local setup and offline screens, and the +// kiosk watchdog and thermals (MAD-806, ADR 0008). package main import ( diff --git a/boxd/internal/daemon/bridge.go b/boxd/internal/daemon/bridge.go index 890cc5c..68f8178 100644 --- a/boxd/internal/daemon/bridge.go +++ b/boxd/internal/daemon/bridge.go @@ -109,12 +109,26 @@ func (d *Daemon) guard(next http.HandlerFunc) http.HandlerFunc { } func (d *Daemon) handleInfo(w http.ResponseWriter, r *http.Request) { - w.Header().Set("Content-Type", "application/json") - json.NewEncoder(w).Encode(map[string]any{ + d.mu.Lock() + th := d.therm + d.mu.Unlock() + // The ADR 0016 core; the thermal fields (MAD-806) ride along once a + // sample exists — stage 4's Screens list reads them ("52 °C", + // "running hot / throttled"), and everyone else ignores them. + info := map[string]any{ "version": d.version, "name": d.displayName(), "capabilities": []string{"reboot", "reload"}, // cec arrives in stage 3 - }) + } + if th.valid() { + // No sample (no vcgencmd here) means no fields: absent reads + // better than a misleading 0 °C. + for k, v := range th.infoFields() { + info[k] = v + } + } + w.Header().Set("Content-Type", "application/json") + json.NewEncoder(w).Encode(info) } var eventTypes = map[string]bool{ @@ -181,8 +195,6 @@ func (d *Daemon) handleReload(w http.ResponseWriter, r *http.Request) { w.Write([]byte(`{"ok":true}` + "\n")) go func() { time.Sleep(250 * time.Millisecond) - if err := d.run("systemctl", "restart", "syncphony-kiosk.service"); err != nil { - d.log.Printf("boxd: reload failed: %v", err) - } + d.restartKiosk("reload (remote care)", false) }() } diff --git a/boxd/internal/daemon/daemon.go b/boxd/internal/daemon/daemon.go index ee7616d..7252332 100644 --- a/boxd/internal/daemon/daemon.go +++ b/boxd/internal/daemon/daemon.go @@ -1,6 +1,8 @@ // Package daemon is boxd's running half: the box bridge on loopback -// (ADR 0016) and the local screens the kiosk shows when there's no server -// to display — setup with a QR code, and the offline "can't reach" screen. +// (ADR 0016), the local screens the kiosk shows when there's no server +// to display — setup with a QR code, and the offline "can't reach" +// screen — and the watchdogs and thermals that make the box heal itself +// (MAD-806, ADR 0008). package daemon import ( @@ -24,16 +26,6 @@ const ( setupWindow = 10 * time.Minute // after SIGUSR1 (stage 3 wires the OK long-press) healthzCache = 2 * time.Second - // The offline screen takes over when the server has been unreachable - // for this long while the page is alive (heartbeats fresh): the kiosk is - // restarted through the local start page, which shows the offline - // screen and returns to /tv on its own once /healthz answers. After the - // restart there are no heartbeats (the page is local), so this cannot - // loop; the cooldown is a second guard. - offlineGrace = 2 * time.Minute - heartbeatFresh = 90 * time.Second - kioskRestartGuard = 10 * time.Minute - // Pairing durability (MAD-804, ADR 0006): Chromium writes its cookie // store and localStorage lazily, so the writes that make this box stay // paired sit in memory and the page cache for a while. After the @@ -88,25 +80,35 @@ type Daemon struct { // a func field so tests can count calls instead of syncing. syncDisk func() error - setupUntil time.Time // LAN setup window open until then (zero = closed) - lastBeat time.Time // last /v1/heartbeat from the page - lastEvent string - lastEventAt time.Time - lastOK bool // last /healthz answer - lastAt time.Time // when it was checked - offlineSince time.Time // server unreachable continuously since - lastRestart time.Time // last kiosk restart triggered by the offline watch - lastFlush time.Time // last pairing-durability sync burst + // vcgencmd samples the hardware thermals; a func field so tests can + // feed readings without a Pi. + vcgencmd func(args ...string) (string, error) + + startedAt time.Time // boxd start; watchdog graces count from here + setupUntil time.Time // LAN setup window open until then (zero = closed) + lastBeat time.Time // last /v1/heartbeat from the page + lastEvent string // last /v1/events type (the nightly reload waits out "playing") + lastEventAt time.Time // when it arrived + lastOK bool // last /healthz answer + lastAt time.Time // when it was checked + offlineSince time.Time // server unreachable continuously since + lastRestart time.Time // last kiosk restart boxd performed (settle window anchor) + restarts []time.Time // watchdog-caused kiosk restarts inside watchdogWindow + lastNightlyDay string // local day the nightly reload last ran ("2006-01-02") + therm thermals // last thermal sample (zero = none yet) + lastFlush time.Time // last pairing-durability sync burst } // New builds a daemon around the current syncphony.txt (the SB_* env hooks // from the config package apply). func New(version string, log *log.Logger, run config.RunCommand) *Daemon { d := &Daemon{ - version: version, - log: log, - run: run, - paths: config.PathsFromEnv(), + version: version, + log: log, + run: run, + paths: config.PathsFromEnv(), + startedAt: time.Now(), + vcgencmd: realVcgencmd, syncDisk: func() error { syscall.Sync() return nil @@ -196,6 +198,7 @@ func (d *Daemon) Run() error { d.log.Printf("boxd: %s bridge on http://%s (server: %s)", d.version, bridgeAddr, d.serverOrNone()) go d.watch() + go d.sampleThermalsLoop() sig := make(chan os.Signal, 4) signal.Notify(sig, syscall.SIGTERM, syscall.SIGINT, syscall.SIGUSR1, syscall.SIGUSR2) @@ -313,42 +316,6 @@ func (d *Daemon) flushDisk(event string) { }() } -// watch is the background loop: the offline screen trigger. While the page -// is alive (fresh heartbeats) but the server stays unreachable past the -// grace period, restart the kiosk so it re-enters through the local start -// page — which shows the offline screen and returns on its own. -func (d *Daemon) watch() { - tick := time.NewTicker(5 * time.Second) - defer tick.Stop() - for range tick.C { - if !d.configured() { - continue - } - ok := d.serverReachable() - d.mu.Lock() - now := time.Now() - if ok { - d.offlineSince = time.Time{} - } else if d.offlineSince.IsZero() { - d.offlineSince = now - } - restart := !d.offlineSince.IsZero() && - now.Sub(d.offlineSince) >= offlineGrace && - !d.lastBeat.IsZero() && now.Sub(d.lastBeat) < heartbeatFresh && - (d.lastRestart.IsZero() || now.Sub(d.lastRestart) > kioskRestartGuard) - if restart { - d.lastRestart = now - } - d.mu.Unlock() - if restart { - d.log.Printf("boxd: server unreachable for %s with a live page; restarting the kiosk onto the offline screen", offlineGrace) - if err := d.run("systemctl", "restart", "syncphony-kiosk.service"); err != nil { - d.log.Printf("boxd: kiosk restart failed: %v", err) - } - } - } -} - // storageHealthy reports whether /data — the only partition that survives // the overlay (MAD-805, ADR 0007) — is mounted. Without it the box boots // anyway, but the pairing, the applied config and the logs are gone (they diff --git a/boxd/internal/daemon/screens.go b/boxd/internal/daemon/screens.go index 0f4dcaa..3e95d5e 100644 --- a/boxd/internal/daemon/screens.go +++ b/boxd/internal/daemon/screens.go @@ -12,8 +12,10 @@ import ( // handleStatus reports everything the local screens (and tests) need. func (d *Daemon) handleStatus(w http.ResponseWriter, r *http.Request) { server := d.serverURL() - w.Header().Set("Content-Type", "application/json") - json.NewEncoder(w).Encode(map[string]any{ + d.mu.Lock() + th := d.therm + d.mu.Unlock() + status := map[string]any{ "configured": server != "", "server": server, "reachable": d.serverReachable(), @@ -24,7 +26,14 @@ func (d *Daemon) handleStatus(w http.ResponseWriter, r *http.Request) { "ips": boxIPs(), "network": networkState(), "storage": storageHealthy(), - }) + } + if th.valid() { + for k, v := range th.infoFields() { + status[k] = v + } + } + w.Header().Set("Content-Type", "application/json") + json.NewEncoder(w).Encode(status) } func hostname() string { diff --git a/boxd/internal/daemon/setup.go b/boxd/internal/daemon/setup.go index 5dd59f2..4d54fc9 100644 --- a/boxd/internal/daemon/setup.go +++ b/boxd/internal/daemon/setup.go @@ -94,9 +94,7 @@ func (d *Daemon) handleSetupSave(w http.ResponseWriter, r *http.Request) { d.run("systemctl", "try-restart", "avahi-daemon.service") } d.closeSetupWindow() - if err := d.run("systemctl", "restart", "syncphony-kiosk.service"); err != nil { - d.log.Printf("boxd: setup: kiosk restart failed: %v", err) - } + d.restartKiosk("setup saved; entering the server through the start page", false) writePage(w, savedPageHTML(serverURL)) } diff --git a/boxd/internal/daemon/thermals.go b/boxd/internal/daemon/thermals.go new file mode 100644 index 0000000..fd1461e --- /dev/null +++ b/boxd/internal/daemon/thermals.go @@ -0,0 +1,133 @@ +package daemon + +import ( + "context" + "errors" + "math" + "os/exec" + "strconv" + "strings" + "time" +) + +// Thermals (MAD-806): the SoC temperature and the firmware's throttling +// flags, sampled once a minute into the journal (one line per sample, so +// `journalctl -u boxd` doubles as a temperature log) and reported through +// /v1/info for the room's Screens list (stage 4). vcgencmd is the +// Raspberry Pi firmware tool (libraspberrypi-bin); where it is missing — +// a dev machine, a test — thermals simply go unreported and everything +// downstream tolerates their absence. +const thermalsEvery = time.Minute + +// thermals is one sample. The flags are vcgencmd get_throttled's bitmap: +// four conditions, current in bits 0-3 and the same four again as sticky +// "has occurred" flags in bits 16-19 (never cleared until reboot — the +// classic value a Pi on a weak power supply reports is 0x50000): +// +// bit 0 under-voltage now bit 16 under-voltage occurred +// bit 1 frequency capped now bit 17 frequency capped occurred +// bit 2 throttled now bit 18 throttling occurred +// bit 3 soft temperature limit bit 19 soft temp limit occurred +type thermals struct { + at time.Time + tempC float64 + rawFlags uint64 +} + +func (t thermals) valid() bool { return !t.at.IsZero() } + +// limitedNow: the firmware is holding the SoC back right now — throttled, +// frequency-capped or at the soft temperature limit. +func (t thermals) limitedNow() bool { + return t.rawFlags&(1<<1|1<<2|1<<3) != 0 +} + +func (t thermals) underVoltage() bool { return t.rawFlags&(1<<0) != 0 } +func (t thermals) throttleOccurred() bool { return t.rawFlags&(0xF<<16) != 0 } + +// infoFields is what /v1/info (and /status) add once a sample exists. +func (t thermals) infoFields() map[string]any { + return map[string]any{ + "temperatureC": math.Round(t.tempC*10) / 10, + "throttled": t.limitedNow(), + "underVoltage": t.underVoltage(), + "throttleOccurred": t.throttleOccurred(), + } +} + +// realVcgencmd runs the firmware tool (on the box; the daemon field is +// stubbed in tests). Bounded, so a wedged firmware call can't stall the +// sampling loop forever. +func realVcgencmd(args ...string) (string, error) { + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + out, err := exec.CommandContext(ctx, "vcgencmd", args...).Output() + return string(out), err +} + +// sampleThermalsLoop samples immediately and then once a minute. It +// returns for good where vcgencmd doesn't exist (not a Pi): thermals are +// optional and absent-by-default everywhere they're reported. +func (d *Daemon) sampleThermalsLoop() { + if !d.sampleThermals() { + return + } + tick := time.NewTicker(thermalsEvery) + defer tick.Stop() + for range tick.C { + d.sampleThermals() + } +} + +// sampleThermals takes one sample, stores it and logs it. It reports +// false only for "there is no vcgencmd here" (stop sampling); a +// transient failure keeps the loop alive and the journal informed. +func (d *Daemon) sampleThermals() bool { + tempOut, err := d.vcgencmd("measure_temp") + if err != nil { + if errors.Is(err, exec.ErrNotFound) { + d.log.Printf("boxd: thermals: no vcgencmd; temperature and throttling go unreported") + return false + } + d.log.Printf("boxd: thermals: %v", err) + return true + } + temp, tempOK := parseTempC(tempOut) + flags := uint64(0) + if out, err := d.vcgencmd("get_throttled"); err == nil { + flags = parseThrottleFlags(out) + } + now := time.Now() + d.mu.Lock() + d.therm = thermals{at: now, tempC: temp, rawFlags: flags} + d.mu.Unlock() + if tempOK { + d.log.Printf("boxd: thermals temp=%.1f'C throttled=0x%x", temp, flags) + } else { + d.log.Printf("boxd: thermals temp=unknown throttled=0x%x", flags) + } + return true +} + +// parseTempC reads vcgencmd measure_temp's "temp=52.3'C" into a float. +func parseTempC(out string) (float64, bool) { + s := strings.TrimSpace(out) + s = strings.TrimPrefix(s, "temp=") + s = strings.TrimSuffix(s, "'C") + f, err := strconv.ParseFloat(s, 64) + if err != nil { + return 0, false + } + return f, true +} + +// parseThrottleFlags reads vcgencmd get_throttled's "throttled=0x50001" +// into the bitmap; anything unparsable is no flags. +func parseThrottleFlags(out string) uint64 { + s := strings.TrimSpace(strings.TrimPrefix(strings.TrimSpace(out), "throttled=")) + f, err := strconv.ParseUint(s, 0, 64) + if err != nil { + return 0 + } + return f +} diff --git a/boxd/internal/daemon/thermals_test.go b/boxd/internal/daemon/thermals_test.go new file mode 100644 index 0000000..0c51942 --- /dev/null +++ b/boxd/internal/daemon/thermals_test.go @@ -0,0 +1,145 @@ +package daemon + +import ( + "encoding/json" + "errors" + "net/http" + "os/exec" + "testing" + "time" +) + +func TestParseTempC(t *testing.T) { + for _, tc := range []struct { + out string + want float64 + ok bool + }{ + {"temp=52.3'C\n", 52.3, true}, + {"temp=61.9'C", 61.9, true}, + {"temp=0.0'C", 0, true}, + {"garbage", 0, false}, + {"", 0, false}, + } { + got, ok := parseTempC(tc.out) + if ok != tc.ok || (ok && got != tc.want) { + t.Errorf("parseTempC(%q) = %v,%v want %v,%v", tc.out, got, ok, tc.want, tc.ok) + } + } +} + +func TestParseThrottleFlags(t *testing.T) { + for _, tc := range []struct { + out string + want uint64 + }{ + {"throttled=0x0\n", 0}, + {"throttled=0x50001", 0x50001}, + {"throttled=0x4", 0x4}, + {"throttled=0", 0}, + {"garbage", 0}, + } { + if got := parseThrottleFlags(tc.out); got != tc.want { + t.Errorf("parseThrottleFlags(%q) = %#x want %#x", tc.out, got, tc.want) + } + } +} + +func TestThermalsDecodesFlags(t *testing.T) { + // 0x50001: under-voltage now (bit 0), under-voltage occurred (bit 16), + // throttling occurred (bit 18) — nothing limiting *right now*. + th := thermals{at: time.Now(), rawFlags: 0x50001} + if !th.underVoltage() || !th.throttleOccurred() || th.limitedNow() { + t.Fatalf("0x50001 decoded wrong: %+v", th.infoFields()) + } + // 0x4: throttled right now. + th.rawFlags = 0x4 + if !th.limitedNow() || th.underVoltage() || th.throttleOccurred() { + t.Fatalf("0x4 decoded wrong: %+v", th.infoFields()) + } + // 0x10000: under-voltage has happened, is not happening. + th.rawFlags = 0x10000 + if th.limitedNow() || th.underVoltage() || !th.throttleOccurred() { + t.Fatalf("0x10000 decoded wrong: %+v", th.infoFields()) + } +} + +// stubVcgencmd answers like the firmware tool; the daemon field swap is +// what tests use instead of a Pi. +func stubVcgencmd(temp string, flags string) func(args ...string) (string, error) { + return func(args ...string) (string, error) { + if len(args) == 0 { + return "", errors.New("no args") + } + switch args[0] { + case "measure_temp": + return temp, nil + case "get_throttled": + return flags, nil + } + return "", errors.New("unknown: " + args[0]) + } +} + +func TestSampleThermalsStoresAndReports(t *testing.T) { + d, _ := newTestDaemon(t, "server_url=https://s.example\n") + d.vcgencmd = stubVcgencmd("temp=48.5'C\n", "throttled=0x0\n") + if !d.sampleThermals() { + t.Fatal("sampling reported vcgencmd missing") + } + + for _, target := range []string{"/v1/info", "/status"} { + rec := get(t, d.bridgeMux(), target) + if rec.Code != http.StatusOK { + t.Fatalf("%s: status %d", target, rec.Code) + } + var m map[string]any + json.Unmarshal(rec.Body.Bytes(), &m) + if m["temperatureC"] != 48.5 { + t.Fatalf("%s: temperatureC = %v", target, m["temperatureC"]) + } + if m["throttled"] != false { + t.Fatalf("%s: throttled = %v", target, m["throttled"]) + } + } + + // A throttled sample shows up as such. + d.vcgencmd = stubVcgencmd("temp=85.0'C\n", "throttled=0x40004\n") + d.sampleThermals() + rec := get(t, d.bridgeMux(), "/v1/info") + var m map[string]any + json.Unmarshal(rec.Body.Bytes(), &m) + if m["throttled"] != true || m["underVoltage"] != false { + t.Fatalf("throttled sample: %v", m) + } +} + +func TestThermalsAbsentWithoutVcgencmd(t *testing.T) { + // Fresh daemon, no sample yet: /v1/info keeps its ADR 0016 shape. + d, _ := newTestDaemon(t, "server_url=https://s.example\n") + rec := get(t, d.bridgeMux(), "/v1/info") + var m map[string]any + json.Unmarshal(rec.Body.Bytes(), &m) + for _, key := range []string{"temperatureC", "throttled", "underVoltage", "throttleOccurred"} { + if _, ok := m[key]; ok { + t.Fatalf("unsampled daemon reports %q", key) + } + } + if m["version"] != "test-version" { + t.Fatalf("info lost its core: %v", m) + } + + // And where vcgencmd doesn't exist (not a Pi), the loop stops and + // nothing is ever reported. + d2, _ := newTestDaemon(t, "server_url=https://s.example\n") + d2.vcgencmd = func(args ...string) (string, error) { + return "", exec.ErrNotFound + } + d2.sampleThermalsLoop() // returns immediately + rec = get(t, d2.bridgeMux(), "/v1/info") + m = nil + json.Unmarshal(rec.Body.Bytes(), &m) + if _, ok := m["temperatureC"]; ok { + t.Fatal("no-vcgencmd daemon must not report thermals") + } +} diff --git a/boxd/internal/daemon/watchdog.go b/boxd/internal/daemon/watchdog.go new file mode 100644 index 0000000..5f4a3f1 --- /dev/null +++ b/boxd/internal/daemon/watchdog.go @@ -0,0 +1,234 @@ +package daemon + +import ( + "fmt" + "time" +) + +// The watchdog policy (MAD-806, ADR 0008): the box notices when it is +// stuck and fixes itself, without anyone walking over to it. Two +// independent watchdogs: +// +// - The kernel's hardware watchdog, armed by systemd's +// RuntimeWatchdogSec=15s (stage-syncphony/07-watchdog), catches what +// no userspace loop can: a kernel or PID-1 hang hard-resets the Pi +// within 15 s. +// - The loop here, in boxd, catches a wedged page: the /tv heartbeat +// (ADR 0016, every 30 s while the page is healthy) stops for 90 s +// while the server's /healthz still answers → restart +// syncphony-kiosk.service. Chromium comes back, the page reloads, the +// speaker resumes. If that happens 3 times in 15 minutes, the box +// reboots — a kiosk boxd cannot keep alive is declared boot-sick. +// +// When the server is unreachable it is the offline screen's job, not the +// watchdog's: a missing heartbeat then means "there is no /tv page" (the +// kiosk sits on the local start screen), so the stall trigger requires +// /healthz to answer and the offline trigger keeps its own restart guard +// — no restart loops. The storage-lost screen is terminal the same way: +// rebooting cannot fix a dead /data, so it never feeds the stall trigger. +const ( + // heartbeatWindow is how long without a /v1/heartbeat counts as a + // stalled page (three missed 30 s beats) — and, in the offline + // trigger, how fresh a beat must be for the page to count as live. + heartbeatWindow = 90 * time.Second + + // offlineGrace and kioskRestartGuard govern the offline screen + // trigger: after the server has been unreachable this long while the + // page is alive, restart the kiosk onto the local start page (the + // offline screen), at most once per guard window. After the restart + // there are no heartbeats (the page is local), so it cannot loop; + // the guard is a second lock on that. + offlineGrace = 2 * time.Minute + kioskRestartGuard = 10 * time.Minute + + // watchdogSettle is how long after a kiosk (re)start the stall + // trigger stays quiet: cage, Chromium and the page need a minute or + // two before their first heartbeat, and a fresh page must not be + // restarted out from under itself. Short enough that three stalled + // restarts still land inside watchdogWindow. + watchdogSettle = 4 * time.Minute + + // watchdogRestarts kiosk restarts within watchdogWindow mean the box + // cannot keep a kiosk alive: reboot instead of restarting again. + watchdogRestarts = 3 + watchdogWindow = 15 * time.Minute + + // nightlyReloadHour is the quiet hour, local time, at which the + // kiosk is reloaded once a day: a long-lived Chromium with the + // visualizer on slowly grows its heap, and a fresh page gives the + // memory back. Only while the page is up and idle (never while + // music is playing, per the bridge events), and only for a session + // that has actually been running — a box booted at 04:50 does not + // need a 05:00 reload. + nightlyReloadHour = 5 + nightlyMinUptime = 6 * time.Hour +) + +// watchView is the locked snapshot the watchdog decision runs on — a +// plain struct so the policy is unit-testable without goroutines or +// clocks. +type watchView struct { + now time.Time + storageOK bool + serverOK bool + startedAt time.Time // boxd start; the first-heartbeat grace counts from here + lastBeat time.Time // zero = the page has never beaten + lastEvent string + offlineSince time.Time + lastRestart time.Time + restarts []time.Time // watchdog-caused kiosk restarts inside watchdogWindow + localHour int + localDay string // "2006-01-02" in local time + nightlyDay string // the last day the nightly reload ran +} + +// watchAction is what the policy decided for one tick. +type watchAction struct { + restart bool // restart syncphony-kiosk.service (a symptom of a stuck box) + nightly bool // restart the kiosk (care: a fresh page) + reboot bool + reason string +} + +// decide is the whole watchdog policy, pure on its input (see the const +// block above for the reasoning). The watch loop applies the result. +func decide(v watchView) watchAction { + if !v.serverOK { + // The offline screen trigger: a live page whose server went + // away — move the TV to the offline screen (the local start + // page), guarded so it can never loop. + if !v.offlineSince.IsZero() && + v.now.Sub(v.offlineSince) >= offlineGrace && + !v.lastBeat.IsZero() && v.now.Sub(v.lastBeat) < heartbeatWindow && + (v.lastRestart.IsZero() || v.now.Sub(v.lastRestart) > kioskRestartGuard) { + return watchAction{ + restart: true, + reason: fmt.Sprintf("server unreachable for %s with a live page; moving to the offline screen", offlineGrace), + } + } + return watchAction{} + } + + // The heartbeat watchdog: a page that stopped beating (or never + // started) while the server answers is stuck — unless the kiosk was + // just (re)started and is still loading, or /data is gone (that + // screen is terminal; a reboot cannot fix the card). + settled := v.now.Sub(v.lastRestart) >= watchdogSettle && + v.now.Sub(v.startedAt) >= watchdogSettle + stalled := !v.lastBeat.IsZero() && v.now.Sub(v.lastBeat) >= heartbeatWindow + if v.storageOK && settled && (stalled || v.lastBeat.IsZero()) { + // Only restarts inside the window count toward the reboot. + recent := pruneWindow(v.restarts, v.now, watchdogWindow) + if len(recent) >= watchdogRestarts-1 { + return watchAction{ + reboot: true, + reason: fmt.Sprintf("%d kiosk restarts within %s and the page is still stuck; rebooting", + len(recent), watchdogWindow), + } + } + return watchAction{restart: true, reason: "page heartbeats stopped while the server answers"} + } + + // The nightly reload, at the quiet hour, only for a page that is up + // and not playing. Skipped (not delayed) when a party runs through + // the whole hour; there is always the next night. + if v.localHour == nightlyReloadHour && + v.localDay != v.nightlyDay && + v.now.Sub(v.startedAt) >= nightlyMinUptime && + !v.lastBeat.IsZero() && v.now.Sub(v.lastBeat) < heartbeatWindow && + v.lastEvent != "playing" { + return watchAction{nightly: true, reason: "nightly reload at the quiet hour"} + } + return watchAction{} +} + +// watch is the background loop: sample the world, decide, act. Every 5 s +// is fine-grained enough for a 90 s heartbeat window. +func (d *Daemon) watch() { + tick := time.NewTicker(5 * time.Second) + defer tick.Stop() + for range tick.C { + if !d.configured() { + // No server, no /tv: the setup screen is the state, not a + // fault. (Thermals run in their own loop regardless.) + continue + } + ok := d.serverReachable() + now := time.Now() + d.mu.Lock() + if ok { + d.offlineSince = time.Time{} + } else if d.offlineSince.IsZero() { + d.offlineSince = now + } + d.restarts = pruneWindow(d.restarts, now, watchdogWindow) + day := now.Format("2006-01-02") + act := decide(watchView{ + now: now, + storageOK: storageHealthy(), + serverOK: ok, + startedAt: d.startedAt, + lastBeat: d.lastBeat, + lastEvent: d.lastEvent, + offlineSince: d.offlineSince, + lastRestart: d.lastRestart, + restarts: d.restarts, + localHour: now.Hour(), + localDay: day, + nightlyDay: d.lastNightlyDay, + }) + if act.nightly { + // Marked under the same lock as the decision, so a slow + // restart can't let the next tick fire it twice. + d.lastNightlyDay = day + } + d.mu.Unlock() + + switch { + case act.reboot: + d.log.Printf("boxd: watchdog: %s", act.reason) + if err := d.run("systemctl", "reboot"); err != nil { + d.log.Printf("boxd: watchdog: reboot failed: %v", err) + } + case act.nightly: + d.log.Printf("boxd: watchdog: %s", act.reason) + d.restartKiosk("nightly reload", false) + case act.restart: + d.log.Printf("boxd: watchdog: %s", act.reason) + d.restartKiosk(act.reason, true) + } + } +} + +// restartKiosk restarts the kiosk unit and records the restart so the +// settle window applies to the fresh page too. count=true is for +// watchdog restarts — the symptoms the 3-in-15-minutes reboot counts; +// intentional restarts (remote reload, the setup save, the nightly +// reload) settle but don't count: they are care, not symptoms. +func (d *Daemon) restartKiosk(reason string, count bool) error { + now := time.Now() + d.mu.Lock() + d.lastRestart = now + if count { + d.restarts = append(d.restarts, now) + } + d.mu.Unlock() + d.log.Printf("boxd: restarting the kiosk (%s)", reason) + if err := d.run("systemctl", "restart", "syncphony-kiosk.service"); err != nil { + d.log.Printf("boxd: kiosk restart failed: %v", err) + return err + } + return nil +} + +// pruneWindow drops timestamps older than window: the restart history +// only ever answers "how many within the window". +func pruneWindow(ts []time.Time, now time.Time, window time.Duration) []time.Time { + kept := ts[:0] + for _, t := range ts { + if now.Sub(t) < window { + kept = append(kept, t) + } + } + return kept +} diff --git a/boxd/internal/daemon/watchdog_test.go b/boxd/internal/daemon/watchdog_test.go new file mode 100644 index 0000000..e4f548e --- /dev/null +++ b/boxd/internal/daemon/watchdog_test.go @@ -0,0 +1,220 @@ +package daemon + +import ( + "strings" + "testing" + "time" +) + +// healthyView is a configured box with everything fine: storage up, +// server answering, a page that beat 10 s ago and is idle, running for +// 10 minutes. Each test bends one input. +func healthyView(now time.Time) watchView { + return watchView{ + now: now, + storageOK: true, + serverOK: true, + startedAt: now.Add(-10 * time.Minute), + lastBeat: now.Add(-10 * time.Second), + lastEvent: "idle", + localHour: 15, + localDay: now.Format("2006-01-02"), + nightlyDay: "", + } +} + +// --- the heartbeat watchdog --------------------------------------------------- + +func TestWatchdogRestartsStalledPage(t *testing.T) { + now := time.Now() + v := healthyView(now) + v.lastBeat = now.Add(-2 * time.Minute) // stalled well past the window + act := decide(v) + if !act.restart || act.reboot || act.nightly { + t.Fatalf("want a kiosk restart, got %+v", act) + } +} + +func TestWatchdogIgnoresStallWhenServerUnreachable(t *testing.T) { + // A missing heartbeat with the server gone means "no /tv page" (the + // offline screen has the TV) — not a stuck page. No restart, no loop. + now := time.Now() + v := healthyView(now) + v.serverOK = false + v.offlineSince = now.Add(-3 * time.Minute) + v.lastBeat = now.Add(-2 * time.Minute) + if act := decide(v); act.restart || act.reboot || act.nightly { + t.Fatalf("stall must not trigger while the server is unreachable: %+v", act) + } +} + +func TestOfflineTriggerStillRestartsLivePage(t *testing.T) { + // The MAD-803 behavior, unchanged: live page, gone server → the + // offline screen takes over. + now := time.Now() + v := healthyView(now) + v.serverOK = false + v.offlineSince = now.Add(-3 * time.Minute) + v.lastBeat = now.Add(-30 * time.Second) + act := decide(v) + if !act.restart { + t.Fatalf("want the offline-screen restart, got %+v", act) + } +} + +func TestWatchdogHonorsSettleWindow(t *testing.T) { + // A kiosk restarted a minute ago is still loading; even a stale + // heartbeat must not restart it again yet. + now := time.Now() + v := healthyView(now) + v.lastBeat = now.Add(-2 * time.Minute) + v.lastRestart = now.Add(-1 * time.Minute) + if act := decide(v); act.restart || act.reboot { + t.Fatalf("restarted within the settle window: %+v", act) + } + v.lastRestart = now.Add(-5 * time.Minute) + if act := decide(v); !act.restart { + t.Fatalf("should restart after the settle window: %+v", act) + } +} + +func TestWatchdogFirstHeartbeatGrace(t *testing.T) { + // A page that has never beaten gets the same grace after boot... + now := time.Now() + v := healthyView(now) + v.lastBeat = time.Time{} + v.startedAt = now.Add(-1 * time.Minute) + if act := decide(v); act.restart || act.reboot { + t.Fatalf("booted a minute ago, still loading: %+v", act) + } + // ...but a box up for ten minutes with no heartbeat ever is stuck. + v.startedAt = now.Add(-10 * time.Minute) + if act := decide(v); !act.restart { + t.Fatalf("no heartbeat ever past the grace: %+v", act) + } +} + +func TestWatchdogSkipsWhenStorageLost(t *testing.T) { + // The storage screen is terminal; rebooting can't fix the card, so + // it never feeds the stall trigger. + now := time.Now() + v := healthyView(now) + v.storageOK = false + v.lastBeat = now.Add(-2 * time.Minute) + if act := decide(v); act.restart || act.reboot || act.nightly { + t.Fatalf("storage-lost box must not be watchdogged: %+v", act) + } +} + +func TestWatchdogRebootsInsteadOfFourthRestart(t *testing.T) { + // Two watchdog restarts inside the window: the next stall is the + // third — reboot rather than restart again. + now := time.Now() + v := healthyView(now) + v.lastBeat = now.Add(-2 * time.Minute) + v.restarts = []time.Time{now.Add(-7 * time.Minute), now.Add(-3 * time.Minute)} + act := decide(v) + if !act.reboot || act.restart { + t.Fatalf("want a reboot on the third stall, got %+v", act) + } +} + +func TestWatchdogRestartsWhenHistoryAgedOut(t *testing.T) { + // Only one of the two restarts is inside the 15-minute window: this + // is the second, not the third — restart, don't reboot. + now := time.Now() + v := healthyView(now) + v.lastBeat = now.Add(-2 * time.Minute) + v.restarts = []time.Time{now.Add(-20 * time.Minute), now.Add(-3 * time.Minute)} + if act := decide(v); !act.restart || act.reboot { + t.Fatalf("want a restart, got %+v", act) + } +} + +// --- the nightly reload -------------------------------------------------------- + +func TestNightlyReloadAtFiveWhenIdle(t *testing.T) { + now := time.Now() + v := healthyView(now) + v.startedAt = now.Add(-8 * time.Hour) + v.localHour = nightlyReloadHour + for _, event := range []string{"idle", "paused", ""} { + v.lastEvent = event + act := decide(v) + if !act.nightly || act.restart || act.reboot { + t.Fatalf("event %q: want the nightly reload, got %+v", event, act) + } + } +} + +func TestNightlyReloadNeverWhilePlaying(t *testing.T) { + now := time.Now() + v := healthyView(now) + v.startedAt = now.Add(-8 * time.Hour) + v.localHour = nightlyReloadHour + v.lastEvent = "playing" + if act := decide(v); act.nightly || act.restart || act.reboot { + t.Fatalf("must never reload while music is playing: %+v", act) + } +} + +func TestNightlyReloadOncePerDayAndOnlyAtFive(t *testing.T) { + now := time.Now() + v := healthyView(now) + v.startedAt = now.Add(-8 * time.Hour) + v.localHour = nightlyReloadHour + v.nightlyDay = v.localDay // already done today + if act := decide(v); act.nightly { + t.Fatalf("already reloaded today: %+v", act) + } + v.nightlyDay = "" + for _, hour := range []int{4, 6} { + v.localHour = hour + if act := decide(v); act.nightly { + t.Fatalf("hour %d must not reload: %+v", hour, act) + } + } +} + +func TestNightlyReloadOnlyForLongSessions(t *testing.T) { + // A box booted at 04:50 doesn't need a 05:00 reload. + now := time.Now() + v := healthyView(now) + v.localHour = nightlyReloadHour + v.startedAt = now.Add(-1 * time.Hour) + if act := decide(v); act.nightly { + t.Fatalf("fresh boot must not reload: %+v", act) + } + v.startedAt = now.Add(-8 * time.Hour) + if act := decide(v); !act.nightly { + t.Fatalf("a long session should reload: %+v", act) + } +} + +// --- restartKiosk bookkeeping --------------------------------------------------- + +func TestRestartKioskRecordsAndCounts(t *testing.T) { + d, rec := newTestDaemon(t, "server_url=https://s.example\n") + if err := d.restartKiosk("test", true); err != nil { + t.Fatalf("restartKiosk: %v", err) + } + waitFor(t, func() bool { + return strings.Contains(strings.Join(rec.calls(), ";"), "systemctl restart syncphony-kiosk.service") + }) + d.mu.Lock() + lastRestart, restarts := d.lastRestart, d.restarts + d.mu.Unlock() + if lastRestart.IsZero() || len(restarts) != 1 { + t.Fatalf("bookkeeping: lastRestart=%v restarts=%v", lastRestart, restarts) + } + // An intentional restart settles but doesn't count. + if err := d.restartKiosk("care", false); err != nil { + t.Fatalf("restartKiosk: %v", err) + } + d.mu.Lock() + restarts = d.restarts + d.mu.Unlock() + if len(restarts) != 1 { + t.Fatalf("counted a care restart: %v", restarts) + } +} diff --git a/docs/adr/0008-watchdogs-and-self-healing.md b/docs/adr/0008-watchdogs-and-self-healing.md new file mode 100644 index 0000000..c2236b7 --- /dev/null +++ b/docs/adr/0008-watchdogs-and-self-healing.md @@ -0,0 +1,108 @@ +# ADR 0008: Watchdogs and self-healing + +- Status: Accepted +- Date: 2026-10-14 +- Deciders: Phase 11 plan (MAD-797), watchdog issue (MAD-806) + +## Context + +The box hangs on a living-room wall: nobody is standing next to it when +Chromium's renderer wedges, the kernel panics, or the clock drifts into +years past. A box that needs someone to walk over and pull the plug has +failed at being an appliance. Stage 2 added the heartbeat (Syncphony +ADR 0016: the page beats every 30 s while healthy) and the offline +screen's restart; this ADR is the rest of the noticing-and-fixing. + +## Decision + +Three watchdogs, layered so each failure has exactly one owner. + +### 1. Hardware watchdog: systemd arms the BCM watchdog at 15 s + +`RuntimeWatchdogSec=15s` (drop-in in `stage-syncphony/07-watchdog`) — +PID 1 pets `/dev/watchdog` at half the interval; a kernel or systemd +hang stops the petting and the hardware hard-resets the Pi. This is the +only watchdog that catches the kernel itself. `RebootWatchdogSec=2min` +bounds a reboot that hangs partway. The read-only root (ADR 0007) is +what makes a hard reset safe: the box comes back clean, paired, playing. + +### 2. Kiosk watchdog in boxd: the heartbeat policy + +`boxd` watches the bridge heartbeat every 5 s +(`boxd/internal/daemon/watchdog.go`). The policy, as one pure decision: + +- **Stall → restart.** The page's heartbeat has been silent for 90 s + (three missed 30 s beats) while the server's `/healthz` answers → + restart `syncphony-kiosk.service`. Chromium comes back, the page + reloads, the speaker resumes on its own (MAD-804). This is the + `kill -STOP`-on-the-renderer case: the room returns within ~2 minutes. +- **Settle window.** A kiosk (re)start is followed by 4 minutes of + quiet: cage, Chromium and the page need a minute or two before their + first heartbeat, and the fresh page must not be restarted out from + under itself. The same grace covers a boot where no heartbeat ever + arrived. +- **3 in 15 minutes → reboot.** Watchdog-caused restarts inside a + 15-minute window, and the page stalls again → reboot instead of + restarting a fourth time. A kiosk boxd cannot keep alive is declared + boot-sick; the hardware watchdog and the read-only root make the + reboot cheap. +- **Server unreachable is not the watchdog's problem.** A missing + heartbeat then means "there is no /tv page" — the kiosk sits on the + local start screen, which is the offline screen's job (MAD-803): that + trigger keeps its own 10-minute guard, so no restart loops. The two + triggers are mutually exclusive on `/healthz`. +- **The storage-lost screen never feeds the watchdog.** Without `/data` + the kiosk cannot produce a page that beats; rebooting cannot fix the + card, so that screen is terminal by design (ADR 0007). +- Intentional restarts (`/v1/reload`, the setup save, the nightly + reload) settle but don't count toward the reboot: care, not symptoms. + +### Nightly reload at 05:00, only when idle + +Chromium with a long-lived page and the visualizer on slowly grows its +heap; a fresh page gives the memory back. Once a day, during the 05:00 +local hour, when the page is up (fresh beats), not playing (bridge +events), and the session has run at least 6 hours, boxd restarts the +kiosk — a reload by the only path that always works. A party running +through the hour skips that night; there is always the next. The +12-hour-visualizer memory check that motivated it is a hardware +procedure in docs/testing.md. + +### 3. Thermals: sample, log, report + +boxd runs `vcgencmd measure_temp` and `vcgencmd get_throttled` +(`libraspberrypi-bin`, installed with this stage) once a minute, logs +one journal line per sample (`journalctl -u boxd` doubles as a +temperature log, and journald is persistent on /data), and reports the +latest values in `GET /v1/info` — `temperatureC`, `throttled`, +`underVoltage`, `throttleOccurred` — for stage 4's Screens list. The +flags are the firmware's bitmap: current conditions in bits 0-3 +(under-voltage, frequency-capped, throttled, soft temperature limit), +the same four as sticky "has occurred" flags in bits 16-19; `throttled` +means limited *right now*. No vcgencmd (a dev machine): thermals are +absent, everything downstream tolerates that. The README recommends a +case with a heatsink or fan; under a long party in a closed case the +soft temperature limit is what silently halves the CPU first. + +### Time: timesyncd before the kiosk + +`systemd-timesyncd` is installed and enabled, and +`syncphony-kiosk.service` orders `After=`/`Wants=` `time-sync.target`: +the server's playback clock sync trusts the browser's clock, and a Pi +has no RTC battery by default (the Pi 5's RTC header ships without a +battery too). The ordering is best-effort by design — a box with no +network must still reach its setup screen. + +## Consequences + +- `echo c > /proc/sysrq-trigger` reboots the box within ~15 s and it + comes back playing; `kill -STOP` on Chromium's renderer brings the + room back within ~2 minutes. Both procedures are in docs/testing.md. +- boxd's `/v1/info` grows thermal fields beyond ADR 0016's core three; + unknown fields are ignorable by contract, and stage 4 (MAD-811) is + their consumer. The bit layout of `get_throttled` is pinned by tests. +- The watchdog restart history lives in memory: a reboot clears the + count, which is exactly right — the reboot *is* the escalation. +- journald gains one line per minute (~few hundred kB/day against the + 50 MB cap); worth it for a temperature history that survives power + cuts. diff --git a/docs/testing.md b/docs/testing.md index 616c6b1..ba389ac 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -6,6 +6,90 @@ image's structure; the checks below are for real hardware. The README's on-Pi checklist covers the everyday list — this page is the destructive one. +## The watchdog gauntlet (MAD-806) + +**Done-when for the watchdogs:** a frozen Chromium renderer brings the +room back within about 2 minutes, and a forced kernel hang reboots the +box, which comes back playing. Run both on a box that is paired and +playing music. + +### Freeze the renderer (the boxd kiosk watchdog) + +```sh +pkill -STOP -f 'type=renderer' # freeze Chromium's renderer process +journalctl -u boxd -f # watch the watchdog work +``` + +Expected timeline: the music keeps playing for a moment (or stops), the +page freezes on screen, and after ~90 s without a heartbeat — while the +server still answers `/healthz` — boxd logs +`page heartbeats stopped while the server answers` and restarts the +kiosk. The room is back, and the speaker resumes by itself, within about +2 minutes of the freeze. Verify the page is interactive again (queue a +song from a phone). + +Repeat the freeze twice more inside 15 minutes: the third stall must +log `...kiosk restarts within 15m0s...; rebooting` and reboot the box +rather than restart a fourth time. After it comes back, everything plays. + +Also worth one pass: pull the server's network cable instead — the +watchdog must stay quiet (it's the offline screen's job), and no restart +loop appears in the journal. + +### Force a kernel hang (the hardware watchdog) + +```sh +sudo sh -c 'echo 1 > /proc/sys/kernel/sysrq' # allow the crash trigger +sudo sh -c 'echo c > /proc/sysrq-trigger' # panic, immediately +``` + +Expected: the box dies instantly (this is a real panic — nothing in +userspace can react), the hardware watchdog hard-resets it within ~15 s +of the hang, and it boots back into the room and resumes playing with +no touch. This works because systemd pets `/dev/watchdog` every ~7.5 s +(`RuntimeWatchdogSec=15s`); the panic stops the petting. + +Both on a Pi 4 and a Pi 5, please — the watchdog device and `sysrq` +behave the same in theory, but this test exists to check that. + +### The 12-hour party soak (Chromium memory) + +The question: does Chromium's memory grow without bound with the +visualizer on, on a 2 GB Pi 4? Procedure: + +1. Pair the box as the room's speaker and play a long queue with the + visualizer on, overnight (~12 h). +2. Sample memory hourly (or read it out of the journal afterwards): + + ```sh + ps -o pid=,rss=,args= -C chromium | sort -k2 -n | tail -3 + ``` + +3. Note the renderer's RSS over time. + +Pass: the total flattens (a sawtooth as pages reload is fine — the +nightly reload at 05:00 only fires while idle, so a party that runs +through the night must not be interrupted; check the journal for when +the reload actually happened). Fail: steady growth past ~1.2 GB total, +OOM kills in the journal (`journalctl -k | grep -i oom`), or audio +dropouts late in the run. If it fails, the finding goes with the +watchdog issue's follow-ups: the nightly reload is already the +mitigation, the next step would be a more aggressive reload policy. + +### Thermals and time sync + +- `journalctl -u boxd | grep thermals` — one line per minute with + `temp=` and `throttled=`. During the soak above, the temperature + should stay under the soft limit (~80 °C on a Pi 4); a persistent + nonzero `throttled=` (e.g. `0x50000`, which means it *has* throttled + or browned out before) points at the case or the power supply. +- `curl -s http://127.0.0.1:8099/v1/info` includes `temperatureC` and + the throttle fields. +- `timedatectl` → "System clock synchronized: yes" while the kiosk is + up; `systemctl status systemd-timesyncd` → active. Check on both a + Pi 4 and a Pi 5 (the Pi 5 has an RTC header, but it has no battery by + default, so it still needs the network). + ## The power-pull gauntlet (50 pulls) **Done-when for the read-only root (MAD-805):** 50 random power pulls — diff --git a/scripts/smoke-test.sh b/scripts/smoke-test.sh index 6d8775f..c731701 100755 --- a/scripts/smoke-test.sh +++ b/scripts/smoke-test.sh @@ -12,11 +12,14 @@ # audio stack is in place (PipeWire user services for the kiosk user with # linger, the sink-selection units, the WirePlumber tuning), boxd (the # Go helper daemon) is installed with its units enabled, the pairing -# persistence pieces agree (one profile path, crash-state prep, reset), and -# the read-only-root layout is complete: three partitions with the third +# persistence pieces agree (one profile path, crash-state prep, reset), the +# read-only-root layout is complete: three partitions with the third # labelled syncphony-data and pre-populated, the overlay mechanism wired # into the initramfs and the kernel command line, every persistent path -# pointing at /data, and the stock root-resize replaced by the data grow. +# pointing at /data, and the stock root-resize replaced by the data grow; +# and the self-healing pieces are wired: the systemd hardware watchdog +# drop-in, timesyncd enabled and ordered before the kiosk, and vcgencmd +# for boxd's thermals. set -euo pipefail @@ -93,7 +96,8 @@ fi # --- packages --------------------------------------------------------------- for pkg in cage chromium fonts-noto-color-emoji libgl1-mesa-dri cloud-init avahi-daemon \ - pipewire pipewire-bin pipewire-alsa pipewire-pulse wireplumber pulseaudio-utils alsa-utils; do + pipewire pipewire-bin pipewire-alsa pipewire-pulse wireplumber pulseaudio-utils alsa-utils \ + libraspberrypi-bin systemd-timesyncd; do if sudo awk -v p="${pkg}" ' $0 == "Package: " p { inpkg = 1 } inpkg && /^Status: / { @@ -112,7 +116,7 @@ done for unit in cloud-config.service cloud-final.service syncphony-kiosk.service syncphony-box-config.service \ avahi-daemon.service syncphony-audio.service syncphony-audio-monitor.service boxd.service \ syncphony-data-grow.service syncphony-data-prepare.service syncphony-data-save.service \ - syncphony-data-save.timer; do + syncphony-data-save.timer systemd-timesyncd.service; do state="$(systemctl --root="${WORK}/root" is-enabled "${unit}" 2>/dev/null || true)" if [ "${state}" = "enabled" ]; then check "unit ${unit} enabled" ok @@ -246,6 +250,33 @@ else check "boxd.service runs boxd serve" "ExecStart mismatch" fi +# --- watchdogs and self-healing (MAD-806) ------------------------------------- +if sudo grep -q '^RuntimeWatchdogSec=15s' \ + "${WORK}/root/etc/systemd/system.conf.d/50-syncphony-watchdog.conf"; then + check "hardware watchdog armed at 15 s" ok +else + check "hardware watchdog armed at 15 s" "drop-in missing or wrong" +fi + +if sudo grep -q '^RebootWatchdogSec=' \ + "${WORK}/root/etc/systemd/system.conf.d/50-syncphony-watchdog.conf"; then + check "reboot watchdog bounded" ok +else + check "reboot watchdog bounded" "no RebootWatchdogSec" +fi + +if sudo grep -q 'time-sync.target' "${WORK}/root/etc/systemd/system/syncphony-kiosk.service"; then + check "kiosk unit waits for time sync" ok +else + check "kiosk unit waits for time sync" "no time-sync.target ordering" +fi + +if sudo test -x "${WORK}/root/usr/bin/vcgencmd"; then + check "vcgencmd installed (thermals)" ok +else + check "vcgencmd installed (thermals)" "missing or not executable" +fi + # The boot config apply is boxd now: same unit, new engine. if sudo grep -q '^ExecStart=/usr/lib/syncphony-box/boxd config apply$' \ "${WORK}/root/etc/systemd/system/syncphony-box-config.service"; then diff --git a/stage-syncphony/01-kiosk/files/syncphony-kiosk.service b/stage-syncphony/01-kiosk/files/syncphony-kiosk.service index f73458b..8b15fd5 100644 --- a/stage-syncphony/01-kiosk/files/syncphony-kiosk.service +++ b/stage-syncphony/01-kiosk/files/syncphony-kiosk.service @@ -1,8 +1,11 @@ [Unit] Description=Syncphony kiosk session (cage + Chromium) Conflicts=getty@tty1.service -After=systemd-user-sessions.service getty@tty1.service network-online.target syncphony-audio.service boxd.service -Wants=network-online.target syncphony-audio.service boxd.service +# time-sync.target: the server's playback clock sync trusts the browser's +# clock, and a Pi has no RTC battery by default — let systemd-timesyncd +# give the box a sane clock before the page starts (MAD-806). +After=systemd-user-sessions.service getty@tty1.service network-online.target syncphony-audio.service boxd.service time-sync.target +Wants=network-online.target syncphony-audio.service boxd.service time-sync.target [Service] User=kiosk diff --git a/stage-syncphony/07-watchdog/00-packages b/stage-syncphony/07-watchdog/00-packages new file mode 100644 index 0000000..a97c6e9 --- /dev/null +++ b/stage-syncphony/07-watchdog/00-packages @@ -0,0 +1,6 @@ +# Watchdogs and self-healing (MAD-806): vcgencmd (libraspberrypi-bin) is +# what boxd samples every minute for the temperature and throttling log, +# and systemd-timesyncd gives the box a sane clock before the kiosk starts +# (a Pi has no RTC battery by default). +libraspberrypi-bin +systemd-timesyncd diff --git a/stage-syncphony/07-watchdog/00-run.sh b/stage-syncphony/07-watchdog/00-run.sh new file mode 100644 index 0000000..12a592f --- /dev/null +++ b/stage-syncphony/07-watchdog/00-run.sh @@ -0,0 +1,25 @@ +#!/bin/bash -e + +# Watchdogs and self-healing (MAD-806, ADR 0008). Two of the three +# watchdogs are image-level; the kiosk watchdog lives in boxd +# (boxd/internal/daemon/watchdog.go). +# +# - Hardware watchdog: systemd's RuntimeWatchdogSec=15s (the drop-in +# below) arms the BCM watchdog. A kernel or systemd hang reboots the +# Pi — the one failure no userspace loop can catch. +# - Time: systemd-timesyncd, enabled here and ordered before the kiosk +# (time-sync.target in syncphony-kiosk.service). The server's +# playback clock sync trusts the browser's clock, and a Pi has no RTC +# battery by default — without sync the box boots with a stale clock. +# - Thermals: boxd runs vcgencmd (installed with this stage's packages) +# every minute; nothing to wire here. + +FILES="$(dirname "$0")/files" + +install -v -d -m 0755 "${ROOTFS_DIR}/etc/systemd/system.conf.d" +install -v -m 0644 "${FILES}/50-syncphony-watchdog.conf" \ + "${ROOTFS_DIR}/etc/systemd/system.conf.d/50-syncphony-watchdog.conf" + +on_chroot << EOF +systemctl enable systemd-timesyncd.service +EOF diff --git a/stage-syncphony/07-watchdog/files/50-syncphony-watchdog.conf b/stage-syncphony/07-watchdog/files/50-syncphony-watchdog.conf new file mode 100644 index 0000000..715ad7a --- /dev/null +++ b/stage-syncphony/07-watchdog/files/50-syncphony-watchdog.conf @@ -0,0 +1,8 @@ +# Hardware watchdog (MAD-806, docs/adr/0008-watchdogs-and-self-healing.md). +# systemd pets /dev/watchdog (the BCM's hardware watchdog) at half the +# runtime interval; a kernel or systemd hang that stops the petting +# hard-resets the Pi within 15 s. RebootWatchdogSec bounds a reboot that +# hangs partway, so the box still comes back on its own. +[Manager] +RuntimeWatchdogSec=15s +RebootWatchdogSec=2min