diff --git a/README.md b/README.md index b9c4af3..1a9a6ae 100644 --- a/README.md +++ b/README.md @@ -36,9 +36,10 @@ The result lands in `deploy/`: `make clean` removes build output and the leftover pi-gen build container. `make test` runs the unit tests — the bats suite (`tests/`) that pins the -`syncphony.txt` behavior against a boxd binary built from the repo, plus -`go test` for boxd — no hardware or image build needed; install bats -(`apt install bats`) and Go first. `make lint` checks gofmt and go vet. +`syncphony.txt` behavior and the `/data` persistence script against +binaries built from the repo, plus `go test` for boxd — no hardware or +image build needed; install bats (`apt install bats`) and Go first. +`make lint` checks gofmt and go vet. CI runs them on every PR and push. ### Continuous integration @@ -47,8 +48,9 @@ GitHub Actions builds the image natively on arm64 runners bats and Go tests, runs the no-hardware smoke test (`scripts/smoke-test.sh` loop-mounts the built image and checks packages, enabled units, the locked first user, the first-boot seed files, the kiosk session, the boot -config file, the audio stack, and boxd), and uploads the image as a workflow -artifact. +config file, the audio stack, boxd, and the read-only-root layout: three +partitions, the overlay wired into the initramfs, and every persistent +path on /data), and uploads the image as a workflow artifact. ## boxd @@ -96,9 +98,9 @@ from there. The box's whole identity is its Chromium profile: the server's `syncphony_display` pairing cookie and the "I'm the speaker" localStorage flag live there, so that's what must survive reboots and power cuts. The -profile path is defined in one place (boxd's `DefaultProfileDir`; -stage 3 moves it to `/data` by changing that constant) and reaches the -kiosk wrapper through `config.env`. See +profile lives on the **/data partition** (`/data/chromium` — see below); +the path is defined in one place (boxd's `DefaultProfileDir`) and reaches +the kiosk wrapper through `config.env`. See [docs/adr/0006-pairing-persistence.md](docs/adr/0006-pairing-persistence.md). Two mechanisms keep it durable: @@ -123,6 +125,45 @@ sudo syncphony-box reset In a dev checkout, `make reset` does the same to a profile (point it with `SB_PROFILE=...`; it defaults to the box's path). +## Safe to unplug: read-only root and /data + +The box is meant to be power-cycled like a TV — pulled at any moment, +mid-song included. The image makes that structurally safe +([docs/adr/0007](docs/adr/0007-readonly-root-and-data.md)): + +- **Three partitions:** boot (FAT), root (ext4), and **data** (ext4, + label `syncphony-data`). On first boot the data partition grows to fill + the card; the root never grows. +- **The root filesystem is read-only behind an overlay** — the mechanism + raspi-config's "Overlay FS" option uses, enabled at build time: the + real root is the lower layer, every runtime write lands in a tmpfs + upper and evaporates at power-off. Nothing on the SD card is ever + half-written. +- **Everything that must persist lives on /data** — the pairing (the + Chromium profile), the applied config, logs, the SSH host keys, the + Imager-created user, Wi-Fi credentials. The maintained list is + [docs/persistent-paths.md](docs/persistent-paths.md). +- **/boot/firmware stays read-only** too; boxd remounts it rw for the + moment it writes `syncphony.txt` (the setup form) or a `resolution=` + change, then ro again. +- **A damaged /data never hangs the boot**: the box comes up unpaired + and the TV says the storage is lost (fix: check or re-flash the card). + +The power-pull test procedure (50 pulls: during boot, during playback, +seconds after pairing) is [docs/testing.md](docs/testing.md). + +**Maintaining the box.** Runtime writes under `/` are discarded by +design, so package updates go to the real root through a chroot (the +overlayroot-chroot role in this image): + +```sh +sudo syncphony-box chroot # shell on the real root, read-write +# (or: sudo syncphony-box chroot apt full-upgrade) +``` + +Reboot afterwards. Changes to the pairing, config and anything else on +`/data` need no chroot — that partition is writable at runtime. + ## Audio The box is the room's speaker: Chromium plays everything through the @@ -175,10 +216,10 @@ the box from booting; bad values are logged and fall back to defaults. | `cec` | `on`, `off` | `on` | switch the TV's input when music starts | On every boot, `syncphony-box-config.service` runs `boxd config apply`: -it parses the file into `/etc/syncphony-box/config.env` before the kiosk -starts; `resolution` and `rotate` also become a `video=HDMI-A-1:…D` kernel -argument (the `D` forces the output on, so a TV that's off at boot still -gets a picture later). +it parses the file into `config.env` (kept on `/data`; `/etc/syncphony-box/config.env` +is a symlink to it) before the kiosk starts; `resolution` and `rotate` +also become a `video=HDMI-A-1:…D` kernel argument (the `D` forces the +output on, so a TV that's off at boot still gets a picture later). Without a valid `server_url` the box shows its setup screen: a QR code you scan with a phone on the same network to set the server — no SD card @@ -186,7 +227,9 @@ editing needed. (Editing the card still works too.) ### On-Pi checklist -CI can't boot a Pi; verify these on hardware when one is at hand: +CI can't boot a Pi; verify these on hardware when one is at hand. The +destructive power-pull gauntlet (the done-when for the read-only root) +has its own page: [docs/testing.md](docs/testing.md). - Boots to the server's `/tv` fullscreen in under ~30 s on a Pi 5. - `sudo systemctl kill --kill-who=all syncphony-kiosk` (or killing @@ -194,13 +237,16 @@ CI can't boot a Pi; verify these on hardware when one is at hand: - No visible cursor — including with a mouse plugged in and wiggled. - No crash bubble, and no "restore pages" prompt after pulling power. - Pair → reboot → the box is back in the room and playing without a touch. -- Pair → pull the power at random points 20 times (some within 10 s of - pairing, some during playback, some during boot) → it comes back paired - every time. If one is ever lost, `journalctl -u boxd` should show the - `paired event; syncing disks` line for that boot — that's the finding - to report. - `sudo syncphony-box reset` wipes the pairing: the box shows a fresh pairing code afterwards, and re-pairs normally. +- Read-only root: `findmnt /` shows an overlay, `/boot/firmware` is `ro`, + `/data` is mounted, `touch /usr/bin/nope` survives only until a reboot, + and the setup form still saves (boxd remounts the boot partition around + its write). +- First boot grows `/data` to fill the card (`lsblk`), root stays at + image size. +- `sudo syncphony-box chroot apt update` writes the real root; the change + is still there after a reboot. - The `/tv` visualizer runs smoothly with GPU rasterization (check `chrome://gpu` on the box); if not, tune the GPU flags in the wrapper. - Edit `syncphony.txt` on a laptop (Notepad is fine), boot, and the server @@ -246,14 +292,20 @@ CI can't boot a Pi; verify these on hardware when one is at hand: ``` pi-gen/ pi-gen submodule (pinned, never edited) -stage-syncphony/ custom pi-gen stage (packages, kiosk session, ...) +stage-syncphony/ custom pi-gen stage (packages, kiosk session, + read-only root, ...) boxd/ the box's Go helper daemon (bridge, config, screens) config pi-gen build configuration -build.sh wraps pi-gen's build-docker.sh (builds boxd first) +build.sh wraps pi-gen's build-docker.sh (builds boxd first, + then appends the /data partition after export) Makefile make image / make test / make lint / make clean +scripts/add-data-partition.sh the /data partition post-export step scripts/smoke-test.sh no-hardware image checks (used by CI) -tests/ bats tests pinning the syncphony.txt behavior (used by CI) +tests/ bats tests pinning the syncphony.txt and /data + behavior (used by CI) docs/adr/ architecture decision records +docs/persistent-paths.md the list of everything that survives a power cut +docs/testing.md the hardware test procedures (power pulls & co) ``` ## License diff --git a/boxd/internal/config/apply.go b/boxd/internal/config/apply.go index 9b09dd0..22b384a 100644 --- a/boxd/internal/config/apply.go +++ b/boxd/internal/config/apply.go @@ -193,7 +193,12 @@ func syncCmdline(resolution, rotate string, p Paths, run RunCommand, log *log.Lo if next == current { return } - if err := os.WriteFile(p.Cmdline, []byte(next+"\n"), 0o644); err != nil { + // /boot/firmware stays read-only (ADR 0007): the write goes through the + // remount dance, so a power pull can never catch the FAT half-written. + err := WithMountWritable(p.Cmdline, run, log, func() error { + return os.WriteFile(p.Cmdline, []byte(next+"\n"), 0o644) + }) + if err != nil { log.Printf("boxd config: cannot write %s; display settings stay as they are: %v", p.Cmdline, err) return } diff --git a/boxd/internal/config/config.go b/boxd/internal/config/config.go index 4a11ea3..f913b03 100644 --- a/boxd/internal/config/config.go +++ b/boxd/internal/config/config.go @@ -16,15 +16,27 @@ import ( // apply path can be tested on a dev machine instead of a box. const ( DefaultTxtPath = "/boot/firmware/syncphony.txt" - DefaultEnvPath = "/etc/syncphony-box/config.env" DefaultCmdlinePath = "/boot/firmware/cmdline.txt" + // Everything that must survive a power pull lives on the /data + // partition (MAD-805, ADR 0007); the root filesystem behind its + // overlay is read-only. The complete list is docs/persistent-paths.md. + DefaultDataDir = "/data" + + // DefaultEnvPath is the applied config. It lives on /data (writable); + // /etc/syncphony-box/config.env is a symlink to it so the kiosk + // wrapper and the reset tool keep reading the path they always did. + // A missing /data only costs the write: apply logs it and the kiosk + // falls back to defaults. + DefaultEnvPath = "/data/syncphony-box/config.env" + // DefaultProfileDir is the Chromium profile: where the paired-display // cookie and the speaker localStorage flag live (MAD-804, ADR 0006). // It is the ONE place the path is defined — config.env carries it to - // the kiosk wrapper and the reset tool. Stage 3 moves it to /data by - // changing this constant. SB_PROFILE overrides it for dev and tests. - DefaultProfileDir = "/var/lib/syncphony-box/chromium" + // the kiosk wrapper and the reset tool, and it lives on /data so the + // pairing survives anything the overlay discards (MAD-805). SB_PROFILE + // overrides it for dev and tests. + DefaultProfileDir = "/data/chromium" ) // Config is the validated result of parsing syncphony.txt. Invalid values diff --git a/boxd/internal/config/mounts.go b/boxd/internal/config/mounts.go new file mode 100644 index 0000000..74ccbee --- /dev/null +++ b/boxd/internal/config/mounts.go @@ -0,0 +1,114 @@ +package config + +import ( + "log" + "os" + "strings" + "syscall" +) + +// The mounts table boxd consults: which filesystem a path lives on, and +// whether that filesystem is read-only. On the box this decides two things +// (MAD-805, ADR 0007): +// +// - writes to /boot/firmware (syncphony.txt, cmdline.txt) sit on a +// read-only vfat mount and need the remount dance of WithMountWritable; +// - /data being mounted is what makes the box's storage (and with it the +// pairing) healthy — see daemon.StorageHealthy. +// +// SB_MOUNTS points the parser at a fake table for tests, in the spirit of +// the other SB_* hooks. +const defaultMountsPath = "/proc/mounts" + +// mountsPath resolves the mounts table to read. +func mountsPath() string { + if p := os.Getenv("SB_MOUNTS"); p != "" { + return p + } + return defaultMountsPath +} + +// Mount is one entry of the mounts table that matters here. +type Mount struct { + Point string // mount point, octal escapes left as-is (no box path has them) + RO bool // mounted read-only +} + +// ReadMounts parses the mounts table; a missing or unreadable table yields +// nil (the callers then behave as "not on a read-only mount"). +func ReadMounts() []Mount { + data, err := os.ReadFile(mountsPath()) + if err != nil { + return nil + } + var mounts []Mount + for _, line := range strings.Split(string(data), "\n") { + // Format: device mountpoint fstype options ... + fields := strings.Fields(line) + if len(fields) < 4 { + continue + } + ro := false + for _, opt := range strings.Split(fields[3], ",") { + if opt == "ro" { + ro = true + } + } + mounts = append(mounts, Mount{Point: fields[1], RO: ro}) + } + return mounts +} + +// MountOf returns the deepest mount whose point contains path — the +// filesystem the path lives on. ok is false when no mount contains it (a +// plain file on a dev machine, say). +func MountOf(path string) (m Mount, ok bool) { + bestLen := -1 + for _, mount := range ReadMounts() { + if mount.Point == "/" || mount.Point == path || strings.HasPrefix(path, mount.Point+"/") { + if len(mount.Point) > bestLen { + m, ok, bestLen = mount, true, len(mount.Point) + } + } + } + return m, ok +} + +// Mounted reports whether the given mount point carries a filesystem. +func Mounted(point string) bool { + for _, mount := range ReadMounts() { + if mount.Point == point { + return true + } + } + return false +} + +// WithMountWritable runs fn while the filesystem holding path is writable, +// then puts it back: remount rw, run fn, sync, remount ro. The box keeps +// /boot/firmware read-only (ADR 0007) so a power pull can never catch the +// FAT partition half-written; the only writes that ever hit it go through +// this dance. +// +// A path on an already-writable filesystem — or on no mount at all (the +// SB_* test hooks point at plain files) — runs fn directly, so dev +// machines and tests never perform (or fail) a remount. +// +// Fail-safe bias: a fn error is returned even if the ro remount also fails +// (the write may have landed), and a failed remount back to ro is logged +// but never discarded silently — the box keeps running either way. +func WithMountWritable(path string, run RunCommand, log *log.Logger, fn func() error) error { + mount, ok := MountOf(path) + if !ok || !mount.RO { + return fn() + } + if err := run("mount", "-o", "remount,rw", mount.Point); err != nil { + return err + } + err := fn() + syscall.Sync() + if roErr := run("mount", "-o", "remount,ro", mount.Point); roErr != nil { + log.Printf("boxd: cannot remount %s read-only after writing: %v (leaving it writable)", mount.Point, roErr) + } + return err +} diff --git a/boxd/internal/config/mounts_test.go b/boxd/internal/config/mounts_test.go new file mode 100644 index 0000000..a10e9cf --- /dev/null +++ b/boxd/internal/config/mounts_test.go @@ -0,0 +1,143 @@ +package config + +import ( + "os" + "path/filepath" + "strings" + "testing" +) + +// fakeMounts writes a /proc/mounts-shaped file and points SB_MOUNTS at it. +func fakeMounts(t *testing.T, content string) { + t.Helper() + p := filepath.Join(t.TempDir(), "mounts") + if err := os.WriteFile(p, []byte(content), 0o644); err != nil { + t.Fatal(err) + } + t.Setenv("SB_MOUNTS", p) +} + +func TestMountOfPicksTheDeepestReadOnlyMount(t *testing.T) { + fakeMounts(t, `/dev/mmcblk0p2 / ext4 rw,relatime 0 0 +/dev/mmcblk0p1 /boot/firmware vfat ro,relatime 0 0 +tmpfs /run tmpfs rw 0 0 +`) + m, ok := MountOf("/boot/firmware/syncphony.txt") + if !ok || m.Point != "/boot/firmware" || !m.RO { + t.Fatalf("boot file: %+v ok=%v", m, ok) + } + m, ok = MountOf("/etc/hostname") + if !ok || m.Point != "/" || m.RO { + t.Fatalf("etc file: %+v ok=%v", m, ok) + } +} + +func TestMountOfRootContainsEverything(t *testing.T) { + fakeMounts(t, "/dev/sda / ext4 ro 0 0\n") + if _, ok := MountOf("/anywhere/file"); !ok { + t.Fatal("/ should contain every absolute path") + } +} + +func TestMountOfMissingTable(t *testing.T) { + t.Setenv("SB_MOUNTS", filepath.Join(t.TempDir(), "nope")) + if _, ok := MountOf("/boot/firmware/cmdline.txt"); ok { + t.Fatal("no table means no mount found") + } +} + +func TestMounted(t *testing.T) { + fakeMounts(t, "/dev/mmcblk0p3 /data ext4 rw,relatime 0 0\n") + if !Mounted("/data") { + t.Fatal("/data should read as mounted") + } + if Mounted("/boot/firmware") { + t.Fatal("/boot/firmware is not in the table") + } +} + +func TestWithMountWritableOnReadOnlyMount(t *testing.T) { + fakeMounts(t, `/dev/mmcblk0p2 / overlay rw,relatime 0 0 +/dev/mmcblk0p1 /boot/firmware vfat ro,relatime 0 0 +`) + var calls []string + run := func(name string, args ...string) error { + calls = append(calls, name+" "+strings.Join(args, " ")) + return nil + } + wrote := false + err := WithMountWritable("/boot/firmware/syncphony.txt", run, testLogger(), func() error { + wrote = true + return nil + }) + if err != nil { + t.Fatal(err) + } + if !wrote { + t.Fatal("fn never ran") + } + want := []string{ + "mount -o remount,rw /boot/firmware", + "mount -o remount,ro /boot/firmware", + } + if strings.Join(calls, ";") != strings.Join(want, ";") { + t.Fatalf("remount dance = %v, want %v", calls, want) + } +} + +func TestWithMountWritableOnWritableMountSkipsTheDance(t *testing.T) { + fakeMounts(t, "/dev/mmcblk0p2 / overlay rw,relatime 0 0\n") + calls := 0 + run := func(name string, args ...string) error { + calls++ + return nil + } + if err := WithMountWritable("/etc/whatever", run, testLogger(), func() error { return nil }); err != nil { + t.Fatal(err) + } + if calls != 0 { + t.Fatalf("writable path triggered %d mount calls", calls) + } +} + +func TestWithMountWritableWithNoTableSkipsTheDance(t *testing.T) { + // The SB_* hooks point at plain files on a dev machine: nothing is + // mounted, nothing is read-only, so the write just happens. This is + // what keeps the bats apply tests (and a dev `boxd config apply`) + // working without a mount table to dance with. + t.Setenv("SB_MOUNTS", filepath.Join(t.TempDir(), "nope")) + calls := 0 + run := func(name string, args ...string) error { + calls++ + return nil + } + wrote := false + err := WithMountWritable("/tmp/cmdline.txt", run, testLogger(), func() error { + wrote = true + return nil + }) + if err != nil || !wrote { + t.Fatalf("err=%v wrote=%v", err, wrote) + } + if calls != 0 { + t.Fatalf("no-table path triggered %d mount calls", calls) + } +} + +func TestWithMountWritableFailsCleanlyWhenRemountFails(t *testing.T) { + fakeMounts(t, "/dev/mmcblk0p1 /boot/firmware vfat ro 0 0\n") + run := func(name string, args ...string) error { + return os.ErrPermission + } + ran := false + err := WithMountWritable("/boot/firmware/syncphony.txt", run, testLogger(), func() error { + ran = true + return nil + }) + if err == nil { + t.Fatal("a failed rw remount must surface") + } + if ran { + t.Fatal("fn must not run against a still-read-only filesystem") + } +} diff --git a/boxd/internal/daemon/bridge.go b/boxd/internal/daemon/bridge.go index 472ed7e..890cc5c 100644 --- a/boxd/internal/daemon/bridge.go +++ b/boxd/internal/daemon/bridge.go @@ -29,6 +29,7 @@ func (d *Daemon) bridgeMux() *http.ServeMux { mux.HandleFunc("GET /status", d.handleStatus) mux.HandleFunc("GET /setup", d.handleSetupScreen) mux.HandleFunc("GET /start", d.handleStartScreen) + mux.HandleFunc("GET /storage", d.handleStorageScreen) return mux } diff --git a/boxd/internal/daemon/daemon.go b/boxd/internal/daemon/daemon.go index ae8a136..ee7616d 100644 --- a/boxd/internal/daemon/daemon.go +++ b/boxd/internal/daemon/daemon.go @@ -349,6 +349,19 @@ func (d *Daemon) watch() { } } +// storageHealthy reports whether /data — the only partition that survives +// the overlay (MAD-805, ADR 0007) — is mounted. Without it the box boots +// anyway, but the pairing, the applied config and the logs are gone (they +// live on tmpfs for the boot), so the kiosk shows the storage-lost screen +// instead of pretending everything is fine. SB_STORAGE pins the answer in +// dev and tests ("ok"/"missing"), like the other SB_* hooks. +func storageHealthy() bool { + if v := os.Getenv("SB_STORAGE"); v != "" { + return v != "missing" + } + return config.Mounted(config.DefaultDataDir) +} + // boxIPs lists the box's LAN addresses (the offline screen shows them, so // whoever is setting the box up can reach the setup form by IP if mDNS // doesn't work on their phone). diff --git a/boxd/internal/daemon/daemon_test.go b/boxd/internal/daemon/daemon_test.go index 5404cfb..70d5597 100644 --- a/boxd/internal/daemon/daemon_test.go +++ b/boxd/internal/daemon/daemon_test.go @@ -27,6 +27,9 @@ func newTestDaemon(t *testing.T, txtContent string) (*Daemon, *recorder) { rec := &recorder{} t.Setenv("SB_TXT", txt) t.Setenv("SB_ENV", env) + // No dev machine has /data mounted: the storage hook pins "healthy" so + // the screens under test don't trip over the host's mount table. + t.Setenv("SB_STORAGE", "ok") d := New("test-version", log.New(log.Writer(), "", 0), rec.run) return d, rec } @@ -325,16 +328,63 @@ func TestStatusReportsServerAndTarget(t *testing.T) { Server string `json:"server"` Reachable bool `json:"reachable"` Target string `json:"target"` + Storage bool `json:"storage"` } json.Unmarshal(rec.Body.Bytes(), &st) if !st.Configured || st.Server != srv.URL || !st.Reachable { t.Fatalf("got %+v", st) } + if !st.Storage { + t.Fatalf("storage hook is ok but /status reports %+v", st) + } if want := config.TargetURL(srv.URL, "test-version", "Den TV"); st.Target != want { t.Fatalf("target %q, want %q", st.Target, want) } } +// --- storage-lost screen (MAD-805) ------------------------------------------- + +// TestStorageScreenTakesOverStartAndSetup: without /data the box still +// boots, but the kiosk must not pretend — the storage screen wins over +// both entry pages, whatever the config says. +func TestStorageScreenTakesOverStartAndSetup(t *testing.T) { + for _, tc := range []struct { + name, txt, path string + }{ + {"unconfigured setup", "", "/setup"}, + {"unconfigured start", "", "/start"}, + {"configured start", "server_url=https://s.example\n", "/start"}, + {"configured setup", "server_url=https://s.example\n", "/setup"}, + } { + t.Run(tc.name, func(t *testing.T) { + d, _ := newTestDaemon(t, tc.txt) + t.Setenv("SB_STORAGE", "missing") // helper pins ok; lose /data here + rec := get(t, d.bridgeMux(), tc.path) + if rec.Code != http.StatusOK { + t.Fatalf("status %d", rec.Code) + } + if !strings.Contains(rec.Body.String(), "lost its storage") { + t.Fatalf("no storage screen: %s", rec.Body) + } + }) + } +} + +func TestStorageScreenNamesWhatToDo(t *testing.T) { + d, _ := newTestDaemon(t, "server_url=https://s.example\nname=Den TV\n") + t.Setenv("SB_STORAGE", "missing") + rec := get(t, d.bridgeMux(), "/storage") + body := rec.Body.String() + if rec.Code != http.StatusOK { + t.Fatalf("status %d", rec.Code) + } + for _, want := range []string{"Den TV", "SD card", "pairing"} { + if !strings.Contains(body, want) { + t.Errorf("storage screen should mention %q", want) + } + } +} + // --- LAN setup form ---------------------------------------------------------- func TestLANClosedWhenConfigured(t *testing.T) { diff --git a/boxd/internal/daemon/pages.go b/boxd/internal/daemon/pages.go index ee50d17..1e64c2a 100644 --- a/boxd/internal/daemon/pages.go +++ b/boxd/internal/daemon/pages.go @@ -82,6 +82,41 @@ type startScreenData struct { Network string } +type storageScreenData struct { + Name string + IPs []string +} + +// storageScreenHTML is the "the SD card's data partition is gone" screen +// (MAD-805): shown instead of setup or start when /data never mounted. +// There is nothing to retry on the TV — the fix is at the card — so the +// page just names it. Pairing-wise nothing is lost that wasn't already: +// the box re-pairs from scratch on the next boot with healthy storage. +func storageScreenHTML(d storageScreenData) string { + b := &strings.Builder{} + t := template.Must(template.New("storage").Parse(` + + + + +Syncphony Box — storage lost + + + +
+

{{.Name}} lost its storage

+

The card's data partition couldn't be read, so this box forgot its room + and its pairing. It is still running, but nothing survives a reboot.

+

Power it off, check or re-flash the SD card, and power it back + on — it will offer a fresh pairing code.

+ {{if .IPs}}

Reach it at {{range .IPs}}http://{{.}} {{end}}

{{end}} +
+ +`)) + t.Execute(b, d) + return b.String() +} + func startScreenHTML(d startScreenData) string { targetJS, err := json.Marshal(d.Target) if err != nil { diff --git a/boxd/internal/daemon/screens.go b/boxd/internal/daemon/screens.go index 0117327..0f4dcaa 100644 --- a/boxd/internal/daemon/screens.go +++ b/boxd/internal/daemon/screens.go @@ -23,6 +23,7 @@ func (d *Daemon) handleStatus(w http.ResponseWriter, r *http.Request) { "hostname": hostname(), "ips": boxIPs(), "network": networkState(), + "storage": storageHealthy(), }) } @@ -36,6 +37,12 @@ func hostname() string { // handleSetupScreen serves the TV setup page: a big QR code and // http://.local, which the phone opens to set the server URL. func (d *Daemon) handleSetupScreen(w http.ResponseWriter, r *http.Request) { + if !storageHealthy() { + // Setup makes no sense without /data: the pairing would die with + // the overlay at the next power cut anyway. Say so instead. + d.handleStorageScreen(w, r) + return + } if d.configured() { // The kiosk only lands here before the first navigate succeeds; if // the server URL is set, the start page is the right entry. @@ -60,6 +67,10 @@ func (d *Daemon) handleSetupScreen(w http.ResponseWriter, r *http.Request) { // shows Chromium's error page — it shows the offline screen and returns to // /tv on its own once /healthz answers. func (d *Daemon) handleStartScreen(w http.ResponseWriter, r *http.Request) { + if !storageHealthy() { + d.handleStorageScreen(w, r) + return + } if !d.configured() { http.Redirect(w, r, "/setup", http.StatusFound) return @@ -73,6 +84,17 @@ func (d *Daemon) handleStartScreen(w http.ResponseWriter, r *http.Request) { writePage(w, startScreenHTML(st)) } +// handleStorageScreen says the quiet part out loud when /data failed to +// mount: the box boots regardless (the overlay keeps it alive), but the +// pairing and everything else that used to live on /data is gone — this +// boot runs on throwaway tmpfs. The screen names it and says what to do. +func (d *Daemon) handleStorageScreen(w http.ResponseWriter, r *http.Request) { + writePage(w, storageScreenHTML(storageScreenData{ + Name: d.displayName(), + IPs: boxIPs(), + })) +} + func qrPNG(content string) (string, error) { png, err := qrcode.Encode(content, qrcode.Medium, 560) if err != nil { diff --git a/boxd/internal/daemon/setup.go b/boxd/internal/daemon/setup.go index 0e5da06..5dd59f2 100644 --- a/boxd/internal/daemon/setup.go +++ b/boxd/internal/daemon/setup.go @@ -65,7 +65,12 @@ func (d *Daemon) handleSetupSave(w http.ResponseWriter, r *http.Request) { if name != "" { pairs = append(pairs, config.KV{Key: "name", Value: name}) } - if err := config.SetKeys(d.paths.Txt, pairs...); err != nil { + // syncphony.txt sits on the read-only boot partition (ADR 0007): the + // save goes through the remount dance (rw, write, sync, ro). + err = config.WithMountWritable(d.paths.Txt, d.run, d.log, func() error { + return config.SetKeys(d.paths.Txt, pairs...) + }) + if err != nil { d.renderFormError(w, "Couldn't save the setting on the box's boot partition. Try again, or edit syncphony.txt on the SD card.") d.log.Printf("boxd: setup: save failed: %v", err) return diff --git a/build.sh b/build.sh index 93e469f..0bad7ae 100755 --- a/build.sh +++ b/build.sh @@ -6,6 +6,8 @@ # to syncphony-box--arm64.img.xz (+ .sha256) # - neutralizes stage2's EXPORT_IMAGE (pi-gen's documented SKIP_IMAGES # mechanism) so only stage-syncphony exports an image +# - adds the third (/data) partition after export (scripts/ +# add-data-partition.sh; pi-gen's export stage can only lay out two) # - bind-mounts stage-syncphony into the build container at /stage-syncphony # so the ../stage-syncphony entry of STAGE_LIST resolves (build-docker.sh # bakes only the pi-gen submodule into the container image) @@ -60,6 +62,12 @@ if [ ! -f "${IMAGE}" ]; then exit 1 fi +# pi-gen's export stage ships a two-partition image; the box's /data +# partition is appended now (needs sudo for losetup/mount/mke2fs, like +# the smoke test). See ADR 0007. +echo "Adding the /data partition..." +./scripts/add-data-partition.sh "${IMAGE}" + sha256sum "${IMAGE}" > "${IMAGE}.sha256" echo diff --git a/docs/adr/0006-pairing-persistence.md b/docs/adr/0006-pairing-persistence.md index f56796c..e721400 100644 --- a/docs/adr/0006-pairing-persistence.md +++ b/docs/adr/0006-pairing-persistence.md @@ -31,14 +31,14 @@ Chromium crashes. Two facts make that non-trivial: ### One place defines the profile path -`config.DefaultProfileDir` (`/var/lib/syncphony-box/chromium`) is the only -definition; the `SB_PROFILE` env hook overrides it for dev and tests. -`boxd config apply` writes it into `config.env` as `PROFILE_DIR=`, the -kiosk wrapper reads it from there (asking `boxd profile path` when -`config.env` is missing, with a last-resort literal the smoke test pins to -boxd's answer), and `sudo syncphony-box reset` uses the same value. When -stage 3 moves the profile to the `/data` partition (MAD-805), changing the -constant is the whole change. +`config.DefaultProfileDir` is the only definition; the `SB_PROFILE` env +hook overrides it for dev and tests. `boxd config apply` writes it into +`config.env` as `PROFILE_DIR=`, the kiosk wrapper reads it from there +(asking `boxd profile path` when `config.env` is missing, with a +last-resort literal the smoke test pins to boxd's answer), and `sudo +syncphony-box reset` uses the same value. MAD-805 (stage 3, ADR 0007) +moved the profile to the `/data` partition — `/data/chromium` — by +changing exactly that constant. ### The bridge `paired` event triggers a sync burst @@ -100,9 +100,12 @@ no-op when the profile doesn't exist. - `sync(2)` blocks for the length of the flush; on an idle Pi that is milliseconds, and it runs on a background goroutine so the bridge never waits for it. -- Hardware verification stays on the on-Pi checklist (README): pair → - reboot → back in the room playing without a touch; 20 power pulls at - random points after pairing; no restore prompt after any of them. +- Hardware verification stays on the on-Pi checklist (README) and the + power-pull gauntlet in docs/testing.md: pair → reboot → back in the + room playing without a touch; pulls at random points after pairing; no + restore prompt after any of them. With MAD-805 the profile sits on + /data behind a journaled ext4, so the bursts flush to the one + partition that persists. ## Considered and rejected diff --git a/docs/adr/0007-readonly-root-and-data.md b/docs/adr/0007-readonly-root-and-data.md new file mode 100644 index 0000000..e25e6b6 --- /dev/null +++ b/docs/adr/0007-readonly-root-and-data.md @@ -0,0 +1,145 @@ +# ADR 0007: read-only root and the /data partition + +- Status: Accepted +- Date: 2026-10-14 +- Deciders: Phase 11 plan (MAD-797), read-only-root issue (MAD-805) + +## Context + +The box sits in a living room: people unplug it, the power blips, nobody +runs `shutdown`. ADR 0006 made the *pairing* durable; this ADR is about +the whole filesystem. A write-mounted ext4 on an SD card will eventually +be caught mid-write by a power pull, and a corrupted root means a box +that does not come back — the one failure mode the box cannot afford. + +## Decision + +### Three partitions; root is read-only behind an overlay + +The image ships three partitions: + +1. **boot** (FAT, as before) — kernel, firmware, `syncphony.txt`; +2. **root** (ext4) — the OS, mounted read-only at runtime; +3. **data** (ext4, label `syncphony-data`, 512 MiB in the image) — + everything that must persist. + +pi-gen's export stage can only lay out two partitions, so +`scripts/add-data-partition.sh` appends the third after export: it grows +the image file, `parted mkpart`s the partition, `mke2fs -d`s the +persistent layout into it (including the kiosk-owned Chromium profile +dir), and resolves the `DATADEV` placeholder in fstab to +`PARTUUID=-03` — the same disk-id spelling pi-gen's own +04-set-partuuid uses. + +The overlay is exactly the mechanism `raspi-config`'s "Overlay FS" option +uses, enabled at build time instead of at runtime: an initramfs script at +`/etc/initramfs-tools/scripts/overlay` redefines `local_mount_root()` so +the real root is mounted read-only and overlaid with a tmpfs upper layer, +and the `boot=overlay` kernel command-line token activates it (the +initramfs `/init` sources `/scripts/$BOOT`). The command line is also +where the stock `resize` token comes *out* — see below. Our copy of the +script keeps the real root visible read-only at `/media/root-ro` for +maintenance, and mounts `/data` itself (after an `e2fsck -p`), so every +systemd unit — journald, cloud-init, the kiosk — sees the persistent +partition from the first moment. + +`/boot/firmware` is mounted read-only too (`defaults,ro` in fstab). The +only runtime writes to it are boxd's own: `syncphony.txt` (the setup +form) and `cmdline.txt` (a changed `resolution=`/`rotate=`). Both go +through one remount dance — remount rw, write, sync, remount ro, and fail +safe either way — in `config.WithMountWritable`. + +### /data holds everything that persists; one document lists it + +The Chromium profile, the applied `config.env`, boxd's state, persistent +journald logs (capped at 50 MB), the machine-id, cloud-init's state, the +account files (the Imager user), SSH host keys, and Wi-Fi connections. +Most are symlinks into `/data`; the account files, SSH keys and +connections can't be (rewritten atomically, read too early), so +`data-persist` restores them from `/data` before cloud-init/sshd/ +NetworkManager start and saves them back after the system settles plus +every 15 minutes. The complete, maintained list is +[docs/persistent-paths.md](../persistent-paths.md). + +Two interactions the overlay forces, and how they're handled: + +- **cloud-init.** With the overlay, `/etc/passwd` changes would vanish at + the first reboot; with `/var/lib/cloud` persisted, cloud-init runs its + first-boot modules exactly once and the identity restore/save pair keeps + the resulting user. Without the pair, the box would lose SSH/console + login after its first power cycle. +- **machine-id.** PID 1 binds it before any service runs, so a + userspace-only restore would be too late: the initramfs overlay script + seeds `/etc/machine-id` from `/data/system/machine-id`, and + `data-persist save` writes the bound id back — the first boot's id + becomes the box's permanent one. + +### First boot grows /data, not root + +The stock Trixie mechanism (`resize_early` in the initramfs + +`rpi-resize.service`) grows partition 2 — the root — which we never want +to grow. The `resize` cmdline token is removed at build time (leaving +both stock scripts inert; a package update reinstalling them changes +nothing), `rpi-resize.service` is disabled, and our own +`local-premount/grow-data` script — the same script, retargeted — grows +the **last** partition to fill the card. `syncphony-data-grow.service` +then runs the online `resize2fs` once (a `/data/system/.grown` marker +skips it afterwards; a reformatted `/data` loses the marker and grows +again). + +### A broken /data never hangs the boot + +`e2fsck -p` in the initramfs, `nofail` in fstab, and every service that +touches `/data` conditioned on it being mounted. If the partition can't +be read, the box boots with everything persistent on throwaway tmpfs: +unpaired, and with boxd's storage screen on the TV saying exactly that +instead of pretending. + +### Maintenance writes the real root + +The overlay makes on-box package updates an illusion that evaporates at +reboot, so the apt/unattended-upgrades timers are disabled. The supported +path is `sudo syncphony-box chroot [cmd]` (the overlayroot-chroot role): +it mounts the real root read-write at `/media/root-rw`, chroots with +`/dev` `/proc` `/sys` bound in, and changes take effect after a reboot. +The stage-4 updater builds on this. + +## Consequences + +- The box can be unplugged at any point — mid-boot, mid-playback, seconds + after pairing — and comes back paired; the worst case is a replayed + journal gap. The 50-pull test procedure is in + [docs/testing.md](../testing.md). +- Anything that writes and isn't on the list in + docs/persistent-paths.md is silently discarded at reboot. That is the + feature; adding a persistent path is a deliberate act. +- The Chromium profile lives on a journaled ext4 mounted rw — it is the + one thing still exposed to power pulls, mitigated by ADR 0006's sync + bursts and by ext4 journaling itself. +- Root filesystem size is fixed at build time (plus pi-gen's margin); + OS growth beyond that margin means rebuilding the image, which is the + update model anyway (stage 4). +- Dev machines and tests never perform a remount dance: the SB_* hooks + point at plain files, and `WithMountWritable` only dances when the + target actually sits on a read-only mount. + +## Considered and rejected + +- **The `overlayroot` package / Ubuntu's tooling.** Raspberry Pi OS's own + overlay mechanism (what raspi-config enables) is simpler, already in + the initramfs, and needs no extra package; `syncphony-box chroot` + covers the maintenance half that overlayroot-chroot would. +- **Keeping the stock `resize` token and shipping an fstab fixup.** The + stock `set_partuuid` initramfs script rewrites the disk id and sed's + fstab *through the overlay* on first boot — the edit lands in tmpfs and + the box breaks on its second boot. Removing the token and growing a + fixed root-free partition sidesteps all of it. +- **Bind-mounting /etc/passwd & friends from /data.** Atomic-replace + writers (passwd, shadow, useradd) rename over the bind mount and fail + with EBUSY; the restore/save pair keeps those tools working. +- **Read-write /boot/firmware.** The FAT has no journal at all; a power + pull mid-write there is the nastiest corruption of the three + partitions. Read-only with a remount dance for the two files boxd owns. +- **A squashfs root instead of an overlay.** More work to build and to + update for the same guarantee; the overlay is the mechanism the OS + already ships and documents. diff --git a/docs/persistent-paths.md b/docs/persistent-paths.md new file mode 100644 index 0000000..344e8fd --- /dev/null +++ b/docs/persistent-paths.md @@ -0,0 +1,38 @@ +# Persistent paths on the box + +The root filesystem is read-only behind an overlay (ADR 0007): everything +written under `/` at runtime lands in a tmpfs and vanishes at power-off. +Anything that must survive lives on `/data`, the third partition (label +`syncphony-data`). This page is the complete list — when a path is added +to or moved off `/data`, this list changes in the same commit. + +| path | lives at | put there by | what it is | +|---|---|---|---| +| Chromium profile (pairing cookie, speaker flag) | `/data/chromium` | kiosk user; path pinned by boxd's `config.DefaultProfileDir` | the box's whole identity — the room pairing | +| applied config | `/data/syncphony-box/config.env` | `boxd config apply` / the setup form | parsed `syncphony.txt`; `/etc/syncphony-box/config.env` is a symlink to it | +| boxd's own state | `/data/boxd` | boxd (as needed) | reserved for daemon state (e.g. learned CEC addresses) | +| journald logs | `/data/journal` | journald (`/var/log/journal` → symlink) | persistent logs, capped at 50 MB (`SystemMaxUse=50M`) | +| machine identity | `/data/system/machine-id` | `data-persist save` (the bound id) | seeded into `/etc/machine-id` by the initramfs overlay script | +| cloud-init state | `/data/system/cloud` | cloud-init (`/var/lib/cloud` → symlink) | without it cloud-init would re-run its first boot every boot | +| account files | `/data/system/identity/{passwd,shadow,group,gshadow,subuid,subgid}` | `data-persist save` → restored by `data-persist restore` | the Imager-created user and password; without this they would vanish with the overlay | +| SSH host keys | `/data/system/ssh/` | `ssh-keygen -A` at restore, saved back by `save` | `/etc/ssh/ssh_host_*` are symlinks here; keeps `known_hosts` stable | +| network connections | `/data/system/network/` | `data-persist save` → restored by `restore` | NetworkManager `system-connections` (Wi-Fi credentials) | + +Not on `/data`, still persistent: + +| path | where | what | +|---|---|---| +| `syncphony.txt` | the FAT boot partition (`/boot/firmware`) | the box's config, editable from any computer; written only through the remount dance (rw → write → sync → ro) | + +Everything else is deliberately throwaway: apt state, Chromium cache, +`$HOME`, `systemd`'s runtime state. The overlay discarding them is the +feature — the SD card can be yanked at any moment without corrupting +anything that matters. + +## If /data can't mount + +The initramfs fsck's `/data` (`e2fsck -p`) and mounts it only when it's +clean enough. If it isn't, the box boots anyway: the pairing is lost, the +kiosk shows the storage-lost screen, and everything above runs on tmpfs +for that boot. The fix is at the card (check, re-flash, or replace it) — +nothing on the TV is recoverable. diff --git a/docs/testing.md b/docs/testing.md new file mode 100644 index 0000000..616c6b1 --- /dev/null +++ b/docs/testing.md @@ -0,0 +1,81 @@ +# Testing the box on hardware + +CI can't boot a Pi. The unit tests (bats + Go, `make test`) and the image +smoke test (`scripts/smoke-test.sh`, runs in CI on every build) pin the +image's structure; the checks below are for real hardware. The README's +on-Pi checklist covers the everyday list — this page is the destructive +one. + +## The power-pull gauntlet (50 pulls) + +**Done-when for the read-only root (MAD-805):** 50 random power pulls — +some during boot, some during playback, some seconds after pairing — +leave a box that boots, is still paired, and plays. + +### Setup + +- A box flashed with the current image, paired to a Syncphony server, + playing. +- Any smart plug that can be power-cycled on a schedule (a cheap Wi-Fi + plug with a repeating timer is enough; this is the "smart plug on a + timer" the issue calls for). Suggested cycle: **on 2 minutes, off 30 + seconds, repeat**. That lands pulls all over the boot/playback + timeline without babysitting. For the pairing-adjacent pulls, cut power + by hand 5–15 s after the pairing code disappears. + +### Procedure + +1. Confirm the layout first: `findmnt /` shows `overlay`, `findmnt /data` + shows the ext4, `findmnt /boot/firmware` shows `ro`. +2. Set the plug cycling. Let it run through **40 cycles** unattended. +3. While it cycles, do **5 pulls by hand during boot** (cut 5–20 s after + applying power) and **5 pulls by hand within 10 s of pairing** (unpair + with `sudo syncphony-box reset` between them; the box re-shows a code). +4. After **every** power restore, check the three things: + - the box boots to the room (no setup screen, no error page); + - it's **still paired** (shows as the room's screen in the app, no new + pairing code); + - playback works (start something from the app; sound comes out). +5. Record the result of each pull (a tick sheet is fine — pass/boot, + pass/paired, pass/plays). + +### Pass criteria + +All 50: boots, paired, plays. If any pull loses the pairing, +`journalctl -u boxd -b` on the *recovered* boot should show the +`paired event; syncing disks` line from the pull that lost it — that's +the finding to report, along with which pull number and what the box was +doing. A box that fails to boot at all is a different, more serious +finding: capture `journalctl -b -1` after a manual recovery if possible. + +### Reading the logs afterwards + +Logs persist across power pulls (journald on `/data`, 50 MB cap), so +`journalctl -b -1` reaches the boot *before* the current one: + +```sh +journalctl -b -1 -u boxd -u syncphony-kiosk # the boot that got cut +journalctl --list-boots | wc -l # one per pull +``` + +## Related one-off checks + +- **Overlay really discards writes:** `sudo touch /usr/bin/nope`, reboot, + `ls /usr/bin/nope` → gone. `findmnt /` says `overlay`. +- **Boot partition stays ro:** `touch /boot/firmware/nope` → + `Read-only file system`. The setup form (QR flow) still saves — boxd + remounts around its write. +- **First-boot grow:** on a fresh flash, `lsblk` shows `/data` grown to + fill the card after the first boot, and + `systemctl status syncphony-data-grow` ran once. +- **A destroyed /data still boots:** (sacrificial card) `dd` garbage over + the third partition from another machine, boot → the TV shows the + storage-lost screen and the box is reachable; fix by re-flashing. +- **Maintenance path:** `sudo syncphony-box chroot`, `apt update` inside, + exit, reboot — the update is on the real root + (`grep -r . /media/root-ro/usr/bin/nope`-style spot check, or a package + version). +- **Imager user survives:** flash with a username/password in the Imager, + boot, reboot, SSH in with it — works (the identity restore/save round + trip). SSH host keys identical across two reboots: + `ssh-keygen -lf /data/system/ssh/ssh_host_ed25519_key.pub`. diff --git a/scripts/add-data-partition.sh b/scripts/add-data-partition.sh new file mode 100755 index 0000000..ac5c86d --- /dev/null +++ b/scripts/add-data-partition.sh @@ -0,0 +1,117 @@ +#!/usr/bin/env bash +# add-data-partition — the /data post-export step (MAD-805, ADR 0007). +# +# pi-gen's export stage lays out two partitions (boot FAT + root ext4) and +# ships the compressed image; nothing inside it can add a third. So after +# pi-gen finishes, this script reopens the built image and appends the +# data partition: +# +# - a 512 MiB ext4 labelled syncphony-data as partition 3 (the grow-data +# initramfs script expands it to fill the card on first boot, and +# syncphony-data-grow.service grows the filesystem to match) +# - the /data directory layout pre-created, with the Chromium profile +# dir owned by the kiosk user (docs/persistent-paths.md) +# - the fstab DATADEV placeholder (written by stage 06) replaced with +# the new partition's PARTUUID — same disk-id spelling pi-gen's own +# 04-set-partuuid uses for partitions 1 and 2 +# +# Needs root for losetup/mount/mke2fs — via sudo, exactly like +# scripts/smoke-test.sh. Runs on the host after ./pi-gen/build-docker.sh; +# ./build.sh calls it. +# +# SB_DATA_PART_SIZE overrides the initial partition size (bytes) for +# tests. + +set -euo pipefail + +if [ "$#" -ne 1 ]; then + echo "usage: $0 .img.xz" >&2 + exit 2 +fi + +ALIGN=$((8 * 1024 * 1024)) # pi-gen aligns all partition starts to 8 MiB +DATA_SIZE="${SB_DATA_PART_SIZE:-$((512 * 1024 * 1024))}" + +IMG_XZ="$(cd "$(dirname "$1")" && pwd)/$(basename "$1")" +WORK="$(mktemp -d /tmp/syncphony-box-data.XXXXXX)" +LOOP="" + +cleanup() { + sudo umount "${WORK}/root" 2>/dev/null || true + if [ -n "${LOOP}" ]; then + sudo losetup -d "${LOOP}" 2>/dev/null || true + fi + rm -rf "${WORK}" +} +trap cleanup EXIT + +echo "add-data-partition: decompressing ${IMG_XZ} ..." +xz -dk -T0 -c "${IMG_XZ}" > "${WORK}/image.img" +IMG="${WORK}/image.img" + +# The MBR disk id is what names partitions in fstab as PARTUUID=-NN. +IMGID="$(dd if="${IMG}" skip=440 bs=1 count=4 2>/dev/null | xxd -e | cut -f 2 -d ' ')" +if [ -z "${IMGID}" ]; then + echo "add-data-partition: cannot read the disk id" >&2 + exit 1 +fi + +# Append partition 3 directly after the last existing one, aligned. +LAST_END="$(parted -m -s "${IMG}" unit B print | awk -F: '$1 ~ /^[0-9]+$/ { e = $3 } END { sub(/B$/, "", e); print e }')" +if [ -z "${LAST_END}" ]; then + echo "add-data-partition: cannot read the partition table" >&2 + exit 1 +fi +DATA_START=$(( (LAST_END + 1 + ALIGN - 1) / ALIGN * ALIGN )) +NEW_SIZE=$(( DATA_START + (DATA_SIZE + ALIGN - 1) / ALIGN * ALIGN )) + +truncate -s "${NEW_SIZE}" "${IMG}" +parted --script "${IMG}" unit B mkpart primary ext4 "${DATA_START}" "$((NEW_SIZE - 1))" + +LOOP="$(sudo losetup --find --show --partscan "${IMG}")" + +# The kiosk user's numeric uid/gid from the image's own account database, +# so the pre-created profile dir lands owned correctly even though the +# user only exists inside the image. +mkdir -p "${WORK}/root" +sudo mount -o ro "${LOOP}p2" "${WORK}/root" +KIOSK_UID="$(sudo awk -F: '$1=="kiosk"{print $3}' "${WORK}/root/etc/passwd")" +KIOSK_GID="$(sudo awk -F: '$1=="kiosk"{print $3}' "${WORK}/root/etc/group")" +sudo umount "${WORK}/root" +if [ -z "${KIOSK_UID}" ] || [ -z "${KIOSK_GID}" ]; then + echo "add-data-partition: no kiosk user in the image" >&2 + exit 1 +fi + +# The persistent layout (docs/persistent-paths.md). data-persist recreates +# anything missing at boot; pre-creating it makes the very first boot +# (journald, ssh-keygen) persistent from the start. +STAGING="${WORK}/data" +mkdir -p \ + "${STAGING}/chromium" \ + "${STAGING}/journal" \ + "${STAGING}/syncphony-box" \ + "${STAGING}/boxd" \ + "${STAGING}/system/cloud" \ + "${STAGING}/system/identity" \ + "${STAGING}/system/ssh" \ + "${STAGING}/system/network" +sudo chown "${KIOSK_UID}:${KIOSK_GID}" "${STAGING}/chromium" + +echo "add-data-partition: making the ${DATA_SIZE}-byte ext4 (label syncphony-data) ..." +sudo mke2fs -q -t ext4 -L syncphony-data -d "${STAGING}" "${LOOP}p3" + +# fstab: the DATADEV placeholder (stage 06 wrote it) becomes the real +# PARTUUID, like pi-gen's 04-set-partuuid does for BOOTDEV/ROOTDEV. +sudo mount -o rw "${LOOP}p2" "${WORK}/root" +sudo sed -i "s/DATADEV/PARTUUID=${IMGID}-03/" "${WORK}/root/etc/fstab" +sudo umount "${WORK}/root" + +sudo losetup -d "${LOOP}" +LOOP="" + +echo "add-data-partition: recompressing ..." +xz --compress -T0 --memlimit-compress=50% -6 -c "${IMG}" > "${WORK}/out.img.xz" +mv "${WORK}/out.img.xz" "${IMG_XZ}" + +echo "add-data-partition: done — ${IMG_XZ} now has three partitions." diff --git a/scripts/smoke-test.sh b/scripts/smoke-test.sh index 52419ee..6d8775f 100755 --- a/scripts/smoke-test.sh +++ b/scripts/smoke-test.sh @@ -3,7 +3,7 @@ # # Usage: scripts/smoke-test.sh deploy/syncphony-box--arm64.img.xz # -# Loop-mounts the image's two partitions and checks that the kiosk packages +# Loop-mounts the image's three partitions and checks that the kiosk packages # are installed, the cloud-init first-boot machinery (which applies Raspberry # Pi Imager settings) is enabled, the first user is locked, the NoCloud # seed files are present on the boot partition, the kiosk session is in @@ -11,10 +11,12 @@ # (syncphony.txt) ships on the boot partition fully commented out, the # audio stack is in place (PipeWire user services for the kiosk user with # linger, the sink-selection units, the WirePlumber tuning), boxd (the -# Go helper daemon) is installed with its units enabled, and the pairing -# persistence pieces agree (one profile path, crash-state prep, reset). -# -# Later issues extend this test as the image grows (boxd, /data, ...). +# Go helper daemon) is installed with its units enabled, the pairing +# persistence pieces agree (one profile path, crash-state prep, reset), and +# the read-only-root layout is complete: three partitions with the third +# labelled syncphony-data and pre-populated, the overlay mechanism wired +# into the initramfs and the kernel command line, every persistent path +# pointing at /data, and the stock root-resize replaced by the data grow. set -euo pipefail @@ -28,7 +30,7 @@ WORK="$(mktemp -d /tmp/syncphony-box-smoke.XXXXXX)" LOOP="" cleanup() { - sudo umount "${WORK}/boot" "${WORK}/root" 2>/dev/null || true + sudo umount "${WORK}/boot" "${WORK}/root" "${WORK}/data" 2>/dev/null || true if [ -n "${LOOP}" ]; then sudo losetup -d "${LOOP}" 2>/dev/null || true fi @@ -52,9 +54,42 @@ echo "Decompressing ${IMAGE_XZ} ..." xz -dk -T0 -c "${IMAGE_XZ}" > "${WORK}/image.img" LOOP="$(sudo losetup --find --show --partscan "${WORK}/image.img")" -mkdir -p "${WORK}/boot" "${WORK}/root" +mkdir -p "${WORK}/boot" "${WORK}/root" "${WORK}/data" sudo mount -o ro "${LOOP}p1" "${WORK}/boot" sudo mount -o ro "${LOOP}p2" "${WORK}/root" +sudo mount -o ro "${LOOP}p3" "${WORK}/data" + +# --- partitions: boot, root, data (MAD-805) ---------------------------------- +part_count="$(parted -m -s "${WORK}/image.img" print | awk -F: '$1 ~ /^[0-9]+$/ { n++ } END { print n }')" +if [ "${part_count}" = "3" ]; then + check "image has 3 partitions" ok +else + check "image has 3 partitions" "found ${part_count}" +fi + +data_label="$(sudo blkid -s LABEL -o value "${LOOP}p3" 2>/dev/null || true)" +if [ "${data_label}" = "syncphony-data" ]; then + check "third partition labelled syncphony-data" ok +else + check "third partition labelled syncphony-data" "label: ${data_label:-none}" +fi + +# The pre-created persistent layout (docs/persistent-paths.md). +for dir in chromium journal syncphony-box boxd system/cloud system/identity system/ssh system/network; do + if sudo test -d "${WORK}/data/${dir}"; then + check "/data/${dir} exists in the image" ok + else + check "/data/${dir} exists in the image" "missing" + fi +done + +kiosk_uid="$(sudo awk -F: '$1=="kiosk"{print $3}' "${WORK}/root/etc/passwd")" +profile_owner="$(sudo stat -c '%u' "${WORK}/data/chromium")" +if [ -n "${kiosk_uid}" ] && [ "${profile_owner}" = "${kiosk_uid}" ]; then + check "/data/chromium owned by the kiosk user" ok +else + check "/data/chromium owned by the kiosk user" "owner ${profile_owner}, kiosk uid ${kiosk_uid}" +fi # --- packages --------------------------------------------------------------- for pkg in cage chromium fonts-noto-color-emoji libgl1-mesa-dri cloud-init avahi-daemon \ @@ -75,7 +110,9 @@ done # --- units ------------------------------------------------------------------ for unit in cloud-config.service cloud-final.service syncphony-kiosk.service syncphony-box-config.service \ - avahi-daemon.service syncphony-audio.service syncphony-audio-monitor.service boxd.service; do + avahi-daemon.service syncphony-audio.service syncphony-audio-monitor.service boxd.service \ + syncphony-data-grow.service syncphony-data-prepare.service syncphony-data-save.service \ + syncphony-data-save.timer; do state="$(systemctl --root="${WORK}/root" is-enabled "${unit}" 2>/dev/null || true)" if [ "${state}" = "enabled" ]; then check "unit ${unit} enabled" ok @@ -160,7 +197,7 @@ if out="$(sudo "${WORK}/root/usr/lib/syncphony-box/boxd" profile path 2>/dev/nul profile_dir="${out}" check "boxd profile path answers" ok else - profile_dir="/var/lib/syncphony-box/chromium" + profile_dir="/data/chromium" check "boxd profile path answers" "skipped (cannot run the arm64 binary here)" fi @@ -306,6 +343,100 @@ else check "config.txt has disable_splash=1" "missing" fi +# --- read-only root (MAD-805) ------------------------------------------------ +# boot=overlay activates the overlay script in the initramfs; the stock +# `resize` token must be gone (it would grow the read-only root). +if [ "${CMDLINE}" != "${CMDLINE/ boot=overlay /}" ]; then + check "cmdline has boot=overlay" ok +else + check "cmdline has boot=overlay" "missing" +fi + +if [ "${CMDLINE}" != "${CMDLINE/ grow-data /}" ]; then + check "cmdline has grow-data" ok +else + check "cmdline has grow-data" "missing" +fi + +case "${CMDLINE}" in + *" resize "*) check "stock resize token removed from cmdline" "still present" ;; + *) check "stock resize token removed from cmdline" ok ;; +esac + +if sudo test -f "${WORK}/root/etc/initramfs-tools/scripts/overlay"; then + check "initramfs overlay script installed" ok +else + check "initramfs overlay script installed" "missing" +fi + +if sudo grep -q 'local_mount_root' "${WORK}/root/etc/initramfs-tools/scripts/overlay" && + sudo grep -q 'root-ro' "${WORK}/root/etc/initramfs-tools/scripts/overlay"; then + check "overlay script mounts the overlay and exposes the real root" ok +else + check "overlay script mounts the overlay and exposes the real root" "unexpected contents" +fi + +if sudo grep -qx 'overlay' "${WORK}/root/etc/initramfs-tools/modules"; then + check "overlay module in the initramfs module list" ok +else + check "overlay module in the initramfs module list" "missing" +fi + +if sudo test -x "${WORK}/root/etc/initramfs-tools/scripts/local-premount/grow-data"; then + check "grow-data initramfs script installed executable" ok +else + check "grow-data initramfs script installed executable" "missing" +fi + +resize_state="$(systemctl --root="${WORK}/root" is-enabled rpi-resize.service 2>/dev/null || true)" +if [ "${resize_state}" != "enabled" ]; then + check "stock rpi-resize (root grow) disabled" ok +else + check "stock rpi-resize (root grow) disabled" "still enabled" +fi + +# fstab: the boot partition is mounted read-only, and /data has its own +# nofail entry pointing at partition 3 (the DATADEV placeholder resolved). +if sudo grep -E '/boot/firmware[[:space:]]+vfat[[:space:]]+[^[:space:]]*ro' \ + "${WORK}/root/etc/fstab" >/dev/null; then + check "fstab mounts /boot/firmware read-only" ok +else + check "fstab mounts /boot/firmware read-only" "no ro flag" +fi + +data_fstab="$(sudo grep -E '/data[[:space:]]+ext4' "${WORK}/root/etc/fstab" || true)" +case "${data_fstab}" in + PARTUUID=*-03*nofail*) check "fstab has the /data entry (partition 3, nofail)" ok ;; + *) check "fstab has the /data entry (partition 3, nofail)" "got: ${data_fstab:-nothing}" ;; +esac + +# Persistent paths resolve onto /data (docs/persistent-paths.md). +for link in "var/log/journal:/data/journal" \ + "var/lib/cloud:/data/system/cloud" \ + "etc/syncphony-box/config.env:/data/syncphony-box/config.env" \ + "etc/ssh/ssh_host_ed25519_key:/data/system/ssh/ssh_host_ed25519_key"; do + path="${link%%:*}" want="${link#*:}" + got="$(sudo readlink "${WORK}/root/${path}" 2>/dev/null || true)" + if [ "${got}" = "${want}" ]; then + check "${path} points at /data" ok + else + check "${path} points at /data" "is ${got:-not a symlink}" + fi +done + +if sudo test -x "${WORK}/root/usr/lib/syncphony-box/data-persist"; then + check "data-persist installed executable" ok +else + check "data-persist installed executable" "missing" +fi + +if sudo grep -q 'SystemMaxUse=50M' \ + "${WORK}/root/etc/systemd/journald.conf.d/50-syncphony-journald.conf"; then + check "journald capped at 50M" ok +else + check "journald capped at 50M" "drop-in missing or no cap" +fi + # --- first user is locked (no password until Imager settings apply) --------- if sudo grep -q '^syncphony:!' "${WORK}/root/etc/shadow"; then check "first user syncphony has locked password" ok diff --git a/stage-syncphony/01-kiosk/files/kiosk b/stage-syncphony/01-kiosk/files/kiosk index 21e31da..2f95d62 100644 --- a/stage-syncphony/01-kiosk/files/kiosk +++ b/stage-syncphony/01-kiosk/files/kiosk @@ -25,15 +25,16 @@ if [ -r "${CONFIG_FILE}" ]; then . "${CONFIG_FILE}" fi -# The Chromium profile (pairing cookie + speaker flag) lives in one place, -# defined by boxd: config.env carries it, and boxd answers it directly when -# config.env is missing. The literal below is only the last resort when -# neither is available; the smoke test asserts it matches boxd's answer. +# The Chromium profile (pairing cookie + speaker flag) lives on the /data +# partition (MAD-805) — one path, defined by boxd: config.env carries it, +# and boxd answers it directly when config.env is missing. The literal +# below is only the last resort when neither is available; the smoke test +# asserts it matches boxd's answer. DATA_DIR="${PROFILE_DIR:-}" if [ -z "${DATA_DIR}" ]; then DATA_DIR="$("${BOXD_BIN}" profile path 2>/dev/null || true)" fi -[ -n "${DATA_DIR}" ] || DATA_DIR=/var/lib/syncphony-box/chromium +[ -n "${DATA_DIR}" ] || DATA_DIR=/data/chromium URL='' if curl -fsS -m 2 "${BOXD_URL}/healthz" >/dev/null 2>&1; then diff --git a/stage-syncphony/01-kiosk/files/syncphony-box b/stage-syncphony/01-kiosk/files/syncphony-box index 69d947b..3918654 100644 --- a/stage-syncphony/01-kiosk/files/syncphony-box +++ b/stage-syncphony/01-kiosk/files/syncphony-box @@ -2,19 +2,23 @@ # syncphony-box — the admin command for a running box (run it with sudo, # from the console or SSH). # -# sudo syncphony-box reset wipe the Chromium profile: the box forgets -# its pairing and shows a new code on the TV. -# Use it when the box moves to another room -# or another server. -# -# The Chromium profile path comes from config.env (boxd's output), so it is -# defined in exactly one place. +# sudo syncphony-box reset wipe the Chromium profile: the box forgets +# its pairing and shows a new code on the TV. +# Use it when the box moves to another room +# or another server. +# sudo syncphony-box chroot [CMD...] work on the REAL root filesystem. +# The box's / is an overlay (writes land in +# tmpfs and vanish at reboot); chroot mounts +# the real root read-write and chroots you +# into it (a shell with no CMD), so package +# updates and config edits stick. Reboot +# afterwards. See docs/adr/0007. set -eu CONFIG_FILE=/etc/syncphony-box/config.env BOXD_BIN=/usr/lib/syncphony-box/boxd -DEFAULT_PROFILE_DIR=/var/lib/syncphony-box/chromium +DEFAULT_PROFILE_DIR=/data/chromium PROFILE_DIR="${DEFAULT_PROFILE_DIR}" if [ -r "${CONFIG_FILE}" ]; then @@ -23,12 +27,24 @@ fi # config.env is written on every boot, but never trust an empty value. [ -n "${PROFILE_DIR}" ] || PROFILE_DIR="${DEFAULT_PROFILE_DIR}" -case "${1:-}" in -reset) +need_root() { if [ "$(id -u)" -ne 0 ]; then - echo "syncphony-box: reset needs root (try: sudo syncphony-box reset)" >&2 + echo "syncphony-box: this needs root (try: sudo syncphony-box $1)" >&2 exit 1 fi +} + +usage() { + cat << 'EOF' +usage: + sudo syncphony-box reset wipe the pairing (new pairing code) + sudo syncphony-box chroot [CMD...] shell/command on the real root fs +EOF +} + +case "${1:-}" in +reset) + need_root reset echo "Wiping the Chromium profile at ${PROFILE_DIR}." echo "This box will forget its room and show a new pairing code." systemctl stop syncphony-kiosk.service @@ -39,11 +55,47 @@ reset) systemctl start syncphony-kiosk.service echo "Done. The box is back on the pairing screen." ;; +chroot) + need_root chroot + # The real root: mounted read-only by the initramfs at /media/root-ro. + # Work on a second, writable mount of the same device; the ro one keeps + # serving the running overlay untouched. + if [ "$(findmnt -n -o FSTYPE /)" != "overlay" ]; then + echo "syncphony-box: the overlay is not active, so / is already the real root." >&2 + echo "Run your command directly." >&2 + exit 1 + fi + SRC="$(findmnt -n -o SOURCE /media/root-ro)" || { + echo "syncphony-box: /media/root-ro is not mounted; the overlay script should have left the real root there." >&2 + exit 1 + } + RW=/media/root-rw + mkdir -p "${RW}" + mount -o rw "${SRC}" "${RW}" + cleanup() { + cd / + umount -l "${RW}/dev" "${RW}/proc" "${RW}/sys" 2>/dev/null || true + umount "${RW}" + } + trap cleanup EXIT + mount --rbind /dev "${RW}/dev" + mount --rbind /proc "${RW}/proc" + mount --rbind /sys "${RW}/sys" + cp /etc/resolv.conf "${RW}/etc/resolv.conf" 2>/dev/null || true + echo "Real root mounted read-write at ${RW} — changes here survive reboots;" + echo "anything you used to change under / was going to tmpfs. Reboot when done." + if [ "$#" -gt 1 ]; then + shift + chroot "${RW}" "$@" + else + chroot "${RW}" /bin/bash + fi + ;; help|-h|--help) - echo "usage: sudo syncphony-box reset wipe the pairing (new pairing code)" + usage ;; *) - echo "usage: sudo syncphony-box reset wipe the pairing (new pairing code)" >&2 + usage >&2 exit 2 ;; esac diff --git a/stage-syncphony/06-readonly-root/00-run.sh b/stage-syncphony/06-readonly-root/00-run.sh new file mode 100644 index 0000000..ccaa3fa --- /dev/null +++ b/stage-syncphony/06-readonly-root/00-run.sh @@ -0,0 +1,117 @@ +#!/bin/bash -e + +# Read-only root and /data (MAD-805, ADR 0007): make the box safe to pull +# the plug on. +# +# - The overlay: raspi-config's "Overlay FS" mechanism, enabled at build +# time — /etc/initramfs-tools/scripts/overlay + the overlay module in +# the initramfs + boot=overlay on the kernel command line. The real +# root becomes a read-only lower layer; runtime writes land in tmpfs. +# - /data: a third partition (created by scripts/add-data-partition.sh +# after export, label syncphony-data) holds everything that must +# persist — the full list is docs/persistent-paths.md. Here we wire +# the paths that reach it. +# - The first-boot grow targets the last partition (our grow-data +# initramfs script, keyed off its own cmdline token); the stock +# `resize` token and rpi-resize (which would grow the read-only root) +# come out. + +FILES="$(dirname "$0")/files" + +# --- the overlay ------------------------------------------------------------- +IIR="${ROOTFS_DIR}/etc/initramfs-tools" +install -v -d -m 0755 "${IIR}/scripts" "${IIR}/scripts/local-premount" +install -v -m 0644 "${FILES}/overlay" "${IIR}/scripts/overlay" +install -v -m 0755 "${FILES}/grow-data" "${IIR}/scripts/local-premount/grow-data" +# The overlay module must be in the initramfs (the script also has an +# insmod fallback off the real root, but don't rely on it). +touch "${IIR}/modules" +grep -qx 'overlay' "${IIR}/modules" || echo overlay >> "${IIR}/modules" +# The initramfs itself is built at export (05-finalise runs +# update-initramfs -k all -c), after this stage, so both of these land +# inside it. No rename/initramfs config.txt line: auto_initramfs=1 loads +# the per-flavour initrd, and boot=overlay (below) activates the script. + +# --- the kernel command line -------------------------------------------------- +# boot=overlay turns the overlay on. `resize` (the stock first-boot token +# that grows the ROOT partition and rewrites the disk id) comes out: root +# stays at its image size and grow-data takes over for /data. +CMDLINE="${ROOTFS_DIR}/boot/firmware/cmdline.txt" +line="$(tr -d '\r\n' < "${CMDLINE}")" +kept="" +for tok in ${line}; do + case "${tok}" in + resize) ;; # stock first-boot root grow — replaced by grow-data + *) kept="${kept:+${kept} }${tok}" ;; + esac +done +line="${kept}" +case " ${line} " in + *" boot=overlay "*) ;; + *) line="boot=overlay ${line}" ;; +esac +case " ${line} " in + *" grow-data "*) ;; + *) line="${line} grow-data" ;; +esac +printf '%s\n' "${line}" > "${CMDLINE}" + +# --- fstab -------------------------------------------------------------------- +# /boot/firmware read-only: the FAT never sits writable at runtime (boxd +# remounts it rw around its syncphony.txt/cmdline.txt writes only). /data +# with nofail: a card with a destroyed data partition still boots (the +# screen says the storage is lost) instead of hanging in emergency mode. +FSTAB="${ROOTFS_DIR}/etc/fstab" +sed -i '\|/boot/firmware| s/defaults/defaults,ro/' "${FSTAB}" +printf 'DATADEV /data ext4 defaults,nofail,x-systemd.device-timeout=15s 0 2\n' >> "${FSTAB}" + +# --- paths that live on /data now --------------------------------------------- +# Symlinks are the whole trick for most of them: read early, followed +# wherever the filesystem points, graceful when /data is missing. The +# account files, SSH keys and NetworkManager connections can't be symlinks +# (rewritten atomically, read too early) — data-persist copies those. +install -v -d -m 0755 "${ROOTFS_DIR}/data" + +rm -rf "${ROOTFS_DIR}/var/log/journal" +ln -v -sfn /data/journal "${ROOTFS_DIR}/var/log/journal" + +rm -rf "${ROOTFS_DIR}/var/lib/cloud" +ln -v -sfn /data/system/cloud "${ROOTFS_DIR}/var/lib/cloud" + +# config.env: boxd writes the real file on /data; the /etc path stays a +# symlink so the kiosk wrapper and the reset tool keep reading it there. +ln -v -sfn /data/syncphony-box/config.env "${ROOTFS_DIR}/etc/syncphony-box/config.env" + +# SSH host keys: dangling until data-persist restore / ssh-keygen -A +# creates the targets on /data (through these symlinks). +install -v -d -m 0755 "${ROOTFS_DIR}/etc/ssh" +for type in rsa ecdsa ed25519; do + for suffix in "" ".pub"; do + key="ssh_host_${type}_key${suffix}" + rm -f "${ROOTFS_DIR}/etc/ssh/${key}" + ln -v -sfn "/data/system/ssh/${key}" "${ROOTFS_DIR}/etc/ssh/${key}" + done +done + +# --- units and tools ------------------------------------------------------------ +install -v -m 0755 "${FILES}/data-persist" "${ROOTFS_DIR}/usr/lib/syncphony-box/data-persist" +install -v -d -m 0755 "${ROOTFS_DIR}/etc/systemd/journald.conf.d" +install -v -m 0644 "${FILES}/50-syncphony-journald.conf" \ + "${ROOTFS_DIR}/etc/systemd/journald.conf.d/50-syncphony-journald.conf" +for unit in syncphony-data-grow.service syncphony-data-prepare.service \ + syncphony-data-save.service syncphony-data-save.timer; do + install -v -m 0644 "${FILES}/${unit}" "${ROOTFS_DIR}/etc/systemd/system/${unit}" +done + +on_chroot << EOF +# The stock first-boot grow targets the root partition; ours targets the +# last one. Disable the stock unit (the resize token is already out of the +# command line above). +systemctl disable rpi-resize.service +# Package downloads/upgrades onto a tmpfs overlay would vanish at reboot +# while looking applied — the one safe update path is writing the real +# root (sudo syncphony-box chroot; ADR 0007), so the timers stay off. +systemctl disable unattended-upgrades.timer apt-daily.timer apt-daily-upgrade.timer || true +systemctl enable syncphony-data-grow.service syncphony-data-prepare.service \ + syncphony-data-save.service syncphony-data-save.timer +EOF diff --git a/stage-syncphony/06-readonly-root/files/50-syncphony-journald.conf b/stage-syncphony/06-readonly-root/files/50-syncphony-journald.conf new file mode 100644 index 0000000..8527c4a --- /dev/null +++ b/stage-syncphony/06-readonly-root/files/50-syncphony-journald.conf @@ -0,0 +1,8 @@ +# Persistent logs for the read-only-root box (MAD-805, ADR 0007). +# /var/log/journal is a symlink to /data/journal, so with /data mounted +# (the normal case — the initramfs mounts it before systemd starts) +# journald stores here; without /data it falls back to volatile. Capped +# so the logs can never eat the card. +[Journal] +Storage=auto +SystemMaxUse=50M diff --git a/stage-syncphony/06-readonly-root/files/data-persist b/stage-syncphony/06-readonly-root/files/data-persist new file mode 100755 index 0000000..5fd8885 --- /dev/null +++ b/stage-syncphony/06-readonly-root/files/data-persist @@ -0,0 +1,143 @@ +#!/bin/sh +# data-persist — move the box's runtime state across the /data boundary +# (MAD-805, ADR 0007). The root filesystem sits behind an overlay with a +# tmpfs upper layer, so everything written under /etc or /var vanishes at +# power-off; /data is the only partition that persists. +# +# Most persistent things reach /data by a plain symlink (the Chromium +# profile, journald's /var/log/journal, config.env — the complete list is +# docs/persistent-paths.md). This script owns the few that cannot be a +# symlink because something reads them too early or another tool rewrites +# them atomically: the account files, the SSH host keys, the +# NetworkManager connections, and the machine-id (seeded from /data by +# the initramfs overlay script, saved back here). +# +# restore runs early on every boot (syncphony-data-prepare.service, +# before cloud-init/sshd/NetworkManager): put the saved state +# back, create any missing SSH keys, ensure the /data layout. +# save runs after the system has settled and then every 15 minutes +# (syncphony-data-save.service/.timer): snapshot the state back +# to /data, so changes made mid-session survive a power pull +# (worst case: the last 15 minutes of account edits are lost). +# +# Dev/test hooks (same spirit as boxd's SB_* hooks): SB_DATA_DIR moves +# /data, SB_ETC_DIR moves /etc, SB_SKIP_SSH=1 skips ssh-keygen. + +set -eu + +DATA="${SB_DATA_DIR:-/data}" +ETC="${SB_ETC_DIR:-/etc}" +IDENT="${DATA}/system/identity" +SSHDIR="${DATA}/system/ssh" +NETDIR="${DATA}/system/network" + +IDENTITY_FILES="passwd shadow group gshadow subuid subgid" +SSH_KEY_TYPES="rsa ecdsa ed25519" + +# A missing or unmountable /data is a supported state (the box boots, the +# TV says "storage lost"): every verb just exits. SB_DATA_DIR points at a +# plain directory in dev, where the mount check makes no sense. +data_usable() { + if [ -n "${SB_DATA_DIR:-}" ]; then + [ -d "${DATA}" ] + return + fi + grep -q " ${DATA} " /proc/mounts +} + +# ensure_layout creates every directory anything on /data expects. It is +# idempotent, so a fresh (or freshly reformatted) data partition is +# bootstrapped by the first boot. +ensure_layout() { + mkdir -p \ + "${DATA}/chromium" \ + "${DATA}/journal" \ + "${DATA}/syncphony-box" \ + "${DATA}/boxd" \ + "${DATA}/system/cloud" \ + "${IDENT}" "${SSHDIR}" "${NETDIR}" + # The Chromium profile belongs to the kiosk user. Harmless where the + # user doesn't exist (dev machines, test runs). + chown kiosk:kiosk "${DATA}/chromium" 2>/dev/null || true +} + +restore() { + data_usable || exit 0 + ensure_layout + + # The account files: whatever the last save had. On a fresh card this + # is a no-op and cloud-init creates the user from the Imager settings. + for f in ${IDENTITY_FILES}; do + if [ -f "${IDENT}/${f}" ]; then + cp -p "${IDENT}/${f}" "${ETC}/${f}" + fi + done + + # SSH host keys: restore first, then fill any that are missing. The + # /etc/ssh paths are symlinks into /data (see stage 06), so ssh-keygen + # -A writes straight to persistent storage. + for t in ${SSH_KEY_TYPES}; do + for suffix in "" ".pub"; do + key="ssh_host_${t}_key${suffix}" + if [ -f "${SSHDIR}/${key}" ]; then + cp -p "${SSHDIR}/${key}" "${ETC}/ssh/${key}" + fi + done + done + # ssh-keygen -A always operates on the real /etc/ssh, so it only runs + # against the box's own layout (never a dev/test SB_ETC_DIR). + if [ -z "${SB_SKIP_SSH:-}" ] && [ -z "${SB_ETC_DIR:-}" ]; then + ssh-keygen -A >/dev/null + fi + + # Wi-Fi (and any other) connections the Imager settings created. + if [ -d "${NETDIR}" ] && [ -d "${ETC}/NetworkManager" ]; then + mkdir -p "${ETC}/NetworkManager/system-connections" + cp -a "${NETDIR}/." "${ETC}/NetworkManager/system-connections/" 2>/dev/null || true + fi +} + +save() { + data_usable || exit 0 + ensure_layout + + for f in ${IDENTITY_FILES}; do + if [ -f "${ETC}/${f}" ]; then + cp -p "${ETC}/${f}" "${IDENT}/${f}" + fi + done + + mkdir -p "${SSHDIR}" + for t in ${SSH_KEY_TYPES}; do + for suffix in "" ".pub"; do + key="ssh_host_${t}_key${suffix}" + if [ -f "${ETC}/ssh/${key}" ]; then + cp -p "${ETC}/ssh/${key}" "${SSHDIR}/${key}" + fi + done + done + + NM="${ETC}/NetworkManager/system-connections" + if [ -d "${NM}" ] && [ -n "$(ls -A "${NM}" 2>/dev/null)" ]; then + cp -a "${NM}/." "${NETDIR}/" + fi + + # The identity the initramfs seeds from on the next boot. + if [ -f "${ETC}/machine-id" ]; then + cp -p "${ETC}/machine-id" "${DATA}/system/machine-id" + fi + + # Pairing-level durability: this is the same sync discipline the + # bridge's pairing burst uses — push it to the card now, not whenever + # the kernel feels like it. + sync +} + +case "${1:-}" in +restore) restore ;; +save) save ;; +*) + echo "usage: data-persist restore|save" >&2 + exit 2 + ;; +esac diff --git a/stage-syncphony/06-readonly-root/files/grow-data b/stage-syncphony/06-readonly-root/files/grow-data new file mode 100755 index 0000000..61e9c00 --- /dev/null +++ b/stage-syncphony/06-readonly-root/files/grow-data @@ -0,0 +1,99 @@ +#!/bin/sh +# Grow the /data partition to fill the card on first boot (MAD-805, +# ADR 0007). This is the stock Trixie resize mechanism +# (raspberrypi-sys-mods' resize_early) retargeted: the stock script grows +# the root partition; this box keeps root at its image size — it is +# read-only behind an overlay — and grows the LAST partition (the data +# one) instead. The filesystem grow follows at boot +# (syncphony-data-grow.service, an online resize2fs). +# +# Stock resize_early is inert here: the build removes its `resize` token +# from the kernel command line, so this script keys off its own token +# (`grow-data`), which stays in the line. The grow is a no-op once the +# partition reaches the end of the device, so the token never needs +# removing. +# +# Runs in the initramfs at local-premount: the table is rewritten while +# nothing is mounted from the card. parted and lsblk are in the initramfs +# (the stock mechanism needs them too; the firstboot hook copies them in). + +set -e + +case "$1" in + prereqs) + exit 0 + ;; +esac + +if ! grep -q ' grow-data' /proc/cmdline; then + exit 0 +fi + +get_variables () { + set +e # This function does not work as intended with -e + local_device_setup "$ROOT" "root file system" + set -e + ROOT_PART_NAME="$(lsblk -no kname "$DEV")" + ROOT_DEV_NAME="$(lsblk -no pkname "$DEV")" + ROOT_DEV="/dev/$ROOT_DEV_NAME" + ROOT_PART_NUM=$(cat "/sys/block/$ROOT_DEV_NAME/$ROOT_PART_NAME/partition") + + # The partition to grow is the LAST one, whatever its number: find it + # in sysfs (the highest partition number on an MBR card image is also + # the last by position; the dir name is the partition's kernel name). + LAST_PART_NUM=0 + for part_entry in "/sys/block/$ROOT_DEV_NAME"/*/partition; do + num=$(cat "$part_entry") + if [ "$num" -gt "$LAST_PART_NUM" ]; then + LAST_PART_NUM="$num" + LAST_PART_NAME="$(basename "$(dirname "$part_entry")")" + fi + done + LAST_START=$(cat "/sys/block/$ROOT_DEV_NAME/$LAST_PART_NAME/start") + LAST_SIZE=$(cat "/sys/block/$ROOT_DEV_NAME/$LAST_PART_NAME/size") + + ROOT_DEV_SIZE=$(cat "/sys/block/$ROOT_DEV_NAME/size") + TARGET_END="$((ROOT_DEV_SIZE - 1))" + if [ "$TARGET_END" -gt 4294967295 ]; then + TARGET_END=4294967295 + fi +} + +do_grow () { + if ! parted -m "$ROOT_DEV" u s resizepart "$LAST_PART_NUM" "$TARGET_END"; then + return 1 + fi +} + +. /scripts/functions +. /scripts/local + +log_begin_msg "Growing the data partition..." + +get_variables +CURRENT_END="$((LAST_START + LAST_SIZE - 1))" +if [ "$LAST_PART_NUM" -le "$ROOT_PART_NUM" ]; then + # Root IS the last partition (a card written before /data existed, or + # a hand-built image): growing it is exactly what we must not do. + log_warning_msg "Grow skipped: no partition beyond root" +elif [ "$CURRENT_END" -ge "$TARGET_END" ]; then + : # already grown — the steady state after the first boot +else + do_grow +fi + +unset ROOT_PART_NAME +unset ROOT_DEV_NAME +unset ROOT_DEV +unset ROOT_PART_NUM +unset LAST_PART_NUM +unset LAST_PART_NAME +unset LAST_START +unset LAST_SIZE +unset ROOT_DEV_SIZE +unset TARGET_END +unset CURRENT_END + +log_end_msg + +exit 0 diff --git a/stage-syncphony/06-readonly-root/files/overlay b/stage-syncphony/06-readonly-root/files/overlay new file mode 100755 index 0000000..362859b --- /dev/null +++ b/stage-syncphony/06-readonly-root/files/overlay @@ -0,0 +1,90 @@ +# Local filesystem mounting -*- shell-script -*- + +# The read-only overlay root for the Syncphony box (MAD-805, ADR 0007). +# +# This is exactly the mechanism raspi-config's "Overlay FS" option uses, +# enabled here at build time: the initramfs /init sources +# /scripts/ after /scripts/local, and BOOT comes from the +# boot=overlay token on the kernel command line — so redefining +# local_mount_root() here replaces how the root filesystem is mounted. +# The real root becomes the read-only lower layer; all runtime writes go +# to a tmpfs upper and vanish on power-off. A power pull can never catch +# the ext4 half-written. +# +# Syncphony additions over the raspi-config original: +# - the real root stays mounted read-only at /media/root-ro, so +# maintenance can still reach it (sudo syncphony-box chroot) +# - the /data partition is fsck'd and mounted here, before the real +# init starts, so every unit (journald, cloud-init, the kiosk) sees +# it. If it fsck's beyond repair the box boots without it: no +# pairing, no persistence, and boxd shows the storage screen. +# - /etc/machine-id is seeded from /data so the box keeps one identity +# across reboots (PID 1 binds the id before any service could +# restore one). + +. /scripts/local + +local_mount_root() +{ + local_top + local_device_setup "${ROOT}" "root file system" + ROOT="${DEV}" + + # Get the root filesystem type if not set + if [ -z "${ROOTFSTYPE}" ]; then + FSTYPE=$(get_fstype "${ROOT}") + else + FSTYPE=${ROOTFSTYPE} + fi + + local_premount + + # CHANGES TO THE ORIGINAL FUNCTION BEGIN HERE + + modprobe ${FSTYPE} + checkfs ${ROOT} root "${FSTYPE}" + + # Create directories for root and the overlay + mkdir /lower /upper + + # Mount read-only root to /lower + if [ "${FSTYPE}" != "unknown" ]; then + mount -r -t ${FSTYPE} ${ROOTFLAGS} ${ROOT} /lower + else + mount -r ${ROOTFLAGS} ${ROOT} /lower + fi + + modprobe overlay || insmod "/lower/lib/modules/$(uname -r)/kernel/fs/overlayfs/overlay.ko" + + # Mount a tmpfs for the overlay in /upper + mount -t tmpfs tmpfs /upper + mkdir /upper/data /upper/work + + # Mount the final overlay-root in $rootmnt + mount -t overlay \ + -olowerdir=/lower,upperdir=/upper/data,workdir=/upper/work \ + overlay ${rootmnt} + + # Keep the real root reachable, read-only, for maintenance. + mkdir -p ${rootmnt}/media/root-ro + mount --move /lower ${rootmnt}/media/root-ro + + # /data: the one partition that persists. fsck it; mount it only when + # it is clean enough (e2fsck -p: 0 = clean, 1 = corrected; anything + # higher means a human should look at it — boot without it instead). + mkdir -p ${rootmnt}/data + fsck_rc=0 + e2fsck -p LABEL=syncphony-data >/dev/null 2>&1 || fsck_rc=$? + if [ "${fsck_rc}" -le 1 ]; then + if mount -t ext4 LABEL=syncphony-data ${rootmnt}/data; then + # A stable identity across reboots: seed the overlay's + # /etc/machine-id from /data. data-persist save writes the + # bound id back on every boot, so the first boot's generated + # id becomes the box's permanent one. + if [ -f ${rootmnt}/data/system/machine-id ] && + ! grep -q '^uninitialized$' ${rootmnt}/data/system/machine-id; then + cp ${rootmnt}/data/system/machine-id ${rootmnt}/etc/machine-id + fi + fi + fi +} diff --git a/stage-syncphony/06-readonly-root/files/syncphony-data-grow.service b/stage-syncphony/06-readonly-root/files/syncphony-data-grow.service new file mode 100644 index 0000000..d964471 --- /dev/null +++ b/stage-syncphony/06-readonly-root/files/syncphony-data-grow.service @@ -0,0 +1,16 @@ +[Unit] +Description=Grow the /data filesystem to fill the card +# The partition itself was grown in the initramfs (grow-data); this is the +# online resize2fs that makes the filesystem notice. The marker keeps it a +# first-boot affair; a reformatted /data loses the marker and grows again. +ConditionPathIsMountPoint=/data +ConditionPathExists=!/data/system/.grown +After=data.mount + +[Service] +Type=oneshot +ExecStart=/usr/sbin/resize2fs /dev/disk/by-label/syncphony-data +ExecStartPost=/bin/touch /data/system/.grown + +[Install] +WantedBy=multi-user.target diff --git a/stage-syncphony/06-readonly-root/files/syncphony-data-prepare.service b/stage-syncphony/06-readonly-root/files/syncphony-data-prepare.service new file mode 100644 index 0000000..8ab883f --- /dev/null +++ b/stage-syncphony/06-readonly-root/files/syncphony-data-prepare.service @@ -0,0 +1,17 @@ +[Unit] +Description=Restore persistent state from /data (read-only-root box) +Documentation=file:///usr/lib/syncphony-box/syncphony.txt +# /data is mounted by the initramfs overlay script (with the fstab entry +# as belt and braces), so this runs ahead of everything that reads the +# state: cloud-init (which would create a fresh user), sshd (host keys), +# NetworkManager (Wi-Fi) and the kiosk (the Chromium profile). +ConditionPathIsMountPoint=/data +After=data.mount +Before=cloud-init-local.service cloud-init.service ssh.service NetworkManager.service boxd.service syncphony-kiosk.service + +[Service] +Type=oneshot +ExecStart=/usr/lib/syncphony-box/data-persist restore + +[Install] +WantedBy=multi-user.target diff --git a/stage-syncphony/06-readonly-root/files/syncphony-data-save.service b/stage-syncphony/06-readonly-root/files/syncphony-data-save.service new file mode 100644 index 0000000..6d0811b --- /dev/null +++ b/stage-syncphony/06-readonly-root/files/syncphony-data-save.service @@ -0,0 +1,14 @@ +[Unit] +Description=Save persistent state to /data (read-only-root box) +ConditionPathIsMountPoint=/data +# After cloud-init's final stage and sshd, so a first boot saves the user +# and host keys they created; the timer keeps saving so mid-session +# changes survive a power pull. +After=cloud-final.service ssh.service + +[Service] +Type=oneshot +ExecStart=/usr/lib/syncphony-box/data-persist save + +[Install] +WantedBy=multi-user.target diff --git a/stage-syncphony/06-readonly-root/files/syncphony-data-save.timer b/stage-syncphony/06-readonly-root/files/syncphony-data-save.timer new file mode 100644 index 0000000..f743921 --- /dev/null +++ b/stage-syncphony/06-readonly-root/files/syncphony-data-save.timer @@ -0,0 +1,9 @@ +[Unit] +Description=Periodically save persistent state to /data + +[Timer] +OnBootSec=10min +OnUnitActiveSec=15min + +[Install] +WantedBy=timers.target diff --git a/tests/syncphony-box-data.bats b/tests/syncphony-box-data.bats new file mode 100644 index 0000000..0236365 --- /dev/null +++ b/tests/syncphony-box-data.bats @@ -0,0 +1,112 @@ +# Tests for data-persist — the script that moves the account files, SSH +# host keys and NetworkManager connections across the /data boundary on a +# read-only-root box (MAD-805, ADR 0007). It runs straight from the stage +# tree; SB_DATA_DIR and SB_ETC_DIR point it at temp dirs (and imply dev +# mode: no mount check, no ssh-keygen). + +BATS_TEST_DIRNAME_ABS="$(cd "${BATS_TEST_DIRNAME}" && pwd)" +SCRIPT="${BATS_TEST_DIRNAME_ABS}/../stage-syncphony/06-readonly-root/files/data-persist" + +setup() { + WORK="$(mktemp -d)" + DATA="${WORK}/data" + ETC="${WORK}/etc" + mkdir -p "${DATA}" "${ETC}/ssh" + export SB_DATA_DIR="${DATA}" SB_ETC_DIR="${ETC}" +} + +teardown() { + rm -rf "${WORK}" +} + +fake_identity() { + printf 'root:x:0:0:root:/root:/bin/bash\nsyncphony:x:1000:1000::/home/syncphony:/bin/bash\n' > "${ETC}/passwd" + printf 'root:*:20000:0:99999:7:::\nsyncphony:$6$rounds:20000:0:99999:7:::\n' > "${ETC}/shadow" + printf 'root:x:0:\n' > "${ETC}/group" + printf 'root:*::\n' > "${ETC}/gshadow" + printf 'syncphony:100000:65536\n' > "${ETC}/subuid" + printf 'syncphony:100000:65536\n' > "${ETC}/subgid" +} + +@test "restore on a fresh /data creates the persistent layout and is a no-op" { + run "${SCRIPT}" restore + [ "$status" -eq 0 ] + for d in chromium journal syncphony-box boxd system/cloud system/identity system/ssh system/network; do + [ -d "${DATA}/${d}" ] + done +} + +@test "save then restore round-trips the account files" { + fake_identity + run "${SCRIPT}" save + [ "$status" -eq 0 ] + [ -f "${DATA}/system/identity/passwd" ] + cmp "${ETC}/passwd" "${DATA}/system/identity/passwd" + cmp "${ETC}/shadow" "${DATA}/system/identity/shadow" + + # Lose the overlay (fresh boot): /etc back to the image's files. + printf 'root:x:0:0:root:/root:/usr/sbin/nologin\n' > "${ETC}/passwd" + printf 'root:*:20000:0:99999:7:::\n' > "${ETC}/shadow" + run "${SCRIPT}" restore + [ "$status" -eq 0 ] + grep -q '^syncphony:' "${ETC}/passwd" + grep -q '^syncphony:' "${ETC}/shadow" +} + +@test "save then restore round-trips the SSH host keys" { + mkdir -p "${DATA}/system/ssh" + printf 'PRIVATE-KEY-DATA' > "${ETC}/ssh/ssh_host_ed25519_key" + printf 'ssh-ed25519 AAAA box' > "${ETC}/ssh/ssh_host_ed25519_key.pub" + run "${SCRIPT}" save + [ "$status" -eq 0 ] + cmp "${ETC}/ssh/ssh_host_ed25519_key" "${DATA}/system/ssh/ssh_host_ed25519_key" + + rm "${ETC}/ssh/ssh_host_ed25519_key" + run "${SCRIPT}" restore + [ "$status" -eq 0 ] + grep -q 'PRIVATE-KEY-DATA' "${ETC}/ssh/ssh_host_ed25519_key" +} + +@test "save persists the machine-id for the initramfs seed" { + printf '0123456789abcdef0123456789abcdef\n' > "${ETC}/machine-id" + run "${SCRIPT}" save + [ "$status" -eq 0 ] + cmp "${ETC}/machine-id" "${DATA}/system/machine-id" +} + +@test "NetworkManager connections round-trip through /data" { + fake_identity + mkdir -p "${ETC}/NetworkManager/system-connections" + printf '[connection]\nid=home\n' > "${ETC}/NetworkManager/system-connections/home.nmconnection" + run "${SCRIPT}" save + [ "$status" -eq 0 ] + [ -f "${DATA}/system/network/home.nmconnection" ] + + rm -rf "${ETC}/NetworkManager/system-connections" + mkdir -p "${ETC}/NetworkManager/system-connections" + run "${SCRIPT}" restore + [ "$status" -eq 0 ] + grep -q 'id=home' "${ETC}/NetworkManager/system-connections/home.nmconnection" +} + +@test "no NetworkManager dir is fine" { + run "${SCRIPT}" save + [ "$status" -eq 0 ] + run "${SCRIPT}" restore + [ "$status" -eq 0 ] +} + +@test "an unmounted /data (real path) is a quiet no-op" { + unset SB_DATA_DIR + # /data does not exist on the test host; even if it did, it would not + # be a mount point here. Both verbs must exit 0 without touching it. + run "${SCRIPT}" restore + [ "$status" -eq 0 ] + run "${SCRIPT}" save + [ "$status" -eq 0 ] +} + +@test "usage errors out" { + run "${SCRIPT}" + [ "$status" -eq 2 ] +} diff --git a/tests/syncphony-box-profile.bats b/tests/syncphony-box-profile.bats index debc43e..df4f03e 100644 --- a/tests/syncphony-box-profile.bats +++ b/tests/syncphony-box-profile.bats @@ -25,7 +25,7 @@ teardown() { @test "path prints the default profile dir" { run "${SCRIPT}" profile path [ "$status" -eq 0 ] - [ "$output" = "/var/lib/syncphony-box/chromium" ] + [ "$output" = "/data/chromium" ] } @test "path honors SB_PROFILE" {