From 786a16b81ab46c55bf134a34262ad8cc8996f7a1 Mon Sep 17 00:00:00 2001 From: Paco Date: Thu, 8 Oct 2026 11:53:01 -0700 Subject: [PATCH 1/8] docs: RFC for slotted system extensions Adds a 'slotted' scope for sysexts/confexts, stored per A/B volume in a configurable store (default /var/trident/sysexts) and activated via an /etc/extensions symlink written during servicing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs/Development/RFCs/0838-slotted-sysexts.md | 205 ++++++++++++++++++ 1 file changed, 205 insertions(+) create mode 100644 docs/Development/RFCs/0838-slotted-sysexts.md diff --git a/docs/Development/RFCs/0838-slotted-sysexts.md b/docs/Development/RFCs/0838-slotted-sysexts.md new file mode 100644 index 0000000000..bddbd860b7 --- /dev/null +++ b/docs/Development/RFCs/0838-slotted-sysexts.md @@ -0,0 +1,205 @@ +# 0838 Slotted system extensions + +- Date: 2026-10-08 +- RFC PR: [microsoft/trident#838](https://github.com/microsoft/trident/pull/838) +- Issue: TBD + +## Summary + +Add a `slotted` scope for sysexts/confexts. Trident stores them per A/B volume +under a configurable store (default `/var/trident/sysexts/{a,b}/`) and creates +the activating `/etc/extensions/.raw` symlink during servicing. + +## Motivation + +`systemd-sysext(8)` merges an extension only if its `extension-release` matches +the running OS: `ID=` must match unless `_any`, then `SYSEXT_LEVEL=` if defined, +else `VERSION_ID=`. + +All three directories Trident accepts today (`/etc/extensions`, +`/var/lib/extensions`, `/.extra/sysext`) are outside the partitions an A/B +update replaces. So an OS-pinned extension silently stops merging after the +first update: no error, no failed unit, the content is just gone. + +Azure Container Linux hits this. Its Azure platform extension (guest agent, +Hyper-V daemons, chrony, `oras`) is `VERSION_ID`-pinned on a non-swapped +partition. Losing `oras` also breaks the distro's extension download path. + +The shape that works already exists in the same image: the container runtime +ships as an extension inside `/usr`, so extension and `os-release` swap +together. This proposal gives that property to anyone who cannot re-seal +dm-verity and re-sign the UKI. + +Goals: correct extension version per booted slot; rollback restores the previous +set; no new partitions, no `/usr` change, no re-signing; existing configs +unaffected. + +Non-goals: changing systemd matching semantics; ESP-based extensions. + +## Public API Design + +### Today + +```rust +pub struct Extension { + pub url: Url, + pub sha384: Sha384Hash, + pub path: Option, // absolute, allow-listed dir, *.raw +} +``` + +`path` conflates *where the file is stored* with *what scope it is active in*. +Slotting separates them: storage becomes Trident-owned and derived from the +target volume. + +### Proposed + +```rust +pub struct Extension { + pub url: Url, + pub sha384: Sha384Hash, + + /// Absolute path in the target OS. Only valid when `scope` is `shared`. + pub path: Option, + + #[serde(default)] + pub scope: ExtensionScope, +} + +pub enum ExtensionScope { + /// Stored at `path`, active on both volumes. For `ID=_any` extensions. + #[default] + Shared, + /// Stored per-volume in the extension store, active only on the volume it + /// was installed to. For OS-version-pinned extensions. + Slotted, +} +``` + +Plus one optional field on `os`: + +```rust +/// Root of the per-volume extension store. Default `/var/trident/sysexts`. +/// Must be on storage not replaced by an A/B update, and must not be a +/// systemd-sysext search path. +pub extension_store: Option, +``` + +Configurable because slotting doubles storage, and large extensions (GPU +drivers, ML runtimes) may need a dedicated volume. Confexts use a sibling +`confexts/` under the same root. + +```yaml +os: + extensionStore: /mnt/bigdisk/trident/sysexts # optional + sysexts: + - url: https://example.com/tooling.raw # unchanged, still default + sha384: + - url: https://example.com/platform.raw + sha384: + scope: slotted +``` + +### Validation + +| Condition | Result | +|---|---| +| `scope` absent | `shared`. Identical to today. | +| `scope: shared` + `path` | Existing allow-list check. | +| `scope: slotted` + `path` | Reject. Path is Trident-owned; accepting it would silently ignore it. | +| `scope: slotted`, no `path` | Filename from `url` basename, as today's default. | +| `scope: slotted`, no A/B volumes | Reject at validation, not at provision. | +| `extensionStore` set, no `slotted` entries | Accept, unused. | +| `extensionStore` inside a sysext search path | Reject: both copies would be discovered. | + +### Backward compatibility + +Both fields are `#[serde(default)]`. Existing configs parse and behave +identically. Schema gains two optional properties. + +### Alternatives rejected + +- **Sentinel `path`** (`/var/trident/sysexts//x.raw`): leaks internal + layout, invites a literal `a`/`b`. +- **Separate `os.slottedSysexts` list**: duplicates the type and its + duplicate-hash/path validation. +- **Infer from `extension-release`**: metadata is inside the image, unavailable + at static validation, and removes the operator's choice to slot an `_any` + extension deliberately. + +## Implementation + +``` +/var/trident/sysexts/{a,b}/platform.raw +/var/trident/confexts/{a,b}/ +``` + +Both directories persist across the update, which is what makes rollback a +symlink flip rather than a re-download. + +The extensions subsystem already runs on `CleanInstall` and `AbUpdate` via +`provision()`, documented as "migrate state from A-partition to B-partition (or +vice versa)". + +1. **prepare** — download, verify SHA-384. Unchanged. +2. **provision** — resolve target volume via + `EngineContext::get_ab_update_volume()` (already used for per-slot verity + addons); place into `//`. +3. **activate** — write `/etc/extensions/.raw` → + `//.raw`. `/etc` writes are already supported via + `Subsystem::writable_etc_overlay()`. + +The symlink supplies the `extension-release.NAME`-matching filename; the stored +file may be named freely. Same indirection distributions already use. + +**Shared `/etc`.** The symlink is not slotted, so between servicing and reboot +the running OS points at an extension whose `VERSION_ID` does not match it. This +is inert: already-merged extensions stay merged, the mismatched one is not +merged. + +**Rollback** repoints symlinks at the previous volume's directory. Inverse of +step 3. + +**Ordering.** Symlinks are written before reboot, so they are correct before +`systemd-sysext.service` runs at `Before=sysinit.target`. + +## Testing + +- Clean install: file in correct volume dir, symlink correct, merged after boot. +- A/B update: symlink repoints and **merged content differs from the previous + slot**. A presence-only assertion passes in the inert case and must not be + used. +- Rollback: symlinks and merged content return to the previous slot. +- Validation: each reject row above. +- Compat: config with no `scope` places byte-identically to today. +- Non-default `extensionStore` on a separate mount. + +## Counter-arguments + +- **"Put it in `/usr`."** Strictly better where possible, but requires re-sealing + verity and re-signing the UKI. This targets that gap. +- **"Use the ESP."** `.efi.extra.d/*.sysext.raw` is genuinely slot-scoped, + but ESPs are 128-256 MiB against payloads in the tens to hundreds of MB, and + `systemd-stub` places them in the initrd, not the running system. +- **"Add A/B extension partitions."** Works, but layout changes are breaking + after the first A/B image ships. Directories achieve the same isolation. +- **"Doubles storage."** Yes. Mitigated by `extensionStore`. + +## Open questions + +1. **Does the rollback path invoke subsystem steps?** Largest unknown; gates the + design. +2. **Should Host Status report slotted extensions per volume?** Derivable from + `ab_active_volume` + store, but not inspectable without reading the FS. +3. **Retention.** Nothing prunes a slot directory when an entry is dropped from + the config. Should `provision()` reconcile, and does that break rollback? +4. **SELinux.** Extensions are already documented as unsupported with SELinux + enabled; a Trident-created `/etc` symlink may additionally need a label. +5. **Default path.** `/var/trident/sysexts` vs `/var/lib/trident/sysexts`; the + datastore already lives under `/var/lib/trident/`. + +## Future possibilities + +- Per-volume scoping for other Trident-managed state that is shared today. +- Changing an extension's scope between servicing operations, which currently + strands a copy in the old location. From 9208499ddd3eb3d298710cbcfce55e501d427789 Mon Sep 17 00:00:00 2001 From: Paco Date: Thu, 8 Oct 2026 11:57:30 -0700 Subject: [PATCH 2/8] docs: resolve rollback question in slotted sysext RFC The A/B rollback path invokes no subsystems: automatic rollback is firmware-driven and validate_boot only reconciles Host Status, while manual_rollback::stage_rollback touches only the pcrlock policy. A symlink written during servicing would therefore survive a rollback pointing at the wrong slot. Activation now resolves at boot from the booted volume, via a single /current flip ordered before systemd-sysext.service. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs/Development/RFCs/0838-slotted-sysexts.md | 108 ++++++++++++------ 1 file changed, 76 insertions(+), 32 deletions(-) diff --git a/docs/Development/RFCs/0838-slotted-sysexts.md b/docs/Development/RFCs/0838-slotted-sysexts.md index bddbd860b7..8695c3b49a 100644 --- a/docs/Development/RFCs/0838-slotted-sysexts.md +++ b/docs/Development/RFCs/0838-slotted-sysexts.md @@ -7,8 +7,8 @@ ## Summary Add a `slotted` scope for sysexts/confexts. Trident stores them per A/B volume -under a configurable store (default `/var/trident/sysexts/{a,b}/`) and creates -the activating `/etc/extensions/.raw` symlink during servicing. +under a configurable store (default `/var/trident/sysexts/{a,b}/`) and activates +the correct slot at boot, before `systemd-sysext` merges. ## Motivation @@ -31,8 +31,8 @@ together. This proposal gives that property to anyone who cannot re-seal dm-verity and re-sign the UKI. Goals: correct extension version per booted slot; rollback restores the previous -set; no new partitions, no `/usr` change, no re-signing; existing configs -unaffected. +set without a Trident servicing operation; no new partitions, no `/usr` change, +no re-signing; existing configs unaffected. Non-goals: changing systemd matching semantics; ESP-based extensions. @@ -130,48 +130,87 @@ identically. Schema gains two optional properties. ## Implementation ``` -/var/trident/sysexts/{a,b}/platform.raw -/var/trident/confexts/{a,b}/ +/var/trident/sysexts/ +├── a/platform.raw +├── b/platform.raw +└── current -> a # flipped at boot ``` -Both directories persist across the update, which is what makes rollback a -symlink flip rather than a re-download. +Both slot directories persist across an update, so a rollback needs no +re-download. Confexts use a sibling `confexts/` under the same root. + +### Activation is resolved at boot, not at servicing + +`/etc` is shared between volumes, so an activating symlink written during +servicing would still point at the new slot after a rollback. This cannot be +corrected by an inverse servicing step, because **the A/B rollback path does not +invoke subsystems**: + +- Automatic rollback is firmware-driven. `rollback::validate_boot` only observes + which root actually booted and reconciles Host Status (`spec = spec_old`). +- `manual_rollback::stage_rollback` touches only the pcrlock policy for + `ManualRollbackAbStaged`. Only the *runtime* path calls + `runtime_update::rollback(&mut subsystems, ..)`. + +Activation must therefore key off what booted, not what was staged: + +1. `/etc/extensions/.raw` → `/current/.raw`. Written once at + clean install, never rewritten. +2. `/current` → `a` | `b`, flipped at boot by a Trident unit from the + booted volume. + +One atomic symlink flip activates the whole set, and it is correct after a +rollback Trident never participated in. + +### Servicing flow The extensions subsystem already runs on `CleanInstall` and `AbUpdate` via `provision()`, documented as "migrate state from A-partition to B-partition (or vice versa)". 1. **prepare** — download, verify SHA-384. Unchanged. -2. **provision** — resolve target volume via +2. **provision** — resolve the target volume via `EngineContext::get_ab_update_volume()` (already used for per-slot verity addons); place into `//`. -3. **activate** — write `/etc/extensions/.raw` → - `//.raw`. `/etc` writes are already supported via - `Subsystem::writable_etc_overlay()`. +3. **configure**, clean install only — write the `/etc/extensions` symlinks. + `/etc` writes are already supported via `Subsystem::writable_etc_overlay()`. + +The symlink supplies the `extension-release.NAME`-matching filename; stored +files may be named freely. + +### Boot unit -The symlink supplies the `extension-release.NAME`-matching filename; the stored -file may be named freely. Same indirection distributions already use. +``` +After=local-fs.target # /var mounted, store readable +Before=systemd-sysext.service +``` -**Shared `/etc`.** The symlink is not slotted, so between servicing and reboot -the running OS points at an extension whose `VERSION_ID` does not match it. This -is inert: already-merged extensions stay merged, the mismatched one is not -merged. +`systemd-sysext.service` is `After=local-fs.target Before=sysinit.target`, so +this window exists. It is also gated on +`ConditionDirectoryNotEmpty=|/etc/extensions`, which the install-time symlinks +satisfy. -**Rollback** repoints symlinks at the previous volume's directory. Inverse of -step 3. +This cannot be `trident.service`, which is `WantedBy=multi-user.target` and runs +long after the merge. -**Ordering.** Symlinks are written before reboot, so they are correct before -`systemd-sysext.service` runs at `Before=sysinit.target`. +Slot discovery: compare the booted `/usr` (or root) PARTUUID against the volume +mapping Trident recorded at install. Not the datastore, which is larger +machinery than this needs. See Open Questions. ## Testing -- Clean install: file in correct volume dir, symlink correct, merged after boot. -- A/B update: symlink repoints and **merged content differs from the previous +- Clean install: file in the correct slot dir, `current` and `/etc/extensions` + symlinks correct, merged after boot. +- A/B update: `current` flips and **merged content differs from the previous slot**. A presence-only assertion passes in the inert case and must not be used. -- Rollback: symlinks and merged content return to the previous slot. +- Rollback, both automatic and manual: `current` returns to the previous slot + and the previous content is merged, **with no Trident servicing operation in + between**. This is the case the design exists for. +- Boot ordering: the flip lands before `systemd-sysext.service` merges, asserted + on first boot after an update rather than on a steady-state boot. - Validation: each reject row above. -- Compat: config with no `scope` places byte-identically to today. +- Compat: a config with no `scope` places byte-identically to today. - Non-default `extensionStore` on a separate mount. ## Counter-arguments @@ -187,16 +226,21 @@ step 3. ## Open questions -1. **Does the rollback path invoke subsystem steps?** Largest unknown; gates the - design. -2. **Should Host Status report slotted extensions per volume?** Derivable from - `ab_active_volume` + store, but not inspectable without reading the FS. +1. **Slot discovery in the boot unit.** Comparing the booted PARTUUID against a + small install-time mapping file is proposed, but the mapping must survive the + update that rewrites it. Reading the datastore this early is the alternative. +2. **Does the flip need to be a Trident binary at all?** The operation is one + `readlink` and one `symlink`. A generator or a small shipped script may be a + better fit than adding a Trident invocation to early boot. 3. **Retention.** Nothing prunes a slot directory when an entry is dropped from - the config. Should `provision()` reconcile, and does that break rollback? + the config. Should `provision()` reconcile, and does that break the rollback + guarantee that the previous slot's files are still present? 4. **SELinux.** Extensions are already documented as unsupported with SELinux - enabled; a Trident-created `/etc` symlink may additionally need a label. + enabled; Trident-created symlinks in `/etc` may additionally need a label. 5. **Default path.** `/var/trident/sysexts` vs `/var/lib/trident/sysexts`; the datastore already lives under `/var/lib/trident/`. +6. **Host Status.** Should it report slotted extensions per volume? Derivable + from `ab_active_volume` plus the store, but not otherwise inspectable. ## Future possibilities From f4383d0650e16ac26d3ed1caacdef7494cd2f587 Mon Sep 17 00:00:00 2001 From: Paco Date: Thu, 8 Oct 2026 12:05:08 -0700 Subject: [PATCH 3/8] docs: express the API surface as Host Configuration YAML The Rust types are an implementation detail; the public surface is the Host Configuration document. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs/Development/RFCs/0838-slotted-sysexts.md | 95 ++++++++----------- 1 file changed, 42 insertions(+), 53 deletions(-) diff --git a/docs/Development/RFCs/0838-slotted-sysexts.md b/docs/Development/RFCs/0838-slotted-sysexts.md index 8695c3b49a..71a0bb5557 100644 --- a/docs/Development/RFCs/0838-slotted-sysexts.md +++ b/docs/Development/RFCs/0838-slotted-sysexts.md @@ -40,82 +40,71 @@ Non-goals: changing systemd matching semantics; ESP-based extensions. ### Today -```rust -pub struct Extension { - pub url: Url, - pub sha384: Sha384Hash, - pub path: Option, // absolute, allow-listed dir, *.raw -} +```yaml +os: + sysexts: + - url: https://example.com/tooling.raw # http(s) | file | oci + sha384: + path: /var/lib/extensions/tooling.raw # optional; allow-listed dir, *.raw ``` -`path` conflates *where the file is stored* with *what scope it is active in*. +`path` defaults to `/var/lib/extensions/`, and is restricted to +`/etc/extensions`, `/var/lib/extensions` or `/.extra/sysext`. + +It conflates *where the file is stored* with *what scope it is active in*. Slotting separates them: storage becomes Trident-owned and derived from the target volume. ### Proposed -```rust -pub struct Extension { - pub url: Url, - pub sha384: Sha384Hash, - - /// Absolute path in the target OS. Only valid when `scope` is `shared`. - pub path: Option, - - #[serde(default)] - pub scope: ExtensionScope, -} - -pub enum ExtensionScope { - /// Stored at `path`, active on both volumes. For `ID=_any` extensions. - #[default] - Shared, - /// Stored per-volume in the extension store, active only on the volume it - /// was installed to. For OS-version-pinned extensions. - Slotted, -} -``` - -Plus one optional field on `os`: - -```rust -/// Root of the per-volume extension store. Default `/var/trident/sysexts`. -/// Must be on storage not replaced by an A/B update, and must not be a -/// systemd-sysext search path. -pub extension_store: Option, -``` - -Configurable because slotting doubles storage, and large extensions (GPU -drivers, ML runtimes) may need a dedicated volume. Confexts use a sibling -`confexts/` under the same root. +Two new optional fields. ```yaml os: - extensionStore: /mnt/bigdisk/trident/sysexts # optional + # Root of the per-volume extension store. + # Default /var/trident/sysexts. Must be on storage an A/B update does not + # replace, and must not be a systemd-sysext search path. + extensionStore: /mnt/bigdisk/trident/sysexts + sysexts: - - url: https://example.com/tooling.raw # unchanged, still default + # unchanged, and still the default + - url: https://example.com/tooling.raw sha384: + + # new - url: https://example.com/platform.raw sha384: scope: slotted ``` +`scope`: + +| Value | Meaning | +|---|---| +| `shared` (default) | Stored at `path`, active on both volumes. For `ID=_any` extensions. Today's behaviour. | +| `slotted` | Stored per-volume under `extensionStore`, active only on the volume it was installed to. For OS-version-pinned extensions. | + +`extensionStore` is configurable because slotting doubles storage, and large +extensions (GPU drivers, ML runtimes) may need a dedicated volume. Confexts use +a sibling `confexts/` under the same root. + ### Validation | Condition | Result | |---|---| | `scope` absent | `shared`. Identical to today. | -| `scope: shared` + `path` | Existing allow-list check. | -| `scope: slotted` + `path` | Reject. Path is Trident-owned; accepting it would silently ignore it. | -| `scope: slotted`, no `path` | Filename from `url` basename, as today's default. | -| `scope: slotted`, no A/B volumes | Reject at validation, not at provision. | +| `scope: shared` with `path` | Existing allow-list check. | +| `scope: slotted` with `path` | Reject. The path is Trident-owned; accepting it would silently ignore it. | +| `scope: slotted`, no `path` | Filename from the `url` basename, as today's default. | +| `scope: slotted`, no A/B volumes configured | Reject at validation, not at provision. | | `extensionStore` set, no `slotted` entries | Accept, unused. | -| `extensionStore` inside a sysext search path | Reject: both copies would be discovered. | +| `extensionStore` inside a sysext search path | Reject: both slots' copies would be discovered. | ### Backward compatibility -Both fields are `#[serde(default)]`. Existing configs parse and behave -identically. Schema gains two optional properties. +Both fields default, so existing Host Configurations parse and behave +identically. The schema gains two optional properties and no field changes +meaning. ### Alternatives rejected @@ -123,9 +112,9 @@ identically. Schema gains two optional properties. layout, invites a literal `a`/`b`. - **Separate `os.slottedSysexts` list**: duplicates the type and its duplicate-hash/path validation. -- **Infer from `extension-release`**: metadata is inside the image, unavailable - at static validation, and removes the operator's choice to slot an `_any` - extension deliberately. +- **Infer from `extension-release`**: the metadata is inside the image, so it is + unavailable at static validation, and it removes the operator's choice to slot + an `_any` extension deliberately. ## Implementation From 60697c7be19fa473f263d4e6a341774e9aef2b24 Mon Sep 17 00:00:00 2001 From: Paco Date: Thu, 8 Oct 2026 17:34:07 -0700 Subject: [PATCH 4/8] docs: rewrite 0838 around the validation change The original premise was wrong. Trident does not place extensions on non-slotted storage: validate_extension_images_locations already rejects any extension directory not on an A/B volume. The real cost is that satisfying it requires a dedicated A/B volume pair, which the reference config pays for with four partitions and 2.4 GB, and which a layout whose only A/B volume is a verity-sealed /usr cannot pay at all. The proposal is now to relax that check: slot-scoping as a property of the extension rather than of the partition table, satisfied by an A/B volume as today or by a Trident-owned per-slot store. Activation no longer needs a boot-time unit. The extensions subsystem gains ManualRollbackAb in runs_on, and automatic rollback is corrected at commit via systemd-sysext refresh; version matching keeps the intervening window absent rather than wrong. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs/Development/RFCs/0838-slotted-sysexts.md | 339 ++++++++++-------- 1 file changed, 190 insertions(+), 149 deletions(-) diff --git a/docs/Development/RFCs/0838-slotted-sysexts.md b/docs/Development/RFCs/0838-slotted-sysexts.md index 71a0bb5557..097bad9203 100644 --- a/docs/Development/RFCs/0838-slotted-sysexts.md +++ b/docs/Development/RFCs/0838-slotted-sysexts.md @@ -1,4 +1,4 @@ -# 0838 Slotted system extensions +# 0838 Slotted Extension Storage - Date: 2026-10-08 - RFC PR: [microsoft/trident#838](https://github.com/microsoft/trident/pull/838) @@ -6,35 +6,58 @@ ## Summary -Add a `slotted` scope for sysexts/confexts. Trident stores them per A/B volume -under a configurable store (default `/var/trident/sysexts/{a,b}/`) and activates -the correct slot at boot, before `systemd-sysext` merges. +Trident requires every extension destination to sit on an A/B volume when A/B is +configured. That is one way to make an extension follow the slot, and it costs a +partition pair per extension directory. -## Motivation +This adds a second. An extension marked `scope: slotted` is stored per volume +under a Trident-owned store, with a symlink at its normal destination. +Slot-scoping becomes a property of the extension rather than of the partition +table. -`systemd-sysext(8)` merges an extension only if its `extension-release` matches -the running OS: `ID=` must match unless `_any`, then `SYSEXT_LEVEL=` if defined, -else `VERSION_ID=`. +## Motivation and Goals -All three directories Trident accepts today (`/etc/extensions`, -`/var/lib/extensions`, `/.extra/sysext`) are outside the partitions an A/B -update replaces. So an OS-pinned extension silently stops merging after the -first update: no error, no failed unit, the content is just gone. +`validate_extension_images_locations` (`config/host/mod.rs:218`) rejects any +extension directory that does not resolve to a device with A/B capabilities, +raising `ExtensionImageNotOnABVolume`. The reasoning is sound: extensions on +shared storage change the running slot immediately and do not roll back. -Azure Container Linux hits this. Its Azure platform extension (guest agent, -Hyper-V daemons, chrony, `oras`) is `VERSION_ID`-pinned on a non-swapped -partition. Losing `oras` also breaks the distro's extension download path. +The cost is that the only way to satisfy it is a dedicated A/B volume pair. +Trident's own reference configuration +(`tests/e2e_tests/trident_configurations/root-verity/trident-config.yaml`) +spends four partitions and 2.4 GB on it: `sysexts-a`/`sysexts-b` at 1 GB each +mounted at `/var/lib/extensions`, and `confexts-a`/`confexts-b` at 200 MB each +at `/var/lib/confexts`. -The shape that works already exists in the same image: the container runtime -ships as an extension inside `/usr`, so extension and `os-release` swap -together. This proposal gives that property to anyone who cannot re-seal -dm-verity and re-sign the UKI. +That price buys only slot-scoping, and it is charged in the currency hardest to +change: -Goals: correct extension version per booted slot; rollback restores the previous -set without a Trident servicing operation; no new partitions, no `/usr` change, -no re-signing; existing configs unaffected. +- **Partitions are fixed at install.** The 1 GB is a guess made before anyone + knows what the host will carry; growing it means repartitioning. +- **Some layouts cannot pay it.** Where the partition table is published, signed + or attested, adding a pair is not a configuration change. A layout whose only + A/B volume is a verity-sealed `/usr` has none to offer on any terms. +- **It is all-or-nothing per directory.** An extension that should follow the + slot and one that should persist cannot share a location. -Non-goals: changing systemd matching semantics; ESP-based extensions. +The check conflates *being slot-scoped*, the property that matters, with *being +on an A/B block device*, one implementation of it. Trident can provide the same +property with directories, on storage the host already has. + +**Goals.** Slot-scoped extensions without a dedicated volume pair; rollback +restores the previous set; shared and slotted coexist in one directory; no new +partitions, no sealed-`/usr` change, no re-signing; existing configurations +unaffected. + +**Non-goals.** Changing `systemd-sysext` matching semantics, which still decides +what merges. Removing the A/B volume route, which stays better where a pair is +affordable, since the filesystem provides the isolation and Trident does +nothing. Extensions on the ESP. + +## Scope + +Sysexts and confexts under `os.sysexts` and `os.confexts`, on clean install, A/B +update and rollback. Runtime update does not change slot and is unaffected. ## Public API Design @@ -48,23 +71,20 @@ os: path: /var/lib/extensions/tooling.raw # optional; allow-listed dir, *.raw ``` -`path` defaults to `/var/lib/extensions/`, and is restricted to -`/etc/extensions`, `/var/lib/extensions` or `/.extra/sysext`. - -It conflates *where the file is stored* with *what scope it is active in*. -Slotting separates them: storage becomes Trident-owned and derived from the -target volume. +`path` is the destination, defaulting to `/var/lib/extensions/{name}.raw`. When +`storage.abUpdate` is configured it must also resolve to an A/B volume. ### Proposed -Two new optional fields. +Two optional fields. `path` keeps one meaning in both cases: where the extension +appears to systemd. ```yaml os: # Root of the per-volume extension store. # Default /var/trident/sysexts. Must be on storage an A/B update does not # replace, and must not be a systemd-sysext search path. - extensionStore: /mnt/bigdisk/trident/sysexts + extensionStore: /mnt/data/trident/sysexts sysexts: # unchanged, and still the default @@ -77,162 +97,183 @@ os: scope: slotted ``` -`scope`: - -| Value | Meaning | -|---|---| -| `shared` (default) | Stored at `path`, active on both volumes. For `ID=_any` extensions. Today's behaviour. | -| `slotted` | Stored per-volume under `extensionStore`, active only on the volume it was installed to. For OS-version-pinned extensions. | +| `scope` | Where the bytes live | What `path` is | +| --- | --- | --- | +| `shared` (default) | At `path`. | The image itself. Today's behaviour. | +| `slotted` | `/{a,b}/{name}.raw` | A symlink to the booted slot's copy. | `extensionStore` is configurable because slotting doubles storage, and large -extensions (GPU drivers, ML runtimes) may need a dedicated volume. Confexts use -a sibling `confexts/` under the same root. +extensions such as GPU drivers may warrant a dedicated volume. Confexts use a +sibling `confexts/` under the same root. + +### The validation change + +`validate_extension_images_locations` is relaxed, not removed. Its requirement +becomes *the destination is slot-scoped*, satisfied either way: + +| Configuration | Rule | +| --- | --- | +| No `abUpdate` | No check, as today. | +| `abUpdate`, `scope: shared` | `path` must be on an A/B volume. Unchanged. | +| `abUpdate`, `scope: slotted` | `path` need not be; the store provides the scoping. | + +This is the substantive behavioural change. Everything else follows from it. ### Validation | Condition | Result | -|---|---| +| --- | --- | | `scope` absent | `shared`. Identical to today. | -| `scope: shared` with `path` | Existing allow-list check. | -| `scope: slotted` with `path` | Reject. The path is Trident-owned; accepting it would silently ignore it. | -| `scope: slotted`, no `path` | Filename from the `url` basename, as today's default. | -| `scope: slotted`, no A/B volumes configured | Reject at validation, not at provision. | +| `scope: slotted`, no `path` | Default destination, as today. | +| `scope: slotted`, no `abUpdate` | Reject, naming the absent configuration. | +| `extensionStore` inside a search path | Reject: both slots' copies would be discovered. | +| `extensionStore` on an A/B volume | Reject: it must outlive the slot it describes. | | `extensionStore` set, no `slotted` entries | Accept, unused. | -| `extensionStore` inside a sysext search path | Reject: both slots' copies would be discovered. | ### Backward compatibility Both fields default, so existing Host Configurations parse and behave -identically. The schema gains two optional properties and no field changes -meaning. +identically and no field changes meaning. The relaxed check is strictly more +permissive, so anything valid today stays valid. The schema gains two optional +properties. ### Alternatives rejected - **Sentinel `path`** (`/var/trident/sysexts//x.raw`): leaks internal - layout, invites a literal `a`/`b`. -- **Separate `os.slottedSysexts` list**: duplicates the type and its - duplicate-hash/path validation. -- **Infer from `extension-release`**: the metadata is inside the image, so it is - unavailable at static validation, and it removes the operator's choice to slot - an `_any` extension deliberately. + layout, invites a literal `a` or `b`. +- **Separate `os.slottedSysexts` list**: duplicates the type and its duplicate + hash and path validation. +- **Infer from `extension-release`**: inside the image, so unavailable at static + validation, and it removes the operator's choice to slot an `ID=_any` + extension deliberately. ## Implementation ``` /var/trident/sysexts/ ├── a/platform.raw -├── b/platform.raw -└── current -> a # flipped at boot -``` - -Both slot directories persist across an update, so a rollback needs no -re-download. Confexts use a sibling `confexts/` under the same root. - -### Activation is resolved at boot, not at servicing +└── b/platform.raw -`/etc` is shared between volumes, so an activating symlink written during -servicing would still point at the new slot after a rollback. This cannot be -corrected by an inverse servicing step, because **the A/B rollback path does not -invoke subsystems**: - -- Automatic rollback is firmware-driven. `rollback::validate_boot` only observes - which root actually booted and reconciles Host Status (`spec = spec_old`). -- `manual_rollback::stage_rollback` touches only the pcrlock policy for - `ManualRollbackAbStaged`. Only the *runtime* path calls - `runtime_update::rollback(&mut subsystems, ..)`. - -Activation must therefore key off what booted, not what was staged: +/var/lib/extensions/ +├── platform.raw -> /var/trident/sysexts/a/platform.raw # slotted +└── tooling.raw # shared, unchanged +``` -1. `/etc/extensions/.raw` → `/current/.raw`. Written once at - clean install, never rewritten. -2. `/current` → `a` | `b`, flipped at boot by a Trident unit from the - booted volume. +The store sits outside every search path, so only the symlink is discoverable +and exactly one copy is ever a candidate. Both slot directories persist across +an update, so rollback needs no re-download. -One atomic symlink flip activates the whole set, and it is correct after a -rollback Trident never participated in. +Symlinks are followed: `image_discover` opens each search path with +`chase_and_opendir(path, root, CHASE_PREFIX_ROOT, ..)` and stats entries with +`flags = 0` on a running system, with an upstream comment explicitly permitting +symlinks into the search path (systemd `src/shared/discover-image.c`). -### Servicing flow +### Servicing -The extensions subsystem already runs on `CleanInstall` and `AbUpdate` via +The extensions subsystem already runs on clean install and A/B update via `provision()`, documented as "migrate state from A-partition to B-partition (or -vice versa)". - -1. **prepare** — download, verify SHA-384. Unchanged. -2. **provision** — resolve the target volume via - `EngineContext::get_ab_update_volume()` (already used for per-slot verity - addons); place into `//`. -3. **configure**, clean install only — write the `/etc/extensions` symlinks. - `/etc` writes are already supported via `Subsystem::writable_etc_overlay()`. - -The symlink supplies the `extension-release.NAME`-matching filename; stored -files may be named freely. - -### Boot unit - -``` -After=local-fs.target # /var mounted, store readable -Before=systemd-sysext.service -``` - -`systemd-sysext.service` is `After=local-fs.target Before=sysinit.target`, so -this window exists. It is also gated on -`ConditionDirectoryNotEmpty=|/etc/extensions`, which the install-time symlinks -satisfy. - -This cannot be `trident.service`, which is `WantedBy=multi-user.target` and runs -long after the merge. - -Slot discovery: compare the booted `/usr` (or root) PARTUUID against the volume -mapping Trident recorded at install. Not the datastore, which is larger -machinery than this needs. See Open Questions. +vice versa)". For a slotted entry it resolves the target volume with +`EngineContext::get_ab_update_volume()`, already used to select the per-slot +verity addon, writes into `//`, and points the destination +symlink at it. + +### Rollback + +`RUNS_ON_ALL` is `CleanInstall | AbUpdate | RuntimeUpdate` (`engine/mod.rs:63`); +it excludes both rollback types despite the name. The extensions subsystem adds +`ManualRollbackAb` to its `runs_on` and repoints the symlinks at the previous +volume's directory. The files were never removed, so this is a symlink operation +rather than a re-download. `finalize_rollback` already locks `SUBSYSTEMS` for +the runtime path. + +**Automatic rollback is not a servicing operation.** When the firmware falls +back on its own, Trident regains control only at `trident commit`, which is +`WantedBy=multi-user.target` and so runs after `systemd-sysext` merges at +`sysinit.target`. + +The outcome is still safe, because the matching rules that motivate this RFC act +as an interlock. The stale symlink resolves to the other slot's extension, which +declares the other OS version, so systemd declines to merge it. The host comes +up with the extension **absent, not wrong**. `commit` then detects the rollback +through the existing `validate_boot` path, repoints the symlinks and runs +`systemd-sysext refresh`, which is the unit's own `ExecStart` and needs no +reboot. + +Degraded for part of one boot, never serving content from the wrong OS version, +and self-closing. Where that window is unacceptable, a +[health check](../../Reference/Host-Configuration/API-Reference/Health.md) +asserting the extension is merged gates the update before it reaches this state. + +## Relationship to RFC 0789 + +[0789](https://github.com/microsoft/trident/pull/789) carries extension images in +COSI files. The two are orthogonal: 0789 changes where the bytes come from, this +changes where they land and how they follow the slot. Either is useful alone, +and a bundled extension can be slotted. + +Both touch `validate_extension_images_locations`, in opposite directions: 0789 +tightens it into dynamic validation against the storage graph, this adds a +second way to satisfy it. Whichever lands first should leave the check expressed +as *is this destination slot-scoped*, so the other is an added arm rather than a +rewrite. 0789's rollback argument, that extensions revert because the slot's +filesystem reverts, is the A/B volume mechanism described here and holds +unchanged for `scope: shared`. ## Testing -- Clean install: file in the correct slot dir, `current` and `/etc/extensions` - symlinks correct, merged after boot. -- A/B update: `current` flips and **merged content differs from the previous - slot**. A presence-only assertion passes in the inert case and must not be - used. -- Rollback, both automatic and manual: `current` returns to the previous slot - and the previous content is merged, **with no Trident servicing operation in - between**. This is the case the design exists for. -- Boot ordering: the flip lands before `systemd-sysext.service` merges, asserted - on first boot after an update rather than on a steady-state boot. -- Validation: each reject row above. -- Compat: a config with no `scope` places byte-identically to today. -- Non-default `extensionStore` on a separate mount. +- Clean install: file in the correct slot directory, symlink resolving, merged + after boot. +- A/B update: the symlink repoints and **the merged content differs from the + previous slot**. A presence-only assertion passes in the inert case and must + not be used. +- Manual rollback: symlinks and merged content return to the previous slot. +- Automatic rollback: on the fallback boot the extension is absent rather than + stale, and merges after `commit` without a reboot. Both halves asserted; the + first distinguishes safe degradation from serving wrong content. +- Coexistence: a shared and a slotted extension in `/var/lib/extensions`, both + merged, only the slotted one changing across an update. +- Validation: each reject row, plus a shared extension off an A/B volume still + rejected. +- Compatibility: a configuration with no `scope` places byte-identically to + today. A non-default `extensionStore` on a separate mount. ## Counter-arguments -- **"Put it in `/usr`."** Strictly better where possible, but requires re-sealing - verity and re-signing the UKI. This targets that gap. -- **"Use the ESP."** `.efi.extra.d/*.sysext.raw` is genuinely slot-scoped, - but ESPs are 128-256 MiB against payloads in the tens to hundreds of MB, and - `systemd-stub` places them in the initrd, not the running system. -- **"Add A/B extension partitions."** Works, but layout changes are breaking - after the first A/B image ships. Directories achieve the same isolation. -- **"Doubles storage."** Yes. Mitigated by `extensionStore`. +- **"Just add the partitions."** Better where possible. This is for layouts that + cannot, and for sizing decided after install rather than before. +- **"Put it in `/usr`."** Better still, and unavailable without re-sealing + dm-verity and re-signing the UKI. Upstream also forbids extension storage + under `/usr`: a `lowerdir=` that is a child of another fails with `-ELOOP`. +- **"Use the ESP."** Genuinely slot-scoped via `.efi.extra.d/`, but ESPs + are 128 to 256 MiB against payloads in the tens to hundreds of megabytes, and + `systemd-stub` places them in the initrd. +- **"Doubles storage."** It does. The A/B volume route doubles it too, with the + allocation fixed at install. +- **"A symlink is weaker than a partition boundary."** Yes, though both are + reachable by anything that can already write the extension image. + +**Prior art.** systemd's versioned directories (`.v/` holding +`_.raw`, highest wins) are the same shape: a directory of +candidates, one selected at discovery. They select by version rather than slot, +so they do not serve this case, but the pattern is systemd's own. ## Open questions -1. **Slot discovery in the boot unit.** Comparing the booted PARTUUID against a - small install-time mapping file is proposed, but the mapping must survive the - update that rewrites it. Reading the datastore this early is the alternative. -2. **Does the flip need to be a Trident binary at all?** The operation is one - `readlink` and one `symlink`. A generator or a small shipped script may be a - better fit than adding a Trident invocation to early boot. -3. **Retention.** Nothing prunes a slot directory when an entry is dropped from - the config. Should `provision()` reconcile, and does that break the rollback - guarantee that the previous slot's files are still present? -4. **SELinux.** Extensions are already documented as unsupported with SELinux - enabled; Trident-created symlinks in `/etc` may additionally need a label. -5. **Default path.** `/var/trident/sysexts` vs `/var/lib/trident/sysexts`; the - datastore already lives under `/var/lib/trident/`. -6. **Host Status.** Should it report slotted extensions per volume? Derivable - from `ab_active_volume` plus the store, but not otherwise inspectable. +1. **Retention.** Nothing prunes a slot directory when an entry leaves the + configuration. Should `provision()` reconcile, and does that conflict with + keeping the previous slot's files for rollback? +2. **Default path.** `/var/trident/sysexts` against `/var/lib/trident/sysexts`; + the datastore already lives under `/var/lib/trident/` (`constants.rs:30`). +3. **Should `commit` refresh unconditionally?** Refreshing only on a detected + rollback is narrower, but an unconditional refresh is cheap and removes a + branch. +4. **Host Status.** Should it report the slotted set per volume? +5. **SELinux.** Extensions are already unsupported with SELinux enforcing. + Slotting does not change that, but Trident-created symlinks may need + labelling, and 0789 proposes moving that check to every servicing type. ## Future possibilities - Per-volume scoping for other Trident-managed state that is shared today. -- Changing an extension's scope between servicing operations, which currently - strands a copy in the old location. +- Changing an extension's scope between operations, which would otherwise strand + a copy in the old location. From 8a86bd3d484a8c96378d1ee42665401841e177a2 Mon Sep 17 00:00:00 2001 From: Paco Date: Mon, 12 Oct 2026 15:40:09 -0700 Subject: [PATCH 5/8] docs: rewrite 0838 around boot-time selection Replaces the previous mechanism. Symlinks are no longer written by the extensions subsystem into /var/lib/extensions during servicing; selection is recomputed every boot into /run/extensions from the slot that actually booted. That removes the ManualRollbackAb runs_on change and the commit-time repair, since nothing persists a decision that a rollback could invalidate. Corrections and additions from review: - Before=systemd-sysext.service is the ordering edge. ensure-sysext.service carries BindsTo= and After=systemd-sysext.service, so it runs after the merge and is the wrong anchor. - WantedBy= plus Before= does not fail closed; the merge unit needs a Requires= drop-in. - Per-file symlinking cannot commit a set atomically. - A store manifest replaces globbing, so absence is distinguishable from loss. - Two-phase rollout, since a rollback slot may predate the loader. - Documents why content matching alone is insufficient, and scopes slotted extensions to /usr and /opt because reordering the merge ahead of an /etc overlay would exhaust the kernel filesystem stacking budget. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs/Development/RFCs/0838-slotted-sysexts.md | 407 +++++++++++++----- 1 file changed, 290 insertions(+), 117 deletions(-) diff --git a/docs/Development/RFCs/0838-slotted-sysexts.md b/docs/Development/RFCs/0838-slotted-sysexts.md index 097bad9203..29951c3383 100644 --- a/docs/Development/RFCs/0838-slotted-sysexts.md +++ b/docs/Development/RFCs/0838-slotted-sysexts.md @@ -1,6 +1,6 @@ # 0838 Slotted Extension Storage -- Date: 2026-10-08 +- Date: 2026-10-12 - RFC PR: [microsoft/trident#838](https://github.com/microsoft/trident/pull/838) - Issue: TBD @@ -11,9 +11,9 @@ configured. That is one way to make an extension follow the slot, and it costs a partition pair per extension directory. This adds a second. An extension marked `scope: slotted` is stored per volume -under a Trident-owned store, with a symlink at its normal destination. -Slot-scoping becomes a property of the extension rather than of the partition -table. +under a Trident-owned store. A small boot-time component selects the booted +slot's copy into `/run/extensions` before the merge runs. Slot-scoping becomes a +property of the extension rather than of the partition table. ## Motivation and Goals @@ -25,14 +25,12 @@ shared storage change the running slot immediately and do not roll back. The cost is that the only way to satisfy it is a dedicated A/B volume pair. Trident's own reference configuration (`tests/e2e_tests/trident_configurations/root-verity/trident-config.yaml`) -spends four partitions and 2.4 GB on it: `sysexts-a`/`sysexts-b` at 1 GB each -mounted at `/var/lib/extensions`, and `confexts-a`/`confexts-b` at 200 MB each -at `/var/lib/confexts`. +spends four partitions and 2.4 GB on it. That price buys only slot-scoping, and it is charged in the currency hardest to change: -- **Partitions are fixed at install.** The 1 GB is a guess made before anyone +- **Partitions are fixed at install.** The size is a guess made before anyone knows what the host will carry; growing it means repartitioning. - **Some layouts cannot pay it.** Where the partition table is published, signed or attested, adding a pair is not a configuration change. A layout whose only @@ -41,23 +39,64 @@ change: slot and one that should persist cannot share a location. The check conflates *being slot-scoped*, the property that matters, with *being -on an A/B block device*, one implementation of it. Trident can provide the same -property with directories, on storage the host already has. +on an A/B block device*, one implementation of it. -**Goals.** Slot-scoped extensions without a dedicated volume pair; rollback -restores the previous set; shared and slotted coexist in one directory; no new -partitions, no sealed-`/usr` change, no re-signing; existing configurations -unaffected. +### Why content matching alone is not sufficient -**Non-goals.** Changing `systemd-sysext` matching semantics, which still decides -what merges. Removing the A/B volume route, which stays better where a pair is -affordable, since the filesystem provides the isolation and Trident does -nothing. Extensions on the ESP. +The obvious alternative is to place both slots' copies in one search directory +and let `systemd-sysext` merge whichever matches the booted OS. Version +mismatches are skipped quietly, so this very nearly works, and on a suitable +platform it would need no new code at all. + +It fails for two independent reasons, either of which is sufficient: + +1. **Deploying one image under two filenames requires systemd v256.** + `path_extract_image_name()` truncates an image name at the first `_` or `+` + (`src/basic/os-util.c`), so `platform_1.raw` and `platform_2.raw` can share a + single inner `extension-release.platform`. That function does not exist + before v256; it was added by commit `37543971` (2024-03-28). On earlier + systemd each copy needs its own `extension-release`, so a signed image must + be repacked per slot. The `user.extension-release.strict` xattr relaxes the + name correspondence but must be set inside the image at build time. +2. **The OS version must actually differ between slots.** Matching uses `ID=`, + then `SYSEXT_LEVEL=` when defined on both sides, otherwise `VERSION_ID=`. A + platform whose `VERSION_ID` is coarser than its build cadence, or which can + deploy the same image to both slots, puts identical values in both slots and + both copies then match. + +Both conditions hold on the first intended consumer. Content matching remains +the better mechanism where a platform satisfies both, and this design does not +preclude adopting it there. + +### Goals + +Slot-scoped extensions without a dedicated volume pair; rollback restores the +previous set with no servicing operation; no new partitions, no sealed-`/usr` +change, no re-signing; existing configurations unaffected. + +### Non-goals + +Changing `systemd-sysext` matching semantics. Removing the A/B volume route, +which stays better where a pair is affordable, since the filesystem provides +the isolation and Trident does nothing. Extensions on the ESP. ## Scope -Sysexts and confexts under `os.sysexts` and `os.confexts`, on clean install, A/B -update and rollback. Runtime update does not change slot and is unaffected. +Sysexts and confexts declared under `os.sysexts` and `os.confexts`, on clean +install and A/B update. Runtime update does not change slot and is unaffected. + +**Slotted extensions carry `/usr` and `/opt` content only.** Contributing to +`/etc` is out of scope for this RFC. On a platform that assembles `/etc` as an +overlay whose lowerdir lives inside `/usr`, a sysext can only affect `/etc` if +it is merged before that overlay is built. Reordering the merge ahead of it +would place the `/etc` overlay on top of the `/usr` overlay, consuming the +kernel's entire filesystem stacking budget: `FILESYSTEM_MAX_STACK_DEPTH` is 2 +(`include/linux/fs.h`), enforced in `fs/overlayfs/super.c`, and an overlay's +depth is the maximum of its layers plus one. The `/etc` overlay would reach +depth 2 and any confext merged onto `/etc` afterwards would reach 3 and fail. + +The durable fix is to stop sourcing `/etc` defaults from inside `/usr`, which is +a platform change rather than a Trident one. ## Public API Design @@ -76,14 +115,12 @@ os: ### Proposed -Two optional fields. `path` keeps one meaning in both cases: where the extension -appears to systemd. - ```yaml os: # Root of the per-volume extension store. # Default /var/trident/sysexts. Must be on storage an A/B update does not - # replace, and must not be a systemd-sysext search path. + # replace, must not be a systemd-sysext search path, and must be on a + # filesystem available at local-fs.target. extensionStore: /mnt/data/trident/sysexts sysexts: @@ -97,14 +134,13 @@ os: scope: slotted ``` -| `scope` | Where the bytes live | What `path` is | +| `scope` | Where the bytes live | Where it is activated | | --- | --- | --- | -| `shared` (default) | At `path`. | The image itself. Today's behaviour. | -| `slotted` | `/{a,b}/{name}.raw` | A symlink to the booted slot's copy. | +| `shared` (default) | At `path`. | At `path`. Today's behaviour. | +| `slotted` | `/{a,b}/{name}.raw` | `/run/extensions/`, by the loader | -`extensionStore` is configurable because slotting doubles storage, and large -extensions such as GPU drivers may warrant a dedicated volume. Confexts use a -sibling `confexts/` under the same root. +A slotted extension always surfaces in `/run/extensions`, so `path` does not +apply to it and is rejected. ### The validation change @@ -115,34 +151,36 @@ becomes *the destination is slot-scoped*, satisfied either way: | --- | --- | | No `abUpdate` | No check, as today. | | `abUpdate`, `scope: shared` | `path` must be on an A/B volume. Unchanged. | -| `abUpdate`, `scope: slotted` | `path` need not be; the store provides the scoping. | +| `abUpdate`, `scope: slotted` | `path` need not be; the store plus the loader provide the scoping. | -This is the substantive behavioural change. Everything else follows from it. +This is the substantive behavioural change. It is only safe where the platform +actually provides a loader, so validation must check a **platform capability**, +not merely that A/B is configured. See [Rollout](#rollout). ### Validation | Condition | Result | | --- | --- | | `scope` absent | `shared`. Identical to today. | -| `scope: slotted`, no `path` | Default destination, as today. | +| `scope: slotted` with `path` | Reject. The destination is loader-owned. | | `scope: slotted`, no `abUpdate` | Reject, naming the absent configuration. | +| `scope: slotted`, no loader capability | Reject. See [Rollout](#rollout). | | `extensionStore` inside a search path | Reject: both slots' copies would be discovered. | | `extensionStore` on an A/B volume | Reject: it must outlive the slot it describes. | -| `extensionStore` set, no `slotted` entries | Accept, unused. | +| `extensionStore` not available at `local-fs.target` | Reject: the loader runs before anything later. | +| Two slotted entries resolving to the same basename | Reject: they would collide in `/run/extensions`. | ### Backward compatibility Both fields default, so existing Host Configurations parse and behave -identically and no field changes meaning. The relaxed check is strictly more -permissive, so anything valid today stays valid. The schema gains two optional -properties. +identically. The relaxed check is strictly more permissive, so anything valid +today stays valid. The schema gains two optional properties. ### Alternatives rejected -- **Sentinel `path`** (`/var/trident/sysexts//x.raw`): leaks internal - layout, invites a literal `a` or `b`. -- **Separate `os.slottedSysexts` list**: duplicates the type and its duplicate - hash and path validation. +- **Sentinel `path`** (`//x.raw`): leaks internal layout, invites a + literal `a` or `b`. +- **Separate `os.slottedSysexts` list**: duplicates the type and its validation. - **Infer from `extension-release`**: inside the image, so unavailable at static validation, and it removes the operator's choice to slot an `ID=_any` extension deliberately. @@ -154,126 +192,261 @@ properties. ├── a/platform.raw └── b/platform.raw -/var/lib/extensions/ -├── platform.raw -> /var/trident/sysexts/a/platform.raw # slotted -└── tooling.raw # shared, unchanged +/run/extensions/ +└── platform.raw -> /var/trident/sysexts/a/platform.raw # recreated each boot ``` -The store sits outside every search path, so only the symlink is discoverable -and exactly one copy is ever a candidate. Both slot directories persist across -an update, so rollback needs no re-download. +The store is outside every search path, so its contents are invisible to +`systemd-sysext` until selected. Both slot directories persist across an update, +so rollback needs no re-download. Symlinks are followed: `image_discover` opens each search path with `chase_and_opendir(path, root, CHASE_PREFIX_ROOT, ..)` and stats entries with `flags = 0` on a running system, with an upstream comment explicitly permitting -symlinks into the search path (systemd `src/shared/discover-image.c`). +symlinks into the search path (`src/shared/discover-image.c`). -### Servicing +### Servicing writes the store, and must write it atomically The extensions subsystem already runs on clean install and A/B update via `provision()`, documented as "migrate state from A-partition to B-partition (or vice versa)". For a slotted entry it resolves the target volume with `EngineContext::get_ab_update_volume()`, already used to select the per-slot -verity addon, writes into `//`, and points the destination -symlink at it. - -### Rollback - -`RUNS_ON_ALL` is `CleanInstall | AbUpdate | RuntimeUpdate` (`engine/mod.rs:63`); -it excludes both rollback types despite the name. The extensions subsystem adds -`ManualRollbackAb` to its `runs_on` and repoints the symlinks at the previous -volume's directory. The files were never removed, so this is a symlink operation -rather than a re-download. `finalize_rollback` already locks `SUBSYSTEMS` for -the runtime path. - -**Automatic rollback is not a servicing operation.** When the firmware falls -back on its own, Trident regains control only at `trident commit`, which is -`WantedBy=multi-user.target` and so runs after `systemd-sysext` merges at -`sysinit.target`. - -The outcome is still safe, because the matching rules that motivate this RFC act -as an interlock. The stale symlink resolves to the other slot's extension, which -declares the other OS version, so systemd declines to merge it. The host comes -up with the extension **absent, not wrong**. `commit` then detects the rollback -through the existing `validate_boot` path, repoints the symlinks and runs -`systemd-sysext refresh`, which is the unit's own `ExecStart` and needs no -reboot. - -Degraded for part of one boot, never serving content from the wrong OS version, -and self-closing. Where that window is unacceptable, a -[health check](../../Reference/Host-Configuration/API-Reference/Health.md) -asserting the extension is merged gates the update before it reaches this state. +verity addon, and writes into `//`. + +Placement must be atomic. A malformed `.raw` is not skipped like a version +mismatch: `image_read_metadata` errors propagate in +`image_discover_and_read_metadata` (`src/sysext/sysext.c`) before any per-image +tolerance applies, so **one bad image blocks every extension on the host**. +Directory-type images are tolerated; `.raw` are not. + +Trident today falls back to a non-atomic `fs::copy` when `rename` fails across a +filesystem boundary (`crates/trident/src/subsystems/extensions/mod.rs:525-540`). +Staging must move to a temporary file on the destination's own filesystem, +verified and `fsync`ed, then renamed. This is a prerequisite, not a +consequence. + +### Selection happens at boot + +Selection is recomputed on every boot from the slot that actually booted. This +is what makes unsupervised rollback correct: when firmware falls back on its +own, no Trident code runs, so any decision written during servicing would be +stale with nothing to correct it. + +`/run` is tmpfs, so a stale pointer cannot be represented across a boot. + +The loader resolves the booted slot, then exposes the matching slot directory's +contents in `/run/extensions`. Its unit: + +```ini +[Unit] +Description=Select Trident system extensions for the booted slot +DefaultDependencies=no +After=local-fs.target +Before=systemd-sysext.service +[Service] +Type=oneshot +RemainAfterExit=yes +ExecStart=/usr/bin/trident-sysext-loader +[Install] +WantedBy=sysinit.target +``` + +`DefaultDependencies=no` is required; the implicit `After=sysinit.target` would +place it after the merge and create a cycle. + +`Before=systemd-sysext.service` is the correct edge even on platforms that do +not enable that unit directly. Flatcar-derived systems wire +`ensure-sysext.service` into `sysinit.target.wants` instead, but that unit +carries `BindsTo=systemd-sysext.service` and `After=systemd-sysext.service`: it +runs *after* the merge, and exists to `daemon-reload` and restart targets so +that units shipped inside an extension are picked up. Ordering before it would +therefore permit running after the merge. + +This works because systemd evaluates conditions in `unit_start()` +(`src/core/unit.c`), not at transaction creation, so the loader populates +`/run/extensions` before the merge unit's own `ConditionDirectoryNotEmpty=` is +tested. The same fact rules out an `ExecStartPre=` drop-in, since that condition +gates the whole unit and would skip it while the directories are still empty. + +### Failing closed + +`WantedBy=` creates a `Wants=`, which `systemd.unit(5)` states has no effect on +the transaction when it fails, and `Before=` carries no requirement. The unit +above therefore does **not** fail closed on its own: a loader failure would +leave boot to continue with extensions absent or partially present. + +The merge unit must carry `Requires=trident-sysext-loader.service` via a +drop-in. This must be a deliberate requirement edge rather than forcing +emergency mode, because the store is on shared storage and a storage fault must +not be able to fail both slots and defeat A/B rollback. + +### Selection is a set, not a sequence + +Creating N symlinks exposes a prefix of the intended set if it fails partway, +which matters where a driver and its userspace libraries must move together. +The loader must therefore commit the set atomically: prepare the complete set +in a staging directory under `/run` and rename it into place, or select the +whole directory at once. + +Selecting the whole directory is simpler but means everything at that search +path comes from one slot, so a shared extension can no longer sit beside a +slotted one there. That trade should be settled before implementation. + +### Slot detection + +The loader resolves the slot per platform, selected by a `LOADER_MODE` +environment variable so that Trident does not hardcode one platform's +mechanism. The first mode reads a slot label from the kernel command line, +supplied by a per-slot signed boot artefact. + +A label is preferable to a partition UUID because no mapping table has to stay +in sync with a layout that changes. However, a label is an *assertion*: a +packaging error could pair it with the other slot's device. The loader should +cross-check it against the booted `/usr` backing device and refuse to proceed on +a mismatch. + +Platform-specific conditions belong in platform-specific units. A single unit +carrying `ConditionKernelCommandLine=` for one platform's parameter would +prevent every other mode from ever running, since the condition is evaluated +before the binary reads `LOADER_MODE`. + +### The store needs a manifest + +Globbing `*.raw` cannot distinguish "no extensions configured" from "storage +lost", "mount failed" or "update interrupted", and treating all of those as +success produces a quietly extension-less host. + +Each slot directory should carry a manifest naming the expected images and their +hashes. An absent manifest means the feature is not configured; a present +manifest with missing or unexpected content is an error. This also gives +garbage collection something to act on, which globbing does not. + +## Rollout + +The loader ships in `/usr` and therefore rides the A/B swap. A rollback into a +slot whose `/usr` predates the loader has no selector, and every slotted +extension is absent for that boot. + +Enabling `scope: slotted` therefore requires two phases: first deploy a release +carrying the loader and let it reach both slots, then permit slotted +configuration. Validation must verify that capability rather than assume it, +which is why the relaxed check above tests for a loader and not merely for A/B. ## Relationship to RFC 0789 -[0789](https://github.com/microsoft/trident/pull/789) carries extension images in -COSI files. The two are orthogonal: 0789 changes where the bytes come from, this -changes where they land and how they follow the slot. Either is useful alone, -and a bundled extension can be slotted. +[0789](https://github.com/microsoft/trident/pull/789) carries extension images +in COSI files. The two are orthogonal: 0789 changes where the bytes come from, +this changes where they land and how they follow the slot. Either is useful +alone, and a bundled extension can be slotted. Both touch `validate_extension_images_locations`, in opposite directions: 0789 tightens it into dynamic validation against the storage graph, this adds a second way to satisfy it. Whichever lands first should leave the check expressed as *is this destination slot-scoped*, so the other is an added arm rather than a rewrite. 0789's rollback argument, that extensions revert because the slot's -filesystem reverts, is the A/B volume mechanism described here and holds -unchanged for `scope: shared`. +filesystem reverts, is the A/B volume mechanism and holds unchanged for +`scope: shared`. ## Testing -- Clean install: file in the correct slot directory, symlink resolving, merged - after boot. -- A/B update: the symlink repoints and **the merged content differs from the +- Clean install: image in the correct slot directory, `/run/extensions` link + resolving, merged after boot. +- A/B update: the selection repoints and **the merged content differs from the previous slot**. A presence-only assertion passes in the inert case and must not be used. -- Manual rollback: symlinks and merged content return to the previous slot. -- Automatic rollback: on the fallback boot the extension is absent rather than - stale, and merges after `commit` without a reboot. Both halves asserted; the - first distinguishes safe degradation from serving wrong content. -- Coexistence: a shared and a slotted extension in `/var/lib/extensions`, both - merged, only the slotted one changing across an update. +- **Unsupervised rollback: with no Trident servicing operation in between**, the + previous slot's content is merged again. This is the case the design exists + for. +- Loader failure: boot does not proceed silently with a partial set. +- Ordering: the selection lands before the merge, asserted on the first boot + after an update rather than on a steady-state boot. +- A malformed image in a slot directory is rejected before it reaches + `/run/extensions`. - Validation: each reject row, plus a shared extension off an A/B volume still rejected. - Compatibility: a configuration with no `scope` places byte-identically to - today. A non-default `extensionStore` on a separate mount. + today. ## Counter-arguments -- **"Just add the partitions."** Better where possible. This is for layouts that - cannot, and for sizing decided after install rather than before. +- **"Just add the partitions."** Better where possible: the filesystem provides + the isolation, Trident does nothing at boot, and capacity is contained. This + is for layouts that cannot. Directory slotting gives equivalent naming, not + equivalent failure containment: a store on shared storage can fill and affect + the running slot, which a dedicated pair cannot. +- **"Do it in the initrd."** Flatcar does exactly this and it is the stronger + position in principle, since the merge then precedes unit enumeration and the + `/etc` overlay. It requires rebuilding and re-signing the UKI, which is a + delivery cost rather than a correctness argument, and it does not fit a + platform that cannot modify its initrd. Worth revisiting for extensions needed + before `sysinit.target`, which this design cannot serve. - **"Put it in `/usr`."** Better still, and unavailable without re-sealing - dm-verity and re-signing the UKI. Upstream also forbids extension storage - under `/usr`: a `lowerdir=` that is a child of another fails with `-ELOOP`. + dm-verity and re-signing. Upstream also forbids extension storage under + `/usr`: a `lowerdir=` that is a child of another fails with `-ELOOP`. - **"Use the ESP."** Genuinely slot-scoped via `.efi.extra.d/`, but ESPs are 128 to 256 MiB against payloads in the tens to hundreds of megabytes, and `systemd-stub` places them in the initrd. - **"Doubles storage."** It does. The A/B volume route doubles it too, with the allocation fixed at install. -- **"A symlink is weaker than a partition boundary."** Yes, though both are - reachable by anything that can already write the extension image. -**Prior art.** systemd's versioned directories (`.v/` holding -`_.raw`, highest wins) are the same shape: a directory of -candidates, one selected at discovery. They select by version rather than slot, -so they do not serve this case, but the pattern is systemd's own. +### Rejected mechanisms + +- **`.v` versioned directories.** They select the highest version and stop, with + no fallback when that image fails OS matching, so they cannot hold per-slot + copies. Verified empirically; Flatcar reached the same conclusion. +- **`systemd-tmpfiles`.** No specifier identifies the booted device, partition + or slot, symlink targets are literal text plus specifier expansion, and the + main pass is ordered after the merge. +- **A systemd generator.** Generators run sandboxed with most of the filesystem + read-only, and `/var` is not mounted when they run. +- **Repair after boot.** Any agent running at `multi-user.target` is far too + late to influence the merge. + +### Prior art + +Three shipping systems converge on recomputing the selection at boot rather than +persisting it: + +- **Flatcar**, whose layout is closest to the first consumer, stamps the version + into the filename on shared storage and rewrites the activating symlink in the + initrd from the booted `/usr`'s `os-release`. +- **ChromeOS** stores `dlc_a`/`dlc_b` on shared stateful storage and derives the + slot from the booted partition number on every service start. It also clears + the inactive slot's validity marker *before* writing and re-sets it after, so + a rollback into a half-written slot finds an honest failure rather than a + plausible file. That is worth adopting. +- **Fedora's `sysexts-manager`** filters candidates by the running `VERSION_ID` + and symlinks the winner into `/run/extensions`. + +The counter-example is instructive: Fedora CoreOS activates sysexts statically +across deployments, and its documentation records that after a major rebase the +extensions silently stop matching and are not loaded. ## Open questions -1. **Retention.** Nothing prunes a slot directory when an entry leaves the - configuration. Should `provision()` reconcile, and does that conflict with - keeping the previous slot's files for rollback? -2. **Default path.** `/var/trident/sysexts` against `/var/lib/trident/sysexts`; - the datastore already lives under `/var/lib/trident/` (`constants.rs:30`). -3. **Should `commit` refresh unconditionally?** Refreshing only on a detected - rollback is narrower, but an unconditional refresh is cheap and removes a - branch. -4. **Host Status.** Should it report the slotted set per volume? -5. **SELinux.** Extensions are already unsupported with SELinux enforcing. - Slotting does not change that, but Trident-created symlinks may need +1. **Whole-directory selection or a prepared set?** Both commit atomically. The + former is simpler; the latter allows shared and slotted extensions to share a + search path. +2. **Manifest format and whether it is Trident-internal or part of the API.** +3. **Retention.** Nothing prunes a slot directory when an entry leaves the + configuration. Reconciling in `provision()` conflicts with keeping the + previous slot's files for rollback. +4. **Default store path.** `/var/trident/sysexts` against + `/var/lib/trident/sysexts`; the datastore already lives under the latter. +5. **Confexts.** The same mechanism applies with different search paths, but + `/etc` content is out of scope here, so whether confexts are in scope at all + needs deciding. +6. **SELinux.** Extensions are already unsupported with SELinux enforcing. + Slotting does not change that, but loader-created symlinks in `/run` need labelling, and 0789 proposes moving that check to every servicing type. +7. **Measured boot.** The selected image is outside the UKI's measurements, so + identical measurements can yield different `/usr` contents. Whether that is + acceptable depends on the platform's attestation claims. ## Future possibilities +- **Initrd-phase selection**, for extensions that must be merged before unit + enumeration or before an `/etc` overlay is assembled. +- **Content matching** on platforms with systemd v256 or later and a + build-granular `VERSION_ID`, which needs no loader at all. - Per-volume scoping for other Trident-managed state that is shared today. -- Changing an extension's scope between operations, which would otherwise strand - a copy in the old location. From 2d58c9f333c7da6096a99398f511f458b159262c Mon Sep 17 00:00:00 2001 From: Paco Date: Tue, 13 Oct 2026 16:15:22 -0700 Subject: [PATCH 6/8] docs: unify sysext and /etc selection in one initrd component Both halves resolve the booted slot and expose that slot's content before a consumer reads it, and both must act before the /etc overlay is mounted in the initrd. /run is bind-moved across switch-root, so the sysext symlinks written there remain valid for the later merge in the real root. Resolving the slot once means the two halves cannot disagree about which slot they came from, and it removes the unit-ordering, condition and fail-closed problems a real-root systemd unit carries. /etc payloads are confext-format images mounted as additional lowerdirs on the existing overlay, so /etc stays mutable on systemd 255. Keeping the confext format rather than inventing a tarball means the artefact is already what native systemd-confext consumes on 256 or later; Azure Linux 4.0 ships 258.4. Payloads must declare ID=_any, because native mounting enforces extension-release matching that the custom mount bypasses, and a concrete ID would silently stop merging at the transition. Adds a change inventory across the new component, Trident and ACL with T-shirt estimates for design and implementation, plus suggested sequencing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs/Development/RFCs/0838-slotted-sysexts.md | 485 ++++++++++-------- 1 file changed, 281 insertions(+), 204 deletions(-) diff --git a/docs/Development/RFCs/0838-slotted-sysexts.md b/docs/Development/RFCs/0838-slotted-sysexts.md index 29951c3383..d2ac694860 100644 --- a/docs/Development/RFCs/0838-slotted-sysexts.md +++ b/docs/Development/RFCs/0838-slotted-sysexts.md @@ -1,6 +1,6 @@ # 0838 Slotted Extension Storage -- Date: 2026-10-12 +- Date: 2026-10-13 - RFC PR: [microsoft/trident#838](https://github.com/microsoft/trident/pull/838) - Issue: TBD @@ -11,24 +11,25 @@ configured. That is one way to make an extension follow the slot, and it costs a partition pair per extension directory. This adds a second. An extension marked `scope: slotted` is stored per volume -under a Trident-owned store. A small boot-time component selects the booted -slot's copy into `/run/extensions` before the merge runs. Slot-scoping becomes a -property of the extension rather than of the partition table. +under a Trident-owned store. A small component resolves the booted slot early in +boot and exposes that slot's content before anything consumes it. Slot-scoping +becomes a property of the extension rather than of the partition table. + +The same component serves `/etc` layers, which otherwise have no viable +mechanism on systemd 255. ## Motivation and Goals `validate_extension_images_locations` (`config/host/mod.rs:218`) rejects any -extension directory that does not resolve to a device with A/B capabilities, -raising `ExtensionImageNotOnABVolume`. The reasoning is sound: extensions on -shared storage change the running slot immediately and do not roll back. +extension directory that does not resolve to a device with A/B capabilities. +The reasoning is sound: extensions on shared storage change the running slot +immediately and do not roll back. The cost is that the only way to satisfy it is a dedicated A/B volume pair. Trident's own reference configuration (`tests/e2e_tests/trident_configurations/root-verity/trident-config.yaml`) -spends four partitions and 2.4 GB on it. - -That price buys only slot-scoping, and it is charged in the currency hardest to -change: +spends four partitions and 2.4 GB on it. That price buys only slot-scoping, and +it is charged in the currency hardest to change: - **Partitions are fixed at install.** The size is a guess made before anyone knows what the host will carry; growing it means repartitioning. @@ -46,57 +47,145 @@ on an A/B block device*, one implementation of it. The obvious alternative is to place both slots' copies in one search directory and let `systemd-sysext` merge whichever matches the booted OS. Version mismatches are skipped quietly, so this very nearly works, and on a suitable -platform it would need no new code at all. +platform it needs no new code at all. -It fails for two independent reasons, either of which is sufficient: +It fails for two independent reasons, either sufficient on its own: 1. **Deploying one image under two filenames requires systemd v256.** `path_extract_image_name()` truncates an image name at the first `_` or `+` - (`src/basic/os-util.c`), so `platform_1.raw` and `platform_2.raw` can share a - single inner `extension-release.platform`. That function does not exist - before v256; it was added by commit `37543971` (2024-03-28). On earlier - systemd each copy needs its own `extension-release`, so a signed image must - be repacked per slot. The `user.extension-release.strict` xattr relaxes the - name correspondence but must be set inside the image at build time. + (`src/basic/os-util.c`), letting `platform_1.raw` and `platform_2.raw` share + one inner `extension-release.platform`. That function does not exist before + v256; it arrived in commit `37543971` (2024-03-28). Earlier systemd requires + each copy to carry its own `extension-release`, so a signed image must be + repacked per slot. 2. **The OS version must actually differ between slots.** Matching uses `ID=`, then `SYSEXT_LEVEL=` when defined on both sides, otherwise `VERSION_ID=`. A platform whose `VERSION_ID` is coarser than its build cadence, or which can - deploy the same image to both slots, puts identical values in both slots and - both copies then match. + deploy the same image to both slots, puts identical values in both and both + copies match. Both conditions hold on the first intended consumer. Content matching remains the better mechanism where a platform satisfies both, and this design does not -preclude adopting it there. +preclude adopting it there. See [Future possibilities](#future-possibilities). ### Goals Slot-scoped extensions without a dedicated volume pair; rollback restores the -previous set with no servicing operation; no new partitions, no sealed-`/usr` -change, no re-signing; existing configurations unaffected. +previous set with no servicing operation; a workable `/etc` story on systemd +255; no new partitions, no sealed-`/usr` change, no modification to systemd. ### Non-goals Changing `systemd-sysext` matching semantics. Removing the A/B volume route, -which stays better where a pair is affordable, since the filesystem provides -the isolation and Trident does nothing. Extensions on the ESP. +which stays better where a pair is affordable. Extensions on the ESP. ## Scope -Sysexts and confexts declared under `os.sysexts` and `os.confexts`, on clean +Sysexts, confexts and `/etc` layers declared in the Host Configuration, on clean install and A/B update. Runtime update does not change slot and is unaffected. -**Slotted extensions carry `/usr` and `/opt` content only.** Contributing to -`/etc` is out of scope for this RFC. On a platform that assembles `/etc` as an -overlay whose lowerdir lives inside `/usr`, a sysext can only affect `/etc` if -it is merged before that overlay is built. Reordering the merge ahead of it -would place the `/etc` overlay on top of the `/usr` overlay, consuming the -kernel's entire filesystem stacking budget: `FILESYSTEM_MAX_STACK_DEPTH` is 2 -(`include/linux/fs.h`), enforced in `fs/overlayfs/super.c`, and an overlay's -depth is the maximum of its layers plus one. The `/etc` overlay would reach -depth 2 and any confext merged onto `/etc` afterwards would reach 3 and fail. +## Design + +### Selection is recomputed every boot, never persisted + +When firmware falls back to the previous slot on its own, no Trident code runs. +Anything written during servicing would be stale with nothing to correct it, so +the selection must be derived from what actually booted. + +Verified: Trident's A/B rollback path invokes no subsystems. +`rollback::validate_boot` only reconciles Host Status, and +`manual_rollback::stage_rollback` touches only the pcrlock policy. Only the +runtime path passes `&mut subsystems`. + +### Both halves run in the initrd -The durable fix is to stop sourcing `/etc` defaults from inside `/usr`, which is -a platform change rather than a Trident one. +The `/etc` half has no choice. On a platform that assembles `/etc` as an overlay +whose lowerdir lives inside `/usr`, anything contributing to `/etc` must act +before that overlay is mounted, which happens in the initrd. + +The sysext half can then ride along, because `/run` is bind-moved across +switch-root (`src/shared/switch-root.c`: `{ "/run", MS_BIND, MS_BIND|MS_REC }`). +Symlinks written to `/run/extensions` in the initrd are still present when +`systemd-sysext` merges later in the real root. Their targets must be written as +post-switch-root paths, so they are dangling in the initrd and correct +afterwards. + +Unifying is not merely possible, it is better. The slot is resolved **once**, so +the sysext set and the `/etc` layer cannot disagree about which slot they came +from. It also removes a large amount of accidental complexity that a real-root +systemd unit would carry: + +| Problem with a real-root unit | In the initrd | +| --- | --- | +| `WantedBy=` creates a `Wants=`, which `systemd.unit(5)` states cannot fail the transaction, so the unit does not fail closed | No unit; failing in dracut is native | +| Ordering anchor is subtle: `ensure-sysext.service` carries `BindsTo=` and `After=systemd-sysext.service`, so it runs *after* the merge and is the wrong edge | Nothing to order against | +| A platform-specific `ConditionKernelCommandLine=` would block every other mode from running | No unit condition | +| Unit enablement needs a preset or scriptlet | No unit to enable | +| Store must be available at `local-fs.target` | `/sysroot` is mounted by definition | + +A real-root unit remains the fallback for platforms that cannot modify their +initrd. The mode selector therefore chooses *where* the loader runs as well as +how it detects the slot. + +### Why `/etc` payloads are confext images + +`systemd-confext` is the natural mechanism and is unusable here: on systemd 255 +it mounts `/etc` read-only. Mutable overlays arrived in v256; Flatcar's backport +is 1310 lines across five files, and modifying systemd is out of scope. + +The resolution is to keep the **format** and replace the **mount**. Payloads are +confext-format DDIs, carrying +`/etc/extension-release.d/extension-release.` (`src/basic/os-util.c`). +The loader mounts them read-only and adds them as additional lowerdirs to the +existing `/etc` overlay, whose upperdir is unchanged, so `/etc` stays mutable. + +This is preferred over tarballs or plain directory trees because: + +- **It is forward-compatible.** The payload is already a valid confext. When a + platform reaches systemd 256 or later, the identical artefact is handed to + native `systemd-confext --mutable=yes` and the custom mounting is deleted. The + custom mounting is a bridge, not a fork. Azure Linux 4.0 ships systemd 258.4, + so this transition is concrete rather than hypothetical. + + Two things do not change at that transition, and both matter. The loader + itself remains, because systemd has no concept of A/B slots: *selection* stays + the platform's job, and only *mounting* becomes native. And native mounting + reintroduces `extension-release` matching, which the custom lowerdir mount + bypasses entirely. +- **It reuses everything.** Same build and signing pipeline as sysexts, same + Trident download and hash-verification path, verity available by DDI + convention. +- **It is atomic.** A loop mount either appears or does not. A tarball + extraction can half-fail, and would also pay extraction cost on every boot. +- **SELinux labels travel in the filesystem image**, rather than in tar metadata. + +#### Payloads must declare `ID=_any` + +Because slot selection is the platform's job and not systemd's, the payload must +not additionally be subject to OS-version matching. A payload declaring a +concrete `ID=` would merge under the custom mount, which ignores matching, and +then silently stop merging the day a platform switches to native confext +mounting. That is the failure mode this RFC exists to prevent, reappearing at +the point of transition. + +Declaring `ID=_any` bypasses matching on every systemd version. Selection +remains entirely with the loader, which is what slot-scoping requires anyway, +and the move to native mounting changes no artefact. Deciding this late would +mean rebuilding every payload at transition time, so it is specified here. + +Adding lowerdirs does not increase filesystem stacking depth, because depth is +the maximum of the layers plus one and every lowerdir sits on a plain +filesystem. This matters: *reordering* the sysext merge ahead of the `/etc` +overlay would instead place `/etc` on top of the `/usr` overlay, reaching depth +2, and any confext merged afterwards would reach 3 and fail. +`FILESYSTEM_MAX_STACK_DEPTH` is 2 (`include/linux/fs.h`), enforced in +`fs/overlayfs/super.c`. + +### Selection is a set, not a sequence + +Creating N symlinks exposes a prefix of the intended set if it fails partway, +which matters where a driver and its userspace libraries must move together. +The loader prepares the complete set and commits it with a single rename. ## Public API Design @@ -118,10 +207,9 @@ os: ```yaml os: # Root of the per-volume extension store. - # Default /var/trident/sysexts. Must be on storage an A/B update does not - # replace, must not be a systemd-sysext search path, and must be on a - # filesystem available at local-fs.target. - extensionStore: /mnt/data/trident/sysexts + # Default /var/trident/extensions. Must be on storage an A/B update does not + # replace, and must not be a systemd-sysext search path. + extensionStore: /mnt/data/trident/extensions sysexts: # unchanged, and still the default @@ -132,15 +220,26 @@ os: - url: https://example.com/platform.raw sha384: scope: slotted + + # new: confext-format images contributed as /etc overlay layers + etcLayers: + - url: https://example.com/platform-etc.raw + sha384: + scope: slotted ``` | `scope` | Where the bytes live | Where it is activated | | --- | --- | --- | | `shared` (default) | At `path`. | At `path`. Today's behaviour. | -| `slotted` | `/{a,b}/{name}.raw` | `/run/extensions/`, by the loader | +| `slotted` | `/{a,b}/` | `/run/extensions`, or as an `/etc` lowerdir | + +A slotted extension always surfaces in a loader-owned location, so `path` does +not apply to it and is rejected. -A slotted extension always surfaces in `/run/extensions`, so `path` does not -apply to it and is rejected. +`etcLayers` is a separate list rather than a `scope` on `os.confexts`, because +the two are activated by different mechanisms and only one of them is +`systemd-confext`. If a platform later gains mutable confext support, entries +can migrate to `os.confexts` unchanged. ### The validation change @@ -153,9 +252,8 @@ becomes *the destination is slot-scoped*, satisfied either way: | `abUpdate`, `scope: shared` | `path` must be on an A/B volume. Unchanged. | | `abUpdate`, `scope: slotted` | `path` need not be; the store plus the loader provide the scoping. | -This is the substantive behavioural change. It is only safe where the platform -actually provides a loader, so validation must check a **platform capability**, -not merely that A/B is configured. See [Rollout](#rollout). +This is only safe where the platform actually provides a loader, so validation +must test a **platform capability**, not merely that A/B is configured. ### Validation @@ -167,49 +265,38 @@ not merely that A/B is configured. See [Rollout](#rollout). | `scope: slotted`, no loader capability | Reject. See [Rollout](#rollout). | | `extensionStore` inside a search path | Reject: both slots' copies would be discovered. | | `extensionStore` on an A/B volume | Reject: it must outlive the slot it describes. | -| `extensionStore` not available at `local-fs.target` | Reject: the loader runs before anything later. | -| Two slotted entries resolving to the same basename | Reject: they would collide in `/run/extensions`. | +| Two slotted entries resolving to the same basename | Reject: they would collide. | +| `etcLayers` entry without confext metadata | Reject at deploy time, after mount. | ### Backward compatibility -Both fields default, so existing Host Configurations parse and behave +All new fields default, so existing Host Configurations parse and behave identically. The relaxed check is strictly more permissive, so anything valid -today stays valid. The schema gains two optional properties. - -### Alternatives rejected - -- **Sentinel `path`** (`//x.raw`): leaks internal layout, invites a - literal `a` or `b`. -- **Separate `os.slottedSysexts` list**: duplicates the type and its validation. -- **Infer from `extension-release`**: inside the image, so unavailable at static - validation, and it removes the operator's choice to slot an `ID=_any` - extension deliberately. +today stays valid. The schema gains three optional properties. ## Implementation ``` -/var/trident/sysexts/ -├── a/platform.raw -└── b/platform.raw - -/run/extensions/ -└── platform.raw -> /var/trident/sysexts/a/platform.raw # recreated each boot +/ +├── a/ +│ ├── manifest.json +│ ├── sysext/platform.raw +│ └── etc/platform-etc.raw +└── b/ ... + +/run/extensions/platform.raw -> /a/sysext/platform.raw +/etc overlay lowerdir list += ``` The store is outside every search path, so its contents are invisible to `systemd-sysext` until selected. Both slot directories persist across an update, so rollback needs no re-download. -Symlinks are followed: `image_discover` opens each search path with -`chase_and_opendir(path, root, CHASE_PREFIX_ROOT, ..)` and stats entries with -`flags = 0` on a running system, with an upstream comment explicitly permitting -symlinks into the search path (`src/shared/discover-image.c`). - -### Servicing writes the store, and must write it atomically +### Servicing writes the store, atomically The extensions subsystem already runs on clean install and A/B update via `provision()`, documented as "migrate state from A-partition to B-partition (or -vice versa)". For a slotted entry it resolves the target volume with +vice versa)". It resolves the target volume with `EngineContext::get_ab_update_volume()`, already used to select the per-slot verity addon, and writes into `//`. @@ -221,116 +308,106 @@ Directory-type images are tolerated; `.raw` are not. Trident today falls back to a non-atomic `fs::copy` when `rename` fails across a filesystem boundary (`crates/trident/src/subsystems/extensions/mod.rs:525-540`). -Staging must move to a temporary file on the destination's own filesystem, -verified and `fsync`ed, then renamed. This is a prerequisite, not a -consequence. - -### Selection happens at boot - -Selection is recomputed on every boot from the slot that actually booted. This -is what makes unsupervised rollback correct: when firmware falls back on its -own, no Trident code runs, so any decision written during servicing would be -stale with nothing to correct it. - -`/run` is tmpfs, so a stale pointer cannot be represented across a boot. - -The loader resolves the booted slot, then exposes the matching slot directory's -contents in `/run/extensions`. Its unit: - -```ini -[Unit] -Description=Select Trident system extensions for the booted slot -DefaultDependencies=no -After=local-fs.target -Before=systemd-sysext.service -[Service] -Type=oneshot -RemainAfterExit=yes -ExecStart=/usr/bin/trident-sysext-loader -[Install] -WantedBy=sysinit.target -``` - -`DefaultDependencies=no` is required; the implicit `After=sysinit.target` would -place it after the merge and create a cycle. - -`Before=systemd-sysext.service` is the correct edge even on platforms that do -not enable that unit directly. Flatcar-derived systems wire -`ensure-sysext.service` into `sysinit.target.wants` instead, but that unit -carries `BindsTo=systemd-sysext.service` and `After=systemd-sysext.service`: it -runs *after* the merge, and exists to `daemon-reload` and restart targets so -that units shipped inside an extension are picked up. Ordering before it would -therefore permit running after the merge. +This is a prerequisite fix, not a consequence of this design. -This works because systemd evaluates conditions in `unit_start()` -(`src/core/unit.c`), not at transaction creation, so the loader populates -`/run/extensions` before the merge unit's own `ConditionDirectoryNotEmpty=` is -tested. The same fact rules out an `ExecStartPre=` drop-in, since that condition -gates the whole unit and would skip it while the directories are still empty. - -### Failing closed - -`WantedBy=` creates a `Wants=`, which `systemd.unit(5)` states has no effect on -the transaction when it fails, and `Before=` carries no requirement. The unit -above therefore does **not** fail closed on its own: a loader failure would -leave boot to continue with extensions absent or partially present. - -The merge unit must carry `Requires=trident-sysext-loader.service` via a -drop-in. This must be a deliberate requirement edge rather than forcing -emergency mode, because the store is on shared storage and a storage fault must -not be able to fail both slots and defeat A/B rollback. - -### Selection is a set, not a sequence +### The store carries a manifest -Creating N symlinks exposes a prefix of the intended set if it fails partway, -which matters where a driver and its userspace libraries must move together. -The loader must therefore commit the set atomically: prepare the complete set -in a staging directory under `/run` and rename it into place, or select the -whole directory at once. +Globbing `*.raw` cannot distinguish "no extensions configured" from "storage +lost", "mount failed" or "update interrupted", and treating all of those as +success produces a quietly extension-less host. -Selecting the whole directory is simpler but means everything at that search -path comes from one slot, so a shared extension can no longer sit beside a -slotted one there. That trade should be settled before implementation. +Each slot directory carries a manifest naming the expected images and hashes. An +absent manifest means the feature is not configured; a present manifest with +missing or unexpected content is an error. This also gives garbage collection +something to act on. ### Slot detection -The loader resolves the slot per platform, selected by a `LOADER_MODE` -environment variable so that Trident does not hardcode one platform's -mechanism. The first mode reads a slot label from the kernel command line, -supplied by a per-slot signed boot artefact. +The loader resolves the slot per platform, selected by a mode setting so that +Trident does not hardcode one platform's mechanism. The first mode reads a slot +label from the kernel command line, supplied by a per-slot signed boot artefact. A label is preferable to a partition UUID because no mapping table has to stay in sync with a layout that changes. However, a label is an *assertion*: a -packaging error could pair it with the other slot's device. The loader should -cross-check it against the booted `/usr` backing device and refuse to proceed on -a mismatch. +packaging error could pair it with the other slot's device. The loader +cross-checks it against the booted `/usr` backing device and refuses to proceed +on a mismatch. + +## Change inventory + +Sizes are T-shirt estimates covering design and implementation separately. Risk +reflects how much is unknown rather than how much work there is. + +### New component + +| Change | Design | Impl | Risk | Notes | +| --- | :-: | :-: | :-: | --- | +| `trident-sysext-loader` crate, slot detection and set commit | S | M | Low | std plus `anyhow`; logic is pure given cmdline, store and target, so unit-testable without root or a VM | +| `/etc` layer mounting and lowerdir emission | M | M | Med | Loop-mount in the initrd; interaction with the existing overlay needs care | +| Manifest format and verification | M | S | Low | Decide internal versus API-visible | + +### Trident + +| Change | Design | Impl | Risk | Notes | +| --- | :-: | :-: | :-: | --- | +| **Atomic extension placement** | S | S | Low | Prerequisite bug fix; live defect today, independent of this RFC | +| `scope` and `extensionStore` API | S | S | Low | Additive, both default | +| `etcLayers` API | M | S | Med | New concept; shape depends on confext migration path | +| Relax `validate_extension_images_locations` | M | S | Med | Must test loader capability, not just A/B | +| Subsystem writes slotted entries per volume | S | M | Low | `provision()` already has the target volume | +| Manifest generation | S | S | Low | | +| Package loader and dracut module in the RPM | S | S | Low | `%files acl` already ships a second binary | +| Schema regeneration | S | S | Low | Mechanical | + +### ACL + +| Change | Design | Impl | Risk | Notes | +| --- | :-: | :-: | :-: | --- | +| New dracut module to install and run the loader | S | S | Low | `dracut-acl-selinux-toggle` is an existing precedent | +| Patch `bootengine` to add `/etc` lowerdirs | S | M | Med | Third patch alongside two already carried; the mount itself is one line, validation is the work | +| Order the loader before the `/etc` overlay mount | S | S | Low | Same initrd window as `initrd-setup-root` | +| Review `overlaycleanup()` interaction | M | S | Med | It compares only against the distro lower layer, so extension-provided defaults will not auto-prune | +| Package catalog entry | S | S | Low | | +| Initrd size and UKI rebuild | S | S | Med | Re-signing already happens every build; size growth is the open part | + +### Not affected + +| Component | Why | +| --- | --- | +| systemd / Azure Linux | Nothing. Avoiding a systemd change is the point of this route | +| AgentBaker, AKS RP | No interaction | +| Image Customizer | None unless RFC 0789 lands, and then only at the shared validation | -Platform-specific conditions belong in platform-specific units. A single unit -carrying `ConditionKernelCommandLine=` for one platform's parameter would -prevent every other mode from ever running, since the condition is evaluated -before the binary reads `LOADER_MODE`. +### Cross-cutting -### The store needs a manifest +| Change | Design | Impl | Risk | Notes | +| --- | :-: | :-: | :-: | --- | +| Rollout and capability gating | M | S | Med | Depends on whether the initrd is per-slot; see [Rollout](#rollout) | +| Boot and rollback end-to-end tests | M | L | High | Unsupervised rollback cannot be asserted from unit tests | +| SELinux policy | M | M | High | Extensions are already unsupported with SELinux enforcing | -Globbing `*.raw` cannot distinguish "no extensions configured" from "storage -lost", "mount failed" or "update interrupted", and treating all of those as -success produces a quietly extension-less host. +### Suggested sequencing -Each slot directory should carry a manifest naming the expected images and their -hashes. An absent manifest means the feature is not configured; a present -manifest with missing or unexpected content is an error. This also gives -garbage collection something to act on, which globbing does not. +1. **Atomic placement fix.** Standalone, lands independently, worth doing + regardless of this RFC. +2. Loader crate and dracut module, sysext half only, behind no API change. +3. Trident API and the validation relaxation. +4. `/etc` layer half, including the `bootengine` patch. Separable; ships after. +5. Rollout gating, once the per-slot-initrd question is settled. ## Rollout -The loader ships in `/usr` and therefore rides the A/B swap. A rollback into a -slot whose `/usr` predates the loader has no selector, and every slotted -extension is absent for that boot. +If the initrd is shared between slots, loader capability is never +slot-dependent and no staged rollout is required. + +If it is per-slot, a rollback into a slot whose initrd predates the loader has +no selector, and every slotted extension is absent for that boot. Enabling +`scope: slotted` then requires two phases: deploy a release carrying the loader +and let it reach both slots, then permit slotted configuration. Validation must +verify that capability rather than assume it. -Enabling `scope: slotted` therefore requires two phases: first deploy a release -carrying the loader and let it reach both slots, then permit slotted -configuration. Validation must verify that capability rather than assume it, -which is why the relaxed check above tests for a loader and not merely for A/B. +This is an open question, listed below, and it changes the work in the table +above. ## Relationship to RFC 0789 @@ -343,25 +420,22 @@ Both touch `validate_extension_images_locations`, in opposite directions: 0789 tightens it into dynamic validation against the storage graph, this adds a second way to satisfy it. Whichever lands first should leave the check expressed as *is this destination slot-scoped*, so the other is an added arm rather than a -rewrite. 0789's rollback argument, that extensions revert because the slot's -filesystem reverts, is the A/B volume mechanism and holds unchanged for -`scope: shared`. +rewrite. ## Testing -- Clean install: image in the correct slot directory, `/run/extensions` link - resolving, merged after boot. +- Clean install: image in the correct slot directory, selection resolving, + merged after boot. - A/B update: the selection repoints and **the merged content differs from the previous slot**. A presence-only assertion passes in the inert case and must not be used. -- **Unsupervised rollback: with no Trident servicing operation in between**, the +- **Unsupervised rollback, with no Trident servicing operation in between**: the previous slot's content is merged again. This is the case the design exists for. +- `/etc` layer: a slotted layer is visible in `/etc`, `/etc` remains writable, + and a write to a file the layer provides persists across reboot. - Loader failure: boot does not proceed silently with a partial set. -- Ordering: the selection lands before the merge, asserted on the first boot - after an update rather than on a steady-state boot. -- A malformed image in a slot directory is rejected before it reaches - `/run/extensions`. +- A malformed image in a slot directory is rejected before it is selected. - Validation: each reject row, plus a shared extension off an A/B volume still rejected. - Compatibility: a configuration with no `scope` places byte-identically to @@ -370,22 +444,19 @@ filesystem reverts, is the A/B volume mechanism and holds unchanged for ## Counter-arguments - **"Just add the partitions."** Better where possible: the filesystem provides - the isolation, Trident does nothing at boot, and capacity is contained. This - is for layouts that cannot. Directory slotting gives equivalent naming, not + the isolation, nothing runs at boot, and capacity is contained. This is for + layouts that cannot. Directory slotting gives equivalent naming, not equivalent failure containment: a store on shared storage can fill and affect the running slot, which a dedicated pair cannot. -- **"Do it in the initrd."** Flatcar does exactly this and it is the stronger - position in principle, since the merge then precedes unit enumeration and the - `/etc` overlay. It requires rebuilding and re-signing the UKI, which is a - delivery cost rather than a correctness argument, and it does not fit a - platform that cannot modify its initrd. Worth revisiting for extensions needed - before `sysinit.target`, which this design cannot serve. - **"Put it in `/usr`."** Better still, and unavailable without re-sealing dm-verity and re-signing. Upstream also forbids extension storage under `/usr`: a `lowerdir=` that is a child of another fails with `-ELOOP`. - **"Use the ESP."** Genuinely slot-scoped via `.efi.extra.d/`, but ESPs are 128 to 256 MiB against payloads in the tens to hundreds of megabytes, and `systemd-stub` places them in the initrd. +- **"Backport mutable confexts."** It exists, and it is 1310 lines across five + files of someone else's component. This route reaches the same place without + carrying that, and converges on native confexts later at zero payload cost. - **"Doubles storage."** It does. The A/B volume route doubles it too, with the allocation fixed at install. @@ -399,6 +470,8 @@ filesystem reverts, is the A/B volume mechanism and holds unchanged for main pass is ordered after the merge. - **A systemd generator.** Generators run sandboxed with most of the filesystem read-only, and `/var` is not mounted when they run. +- **Reordering the sysext merge ahead of the `/etc` overlay.** Exhausts the + filesystem stacking budget and breaks confexts outright. - **Repair after boot.** Any agent running at `multi-user.target` is far too late to influence the merge. @@ -409,12 +482,13 @@ persisting it: - **Flatcar**, whose layout is closest to the first consumer, stamps the version into the filename on shared storage and rewrites the activating symlink in the - initrd from the booted `/usr`'s `os-release`. + initrd from the booted `/usr`'s `os-release`. It has since moved extension + mounting wholesale into the initrd and deleted its late-merge workaround. - **ChromeOS** stores `dlc_a`/`dlc_b` on shared stateful storage and derives the - slot from the booted partition number on every service start. It also clears - the inactive slot's validity marker *before* writing and re-sets it after, so - a rollback into a half-written slot finds an honest failure rather than a - plausible file. That is worth adopting. + slot from the booted partition number on every service start. It clears the + inactive slot's validity marker *before* writing and re-sets it after, so a + rollback into a half-written slot finds an honest failure rather than a + plausible file. Worth adopting. - **Fedora's `sysexts-manager`** filters candidates by the running `VERSION_ID` and symlinks the winner into `/run/extensions`. @@ -424,29 +498,32 @@ extensions silently stop matching and are not loaded. ## Open questions -1. **Whole-directory selection or a prepared set?** Both commit atomically. The +1. **Is the initrd per-slot or shared?** Decides whether a staged rollout is + needed at all, and changes the estimates above. +2. **Whole-directory selection or a prepared set?** Both commit atomically. The former is simpler; the latter allows shared and slotted extensions to share a search path. -2. **Manifest format and whether it is Trident-internal or part of the API.** -3. **Retention.** Nothing prunes a slot directory when an entry leaves the +3. **Manifest format**, and whether it is Trident-internal or API-visible. +4. **Retention.** Nothing prunes a slot directory when an entry leaves the configuration. Reconciling in `provision()` conflicts with keeping the previous slot's files for rollback. -4. **Default store path.** `/var/trident/sysexts` against - `/var/lib/trident/sysexts`; the datastore already lives under the latter. -5. **Confexts.** The same mechanism applies with different search paths, but - `/etc` content is out of scope here, so whether confexts are in scope at all - needs deciding. -6. **SELinux.** Extensions are already unsupported with SELinux enforcing. - Slotting does not change that, but loader-created symlinks in `/run` need - labelling, and 0789 proposes moving that check to every servicing type. -7. **Measured boot.** The selected image is outside the UKI's measurements, so +5. **Default store path.** `/var/trident/extensions` against + `/var/lib/trident/extensions`; the datastore already lives under the latter. +6. **Measured boot.** The selected image is outside the UKI's measurements, so identical measurements can yield different `/usr` contents. Whether that is acceptable depends on the platform's attestation claims. +7. **SELinux.** Extensions are already unsupported with SELinux enforcing, which + is a precondition this RFC does not resolve. ## Future possibilities -- **Initrd-phase selection**, for extensions that must be merged before unit - enumeration or before an `/etc` overlay is assembled. +- **Native confext mounting.** On systemd 256 or later, the loader places the + selected payload in a confext search path and `systemd-confext --mutable=yes` + mounts it, replacing the custom lowerdir handling. The loader still performs + selection, and the payloads do not change provided they declare `ID=_any`. + Azure Linux 4.0 ships systemd 258.4. - **Content matching** on platforms with systemd v256 or later and a build-granular `VERSION_ID`, which needs no loader at all. +- **Initrd-phase sysext merging**, as Flatcar now does, for extensions that must + be present before unit enumeration. - Per-volume scoping for other Trident-managed state that is shared today. From 710d82772ef107e2d312169f798f98db4daa48fc Mon Sep 17 00:00:00 2001 From: Paco Date: Wed, 14 Oct 2026 10:23:11 -0700 Subject: [PATCH 7/8] docs: rebase 0838 on RFC 0789 as a dependency 0789 was rewritten and now names this RFC in three places: the optional path discussion, destination validation, and an open question asking how a bundled extension requests slotted placement. This RFC now answers that question rather than sitting beside it. The common case is a COSI-bundled extension that is slot-scoped, since an extension shipped with the OS image is already tied to it. An independently hosted extension placed per slot is the unusual combination. Removes what 0789 delivers: the invented manifest.json is replaced by 0789's Deployed Extension Inventory, the store is keyed by (kind, id) instead of filenames, and the effective set, resolved-placement validation, atomic staging and SELinux check relocation are referenced rather than restated. Adds scope: shared|slotted on both the COSI ExtensionImage and the Host Configuration Extension, spelled identically so one concept has one name, and mutually exclusive with path so placement never changes implicitly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs/Development/RFCs/0838-slotted-sysexts.md | 432 +++++++++--------- 1 file changed, 216 insertions(+), 216 deletions(-) diff --git a/docs/Development/RFCs/0838-slotted-sysexts.md b/docs/Development/RFCs/0838-slotted-sysexts.md index d2ac694860..efbc05877b 100644 --- a/docs/Development/RFCs/0838-slotted-sysexts.md +++ b/docs/Development/RFCs/0838-slotted-sysexts.md @@ -1,35 +1,45 @@ # 0838 Slotted Extension Storage -- Date: 2026-10-13 +- Date: 2026-10-14 - RFC PR: [microsoft/trident#838](https://github.com/microsoft/trident/pull/838) -- Issue: TBD +- Builds on: [RFC 0789](https://github.com/microsoft/trident/pull/789) ## Summary -Trident requires every extension destination to sit on an A/B volume when A/B is -configured. That is one way to make an extension follow the slot, and it costs a -partition pair per extension directory. +[RFC 0789](https://github.com/microsoft/trident/pull/789) carries extension +images inside the COSI, so an OS-coupled extension ships and rolls out with the +OS. It leaves one question open: such an extension still has to land somewhere +that follows the A/B slot, and today the only way to satisfy that is a dedicated +A/B partition pair. -This adds a second. An extension marked `scope: slotted` is stored per volume -under a Trident-owned store. A small component resolves the booted slot early in -boot and exposes that slot's content before anything consumes it. Slot-scoping -becomes a property of the extension rather than of the partition table. - -The same component serves `/etc` layers, which otherwise have no viable +This RFC answers it. A bundled extension can request Trident-managed, per-slot +storage, and a boot-time component selects the booted slot's copy before +anything consumes it. It also supplies `/etc` layers, which have no viable mechanism on systemd 255. +An extension bundled in the COSI is already tied to the OS image it shipped +with. Slotted placement is the natural destination for it, and the two features +are expected to be used together in the common case. An independently hosted +extension fetched over OCI and placed per slot is the unusual combination, not +the default. + ## Motivation and Goals -`validate_extension_images_locations` (`config/host/mod.rs:218`) rejects any -extension directory that does not resolve to a device with A/B capabilities. -The reasoning is sound: extensions on shared storage change the running slot -immediately and do not roll back. +0789 establishes that OS-coupled extensions belong in the COSI. It also +establishes, in [Destination +Validation](https://github.com/microsoft/trident/pull/789), that a bundled image +written to shared storage changes the running slot immediately and defeats +rollback, so placement must be slot-scoped. -The cost is that the only way to satisfy it is a dedicated A/B volume pair. -Trident's own reference configuration +Trident has exactly one mechanism for that today, and +`validate_extension_images_locations` (`config/host/mod.rs:218`) enforces it: the +destination must resolve to a device with A/B capabilities. The only way to +satisfy it is a dedicated A/B volume pair. Trident's own reference configuration (`tests/e2e_tests/trident_configurations/root-verity/trident-config.yaml`) -spends four partitions and 2.4 GB on it. That price buys only slot-scoping, and -it is charged in the currency hardest to change: +spends four partitions and 2.4 GB on exactly this. + +That price buys only slot-scoping, and it is charged in the currency hardest to +change: - **Partitions are fixed at install.** The size is a guess made before anyone knows what the host will carry; growing it means repartitioning. @@ -44,7 +54,7 @@ on an A/B block device*, one implementation of it. ### Why content matching alone is not sufficient -The obvious alternative is to place both slots' copies in one search directory +A tempting alternative is to place both slots' copies in one search directory and let `systemd-sysext` merge whichever matches the booted OS. Version mismatches are skipped quietly, so this very nearly works, and on a suitable platform it needs no new code at all. @@ -53,11 +63,10 @@ It fails for two independent reasons, either sufficient on its own: 1. **Deploying one image under two filenames requires systemd v256.** `path_extract_image_name()` truncates an image name at the first `_` or `+` - (`src/basic/os-util.c`), letting `platform_1.raw` and `platform_2.raw` share - one inner `extension-release.platform`. That function does not exist before - v256; it arrived in commit `37543971` (2024-03-28). Earlier systemd requires - each copy to carry its own `extension-release`, so a signed image must be - repacked per slot. + (`src/basic/os-util.c`), letting two copies share one inner + `extension-release`. That function arrived in commit `37543971` (2024-03-28). + Earlier systemd requires each copy to carry its own, so a signed image must + be repacked per slot. 2. **The OS version must actually differ between slots.** Matching uses `ID=`, then `SYSEXT_LEVEL=` when defined on both sides, otherwise `VERSION_ID=`. A platform whose `VERSION_ID` is coarser than its build cadence, or which can @@ -65,24 +74,44 @@ It fails for two independent reasons, either sufficient on its own: copies match. Both conditions hold on the first intended consumer. Content matching remains -the better mechanism where a platform satisfies both, and this design does not -preclude adopting it there. See [Future possibilities](#future-possibilities). +the better mechanism where a platform satisfies both; see +[Future possibilities](#future-possibilities). ### Goals -Slot-scoped extensions without a dedicated volume pair; rollback restores the -previous set with no servicing operation; a workable `/etc` story on systemd -255; no new partitions, no sealed-`/usr` change, no modification to systemd. +Give bundled extensions a slot-scoped destination without a dedicated volume +pair; restore the previous set on rollback with no servicing operation; provide +a workable `/etc` story on systemd 255; require no new partitions, no +sealed-`/usr` change and no modification to systemd. ### Non-goals -Changing `systemd-sysext` matching semantics. Removing the A/B volume route, -which stays better where a pair is affordable. Extensions on the ESP. +Changing `systemd-sysext` matching semantics. Replacing the A/B volume route, +which stays better where a pair is affordable. Extensions on the ESP. Anything +0789 already covers, including the effective set, the deployed inventory and +extension-only updates. + +## Dependencies + +**RFC 0789 is a prerequisite.** This RFC adds a placement policy to the model +0789 defines and does not restate it. In particular it relies on: + +| From 0789 | Used here as | +| --- | --- | +| `extensions.sysexts` / `extensions.confexts` in COSI 1.3 | The delivery mechanism for slotted content | +| `(kind, id)` as stable extension identity | The store key, instead of a filename | +| Deployed Extension Inventory in Host Status | The record of what is deployed per slot; this RFC adds no separate manifest | +| Effective set computed from COSI plus Host Configuration | The set the loader selects from | +| Resolved-placement destination validation | The hook where the slotted arm is added | +| Atomic staging on the destination filesystem | A prerequisite shared by both RFCs | +| SELinux check moved into validation for every servicing type | Inherited unchanged | -## Scope +0789's prerequisites apply transitively, including the `set_up_extensions` +replacement bug and durable staged payloads. -Sysexts, confexts and `/etc` layers declared in the Host Configuration, on clean -install and A/B update. Runtime update does not change slot and is unaffected. +Bundled extensions under +[disk streaming](../../Explanation/Disk-Streaming.md) are refused by 0789; that +restriction is inherited. ## Design @@ -90,7 +119,7 @@ install and A/B update. Runtime update does not change slot and is unaffected. When firmware falls back to the previous slot on its own, no Trident code runs. Anything written during servicing would be stale with nothing to correct it, so -the selection must be derived from what actually booted. +the selection must derive from what actually booted. Verified: Trident's A/B rollback path invokes no subsystems. `rollback::validate_boot` only reconciles Host Status, and @@ -99,33 +128,31 @@ runtime path passes `&mut subsystems`. ### Both halves run in the initrd -The `/etc` half has no choice. On a platform that assembles `/etc` as an overlay -whose lowerdir lives inside `/usr`, anything contributing to `/etc` must act -before that overlay is mounted, which happens in the initrd. +The `/etc` half has no choice. Where `/etc` is assembled as an overlay whose +lowerdir lives inside `/usr`, anything contributing to it must act before that +overlay is mounted, which happens in the initrd. -The sysext half can then ride along, because `/run` is bind-moved across -switch-root (`src/shared/switch-root.c`: `{ "/run", MS_BIND, MS_BIND|MS_REC }`). -Symlinks written to `/run/extensions` in the initrd are still present when -`systemd-sysext` merges later in the real root. Their targets must be written as -post-switch-root paths, so they are dangling in the initrd and correct -afterwards. +The sysext half rides along, because `/run` is bind-moved across switch-root +(`src/shared/switch-root.c`: `{ "/run", MS_BIND, MS_BIND|MS_REC }`). Symlinks +written to `/run/extensions` in the initrd are still present when +`systemd-sysext` merges in the real root. Their targets are written as +post-switch-root paths, so they are dangling in the initrd and correct after. -Unifying is not merely possible, it is better. The slot is resolved **once**, so -the sysext set and the `/etc` layer cannot disagree about which slot they came -from. It also removes a large amount of accidental complexity that a real-root -systemd unit would carry: +Unifying is better than running them separately: the slot is resolved **once**, +so the sysext set and the `/etc` layer cannot disagree about which slot they +came from. It also removes complexity a real-root systemd unit would carry: | Problem with a real-root unit | In the initrd | | --- | --- | | `WantedBy=` creates a `Wants=`, which `systemd.unit(5)` states cannot fail the transaction, so the unit does not fail closed | No unit; failing in dracut is native | -| Ordering anchor is subtle: `ensure-sysext.service` carries `BindsTo=` and `After=systemd-sysext.service`, so it runs *after* the merge and is the wrong edge | Nothing to order against | -| A platform-specific `ConditionKernelCommandLine=` would block every other mode from running | No unit condition | +| Ordering is subtle: `ensure-sysext.service` carries `BindsTo=` and `After=systemd-sysext.service`, so it runs *after* the merge and is the wrong anchor | Nothing to order against | +| A platform-specific `ConditionKernelCommandLine=` would block every other mode | No unit condition | | Unit enablement needs a preset or scriptlet | No unit to enable | | Store must be available at `local-fs.target` | `/sysroot` is mounted by definition | A real-root unit remains the fallback for platforms that cannot modify their -initrd. The mode selector therefore chooses *where* the loader runs as well as -how it detects the slot. +initrd, so the mode selector chooses *where* the loader runs as well as how it +detects the slot. ### Why `/etc` payloads are confext images @@ -133,51 +160,47 @@ how it detects the slot. it mounts `/etc` read-only. Mutable overlays arrived in v256; Flatcar's backport is 1310 lines across five files, and modifying systemd is out of scope. -The resolution is to keep the **format** and replace the **mount**. Payloads are -confext-format DDIs, carrying -`/etc/extension-release.d/extension-release.` (`src/basic/os-util.c`). -The loader mounts them read-only and adds them as additional lowerdirs to the -existing `/etc` overlay, whose upperdir is unchanged, so `/etc` stays mutable. - -This is preferred over tarballs or plain directory trees because: - -- **It is forward-compatible.** The payload is already a valid confext. When a - platform reaches systemd 256 or later, the identical artefact is handed to - native `systemd-confext --mutable=yes` and the custom mounting is deleted. The - custom mounting is a bridge, not a fork. Azure Linux 4.0 ships systemd 258.4, - so this transition is concrete rather than hypothetical. - - Two things do not change at that transition, and both matter. The loader - itself remains, because systemd has no concept of A/B slots: *selection* stays - the platform's job, and only *mounting* becomes native. And native mounting - reintroduces `extension-release` matching, which the custom lowerdir mount - bypasses entirely. +Keep the **format** and replace the **mount**. Payloads are confext-format DDIs +carrying `/etc/extension-release.d/extension-release.` +(`src/basic/os-util.c`). The loader mounts them read-only and adds them as +additional lowerdirs to the existing `/etc` overlay, whose upperdir is +unchanged, so `/etc` stays mutable. + +Preferred over tarballs or plain trees because: + +- **It is forward-compatible.** The payload is already a valid confext. Azure + Linux 4.0 ships systemd 258.4, where the identical artefact is handed to + native `systemd-confext --mutable=yes` and the custom mounting is deleted. + Two things do not change at that transition: the loader remains, because + systemd has no concept of A/B slots and selection stays the platform's job; + and native mounting reintroduces `extension-release` matching that the custom + lowerdir mount bypasses. - **It reuses everything.** Same build and signing pipeline as sysexts, same - Trident download and hash-verification path, verity available by DDI - convention. + 0789 delivery path, verity available by DDI convention. - **It is atomic.** A loop mount either appears or does not. A tarball - extraction can half-fail, and would also pay extraction cost on every boot. -- **SELinux labels travel in the filesystem image**, rather than in tar metadata. + extraction can half-fail and pays extraction cost on every boot. +- **SELinux labels travel in the filesystem image**, not in tar metadata. #### Payloads must declare `ID=_any` Because slot selection is the platform's job and not systemd's, the payload must not additionally be subject to OS-version matching. A payload declaring a -concrete `ID=` would merge under the custom mount, which ignores matching, and -then silently stop merging the day a platform switches to native confext -mounting. That is the failure mode this RFC exists to prevent, reappearing at -the point of transition. +concrete `ID=` would merge under the custom mount, which ignores matching, then +silently stop merging the day a platform switches to native confext mounting. +That is the failure mode this RFC exists to prevent, reappearing at the point of +transition. -Declaring `ID=_any` bypasses matching on every systemd version. Selection -remains entirely with the loader, which is what slot-scoping requires anyway, -and the move to native mounting changes no artefact. Deciding this late would -mean rebuilding every payload at transition time, so it is specified here. +`ID=_any` bypasses matching on every systemd version. Selection remains with the +loader, which is what slot-scoping requires anyway, and the move to native +mounting changes no artefact. Deciding this late would mean rebuilding every +payload, so it is specified here. 0789 already states that `ID=_any` must not be +warned on, since it declares compatibility rather than delivery method. Adding lowerdirs does not increase filesystem stacking depth, because depth is the maximum of the layers plus one and every lowerdir sits on a plain -filesystem. This matters: *reordering* the sysext merge ahead of the `/etc` -overlay would instead place `/etc` on top of the `/usr` overlay, reaching depth -2, and any confext merged afterwards would reach 3 and fail. +filesystem. *Reordering* the sysext merge ahead of the `/etc` overlay would +instead place `/etc` on top of the `/usr` overlay, reaching depth 2, and any +confext merged afterwards would reach 3 and fail. `FILESYSTEM_MAX_STACK_DEPTH` is 2 (`include/linux/fs.h`), enforced in `fs/overlayfs/super.c`. @@ -189,20 +212,23 @@ The loader prepares the complete set and commits it with a single rename. ## Public API Design -### Today +### Answering 0789's open question -```yaml -os: - sysexts: - - url: https://example.com/tooling.raw # http(s) | file | oci - sha384: - path: /var/lib/extensions/tooling.raw # optional; allow-listed dir, *.raw -``` +0789 asks how a COSI entry requests slotted placement, and states that omitting +`path` must not opt in. The answer is an explicit `scope`, spelled identically +on both sources so one concept has one name. + +**COSI metadata**, extending 0789's `ExtensionImage`: -`path` is the destination, defaulting to `/var/lib/extensions/{name}.raw`. When -`storage.abUpdate` is configured it must also resolve to an A/B volume. +```json +{ + "id": "gpu-driver", + "image": { "path": "images/extensions/gpu-driver.rawzst", "...": "..." }, + "scope": "slotted" +} +``` -### Proposed +**Host Configuration**, extending `Extension`: ```yaml os: @@ -212,16 +238,13 @@ os: extensionStore: /mnt/data/trident/extensions sysexts: - # unchanged, and still the default - - url: https://example.com/tooling.raw + - url: https://example.com/tooling.raw # unchanged, still the default sha384: - - # new - url: https://example.com/platform.raw sha384: scope: slotted - # new: confext-format images contributed as /etc overlay layers + # confext-format images contributed as /etc overlay layers etcLayers: - url: https://example.com/platform-etc.raw sha384: @@ -231,29 +254,32 @@ os: | `scope` | Where the bytes live | Where it is activated | | --- | --- | --- | | `shared` (default) | At `path`. | At `path`. Today's behaviour. | -| `slotted` | `/{a,b}/` | `/run/extensions`, or as an `/etc` lowerdir | +| `slotted` | `/{a,b}/`, keyed by `(kind, id)` | `/run/extensions`, or as an `/etc` lowerdir | -A slotted extension always surfaces in a loader-owned location, so `path` does -not apply to it and is rejected. +`scope` and `path` are mutually exclusive: a slotted extension surfaces in a +loader-owned location, so a caller-supplied destination would be silently +ignored. This satisfies 0789's requirement that placement never change +implicitly. -`etcLayers` is a separate list rather than a `scope` on `os.confexts`, because -the two are activated by different mechanisms and only one of them is -`systemd-confext`. If a platform later gains mutable confext support, entries -can migrate to `os.confexts` unchanged. +`etcLayers` is a separate list rather than a `scope` on `os.confexts` because +the two are activated by different mechanisms, only one of which is +`systemd-confext`. The COSI side needs no equivalent split: a confext entry with +`scope: slotted` is an `/etc` layer. ### The validation change `validate_extension_images_locations` is relaxed, not removed. Its requirement -becomes *the destination is slot-scoped*, satisfied either way: +becomes *the destination is slot-scoped*, satisfied by either mechanism. This is +the arm 0789's Destination Validation anticipates: | Configuration | Rule | | --- | --- | | No `abUpdate` | No check, as today. | -| `abUpdate`, `scope: shared` | `path` must be on an A/B volume. Unchanged. | -| `abUpdate`, `scope: slotted` | `path` need not be; the store plus the loader provide the scoping. | +| `abUpdate`, `scope: shared` | Destination must be on an A/B volume. Unchanged. | +| `abUpdate`, `scope: slotted` | Satisfied by the store, **only when loader capability is present**. | -This is only safe where the platform actually provides a loader, so validation -must test a **platform capability**, not merely that A/B is configured. +0789 states the same constraint from its side: slotted storage qualifies only +with loader capability, and omitting `path` must not bypass the A/B check. ### Validation @@ -265,67 +291,59 @@ must test a **platform capability**, not merely that A/B is configured. | `scope: slotted`, no loader capability | Reject. See [Rollout](#rollout). | | `extensionStore` inside a search path | Reject: both slots' copies would be discovered. | | `extensionStore` on an A/B volume | Reject: it must outlive the slot it describes. | -| Two slotted entries resolving to the same basename | Reject: they would collide. | | `etcLayers` entry without confext metadata | Reject at deploy time, after mount. | +| `etcLayers` entry without `ID=_any` | Reject; see above. | + +Duplicate `(kind, id)` and destination collisions are already covered by 0789 +and are not restated. ### Backward compatibility -All new fields default, so existing Host Configurations parse and behave -identically. The relaxed check is strictly more permissive, so anything valid -today stays valid. The schema gains three optional properties. +All new fields default, so existing Host Configurations and 1.3 COSI files +without `scope` behave identically. The relaxed check is strictly more +permissive, so anything valid today stays valid. ## Implementation ``` / ├── a/ -│ ├── manifest.json -│ ├── sysext/platform.raw -│ └── etc/platform-etc.raw +│ ├── sysext/.raw +│ └── etc/.raw └── b/ ... -/run/extensions/platform.raw -> /a/sysext/platform.raw -/etc overlay lowerdir list += +/run/extensions/.raw -> /a/sysext/.raw +/etc overlay lowerdir list += .raw> ``` -The store is outside every search path, so its contents are invisible to -`systemd-sysext` until selected. Both slot directories persist across an update, -so rollback needs no re-download. +Keyed by `(kind, id)` from 0789 rather than by filename, so replacement and +comparison use the same identity the inventory does. The store is outside every +search path, so its contents are invisible to `systemd-sysext` until selected. +Both slot directories persist across an update, so rollback needs no +re-download. -### Servicing writes the store, atomically +### Servicing -The extensions subsystem already runs on clean install and A/B update via -`provision()`, documented as "migrate state from A-partition to B-partition (or -vice versa)". It resolves the target volume with -`EngineContext::get_ab_update_volume()`, already used to select the per-slot -verity addon, and writes into `//`. - -Placement must be atomic. A malformed `.raw` is not skipped like a version -mismatch: `image_read_metadata` errors propagate in -`image_discover_and_read_metadata` (`src/sysext/sysext.c`) before any per-image -tolerance applies, so **one bad image blocks every extension on the host**. -Directory-type images are tolerated; `.raw` are not. +0789's extensions subsystem already computes the effective set and places +images. For a slotted entry the destination becomes `//`, with +the target volume from `EngineContext::get_ab_update_volume()`, already used to +select the per-slot verity addon. -Trident today falls back to a non-atomic `fs::copy` when `rename` fails across a -filesystem boundary (`crates/trident/src/subsystems/extensions/mod.rs:525-540`). -This is a prerequisite fix, not a consequence of this design. +What is deployed per slot is recorded in 0789's Deployed Extension Inventory. +This RFC adds no separate manifest: the inventory already carries kind, ID, +name, resolved placement, source and source hash, which is what the loader needs +to distinguish "nothing configured" from "storage lost". -### The store carries a manifest - -Globbing `*.raw` cannot distinguish "no extensions configured" from "storage -lost", "mount failed" or "update interrupted", and treating all of those as -success produces a quietly extension-less host. - -Each slot directory carries a manifest naming the expected images and hashes. An -absent manifest means the feature is not configured; a present manifest with -missing or unexpected content is an error. This also gives garbage collection -something to act on. +Atomic placement is a shared prerequisite. A malformed `.raw` is not skipped like +a version mismatch: `image_read_metadata` errors propagate in +`image_discover_and_read_metadata` (`src/sysext/sysext.c`) before any per-image +tolerance applies, so one bad image blocks every extension on the host. ### Slot detection -The loader resolves the slot per platform, selected by a mode setting so that -Trident does not hardcode one platform's mechanism. The first mode reads a slot -label from the kernel command line, supplied by a per-slot signed boot artefact. +The loader resolves the slot per platform, selected by a mode setting so Trident +does not hardcode one platform's mechanism. The first mode reads a slot label +from the kernel command line, supplied by a per-slot signed boot artefact. A label is preferable to a partition UUID because no mapping table has to stay in sync with a layout that changes. However, a label is an *assertion*: a @@ -335,27 +353,26 @@ on a mismatch. ## Change inventory -Sizes are T-shirt estimates covering design and implementation separately. Risk -reflects how much is unknown rather than how much work there is. +T-shirt estimates covering design and implementation separately. Risk reflects +how much is unknown rather than how much work there is. Rows delivered by 0789 +are excluded. ### New component | Change | Design | Impl | Risk | Notes | | --- | :-: | :-: | :-: | --- | -| `trident-sysext-loader` crate, slot detection and set commit | S | M | Low | std plus `anyhow`; logic is pure given cmdline, store and target, so unit-testable without root or a VM | +| `trident-sysext-loader` crate, slot detection and set commit | S | M | Low | std plus `anyhow`; pure given cmdline, store and inventory, so unit-testable without root or a VM | | `/etc` layer mounting and lowerdir emission | M | M | Med | Loop-mount in the initrd; interaction with the existing overlay needs care | -| Manifest format and verification | M | S | Low | Decide internal versus API-visible | ### Trident | Change | Design | Impl | Risk | Notes | | --- | :-: | :-: | :-: | --- | -| **Atomic extension placement** | S | S | Low | Prerequisite bug fix; live defect today, independent of this RFC | -| `scope` and `extensionStore` API | S | S | Low | Additive, both default | -| `etcLayers` API | M | S | Med | New concept; shape depends on confext migration path | -| Relax `validate_extension_images_locations` | M | S | Med | Must test loader capability, not just A/B | -| Subsystem writes slotted entries per volume | S | M | Low | `provision()` already has the target volume | -| Manifest generation | S | S | Low | | +| `scope` on COSI `ExtensionImage` and HC `Extension` | S | S | Low | Additive; answers 0789's open question | +| `extensionStore` and `etcLayers` API | M | S | Med | `etcLayers` shape depends on the confext migration path | +| Relax `validate_extension_images_locations` | M | S | Med | Adds the slotted arm to 0789's resolved-placement validation | +| Slotted destination in the extensions subsystem | S | S | Low | 0789 already resolves placement; this adds one destination policy | +| Loader capability detection | M | S | Med | Needed by both RFCs' validation | | Package loader and dracut module in the RPM | S | S | Low | `%files acl` already ships a second binary | | Schema regeneration | S | S | Low | Mechanical | @@ -364,7 +381,7 @@ reflects how much is unknown rather than how much work there is. | Change | Design | Impl | Risk | Notes | | --- | :-: | :-: | :-: | --- | | New dracut module to install and run the loader | S | S | Low | `dracut-acl-selinux-toggle` is an existing precedent | -| Patch `bootengine` to add `/etc` lowerdirs | S | M | Med | Third patch alongside two already carried; the mount itself is one line, validation is the work | +| Patch `bootengine` to add `/etc` lowerdirs | S | M | Med | Third patch alongside two already carried; the mount is one line, validation is the work | | Order the loader before the `/etc` overlay mount | S | S | Low | Same initrd window as `initrd-setup-root` | | Review `overlaycleanup()` interaction | M | S | Med | It compares only against the distro lower layer, so extension-provided defaults will not auto-prune | | Package catalog entry | S | S | Low | | @@ -376,25 +393,28 @@ reflects how much is unknown rather than how much work there is. | --- | --- | | systemd / Azure Linux | Nothing. Avoiding a systemd change is the point of this route | | AgentBaker, AKS RP | No interaction | -| Image Customizer | None unless RFC 0789 lands, and then only at the shared validation | +| Image Customizer | Covered by 0789, which already requires a writer emitting the section | ### Cross-cutting | Change | Design | Impl | Risk | Notes | | --- | :-: | :-: | :-: | --- | -| Rollout and capability gating | M | S | Med | Depends on whether the initrd is per-slot; see [Rollout](#rollout) | +| Rollout and capability gating | M | S | Med | Depends on whether the initrd is per-slot | | Boot and rollback end-to-end tests | M | L | High | Unsupervised rollback cannot be asserted from unit tests | | SELinux policy | M | M | High | Extensions are already unsupported with SELinux enforcing | ### Suggested sequencing -1. **Atomic placement fix.** Standalone, lands independently, worth doing - regardless of this RFC. +1. 0789 steps 1 and 2: prerequisites, COSI 1.3, and reading and installing + bundled extensions. 2. Loader crate and dracut module, sysext half only, behind no API change. -3. Trident API and the validation relaxation. +3. `scope` plumbing and the validation relaxation. 4. `/etc` layer half, including the `bootengine` patch. Separable; ships after. 5. Rollout gating, once the per-slot-initrd question is settled. +The atomic-placement fix is a prerequisite of both RFCs and should land first +and independently; it is a live defect today. + ## Rollout If the initrd is shared between slots, loader capability is never @@ -404,42 +424,26 @@ If it is per-slot, a rollback into a slot whose initrd predates the loader has no selector, and every slotted extension is absent for that boot. Enabling `scope: slotted` then requires two phases: deploy a release carrying the loader and let it reach both slots, then permit slotted configuration. Validation must -verify that capability rather than assume it. - -This is an open question, listed below, and it changes the work in the table -above. - -## Relationship to RFC 0789 - -[0789](https://github.com/microsoft/trident/pull/789) carries extension images -in COSI files. The two are orthogonal: 0789 changes where the bytes come from, -this changes where they land and how they follow the slot. Either is useful -alone, and a bundled extension can be slotted. - -Both touch `validate_extension_images_locations`, in opposite directions: 0789 -tightens it into dynamic validation against the storage graph, this adds a -second way to satisfy it. Whichever lands first should leave the check expressed -as *is this destination slot-scoped*, so the other is an added arm rather than a -rewrite. +verify that capability rather than assume it, which is also what 0789 requires +of its destination validation. ## Testing -- Clean install: image in the correct slot directory, selection resolving, - merged after boot. +Beyond 0789's coverage: + +- A bundled extension with `scope: slotted`: placed in the correct slot + directory, selected at boot, merged. - A/B update: the selection repoints and **the merged content differs from the previous slot**. A presence-only assertion passes in the inert case and must not be used. - **Unsupervised rollback, with no Trident servicing operation in between**: the previous slot's content is merged again. This is the case the design exists for. -- `/etc` layer: a slotted layer is visible in `/etc`, `/etc` remains writable, - and a write to a file the layer provides persists across reboot. +- `/etc` layer: visible in `/etc`, `/etc` remains writable, and a write to a + file the layer provides persists across reboot. - Loader failure: boot does not proceed silently with a partial set. -- A malformed image in a slot directory is rejected before it is selected. - Validation: each reject row, plus a shared extension off an A/B volume still - rejected. -- Compatibility: a configuration with no `scope` places byte-identically to - today. + rejected, and `scope: slotted` rejected without loader capability. ## Counter-arguments @@ -457,8 +461,6 @@ rewrite. - **"Backport mutable confexts."** It exists, and it is 1310 lines across five files of someone else's component. This route reaches the same place without carrying that, and converges on native confexts later at zero payload cost. -- **"Doubles storage."** It does. The A/B volume route doubles it too, with the - allocation fixed at install. ### Rejected mechanisms @@ -466,14 +468,11 @@ rewrite. no fallback when that image fails OS matching, so they cannot hold per-slot copies. Verified empirically; Flatcar reached the same conclusion. - **`systemd-tmpfiles`.** No specifier identifies the booted device, partition - or slot, symlink targets are literal text plus specifier expansion, and the - main pass is ordered after the merge. + or slot, and the main pass is ordered after the merge. - **A systemd generator.** Generators run sandboxed with most of the filesystem read-only, and `/var` is not mounted when they run. - **Reordering the sysext merge ahead of the `/etc` overlay.** Exhausts the filesystem stacking budget and breaks confexts outright. -- **Repair after boot.** Any agent running at `multi-user.target` is far too - late to influence the merge. ### Prior art @@ -492,8 +491,8 @@ persisting it: - **Fedora's `sysexts-manager`** filters candidates by the running `VERSION_ID` and symlinks the winner into `/run/extensions`. -The counter-example is instructive: Fedora CoreOS activates sysexts statically -across deployments, and its documentation records that after a major rebase the +Fedora CoreOS is the counter-example: it activates sysexts statically across +deployments, and its documentation records that after a major rebase the extensions silently stop matching and are not loaded. ## Open questions @@ -503,25 +502,26 @@ extensions silently stop matching and are not loaded. 2. **Whole-directory selection or a prepared set?** Both commit atomically. The former is simpler; the latter allows shared and slotted extensions to share a search path. -3. **Manifest format**, and whether it is Trident-internal or API-visible. -4. **Retention.** Nothing prunes a slot directory when an entry leaves the - configuration. Reconciling in `provision()` conflicts with keeping the - previous slot's files for rollback. -5. **Default store path.** `/var/trident/extensions` against +3. **Retention.** Nothing prunes a slot directory when an entry leaves the + configuration. Reconciling during provisioning conflicts with keeping the + previous slot's files for rollback. 0789's inventory gives this something to + act on, but the policy is undecided. +4. **Default store path.** `/var/trident/extensions` against `/var/lib/trident/extensions`; the datastore already lives under the latter. +5. **Does a slotted extension need an extension-only update path?** 0789's + Runtime Update path writes to the running slot, which is in tension with + slot-scoped placement. 6. **Measured boot.** The selected image is outside the UKI's measurements, so - identical measurements can yield different `/usr` contents. Whether that is - acceptable depends on the platform's attestation claims. -7. **SELinux.** Extensions are already unsupported with SELinux enforcing, which - is a precondition this RFC does not resolve. + identical measurements can yield different `/usr` contents. +7. **SELinux.** Extensions are already unsupported with SELinux enforcing, a + precondition neither RFC resolves. ## Future possibilities - **Native confext mounting.** On systemd 256 or later, the loader places the selected payload in a confext search path and `systemd-confext --mutable=yes` - mounts it, replacing the custom lowerdir handling. The loader still performs - selection, and the payloads do not change provided they declare `ID=_any`. - Azure Linux 4.0 ships systemd 258.4. + mounts it. The loader still performs selection, and payloads do not change + provided they declare `ID=_any`. Azure Linux 4.0 ships systemd 258.4. - **Content matching** on platforms with systemd v256 or later and a build-granular `VERSION_ID`, which needs no loader at all. - **Initrd-phase sysext merging**, as Flatcar now does, for extensions that must From d424a2432037378aeac0d9d2d93d3b3e5f837238 Mon Sep 17 00:00:00 2001 From: Paco Date: Wed, 14 Oct 2026 11:17:33 -0700 Subject: [PATCH 8/8] docs: avoid 'commit' for the loader's atomic activation 'commit' already names the post-boot servicing operation in Trident, so using it for the loader publishing its symlink set is a confusing collision. The loader has no relationship to that operation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs/Development/RFCs/0838-slotted-sysexts.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/Development/RFCs/0838-slotted-sysexts.md b/docs/Development/RFCs/0838-slotted-sysexts.md index efbc05877b..b21c2f42c1 100644 --- a/docs/Development/RFCs/0838-slotted-sysexts.md +++ b/docs/Development/RFCs/0838-slotted-sysexts.md @@ -208,7 +208,8 @@ confext merged afterwards would reach 3 and fail. Creating N symlinks exposes a prefix of the intended set if it fails partway, which matters where a driver and its userspace libraries must move together. -The loader prepares the complete set and commits it with a single rename. +The loader prepares the complete set out of the way and activates it with a +single rename, so the set becomes visible all at once or not at all. ## Public API Design @@ -361,7 +362,7 @@ are excluded. | Change | Design | Impl | Risk | Notes | | --- | :-: | :-: | :-: | --- | -| `trident-sysext-loader` crate, slot detection and set commit | S | M | Low | std plus `anyhow`; pure given cmdline, store and inventory, so unit-testable without root or a VM | +| `trident-sysext-loader` crate, slot detection and atomic set activation | S | M | Low | std plus `anyhow`; pure given cmdline, store and inventory, so unit-testable without root or a VM | | `/etc` layer mounting and lowerdir emission | M | M | Med | Loop-mount in the initrd; interaction with the existing overlay needs care | ### Trident @@ -499,7 +500,7 @@ extensions silently stop matching and are not loaded. 1. **Is the initrd per-slot or shared?** Decides whether a staged rollout is needed at all, and changes the estimates above. -2. **Whole-directory selection or a prepared set?** Both commit atomically. The +2. **Whole-directory selection or a prepared set?** Both activate atomically. The former is simpler; the latter allows shared and slotted extensions to share a search path. 3. **Retention.** Nothing prunes a slot directory when an entry leaves the